dtoro
d80a394b7f
Documentation and repo-hygiene pass following the client/server split:
Plan drift (audited all other active plans against current code):
- oikos-gaps-and-improvements.md: mark Section C and D.5 resolved (both
described cmd/hermes, renamed to cmd/nomos with a real LLM loop since);
refresh ~10 stale file:line citations; fix tool-count (33, not 28).
- liveness-drift-and-ux-cohesion.md: fix stale default-model claim (now
deepseek-v4-pro since 2026-07-10) and "not yet deployed" status.
- nomos-agent-code-review.md: fix C1's citation (one unauthenticated route
to nomos now, not two, after the client/server split).
- wails-desktop-app.md: record the production deploy outcome.
Repo structure: added missing directories to README/CONTRIBUTING layout
tables (checks/, tools/, cmd/webhook/, docs/operations/), fixed a broken
link, added ADR 0015 documenting the auth/CORS/client-split model (there
wasn't one despite CONTRIBUTING's own process requiring it), normalized
ADR 0013/0014's format drift, added an Authentication section to
AGENTS.md/CLIENTS.md (every example call was missing the now-required
bearer header).
Retired the Goose+Nomos workstation flow (bootstrap.sh --with-nomos,
tools/setup-nomos-soul.sh, .agents/operations/nomos-agent.md) and the
Caveman auto-install tooling (tools/setup-caveman.sh, tools/caveman/) —
both superseded by the production containerized Nomos agent, which has
never used either. Kept .agents/shared/caveman.md itself (the terse
writing-style convention agents still follow by reading it).
Deleted the orphaned legacy Python oikos/ directory — nothing imports it,
and bin/homelab (the CLI it was kept for) no longer exists in the repo.
Rewrote .agents/operations/agent-enrollment.md (365 -> ~110 lines) and
commands.md to match the current architecture instead of the retired
`homelab` CLI; migrated the still-true networking prerequisites (Netbird,
split-horizon DNS, SSH key distribution) into the knowledge base as a
runbook via upsert_knowledge rather than duplicating them in markdown.
Updated all 10 .agents/skills/ runbooks referencing the dead CLI with
their real MCP tool / REST API equivalents, or flagged them as needing
verification where no equivalent is confirmed yet.
Two real bugs found and fixed, not just docs:
- The tools/setup-*.sh auto-setup glob was tools/*.setup.sh in THREE
places (tools/post-pull.sh, bootstrap.sh, and internal/httpapi/impl.go's
GetClientContext handler) since the mechanism's introduction on
2026-06-02 — never matched any real filename, so no client has ever
picked up an auto-setup script via git-pull or the context-poller sync.
Fixed all three; the Go server-side fix is the one that actually matters
since it's what the current context-poller mechanism depends on.
- bootstrap.sh removed dead vestigial --gitea-token/--gitea-user flags
(parsed, never consumed) left over from an earlier clone-based model.
Also flagged, not fixed (documented as an open gap in
client-enrollment/SKILL.md): bootstrap.sh tells a freshly-enrolled client
to call POST /api/v1/clients/{slug}/activate to finish enrollment, but
that route doesn't exist in api/openapi.yaml — EnrollClient sets entities
to provisioning and nothing currently transitions them to active.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
118 lines
4.4 KiB
Markdown
118 lines
4.4 KiB
Markdown
---
|
|
name: recover-dpkg-interrupted
|
|
risk_class: reversible_low
|
|
verification: "dpkg --audit (should be clean); apt-get check"
|
|
---
|
|
|
|
# Runbook — recover from dpkg-interrupted state
|
|
|
|
You're here because an apt run got killed mid-transaction and the target now
|
|
has packages that are **unpacked but not configured**. Symptoms:
|
|
|
|
- `apt` refuses to do anything new: `Error: dpkg was interrupted, you must
|
|
manually run 'dpkg --configure -a' to correct the problem.`
|
|
- `dpkg --audit` lists packages with header
|
|
`The following packages have been unpacked but not yet configured.`
|
|
- `dpkg --audit` on the host directly shows unpacked-not-configured packages
|
|
(there's no fleet-wide audit tool anymore — check per-host).
|
|
|
|
The system is still running the **old** binaries (still in memory), but the
|
|
**new** binaries are unpacked and waiting for their postinst to run. Two
|
|
worst-case manifestations from the 2026-05-21 sweep:
|
|
|
|
- LXC 121 caddy: leftover state from a prior aborted apt run; caddy itself was
|
|
still serving but the new caddy binary on disk hadn't been wired up.
|
|
- hubris: ssh master died mid-Wave-6 → 135 packages unpacked-not-configured,
|
|
including `systemd`, `openssh-server`, `sudo`, `netbird`. The half-
|
|
configured netbird daemon dropped the mesh peer, and we got locked out
|
|
until we recovered from the PVE web UI Shell.
|
|
|
|
**Do not reboot until dpkg is clean.** A reboot tries to start the new
|
|
binaries' services, which may fail because postinst never ran (missing users,
|
|
config dirs, capabilities, etc.). The system might not come back up cleanly.
|
|
|
|
## Path A — target is still reachable over ssh (preferred)
|
|
|
|
```
|
|
ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
|
|
```
|
|
|
|
Or for an LXC by name (via the MCP `run` tool, or directly on the Proxmox
|
|
host):
|
|
|
|
```
|
|
pct exec <lxc> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
|
|
```
|
|
|
|
When that returns, confirm:
|
|
|
|
```
|
|
ssh <host> -- dpkg --audit
|
|
```
|
|
|
|
Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
|
|
deferred (kernel/PVE on hubris, 0 elsewhere).
|
|
|
|
## Path B — target locked out (mesh broken / ssh dead)
|
|
|
|
Most common for hubris when netbird itself went half-configured: the daemon
|
|
crashed on the new binary, the mesh peer dropped, port 22022 stopped listening,
|
|
and you can't ssh in.
|
|
|
|
1. Open `https://proxmox.hubris.network` in a browser.
|
|
2. Datacenter → node `hubris` → `>_ Shell` (or `_ Console`). That's a root
|
|
shell on hubris served by the PVE web UI, independent of the netbird mesh.
|
|
3. Run the recovery one-liner:
|
|
|
|
```
|
|
DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
|
|
&& DEBIAN_FRONTEND=noninteractive apt -y -o Dpkg::Options::=--force-confold upgrade \
|
|
&& systemctl restart netbird \
|
|
&& dpkg --audit \
|
|
&& echo RECOVERY_OK
|
|
```
|
|
|
|
Wait for `RECOVERY_OK`. The `systemctl restart netbird` is the bit that
|
|
heals the mesh — once netbird's daemon comes back up clean, your client's
|
|
peer state moves from `Connecting` to `Connected` within ~30 seconds and
|
|
the rest of your tooling works again.
|
|
|
|
4. For an **LXC** that's locked out (less common — LXCs reach the world via
|
|
netbird routed through hubris, so unless hubris itself is broken, you can
|
|
still `pct enter` from the hubris shell):
|
|
|
|
From the PVE web UI shell on hubris:
|
|
|
|
```
|
|
pct enter <id>
|
|
DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y upgrade
|
|
exit
|
|
```
|
|
|
|
## Prevention
|
|
|
|
The old `homelab apt-upgrade` wrapper (retired along with the rest of the
|
|
`homelab` CLI) used to launch apt inside a `systemd-run --collect` unit on
|
|
the target so it survived ssh teardown — that's the failure mode that put
|
|
hubris into this state in the first place. There's no fleet-wide wrapper
|
|
anymore; if you run apt manually over ssh, wrap it yourself the same way:
|
|
|
|
```
|
|
ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
|
|
```
|
|
|
|
Then `systemctl status apt-recovery` from a fresh ssh to check progress.
|
|
|
|
## Related
|
|
|
|
- [Operations cheatsheet](../../operations/commands.md)
|
|
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
|
|
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)
|
|
|
|
## Changelog
|
|
|
|
### 2026-05-21 — initial page
|
|
Documents the dpkg-interrupted recovery path that came out of the
|
|
fleet apt sweep (Wave 6 killed mid-transaction; hubris recovered via PVE
|
|
web Shell).
|