---
name: recover-dpkg-interrupted
risk_class: reversible_low
verification: "dpkg --audit (should be clean); apt-get check"
---

# Runbook — recover from dpkg-interrupted state

You're here because an apt run got killed mid-transaction and the target now
has packages that are **unpacked but not configured**. Symptoms:

- `apt` refuses to do anything new: `Error: dpkg was interrupted, you must
  manually run 'dpkg --configure -a' to correct the problem.`
- `dpkg --audit` lists packages with header
  `The following packages have been unpacked but not yet configured.`
- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.

The system is still running the **old** binaries (still in memory), but the
**new** binaries are unpacked and waiting for their postinst to run. Two
worst-case manifestations from the 2026-05-21 sweep:

- LXC 121 caddy: leftover state from a prior aborted apt run; caddy itself was
  still serving but the new caddy binary on disk hadn't been wired up.
- hubris: ssh master died mid-Wave-6 → 135 packages unpacked-not-configured,
  including `systemd`, `openssh-server`, `sudo`, `netbird`. The half-
  configured netbird daemon dropped the mesh peer, and we got locked out
  until we recovered from the PVE web UI Shell.

**Do not reboot until dpkg is clean.** A reboot tries to start the new
binaries' services, which may fail because postinst never ran (missing users,
config dirs, capabilities, etc.). The system might not come back up cleanly.

## Path A — target is still reachable over ssh (preferred)

```
homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```

Or for an LXC by name:

```
homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
```

When that returns, confirm:

```
homelab apt-audit --target <host>
```

Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
deferred (kernel/PVE on hubris, 0 elsewhere).

## Path B — target locked out (mesh broken / ssh dead)

Most common for hubris when netbird itself went half-configured: the daemon
crashed on the new binary, the mesh peer dropped, port 22022 stopped listening,
and you can't ssh in.

1. Open `https://proxmox.hubris.network` in a browser.
2. Datacenter → node `hubris` → `>_ Shell` (or `_ Console`). That's a root
   shell on hubris served by the PVE web UI, independent of the netbird mesh.
3. Run the recovery one-liner:

```
DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
  && DEBIAN_FRONTEND=noninteractive apt -y -o Dpkg::Options::=--force-confold upgrade \
  && systemctl restart netbird \
  && dpkg --audit \
  && echo RECOVERY_OK
```

   Wait for `RECOVERY_OK`. The `systemctl restart netbird` is the bit that
   heals the mesh — once netbird's daemon comes back up clean, your client's
   peer state moves from `Connecting` to `Connected` within ~30 seconds and
   the rest of your tooling works again.

4. For an **LXC** that's locked out (less common — LXCs reach the world via
   netbird routed through hubris, so unless hubris itself is broken, you can
   still `pct enter` from the hubris shell):

   From the PVE web UI shell on hubris:

   ```
   pct enter <id>
   DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y upgrade
   exit
   ```

## Prevention

The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
unit on the target, so it survives ssh teardown — the failure mode that put
hubris into this state in the first place is no longer reachable through the
standard tool. If you absolutely need to run apt manually over ssh, wrap it:

```
ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
```

Then `systemctl status apt-recovery` from a fresh ssh to check progress.

## Related

- [Operations cheatsheet](../../operations/commands.md)
- [Auto-deploy pipelines](../../../archive/knowledge/infrastructure/auto-deploy.md)
- [Hubris host page](../../../archive/knowledge/hosts/hubris.md)

## Changelog

### 2026-05-21 — initial page
Documents the dpkg-interrupted recovery path that came out of the
fleet apt sweep (Wave 6 killed mid-transaction; hubris recovered via PVE
web Shell).
