dtoro
e32b21072d
- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core - Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only) - Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor - Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift - Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options) - Wire Matrix (matrix:dtoro) as health alert delivery platform - Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml - Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md
116 lines
7.0 KiB
Markdown
116 lines
7.0 KiB
Markdown
# Backups — restic on external drive (DISABLED)
|
|
|
|
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).
|
|
|
|
## Status
|
|
|
|
**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
|
|
- `backup-library@homecloud.timer`
|
|
- `backup-library@images.timer`
|
|
- `backup-library@small.timer`
|
|
- `backup-library-check.timer`
|
|
|
|
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
|
|
|
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
|
|
|
|
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
|
|
|
## Design
|
|
|
|
Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.
|
|
|
|
Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.
|
|
|
|
## Components
|
|
|
|
- **Repo:** `dtoro/backup-library`
|
|
- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
|
|
- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
|
|
- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
|
|
- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.
|
|
|
|
## Mount-on-demand
|
|
|
|
`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
|
|
|
|
## UAS blacklist
|
|
|
|
`/etc/modprobe.d/usb-storage-quirks.conf`:
|
|
```
|
|
options usb-storage quirks=090c:2320:u
|
|
```
|
|
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.
|
|
|
|
## Schedule
|
|
|
|
Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
|
|
|
|
| Timer | When | Include list | Approx size |
|
|
| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
|
|
| `backup-library@homecloud.timer` | day 1 / month | `/etc/restic/include-homecloud.list` | ~315 G |
|
|
| `backup-library@images.timer` | day 10 / month | `/etc/restic/include-images.list` | ~103 G |
|
|
| `backup-library@small.timer` | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |
|
|
|
|
Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.
|
|
|
|
Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).
|
|
|
|
Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.
|
|
|
|
## Thermal caps
|
|
|
|
Baked into the systemd units:
|
|
- `IOReadBandwidthMax=/mnt/library 50M`
|
|
- `IOWriteBandwidthMax=/mnt/backup 30M`
|
|
- `--read-concurrency=1` on restic.
|
|
|
|
## Wrapper
|
|
|
|
`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.
|
|
|
|
## Notifications
|
|
|
|
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
|
|
|
|
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
|
|
|
|
## Recovery
|
|
|
|
Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.
|
|
|
|
## Known SPOF
|
|
|
|
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.
|
|
|
|
## Drive history
|
|
|
|
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
|
|
|
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
|
|
|
|
## Thermal monitoring
|
|
|
|
Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).
|
|
|
|
## Related
|
|
- [Hubris host](../hosts/hubris.md)
|
|
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
|
- [Monitoring](monitoring.md)
|
|
- [Auto-deploy](auto-deploy.md)
|
|
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
|
|
|
## Changelog
|
|
|
|
### 2026-04-28 — wiki entry created
|
|
Initial documentation. Status remains DISABLED.
|
|
|
|
### 2026-04-22 — DISABLED
|
|
Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
|
|
|
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
|
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
|
|
|
### 2026-04-20 — deployed; redesigned for thermal-gentleness
|
|
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.
|