# Backups — restic on external drive (DISABLED)

Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).

## Status

**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
- `backup-library@homecloud.timer`
- `backup-library@images.timer`
- `backup-library@small.timer`
- `backup-library-check.timer`

Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).

**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).

**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.

## Design

Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.

Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.

## Components

- **Repo:** `dtoro/backup-library`
- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.

## Mount-on-demand

`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.

## UAS blacklist

`/etc/modprobe.d/usb-storage-quirks.conf`:
```
options usb-storage quirks=090c:2320:u
```
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.

## Schedule

Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:

| Timer                              | When           | Include list                         | Approx size |
| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
| `backup-library@homecloud.timer`   | day 1 / month  | `/etc/restic/include-homecloud.list` | ~315 G       |
| `backup-library@images.timer`      | day 10 / month | `/etc/restic/include-images.list`    | ~103 G       |
| `backup-library@small.timer`       | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |

Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.

Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).

Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.

## Thermal caps

Baked into the systemd units:
- `IOReadBandwidthMax=/mnt/library 50M`
- `IOWriteBandwidthMax=/mnt/backup 30M`
- `--read-concurrency=1` on restic.

## Wrapper

`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.

## Notifications

POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.

`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.

## Recovery

Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.

## Known SPOF

Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.

## Drive history

The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.

After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).

## Thermal monitoring

Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).

## Related
- [Hubris host](../hosts/hubris.md)
- [claudio-bot (123)](../containers/123-claudio-bot.md)
- [Monitoring](monitoring.md)
- [Auto-deploy](auto-deploy.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)

## Changelog

### 2026-04-28 — wiki entry created
Initial documentation. Status remains DISABLED.

### 2026-04-22 — DISABLED
Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.

### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.

### 2026-04-20 — deployed; redesigned for thermal-gentleness
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.