oikos/knowledge/wiki/infrastructure/backups.md

Backups — restic on external drive (DEPRECATED — superseded)

DEPRECATED 2026-07-01. Superseded by the rclone → Proton Drive off-host mirror on LXC 132 rclone. That job finally closes the off-host / 3-2-1 gap this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been DISABLED since 2026-04-22 and is not coming back in its old form.

Current backup — rclone → Proton Drive (LXC 132)

• Where: LXC 132 rclone (192.168.8.214), /mnt/library mounted read-only.
• What: plain rclone sync (Proton mirrors local; browsable files, no versioning) of the folders listed in /etc/rclone-backup/folders.list, to proton:library-backup/….
• When: monthly — rclone-backup.timer (OnCalendar=*-*-01 03:00).
• UI: rclone Web GUI on 192.168.8.214:5572 (LAN-only, no auth).
• Encryption: Proton's built-in E2E (no rclone crypt overlay).
• Runs / logs: /var/log/rclone-backup/ + runs.jsonl.
• Still a single off-host target (Proton only). Not yet a full 3-2-1 (no second independent copy), but strictly better than the previous "no off-host copy at all."

See 132-rclone for the full design.

---

Legacy — restic on external drive (DISABLED 2026-04-22)

Chunked monthly restic backup of /mnt/library's irreplaceable subset. Disabled 2026-04-22 as part of the hubris crash-loop A/B test.

Status

DISABLED 2026-04-22. All four timers systemctl disable --now'd:

• backup-library@homecloud.timer
• backup-library@images.timer
• backup-library@small.timer
• backup-library-check.timer

Fstab entry commented out. USB drive de-authorized and physically removed. backup-library-deploy.service left enabled (harmless webhook receiver).

Reason: the host hang recurred 2026-04-22 18:42 after 30h despite the cpu-epp fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See investigation.

To re-enable: uncomment fstab line, systemctl enable --now the four timers, re-attach drive.

Design

Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs /mnt/library) into thermal danger.

Retention per tag: --keep-last 3 --keep-monthly 12 --keep-yearly 3 with --group-by host,tags,paths.

Components

• Repo: dtoro/backup-library
• Checkout: /opt/backup-library on the hubris host
• Auto-deploys via gitea webhook → http://192.168.8.77:9798/deploy. See auto-deploy. Gitea (104) app.ini ALLOWED_HOST_LIST includes 192.168.8.77 for this.
• Restic repo: /mnt/backup/restic-library. Passphrase /etc/restic/passphrase (mode 600). Escrow in password manager — loss = permanent data loss.
• External drive: /dev/sda1 ext4 label backup-library UUID ff46e775-1ba1-4892-82c9-e5cac5be933a. Fstab uses noauto + nofail,x-systemd.device-timeout=10s,errors=remount-ro.

Mount-on-demand

/usr/local/sbin/backup-usb.sh attach|detach|status. All three backup units (backup-library.service, backup-library@.service, backup-library-check.service) have ExecStartPre=backup-usb.sh attach and ExecStopPost=backup-usb.sh detach. The helper toggles /sys/bus/usb/devices/*/authorized by matching vendor:product 090c:2320, then mounts/unmounts /mnt/backup. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.

UAS blacklist

/etc/modprobe.d/usb-storage-quirks.conf:

options usb-storage quirks=090c:2320:u

Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with dmesg | grep "UAS is ignored".

Schedule

Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:

Timer	When	Include list	Approx size
backup-library@homecloud.timer	day 1 / month	/etc/restic/include-homecloud.list	~315 G
backup-library@images.timer	day 10 / month	/etc/restic/include-images.list	~103 G
backup-library@small.timer	day 20 / month	/etc/restic/include-small.list (docs / books / music / notes / repos / marimo / heaper)	~11 G

Snapshots tagged chunk-<group> so forget/prune treats each series independently.

Ad-hoc full run (kept for manual use): systemctl start backup-library.service (no arg → uses /etc/restic/include.list, tag monthly).

Yearly integrity: backup-library-check.timer (OnCalendar=yearly) runs full restic check --read-data.

Thermal caps

Baked into the systemd units:

• IOReadBandwidthMax=/mnt/library 50M
• IOWriteBandwidthMax=/mnt/backup 30M
• --read-concurrency=1 on restic.

Wrapper

/usr/local/sbin/backup-library.sh — preflight → unlock → backup → forget/prune (--group-by host,tags,paths) → check --read-data-subset=5% → notify. Takes optional <group> arg or GROUP= env.

Notifications

POST to claudio-bot (123) http://192.168.8.230:9090/notify — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes send_message via Matrix instead.

OnFailure=notify-failure@%n.service on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.

Recovery

Runbook at /usr/share/doc/backup-library/RECOVERY.md (or in the repo at doc/RECOVERY.md). Covers restic snapshots/ls/find/restore/mount, uid/gid gotcha, cross-host recovery.

Known SPOF

Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via restic copy) were not implemented before this job was retired — the off-host copy is now provided by rclone → Proton Drive (LXC 132) instead. A second independent copy is still outstanding.

Drive history

The Silicon Motion Portable SSD (vid:pid 090c:2320) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.

After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by uas_eh_abort_handler storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See investigation.

Thermal monitoring

Moved out of this repo to dtoro/claudio-monitor on 2026-04-21 (commit 50dc213). See monitoring.

Related

• Hubris host
• claudio-bot (123) (destroyed 2026-06-04)
• Monitoring
• Auto-deploy
• Investigation: 2026-04-21 crash loop

Changelog

2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)

Off-host backup moved to a plain rclone sync mirror on the new LXC 132 rclone (/mnt/library → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain disabled (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".

2026-04-28 — wiki entry created

Initial documentation. Status remains DISABLED.

2026-04-22 — DISABLED

Drive removed as the A/B test in the crash investigation. Timers disabled, fstab commented, drive de-authorized.

2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive

Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles /sys/bus/usb/.../authorized so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in backup-library.sh: python3 -c '…' KEY=VAL does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.

2026-04-20 — deployed; redesigned for thermal-gentleness

Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.