f5cd320433
Initial documentation of the hubris Proxmox homelab as a cross-linked markdown wiki. Per-node pages, cross-cutting infrastructure pages, an investigation log, and an operations cheatsheet. Each node and topic ends with a Changelog section so changes can be tracked in-place going forward. Refreshed against live state on 2026-04-28 — 14 active LXCs (109 syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test state of the 2026-04-21 hubris crash-loop investigation.
7.0 KiB
Backups — restic on external drive (DISABLED)
Chunked monthly restic backup of /mnt/library's irreplaceable subset. Disabled 2026-04-22 as part of the hubris crash-loop A/B test.
Status
DISABLED 2026-04-22. All four timers systemctl disable --now'd:
backup-library@homecloud.timerbackup-library@images.timerbackup-library@small.timerbackup-library-check.timer
Fstab entry commented out. USB drive de-authorized and physically removed. backup-library-deploy.service left enabled (harmless webhook receiver).
Reason: the host hang recurred 2026-04-22 18:42 after 30h despite the cpu-epp fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See investigation.
To re-enable: uncomment fstab line, systemctl enable --now the four timers, re-attach drive.
Design
Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs /mnt/library) into thermal danger.
Retention per tag: --keep-last 3 --keep-monthly 12 --keep-yearly 3 with --group-by host,tags,paths.
Components
- Repo:
dtoro/backup-library - Checkout:
/opt/backup-libraryon the hubris host - Auto-deploys via gitea webhook →
http://192.168.8.77:9798/deploy. See auto-deploy. Gitea (104)app.iniALLOWED_HOST_LISTincludes192.168.8.77for this. - Restic repo:
/mnt/backup/restic-library. Passphrase/etc/restic/passphrase(mode 600). Escrow in password manager — loss = permanent data loss. - External drive:
/dev/sda1ext4 labelbackup-libraryUUIDff46e775-1ba1-4892-82c9-e5cac5be933a. Fstab usesnoauto+nofail,x-systemd.device-timeout=10s,errors=remount-ro.
Mount-on-demand
/usr/local/sbin/backup-usb.sh attach|detach|status. All three backup units (backup-library.service, backup-library@.service, backup-library-check.service) have ExecStartPre=backup-usb.sh attach and ExecStopPost=backup-usb.sh detach. The helper toggles /sys/bus/usb/devices/*/authorized by matching vendor:product 090c:2320, then mounts/unmounts /mnt/backup. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
UAS blacklist
/etc/modprobe.d/usb-storage-quirks.conf:
options usb-storage quirks=090c:2320:u
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with dmesg | grep "UAS is ignored".
Schedule
Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
| Timer | When | Include list | Approx size |
|---|---|---|---|
backup-library@homecloud.timer |
day 1 / month | /etc/restic/include-homecloud.list |
~315 G |
backup-library@images.timer |
day 10 / month | /etc/restic/include-images.list |
~103 G |
backup-library@small.timer |
day 20 / month | /etc/restic/include-small.list (docs / books / music / notes / repos / marimo / heaper) |
~11 G |
Snapshots tagged chunk-<group> so forget/prune treats each series independently.
Ad-hoc full run (kept for manual use): systemctl start backup-library.service (no arg → uses /etc/restic/include.list, tag monthly).
Yearly integrity: backup-library-check.timer (OnCalendar=yearly) runs full restic check --read-data.
Thermal caps
Baked into the systemd units:
IOReadBandwidthMax=/mnt/library 50MIOWriteBandwidthMax=/mnt/backup 30M--read-concurrency=1on restic.
Wrapper
/usr/local/sbin/backup-library.sh — preflight → unlock → backup → forget/prune (--group-by host,tags,paths) → check --read-data-subset=5% → notify. Takes optional <group> arg or GROUP= env.
Notifications
POST to claudio-bot (123) http://192.168.8.230:9090/notify with header X-Bot-Token: $(cat /etc/restic/bot.token). Token in /etc/restic/bot.token is the same value as the bot's ipc.token — rotate together.
OnFailure=notify-failure@%n.service on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
Recovery
Runbook at /usr/share/doc/backup-library/RECOVERY.md (or in the repo at doc/RECOVERY.md). Covers restic snapshots/ls/find/restore/mount, uid/gid gotcha, cross-host recovery.
Known SPOF
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via restic copy) are not yet implemented.
Drive history
The Silicon Motion Portable SSD (vid:pid 090c:2320) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by uas_eh_abort_handler storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See investigation.
Thermal monitoring
Moved out of this repo to dtoro/claudio-monitor on 2026-04-21 (commit 50dc213). See monitoring.
Related
Changelog
2026-04-28 — wiki entry created
Initial documentation. Status remains DISABLED.
2026-04-22 — DISABLED
Drive removed as the A/B test in the crash investigation. Timers disabled, fstab commented, drive de-authorized.
2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles /sys/bus/usb/.../authorized so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in backup-library.sh: python3 -c '…' KEY=VAL does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
2026-04-20 — deployed; redesigned for thermal-gentleness
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.