Files

dtoro e32b21072d claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes

- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core
- Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only)
- Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor
- Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift
- Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options)
- Wire Matrix (matrix:dtoro) as health alert delivery platform
- Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml
- Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md

2026-06-04 10:14:56 +02:00

7.0 KiB

Raw Blame History

Backups — restic on external drive (DISABLED)

Chunked monthly restic backup of /mnt/library's irreplaceable subset. Disabled 2026-04-22 as part of the hubris crash-loop A/B test.

Status

DISABLED 2026-04-22. All four timers systemctl disable --now'd:

backup-library@homecloud.timer
backup-library@images.timer
backup-library@small.timer
backup-library-check.timer

Fstab entry commented out. USB drive de-authorized and physically removed. backup-library-deploy.service left enabled (harmless webhook receiver).

Reason: the host hang recurred 2026-04-22 18:42 after 30h despite the cpu-epp fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See investigation.

To re-enable: uncomment fstab line, systemctl enable --now the four timers, re-attach drive.

Design

Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs /mnt/library) into thermal danger.

Retention per tag: --keep-last 3 --keep-monthly 12 --keep-yearly 3 with --group-by host,tags,paths.

Components

Repo: dtoro/backup-library
Checkout: /opt/backup-library on the hubris host
Auto-deploys via gitea webhook → http://192.168.8.77:9798/deploy. See auto-deploy. Gitea (104) app.ini ALLOWED_HOST_LIST includes 192.168.8.77 for this.
Restic repo: /mnt/backup/restic-library. Passphrase /etc/restic/passphrase (mode 600). Escrow in password manager — loss = permanent data loss.
External drive: /dev/sda1 ext4 label backup-library UUID ff46e775-1ba1-4892-82c9-e5cac5be933a. Fstab uses noauto + nofail,x-systemd.device-timeout=10s,errors=remount-ro.

Mount-on-demand

/usr/local/sbin/backup-usb.sh attach|detach|status. All three backup units (backup-library.service, backup-library@.service, backup-library-check.service) have ExecStartPre=backup-usb.sh attach and ExecStopPost=backup-usb.sh detach. The helper toggles /sys/bus/usb/devices/*/authorized by matching vendor:product 090c:2320, then mounts/unmounts /mnt/backup. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.

UAS blacklist

/etc/modprobe.d/usb-storage-quirks.conf:

options usb-storage quirks=090c:2320:u

Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with dmesg | grep "UAS is ignored".

Schedule

Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:

Timer	When	Include list	Approx size
`backup-library@homecloud.timer`	day 1 / month	`/etc/restic/include-homecloud.list`	~315 G
`backup-library@images.timer`	day 10 / month	`/etc/restic/include-images.list`	~103 G
`backup-library@small.timer`	day 20 / month	`/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper)	~11 G

Snapshots tagged chunk-<group> so forget/prune treats each series independently.

Ad-hoc full run (kept for manual use): systemctl start backup-library.service (no arg → uses /etc/restic/include.list, tag monthly).

Yearly integrity: backup-library-check.timer (OnCalendar=yearly) runs full restic check --read-data.

Thermal caps

Baked into the systemd units:

IOReadBandwidthMax=/mnt/library 50M
IOWriteBandwidthMax=/mnt/backup 30M
--read-concurrency=1 on restic.

Wrapper

/usr/local/sbin/backup-library.sh — preflight → unlock → backup → forget/prune (--group-by host,tags,paths) → check --read-data-subset=5% → notify. Takes optional <group> arg or GROUP= env.

Notifications

~~POST to claudio-bot (123) http://192.168.8.230:9090/notify~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes send_message via Matrix instead.

OnFailure=notify-failure@%n.service on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.

Recovery

Runbook at /usr/share/doc/backup-library/RECOVERY.md (or in the repo at doc/RECOVERY.md). Covers restic snapshots/ls/find/restore/mount, uid/gid gotcha, cross-host recovery.

Known SPOF

Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via restic copy) are not yet implemented.

Drive history

The Silicon Motion Portable SSD (vid:pid 090c:2320) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.

After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by uas_eh_abort_handler storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See investigation.

Thermal monitoring

Moved out of this repo to dtoro/claudio-monitor on 2026-04-21 (commit 50dc213). See monitoring.

Hubris host
~~claudio-bot (123)~~ (destroyed 2026-06-04)
Monitoring
Auto-deploy
Investigation: 2026-04-21 crash loop

Changelog

2026-04-28 — wiki entry created

Initial documentation. Status remains DISABLED.

2026-04-22 — DISABLED

Drive removed as the A/B test in the crash investigation. Timers disabled, fstab commented, drive de-authorized.

2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive

Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles /sys/bus/usb/.../authorized so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in backup-library.sh: python3 -c '…' KEY=VAL does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.

2026-04-20 — deployed; redesigned for thermal-gentleness

Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.

7.0 KiB Raw Blame History