dtoro
6b75f7302d
- Migrations 010 (content_hash) + 011 (search tsvector column) - new: internal/knowledge/seed.go — knowledge seed ingest engine - new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge - wire knowledge ingest into oikos seed pipeline - convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml - archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/ - delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py - remove empty knowledge/ directory tree
143 lines
9.0 KiB
Markdown
143 lines
9.0 KiB
Markdown
# Backups — restic on external drive (DEPRECATED — superseded)
|
|
|
|
> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on
|
|
> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap
|
|
> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been
|
|
> **DISABLED since 2026-04-22** and is not coming back in its old form.
|
|
|
|
## Current backup — rclone → Proton Drive (LXC 132)
|
|
|
|
- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library`
|
|
mounted **read-only**.
|
|
- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the
|
|
folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`.
|
|
- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`).
|
|
- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth).
|
|
- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay).
|
|
- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`.
|
|
- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent
|
|
copy), but strictly better than the previous "no off-host copy at all."
|
|
|
|
See [132-rclone](../containers/132-rclone.md) for the full design.
|
|
|
|
---
|
|
|
|
## Legacy — restic on external drive (DISABLED 2026-04-22)
|
|
|
|
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
|
|
|
## Status
|
|
|
|
**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
|
|
- `backup-library@homecloud.timer`
|
|
- `backup-library@images.timer`
|
|
- `backup-library@small.timer`
|
|
- `backup-library-check.timer`
|
|
|
|
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
|
|
|
|
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
|
|
|
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
|
|
|
|
## Design
|
|
|
|
Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.
|
|
|
|
Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.
|
|
|
|
## Components
|
|
|
|
- **Repo:** `dtoro/backup-library`
|
|
- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
|
|
- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
|
|
- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
|
|
- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.
|
|
|
|
## Mount-on-demand
|
|
|
|
`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
|
|
|
|
## UAS blacklist
|
|
|
|
`/etc/modprobe.d/usb-storage-quirks.conf`:
|
|
```
|
|
options usb-storage quirks=090c:2320:u
|
|
```
|
|
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.
|
|
|
|
## Schedule
|
|
|
|
Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
|
|
|
|
| Timer | When | Include list | Approx size |
|
|
| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
|
|
| `backup-library@homecloud.timer` | day 1 / month | `/etc/restic/include-homecloud.list` | ~315 G |
|
|
| `backup-library@images.timer` | day 10 / month | `/etc/restic/include-images.list` | ~103 G |
|
|
| `backup-library@small.timer` | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |
|
|
|
|
Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.
|
|
|
|
Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).
|
|
|
|
Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.
|
|
|
|
## Thermal caps
|
|
|
|
Baked into the systemd units:
|
|
- `IOReadBandwidthMax=/mnt/library 50M`
|
|
- `IOWriteBandwidthMax=/mnt/backup 30M`
|
|
- `--read-concurrency=1` on restic.
|
|
|
|
## Wrapper
|
|
|
|
`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.
|
|
|
|
## Notifications
|
|
|
|
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
|
|
|
|
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
|
|
|
|
## Recovery
|
|
|
|
Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.
|
|
|
|
## Known SPOF
|
|
|
|
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding.
|
|
|
|
## Drive history
|
|
|
|
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
|
|
|
|
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
|
|
|
## Thermal monitoring
|
|
|
|
Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).
|
|
|
|
## Related
|
|
- [Hubris host](../hosts/hubris.md)
|
|
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
|
- [Monitoring](monitoring.md)
|
|
- [Auto-deploy](auto-deploy.md)
|
|
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
|
|
|
## Changelog
|
|
|
|
### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)
|
|
Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".
|
|
|
|
### 2026-04-28 — wiki entry created
|
|
Initial documentation. Status remains DISABLED.
|
|
|
|
### 2026-04-22 — DISABLED
|
|
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
|
|
|
|
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
|
|
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
|
|
|
|
### 2026-04-20 — deployed; redesigned for thermal-gentleness
|
|
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.
|