diff --git a/README.md b/README.md index 8dafc87..388b506 100644 --- a/README.md +++ b/README.md @@ -32,6 +32,7 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links: | 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) | | 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS | | 130 | [grimmory](containers/130-grimmory.md) | 192.168.8.213 | Digital library (Grimmory — fork of Booklore) | +| 132 | [rclone](containers/132-rclone.md) | 192.168.8.214 | Off-host backup → Proton Drive (rclone + Web GUI) | ### Cross-cutting infrastructure - [DNS — split-horizon](infrastructure/dns.md) @@ -40,7 +41,7 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links: - [Monitoring — Hermes health watchdog](infrastructure/monitoring.md) - [Media permissions — `media` GID 10000](infrastructure/media-permissions.md) - [SSH access](infrastructure/ssh-access.md) -- [Backups — restic on external drive (disabled)](infrastructure/backups.md) +- [Backups — rclone → Proton Drive (LXC 132); restic-on-USB deprecated](infrastructure/backups.md) - [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md) - [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md) - [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance diff --git a/containers/132-rclone.md b/containers/132-rclone.md new file mode 100644 index 0000000..9bbc537 --- /dev/null +++ b/containers/132-rclone.md @@ -0,0 +1,149 @@ +# 132 — `rclone` + +Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive** +with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing +and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md). + +Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container, +so this landed on **132**.) + +## At a glance + +- **Hostname:** `rclone` +- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik) +- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree, + incl. `homecloud/` and `documents/`, regardless of owner) +- **Resources:** 1 core / 1 GiB RAM / 8 GiB rootfs (Debian 13) +- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup + job must never be able to write into the library +- **Public hostname:** none — the UI is **LAN-only, no auth** (by design) + +## Service / port map + +| Service | Listen | Notes | +|---------|--------|-------| +| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status | +| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) | + +## Backup design + +- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**). +- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay + browsable in Proton's web UI). +- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line + (`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other + disks once bind-mounted into this LXC. +- **Path mapping:** source `S` → `proton:library-backup/` + (e.g. `/mnt/library/notes` → `proton:library-backup/mnt/library/notes`). +- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept + the path stable) — no arg = every enabled line. Submits each folder as an **async job through the + rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`), + so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on + Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The + rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with + query params 404s. +- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/-.log`; one-line + JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`. +- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today; + **TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md). + +## rclone + Proton Drive + +- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present + (`rclone v1.74.3`). +- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600. + **This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never + commit it.** Escrow the Proton account creds in the password manager. +- **Config gotchas** (from rclone docs/forum): + - Log into Proton via a **browser at least once** first, or key generation fails. + - For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates + codes; obscure with `rclone obscure`. + - Passwords with **extended-ASCII** characters are known to break auth. + - Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner). +- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted + to this LXC's age key, so the remote can be rebuilt after a re-provision. + +## The UI (rclone Web GUI) + +`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first +start). Reach it at **http://192.168.8.214:5572** on the LAN. + +> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data — +> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is +> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik. + +## Tracked config (deferred) + +**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files +directly on the LXC — fully functional, just not version-controlled or auto-deployed. A +`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and +staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like +[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation. +See [auto-deploy](../infrastructure/auto-deploy.md). + +**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G), +`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as +a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free +after this set. + +## Enrollment gotcha: `pct exec` PATH + +`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so +`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own +`command -v sops` post-install check failed under `pct exec` even though the binary installed fine. +Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH), +rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups +gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to +[agent-enrollment.md troubleshooting](../operations/agent-enrollment.md#troubleshooting) if it recurs +on future LXC bootstraps. + +## Related + +- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup +- [Hubris host](../hosts/hubris.md) — owns `/mnt/library` +- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library` +- [Containers index](index.md) + +## Changelog + +### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility) + +The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web +GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote +`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with +`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling +`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this +rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line +as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running, +completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather +than truncating in place, specifically so it wouldn't risk corrupting the still-running original +`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was +unaffected, kept running to completion under the old in-memory script content). The already-running +scheduled sync from before this change is a standalone process and won't retroactively appear in the +GUI; every run after this point will. + +### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync + +Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was +entered instead of the TOTP secret — reconfigured with the secret, auth now works +(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes` +(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not +an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB +plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection: +`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the +actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early +rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by +choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays +staged at `/root/rclone-repo` for later. + +### 2026-07-01 — provisioned; enrolled + +LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3 +installed from the official binary (`protondrive` backend present). Runner + monthly timer + +`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into +homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared +secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST` +updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive +remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain +operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB +backup deprecated in the same change. diff --git a/containers/index.md b/containers/index.md index b0681ca..0fffc88 100644 --- a/containers/index.md +++ b/containers/index.md @@ -18,6 +18,7 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per | 128 | [trmnl](128-trmnl.md) | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running | | 129 | [house](129-house.md) | 192.168.8.212 | unpriv | 1 | 1344 MiB | 8 GiB | — | `house.hubris.network` | running | | 130 | [grimmory](130-grimmory.md) | 192.168.8.213 | priv | 1 | 2 GiB | 16 GiB | `/mnt/library` | `books.hubris.network` | running | +| 132 | [rclone](132-rclone.md) | 192.168.8.214 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` (**ro**) | — (LAN-only UI `:5572`) | running | ## Recently destroyed (kept for archaeology) diff --git a/infrastructure/backups.md b/infrastructure/backups.md index 895a33d..eba2dd7 100644 --- a/infrastructure/backups.md +++ b/infrastructure/backups.md @@ -1,4 +1,28 @@ -# Backups — restic on external drive (DISABLED) +# Backups — restic on external drive (DEPRECATED — superseded) + +> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on +> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap +> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been +> **DISABLED since 2026-04-22** and is not coming back in its old form. + +## Current backup — rclone → Proton Drive (LXC 132) + +- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library` + mounted **read-only**. +- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the + folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`. +- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`). +- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth). +- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay). +- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`. +- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent + copy), but strictly better than the previous "no off-host copy at all." + +See [132-rclone](../containers/132-rclone.md) for the full design. + +--- + +## Legacy — restic on external drive (DISABLED 2026-04-22) Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md). @@ -81,7 +105,7 @@ Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/R ## Known SPOF -Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented. +Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding. ## Drive history @@ -102,6 +126,9 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213 ## Changelog +### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132) +Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy". + ### 2026-04-28 — wiki entry created Initial documentation. Status remains DISABLED. diff --git a/infrastructure/media-permissions.md b/infrastructure/media-permissions.md index a92808e..52b5a56 100644 --- a/infrastructure/media-permissions.md +++ b/infrastructure/media-permissions.md @@ -62,6 +62,7 @@ Every LXC that mounts `/mnt/library` participates in a shared `media` group with | 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data | | 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr | | 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed | +| 132 | [rclone](../containers/132-rclone.md) | priv | **read-only** mount; runs as root → reads all subtrees. No media group needed | > Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).