oikos/knowledge/wiki/containers/132-rclone.md

132 — rclone

Off-host backup appliance. Mirrors selected /mnt/library folders to Proton Drive with a plain rclone sync (monthly), and serves rclone's Web GUI on the LAN for browsing and ad-hoc runs. Replaces the disabled restic-on-USB job — see backups.

Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented teddycloud container, so this landed on 132.)

At a glance

• Hostname: rclone
• IP: 192.168.8.214 (static, set in PVE net0 config — same pattern as grimmory/authentik)
• Privilege: privileged (root in-container = host root → reads every /mnt/library subtree, incl. homecloud/ and documents/, regardless of owner)
• Resources: 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03 after rclone-rcd.service was OOM-killed under real load (see "Known issue" below)
• Mounts: /mnt/library read-only (mp0: /mnt/library,mp=/mnt/library,ro=1) — a backup job must never be able to write into the library
• Public hostname: none — the UI is LAN-only, no auth (by design)

Service / port map

Service	Listen	Notes
rclone Web GUI (rcd)	192.168.8.214:5572	rclone-rcd.service, --rc-no-auth, LAN-only. Browse /mnt/library + proton:, run ad-hoc syncs, live job status
monthly mirror	—	rclone-backup.service + .timer (OnCalendar=*-*-01 03:00)

Backup design

• Mode: plain mirror — rclone sync (Proton mirrors local; deletions propagate; no versioning).
• Encryption: Proton Drive's built-in E2E only (no rclone crypt overlay → files stay browsable in Proton's web UI).
• Selected set: /etc/rclone-backup/folders.list — one absolute source path per line (#/blank ignored). This file is the picked set the monthly timer mirrors. Extensible to other disks once bind-mounted into this LXC.
• Path mapping: source S → proton:library-backup/<S without leading slash> (e.g. /mnt/library/notes → proton:library-backup/mnt/library/notes).
• Runner: /usr/local/sbin/rclone-backup.sh [folder ...] (Python, despite the .sh name — kept the path stable) — no arg = every enabled line. Submits each folder as an async job through the rclone rc API served by rclone-rcd.service (the same daemon backing the Web GUI on :5572), so scheduled/ad-hoc runs show up live in the GUI's Jobs panel, not just in logs. Gentle on Proton's rate limits (Transfers=4, TPSLimit=8, FastList=true via the rc _config payload). The rc API here requires POST for every call including job/status and core/stats — GET with query params 404s.
• Logs / "past runs": per-run logs in /var/log/rclone-backup/<safe>-<ts>.log; one-line JSON summary per run appended to /var/log/rclone-backup/runs.jsonl.
• Failure notify: OnFailure=rclone-backup-notify@%n.service → logs to journal today; TODO wire to Hermes send_message (Matrix) per backups.

rclone + Proton Drive

• rclone installed from the official binary (not apt) so the protondrive backend is present (rclone v1.74.3).
• Remote proton: (type protondrive). Config at /root/.config/rclone/rclone.conf, mode 600. This file is a secret (holds the obscured Proton password + TOTP secret + session) — never commit it. Escrow the Proton account creds in the password manager.
• Config gotchas (from rclone docs/forum):
  • Log into Proton via a browser at least once first, or key generation fails.
  • For unattended runs, store the TOTP secret (not a 6-digit code) so rclone self-generates codes; obscure with rclone obscure.
  • Passwords with extended-ASCII characters are known to break auth.
  • Proton's API is rate-limited → keep --transfers/--tpslimit conservative (baked into the runner).
• DR escrow (pending): store the Proton creds as sops secret secrets/protondrive.yaml, granted to this LXC's age key, so the remote can be rebuilt after a re-provision.

The UI (rclone Web GUI)

rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572 (assets auto-downloaded on first start). Reach it at http://192.168.8.214:5572 on the LAN.

Security note: --rc-no-auth exposes full rclone control — including deleting remote data — to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is not publicly reachable. Harden later by adding --rc-user/--rc-pass or fronting it with Authentik.

Tracked config (deferred)

Not yet tracked. The runner, systemd units, and folders.list currently live as plain files directly on the LXC — fully functional, just not version-controlled or auto-deployed. A dtoro/rclone gitea repo package (runner, units, install.sh, webhook receiver) is pre-built and staged at /root/rclone-repo on the LXC for whenever this gets tracked (Shape A, like caddy). Gitea ALLOWED_HOST_LIST already includes 192.168.8.214 in anticipation. See auto-deploy.

Selected folders (live in /etc/rclone-backup/folders.list): /mnt/library/cloud (287G), /mnt/library/documents (249M), /mnt/library/repos (83M). /mnt/library/notes was synced once as a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free after this set.

Enrollment gotcha: pct exec PATH

pct exec (lxc-attach) does not source /etc/environment or run a login shell, so /usr/local/bin (where bootstrap installs sops) isn't on $PATH by default — bootstrap's own command -v sops post-install check failed under pct exec even though the binary installed fine. Fixed by symlinking /usr/local/bin/{sops,homelab} into /usr/bin (always on the minimal PATH), rather than relying on /etc/environment. Same category as the documented pct exec no-initgroups gotcha — worth adding to agent-enrollment.md troubleshooting if it recurs on future LXC bootstraps.

Known issue: rclone-rcd.service OOM-killed under 1 GiB RAM (root cause, resolved)

What looked like repeated "protondrive silently stalls" was actually rclone-rcd.service (the rc API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting OOM-killed under the original 1 GiB RAM allocation — journalctl confirms A process of this unit has been killed by the OOM killer at the exact moment a transfer had "frozen." systemd's own Restart=on-failure (5s) auto-respawns it, but every in-flight job's state is lost on the kill, which looked identical to a silent backend hang from the outside (frozen core/stats, no new log lines). Fix: bumped the LXC's memory to 2 GiB (pct set 132 -memory 2048 — applies live via the host cgroup, confirmed via cat /sys/fs/cgroup/lxc/132/memory.max on hubris, no container reboot needed). After the bump, the full folder set (cloud 287G, documents 249M, repos 83M) completed cleanly with no further kills.

rclone-backup-watchdog.timer (every 5 min) → rclone-backup-watchdog.sh: if rclone-backup.service is active but total transferred bytes (global core/stats on the rc API) haven't moved for 15 minutes, it restarts both rclone-rcd.service (clears any stuck/orphaned job — this is the actual daemon holding the work, not the thin wrapper) and then rclone-backup.service (--no-block, load-bearing — see below). Kept as a safety net even after the RAM fix, in case memory pressure returns under a larger folder set later. State kept in /var/lib/rclone-backup/watchdog-state.json, cleared whenever the service isn't running.

Two watchdog design bugs found and fixed while chasing this (2026-07-03):

1. Wrong stats-group key. Per-job progress polling queried core/stats under job/<jobid>, but rclone tracks stats under whatever _group name the job was submitted with. Made a perfectly healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same group variable consistently. Lesson: distrust the per-run log's "progress bytes=" line during an incident; cross-check with unfiltered core/stats first.
2. Watchdog restarted only the thin wrapper, and blocked doing it. The actual rclone sync work runs inside rclone-rcd.service, not rclone-backup.service — restarting the wrapper alone left any stuck job orphaned inside rcd while a new wrapper submitted a duplicate job on top. Worse, systemctl restart rclone-backup.service (no --no-block) blocks until the new invocation's long-running ExecStart exits — which could be hours — so the watchdog's own oneshot service never logged "Finished," and OnUnitActiveSec (which schedules relative to the previous run finishing) never fired again. The watchdog silently disabled itself after exactly one use. Fixed: restart rclone-rcd.service first, then rclone-backup.service with --no-block.

Related

• Backups — this job supersedes the disabled restic-on-USB backup
• Hubris host — owns /mnt/library
• Media permissions — read-only consumer of /mnt/library
• Containers index

Changelog

2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed

What looked like repeated silent "protondrive stalls" turned out to be rclone-rcd.service getting OOM-killed under the original 1 GiB RAM allocation — confirmed via journalctl -u rclone-rcd.service showing killed by the OOM killer at the exact freeze point. Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: cloud (287G) completed cleanly (exit 0), documents (249M) completed with 1 minor error (259.7 MB transferred), repos (83M) completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a blocking systemctl restart, causing it to silently disable itself after one use) — see "Known issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM bump addresses the actual root cause.

2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)

The original runner (rclone sync invoked as a standalone CLI subprocess) was invisible to the Web GUI's Jobs panel — the GUI only tracks work submitted through its own rcd process. Rewrote /usr/local/sbin/rclone-backup.sh in Python, submitting each folder via POST /sync/sync with _async: true against http://127.0.0.1:5572 (the running rclone-rcd.service), then polling POST /job/status + POST /core/stats (both must be POST — GET-with-querystring 404s on this rc API) until finished, logging periodic progress snapshots and the same runs.jsonl summary line as before. Verified live: submitted job visible in POST /job/list's runningIds while running, completed cleanly (success: true) once done. Deployed via atomic rename (write-then-mv) rather than truncating in place, specifically so it wouldn't risk corrupting the still-running original cloud+documents+repos sync mid-flight (verified after the fact: that sync's bash process was unaffected, kept running to completion under the old in-memory script content). The already-running scheduled sync from before this change is a standalone process and won't retroactively appear in the GUI; every run after this point will.

2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync

Initial rclone config failed 2FA (422 ... auth/v4/2fa) because a live 6-digit TOTP code was entered instead of the TOTP secret — reconfigured with the secret, auth now works (rclone lsd proton: lists the Drive). Verified end-to-end with a real sync of /mnt/library/notes (219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB plan, 1.945 TiB free) before enabling a large folder. folders.list set to the real selection: cloud (287G), documents (249M), repos (83M); a full sync of that set was kicked off via the actual rclone-backup.service unit (not an ad-hoc call) to validate the real monthly path early rather than waiting for the Aug 1 timer. Tracked-repo step (dtoro/rclone on gitea) deferred by choice — runner/units/folders.list remain plain files on the LXC for now; the repo package stays staged at /root/rclone-repo for later.

2026-07-01 — provisioned; enrolled

LXC 132 created (Debian 13, privileged, 192.168.8.214, /mnt/library read-only). rclone v1.74.3 installed from the official binary (protondrive backend present). Runner + monthly timer + folders.list deployed; rclone Web GUI (rcd, LAN-only no-auth) live on :5572. Enrolled into homelab-context (--no-mesh, LAN-only issuance): age key issued, inventory finalized, shared secrets granted, homelab whoami + homelab secret hello verified. Gitea ALLOWED_HOST_LIST updated to include 192.168.8.214. Hit and fixed a pct exec PATH gotcha (see below). Proton Drive remote, dtoro/rclone tracked repo + webhook, and the secrets/protondrive.yaml escrow remain operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB backup deprecated in the same change.