From 6669feafdc819f9854f323bb7ee989e633a1e107 Mon Sep 17 00:00:00 2001 From: dtoro Date: Fri, 3 Jul 2026 12:32:53 +0200 Subject: [PATCH] docs(rclone): document protondrive silent-stall incident + watchdog Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own --timeout didn't catch a protondrive-specific hang (transfer at 100%, zero bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts rclone-backup.service if transferred bytes are frozen for 15+ min. Also found and fixed a monitoring bug in the runner (wrong stats-group key) that made a healthy sync look falsely stalled for 22h in its own log. --- containers/132-rclone.md | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/containers/132-rclone.md b/containers/132-rclone.md index 9bbc537..bf58753 100644 --- a/containers/132-rclone.md +++ b/containers/132-rclone.md @@ -97,6 +97,32 @@ gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to [agent-enrollment.md troubleshooting](../operations/agent-enrollment.md#troubleshooting) if it recurs on future LXC bootstraps. +## Known issue: silent protondrive stalls + watchdog + +The protondrive backend has been observed (twice in the first 24h) to hang a transfer at 100% (or +mid-percentage) with **zero bytes, zero errors, zero retries** for hours — no timeout ever fires, +including `--timeout 5m`/`--contimeout 30s` set via the rc `_config` (confirmed applied via +`options/get`, made no difference). Signature: `core/stats` `transferring` list shows the exact same +byte counts across repeated polls; `journalctl -u rclone-rcd.service` goes completely silent (no +new lines at all) once it happens. The only reliable fix found is **killing and restarting** +(`systemctl restart rclone-backup.service`) — rclone sync is idempotent on resume, so already- +uploaded bytes aren't re-transferred. + +**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if +`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API) +haven't moved for 15 minutes, it restarts the service automatically. State kept in +`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running so a stale +timestamp doesn't cause a false trigger next time it starts. + +**Runner monitoring bug (fixed 2026-07-03):** the runner's per-job progress polling queried +`core/stats` under group key `job/`, but rclone actually tracks stats under whatever `_group` +name the job was submitted with. This silently returned all-zero stats the entire time, making a +perfectly healthy sync look stalled at 0 bytes for 22+ hours in the per-run log — the real progress +was only visible via `core/stats` with no group filter (global) or a `job/list`+`job/status` cross- +check. Fixed by using the same `group` variable throughout. **Lesson: distrust the per-run log's +"progress bytes=" line at a glance during this incident window; cross-check with unfiltered +`core/stats` before concluding a stall is real.** + ## Related - [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup @@ -106,6 +132,18 @@ on future LXC bootstraps. ## Changelog +### 2026-07-03 — two silent protondrive stalls hit; watchdog added; monitoring bug fixed + +The `cloud` sync stalled silently twice in its first ~24h (see "Known issue" above) — once +pre-timeout-fix (~6.5h with zero progress before being caught), once post-fix (~22h, but that +second one turned out to be **partly a false alarm**: a bug in the runner's stats-group key made a +healthy, actively-transferring sync (real progress 48.6G → 62.4G confirmed via unfiltered +`core/stats`) look completely frozen in its own log. Fixed the group-key bug, then caught and +confirmed a **second, genuine** stall (zero rcd log activity for 15+ min, frozen byte counts) and +restarted again. Added `rclone-backup-watchdog.timer`/`.service` (5-min interval, 15-min stall +threshold) so future stalls auto-recover without manual intervention. Total transferred as of this +entry: ~62.4 GB of the ~145 GB selected set. + ### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility) The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web