docs(rclone): document protondrive silent-stall incident + watchdog

Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.

containers/132-rclone.md

@@ -97,6 +97,32 @@ gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
 [agent-enrollment.md troubleshooting](../operations/agent-enrollment.md#troubleshooting) if it recurs
 on future LXC bootstraps.
 
+## Known issue: silent protondrive stalls + watchdog
+
+The protondrive backend has been observed (twice in the first 24h) to hang a transfer at 100% (or
+mid-percentage) with **zero bytes, zero errors, zero retries** for hours — no timeout ever fires,
+including `--timeout 5m`/`--contimeout 30s` set via the rc `_config` (confirmed applied via
+`options/get`, made no difference). Signature: `core/stats` `transferring` list shows the exact same
+byte counts across repeated polls; `journalctl -u rclone-rcd.service` goes completely silent (no
+new lines at all) once it happens. The only reliable fix found is **killing and restarting**
+(`systemctl restart rclone-backup.service`) — rclone sync is idempotent on resume, so already-
+uploaded bytes aren't re-transferred.
+
+**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
+`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
+haven't moved for 15 minutes, it restarts the service automatically. State kept in
+`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running so a stale
+timestamp doesn't cause a false trigger next time it starts.
+
+**Runner monitoring bug (fixed 2026-07-03):** the runner's per-job progress polling queried
+`core/stats` under group key `job/<jobid>`, but rclone actually tracks stats under whatever `_group`
+name the job was submitted with. This silently returned all-zero stats the entire time, making a
+perfectly healthy sync look stalled at 0 bytes for 22+ hours in the per-run log — the real progress
+was only visible via `core/stats` with no group filter (global) or a `job/list`+`job/status` cross-
+check. Fixed by using the same `group` variable throughout. **Lesson: distrust the per-run log's
+"progress bytes=" line at a glance during this incident window; cross-check with unfiltered
+`core/stats` before concluding a stall is real.**
+
 ## Related
 
 - [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
@@ -106,6 +132,18 @@ on future LXC bootstraps.
 
 ## Changelog
 
+### 2026-07-03 — two silent protondrive stalls hit; watchdog added; monitoring bug fixed
+
+The `cloud` sync stalled silently twice in its first ~24h (see "Known issue" above) — once
+pre-timeout-fix (~6.5h with zero progress before being caught), once post-fix (~22h, but that
+second one turned out to be **partly a false alarm**: a bug in the runner's stats-group key made a
+healthy, actively-transferring sync (real progress 48.6G → 62.4G confirmed via unfiltered
+`core/stats`) look completely frozen in its own log. Fixed the group-key bug, then caught and
+confirmed a **second, genuine** stall (zero rcd log activity for 15+ min, frozen byte counts) and
+restarted again. Added `rclone-backup-watchdog.timer`/`.service` (5-min interval, 15-min stall
+threshold) so future stalls auto-recover without manual intervention. Total transferred as of this
+entry: ~62.4 GB of the ~145 GB selected set.
+
 ### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
 
 The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web