db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion

- Migrations 010 (content_hash) + 011 (search tsvector column) - new: internal/knowledge/seed.go — knowledge seed ingest engine - new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge - wire knowledge ingest into oikos seed pipeline - convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml - archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/ - delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py - remove empty knowledge/ directory tree
2026-07-07 20:22:30 +02:00
parent b2bfa26f64
commit 6b75f7302d
125 changed files with 7557 additions and 1938 deletions
--- a/archive/knowledge/containers/132-rclone.md
+++ b/archive/knowledge/containers/132-rclone.md
@@ -0,0 +1,199 @@
+# 132 — `rclone`
+
+Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive**
+with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing
+and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md).
+
+Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container,
+so this landed on **132**.)
+
+## At a glance
+
+- **Hostname:** `rclone`
+- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik)
+- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree,
+  incl. `homecloud/` and `documents/`, regardless of owner)
+- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03
+  after `rclone-rcd.service` was OOM-killed under real load (see "Known issue" below)
+- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup
+  job must never be able to write into the library
+- **Public hostname:** none — the UI is **LAN-only, no auth** (by design)
+
+## Service / port map
+
+| Service | Listen | Notes |
+|---------|--------|-------|
+| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status |
+| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) |
+
+## Backup design
+
+- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**).
+- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay
+  browsable in Proton's web UI).
+- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line
+  (`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other
+  disks once bind-mounted into this LXC.
+- **Path mapping:** source `S` → `proton:library-backup/<S without leading slash>`
+  (e.g. `/mnt/library/notes` → `proton:library-backup/mnt/library/notes`).
+- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept
+  the path stable) — no arg = every enabled line. Submits each folder as an **async job through the
+  rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`),
+  so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on
+  Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The
+  rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with
+  query params 404s.
+- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/<safe>-<ts>.log`; one-line
+  JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`.
+- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today;
+  **TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md).
+
+## rclone + Proton Drive
+
+- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present
+  (`rclone v1.74.3`).
+- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600.
+  **This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never
+  commit it.** Escrow the Proton account creds in the password manager.
+- **Config gotchas** (from rclone docs/forum):
+  - Log into Proton via a **browser at least once** first, or key generation fails.
+  - For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates
+    codes; obscure with `rclone obscure`.
+  - Passwords with **extended-ASCII** characters are known to break auth.
+  - Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner).
+- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted
+  to this LXC's age key, so the remote can be rebuilt after a re-provision.
+
+## The UI (rclone Web GUI)
+
+`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first
+start). Reach it at **http://192.168.8.214:5572** on the LAN.
+
+> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data —
+> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is
+> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik.
+
+## Tracked config (deferred)
+
+**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files
+directly on the LXC — fully functional, just not version-controlled or auto-deployed. A
+`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and
+staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like
+[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation.
+See [auto-deploy](../infrastructure/auto-deploy.md).
+
+**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G),
+`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as
+a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free
+after this set.
+
+## Enrollment gotcha: `pct exec` PATH
+
+`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so
+`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own
+`command -v sops` post-install check failed under `pct exec` even though the binary installed fine.
+Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
+rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
+gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
+[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
+on future LXC bootstraps.
+
+## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
+
+What looked like repeated "protondrive silently stalls" was actually **`rclone-rcd.service` (the rc
+API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting
+OOM-killed** under the original 1 GiB RAM allocation — `journalctl` confirms
+`A process of this unit has been killed by the OOM killer` at the exact moment a transfer had
+"frozen." systemd's own `Restart=on-failure` (5s) auto-respawns it, but every in-flight job's state
+is lost on the kill, which looked identical to a silent backend hang from the outside (frozen
+`core/stats`, no new log lines). **Fix: bumped the LXC's memory to 2 GiB** (`pct set 132 -memory
+2048` — applies live via the host cgroup, confirmed via `cat /sys/fs/cgroup/lxc/132/memory.max` on
+hubris, no container reboot needed). After the bump, the full folder set (`cloud` 287G, `documents`
+249M, `repos` 83M) completed cleanly with no further kills.
+
+**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
+`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
+haven't moved for 15 minutes, it restarts both `rclone-rcd.service` (clears any stuck/orphaned job —
+this is the actual daemon holding the work, not the thin wrapper) and then `rclone-backup.service`
+(`--no-block`, load-bearing — see below). Kept as a safety net even after the RAM fix, in case
+memory pressure returns under a larger folder set later. State kept in
+`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running.
+
+**Two watchdog design bugs found and fixed while chasing this (2026-07-03):**
+1. **Wrong stats-group key.** Per-job progress polling queried `core/stats` under `job/<jobid>`,
+   but rclone tracks stats under whatever `_group` name the job was submitted with. Made a perfectly
+   healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same
+   `group` variable consistently. **Lesson: distrust the per-run log's "progress bytes=" line during
+   an incident; cross-check with unfiltered `core/stats` first.**
+2. **Watchdog restarted only the thin wrapper, and blocked doing it.** The actual `rclone sync` work
+   runs inside `rclone-rcd.service`, not `rclone-backup.service` — restarting the wrapper alone left
+   any stuck job orphaned inside `rcd` while a new wrapper submitted a duplicate job on top. Worse,
+   `systemctl restart rclone-backup.service` (no `--no-block`) blocks until the *new* invocation's
+   long-running `ExecStart` exits — which could be hours — so the watchdog's own oneshot service
+   never logged "Finished," and `OnUnitActiveSec` (which schedules relative to the previous run
+   *finishing*) never fired again. The watchdog silently disabled itself after exactly one use.
+   Fixed: restart `rclone-rcd.service` first, then `rclone-backup.service` with `--no-block`.
+
+## Related
+
+- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
+- [Hubris host](../hosts/hubris.md) — owns `/mnt/library`
+- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library`
+- [Containers index](index.md)
+
+## Changelog
+
+### 2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed
+
+What looked like repeated silent "protondrive stalls" turned out to be **`rclone-rcd.service`
+getting OOM-killed** under the original 1 GiB RAM allocation — confirmed via
+`journalctl -u rclone-rcd.service` showing `killed by the OOM killer` at the exact freeze point.
+Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: `cloud` (287G) completed cleanly
+(exit 0), `documents` (249M) completed with 1 minor error (259.7 MB transferred), `repos` (83M)
+completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong
+stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a
+blocking `systemctl restart`, causing it to silently disable itself after one use) — see "Known
+issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM
+bump addresses the actual root cause.
+
+### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
+
+The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web
+GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote
+`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with
+`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling
+`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this
+rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line
+as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running,
+completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather
+than truncating in place, specifically so it wouldn't risk corrupting the still-running original
+`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was
+unaffected, kept running to completion under the old in-memory script content). The already-running
+scheduled sync from before this change is a standalone process and won't retroactively appear in the
+GUI; every run after this point will.
+
+### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync
+
+Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was
+entered instead of the TOTP secret — reconfigured with the secret, auth now works
+(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes`
+(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not
+an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB
+plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection:
+`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the
+actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early
+rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by
+choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays
+staged at `/root/rclone-repo` for later.
+
+### 2026-07-01 — provisioned; enrolled
+
+LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3
+installed from the official binary (`protondrive` backend present). Runner + monthly timer +
+`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into
+homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared
+secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST`
+updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive
+remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain
+operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB
+backup deprecated in the same change.