The recurring silent-freeze incidents on LXC 132 were rclone-rcd.service
getting OOM-killed under the original 1 GiB allocation, not a protondrive
backend quirk as first suspected. journalctl confirmed the OOM kill at the
exact freeze point. Bumped LXC memory to 2 GiB (live, no reboot) and the
full folder set (cloud/documents/repos) completed cleanly afterward.
Also documents two watchdog bugs found while chasing this: a wrong
stats-group key that made a healthy sync look falsely frozen, and a
blocking systemctl restart that caused the watchdog to silently disable
itself after firing once. Both fixed; watchdog kept as a safety net.
Two silent stalls hit in LXC 132's first 24h of real traffic: rclone's own
--timeout didn't catch a protondrive-specific hang (transfer at 100%, zero
bytes/errors/retries for hours). Added a 5-min watchdog timer that restarts
rclone-backup.service if transferred bytes are frozen for 15+ min. Also
found and fixed a monitoring bug in the runner (wrong stats-group key) that
made a healthy sync look falsely stalled for 22h in its own log.
New off-host backup job replacing the disabled restic-on-USB backup: LXC 132
`rclone` mirrors selected /mnt/library folders to Proton Drive (plain rclone
sync, Proton's built-in E2E, no crypt overlay) on a monthly timer, with
rclone's Web GUI for LAN-only browsing/ad-hoc runs and live job status.
- containers/132-rclone.md: full design, Proton auth gotcha (TOTP secret vs
live code), pct exec PATH gotcha, rc-API job-visibility runner rewrite,
selected folder set (cloud/documents/repos), deferred tracked-repo note.
- infrastructure/backups.md: restic-on-USB marked DEPRECATED/superseded,
leads with the new job now.
- containers/index.md, README.md, infrastructure/media-permissions.md:
register the new container.
Every other mutating subcommand (secret, refresh-creds, client add/remove)
already re-execs via sudo only when os.geteuid() != 0. cmd_sync was the
one exception, calling sudo unconditionally — fails with "No such file or
directory: 'sudo'" on minimal root-only images (no sudo binary at all),
hit live running `homelab sync` on strong over root SSH.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Standard baseline for an enrolled workstation-class client, matching
mac-mini/republic-laptop/etc: the bootstrap decrypt-test secret plus
the write-scoped Gitea PAT so strong can push to the wiki repo on its
own (homelab client add/remove, wiki edits from that host).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
strong now has its own age key (issued over LAN via --no-mesh),
pubkey recorded in inventory.yaml. Not yet granted to any secrets
file - that's a separate decision.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Running bootstrap without --no-secrets always tried to install and
connect Netbird, even when the mesh-check right after it already knows
how to fall back to plain LAN reachability. On a host nobody's watching
interactively (e.g. driven over SSH), this hangs forever at the
device-code prompt — hit live on strong, had to kill the stuck
`netbird up` process manually. --no-mesh skips netbird install/up while
still allowing the existing LAN-fallback path to satisfy secrets
issuance.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Discovered live re-running bootstrap on strong for secrets issuance:
apt-get install sops fails outright (no such Debian package — matches
what agent-enrollment.md's manual-install recipe already does, fetching
the binary from GitHub releases instead of a package manager). dnf would
have the same problem. Added install_sops_binary(), used on both the
dnf and apt paths; Darwin still installs via brew, which does carry sops.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The pipx/mcp-CLI step and the Hermes goose installer both called
`sudo -u <user> ...` unconditionally. On minimal Linux images reached
via `ssh root@host` (no SUDO_USER, and often no `sudo` binary at all —
seen live on strong), this failed with "sudo: command not found" and
silently no-opped the mcp CLI install. Added a run_as() helper that
only shells out to sudo when there's a real invoking user distinct
from root.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
bootstrap.sh --no-secrets ran clean: sync timer, homelab CLI, and
AGENTS.md are live on strong. Noted two follow-ups: secrets issuance
is reachable over plain LAN (mesh: lan) so age-key enrollment doesn't
actually need Netbird, and bootstrap's pipx/mcp-CLI step silently no-ops
when run as root over SSH (missing `sudo` binary).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it
to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice
yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed
the wiki/inventory identity from ludo-mini to strong to match the OS/
cluster hostname, since bootstrap's client-enrollment lookup depends on
that match. Also regenerated hosts/grimmory.yaml, which was missing from
git despite being referenced by inventory.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Fix subscription path to /budget/subscriptions (not /subscriptions/)
- Document RRULE rejection; use cycle_interval instead
- Add GET /budget/ vs ?month= behavior note
- Add full category/payment-method ID tables
- Add Cookie Share recommendation: 2,650 €/month based on 6-mo analysis
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Step-by-step procedure to import N26 CSV transactions into Yuvomi's
Budget and Subscriptions modules using the yuvomi-mcp tools.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fills in the age public key issued by secrets-issuance after
homelab client enrollment completed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All LXCs use ip=<addr>/24 in pve net0 config directly. No Fritz!Box
lease needed. Updated container doc and migration runbook accordingly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OIDC config migrated from Booklore DB dump — Confidential client (not
PKCE), credentials intact. offline_access scope added to Authentik
provider. Backchannel logout URL set to permanent IP 192.168.8.213.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
LXC 126 stopped and destroyed on hubris. Remove all live references:
inventory, container doc, host file, README, containers index, auto-deploy
pipeline, DNS entry, SSH access table, nfs-export mount list.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of the provision-time 504s: netbird home-lab-network (192.168.8.0/24)
had no active routing peer — mac-mini routing peer's netbird daemon was down, so
all home-backed public services (artifacto/blog/trmnl) 504'd at the VPS edge.
netbird up on mac-mini restored it; verified trmnl public 200/401, artifacto 200.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LXC 128 trmnl hosts the TRMNL plugins middleware (dtoro/terminalito), polled by
TRMNL cloud. trmnl-plugins.service on :9851; Caddy block + LE cert; VPS traefik
router trmnl-public + cert mirror. Public path pending VPS<->home netbird route
recovery (was "No networks available" at provision time, artifacto/blog 504 too).
LAN Technitium record + SOPS enrollment + Google/MVG creds pending. Plan -> In Progress.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Caddyfile updated on caddy LXC with new backend IPs
- inventory.yaml updated to reflect current DHCP assignments
- hosts/{paperless,haos}.yaml regenerated from inventory
