investigations: add moonlight/sunshine WiFi jitter report + index entry

- New investigation doc: 2026-06-03-moonlight-sunshine-wifi-jitter.md - Updated investigations/index.md with link and status - Added .gitignore for .DS_Store - Saved Hermes planning docs from recent sessions
2026-06-04 21:05:20 +02:00
parent e32b21072d
commit 728e501435
5 changed files with 620 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1 @@
+.DS_Store
--- a/.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md
+++ b/.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md
@@ -0,0 +1,277 @@
+# Plan: Migrate library SSD to ludo-mini + Proxmox gaming/media server
+
+## Goal
+
+Split the homelab into two Proxmox hosts:
+
+| Host | Role | Storage |
+|------|------|---------|
+| **hubris** | Core services (reverse-proxy, SSO, Matrix, git, documents, HA) | SSD 1 — boot + LXC rootfs (unchanged) |
+| **ludo-mini** | Gaming server + media/library services | SSD 2 — Samsung 990 EVO Plus 4 TB (moved from hubris) |
+
+The library SSD physically moves from hubris to ludo-mini. hubris LXCs that still need `/mnt/library` access it over NFS from ludo-mini.
+
+## Current state
+
+### hubris hardware
+- GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS, 12 vCPU, ~28 GiB RAM
+- 2× Samsung 990 EVO Plus NVMe:
+  - nvme0: `local` (95G) + `local-lvm` (856G) — boot, ISOs, LXC rootfs
+  - nvme1: `library` LVM (3.7T) — `/mnt/library` ext4 via `/dev/mapper/library-library`
+
+### LXCs binding `/mnt/library` (host-level bind-mount)
+| ID | Name | Role | I/O profile |
+|----|------|------|-------------|
+| 101 | jellyfin | Media streaming | Read-heavy, sequential |
+| 103 | paperless | Document archive | Mixed, OCR writes |
+| 104 | gitea | Git server | Mixed, lots of small files |
+| 105 | apps | Docker (booklore, audiobookshelf, artifacto, MCP) | Mixed, depends on container |
+| 114 | nextcloud | File sync | Mixed, WebDAV |
+| 119 | sophia | Workshop | Low I/O |
+| 120 | mule-images | Photo management | Write-heavy (processing), iGPU |
+| 122 | arriman | *arr stack + downloads | Write-heavy (downloads) |
+| 126 | plato | App (sub-mount: `/mnt/library/documents/plato`) | Light |
+
+### NFS export chain (for VM 100 zimaos)
+```
+/mnt/library (ext4, host) → bind-mount → LXC 102 (nfs-export) → NFSv4 → VM 100 (zimaos)
+```
+
+### ludo-mini current
+- Linux workstation, wired Ethernet 2.5 Gbps, `192.168.178.181` (household LAN)
+- Runs Sunshine for game streaming
+- No Proxmox, no LVM config
+- Connected to SODOLA switch (same switch as hubris eno1)
+
+### Network topology
+```
+Fritz!Box 7590 (192.168.178.1)
+  └── SODOLA 2.5G switch
+       ├── hubris eno1 → vmbr1 (192.168.178.10)
+       │     └── routes to vmbr0 (192.168.8.0/24) — all LXCs
+       └── ludo-mini (192.168.178.181)
+```
+
+hubris routes between `192.168.8.0/24` (vmbr0) and `192.168.178.0/24` (vmbr1). So LXCs can reach ludo-mini via hubris as a router.
+
+## Key decisions
+
+### 1. Service split — what moves, what stays
+
+**Move to ludo-mini** (high I/O, benefits from data locality + GPU):
+- 101 jellyfin — media streaming, GPU transcoding
+- 120 mule-images — photo processing, iGPU passthrough
+- 122 arriman — *arr stack, downloads write to library
+
+**Stay on hubris, NFS-mount library from ludo-mini:**
+- 103 paperless — documents, moderate I/O
+- 104 gitea — git repos (small files, some I/O sensitivity but acceptable over NFS)
+- 105 apps — Docker apps, mixed workloads
+- 114 nextcloud — file sync
+- 119 sophia — workshop, light use
+- 126 plato — app, light use
+- 100 zimaos — NAS frontend, already NFS-mounted
+
+### 2. NFS architecture
+
+Instead of changing every LXC's mount config, keep the bind-mount pattern on hubris:
+
+```
+ludo-mini: /mnt/library (ext4, local NVMe)
+    │
+    └── NFSv4 export to 192.168.8.0/24
+              │
+              └── hubris host: NFS-mount at /mnt/library
+                        │
+                        └── LXCs: bind-mount /mnt/library (unchanged!)
+```
+
+This is transparent to all hubris LXCs — no container config changes needed. Only the hubris host changes from ext4 local mount to NFS mount. The LXC bind-mounts "just work" because `/mnt/library` is still at the same path on the host.
+
+### 3. Network — ludo-mini reachability from hubris LXCs
+
+LXCs on `192.168.8.0/24` reach ludo-mini (`192.168.178.181`) through hubris routing:
+- `vmbr0` → hubris kernel routing → `vmbr1` → SODOLA → ludo-mini
+- Already works (IP forwarding enabled on hubris)
+
+**Alternative (cleaner):** Add a secondary IP `192.168.8.x` on ludo-mini's physical interface so it's directly on the homelab subnet. This avoids the router hop and keeps NFS traffic off kernel forwarding path. Worth considering but not required.
+
+### 4. Gaming on ludo-mini with Proxmox
+
+ludo-mini runs Sunshine (game streaming). Under Proxmox:
+- **Option A:** Gaming VM with GPU passthrough — Sunshine + games in a VM, full GPU access
+- **Option B:** LXC with GPU device passthrough (`/dev/dri`) — lighter, shares kernel
+- **Option C:** Keep Sunshine on the Proxmox host itself (not recommended, but simplest)
+
+Option A is the cleanest for isolation. Games need a full desktop environment and GPU drivers; a VM with GPU passthrough gives them that.
+
+### 5. What about nfs-export (LXC 102)?
+
+Currently exports `/mnt/library` to zimaos. After migration:
+- If zimaos stays on hubris and accesses library via host NFS → bind-mount → LXC 102, that's triple-hop (ludo-mini → NFS → hubris → bind-mount → LXC 102 → NFS → zimaos). Terrible.
+- Better: zimaos NFS-mounts directly from ludo-mini.
+- So LXC 102 gets decommissioned (or repurposed).
+- zimaos gets a new NFS mount pointing directly at ludo-mini.
+
+## Migration phases
+
+### Phase 1 — Preparation (no downtime)
+
+1. **Document current state** on hubris:
+   - `pct list` — full container inventory
+   - `pct config <id>` for every library-mounting LXC
+   - `cat /etc/fstab` — capture the library mount line
+   - `df -h /mnt/library` — confirm space usage
+   - `lsblk -f` — UUID, filesystem
+   - Identify the exact NVMe device (`nvme1n1`)
+
+2. **Pre-flight on ludo-mini:**
+   - Confirm hardware: CPU, RAM, available M.2 slots, GPU model
+   - Confirm it can take the Samsung 990 EVO Plus (M.2 NVMe, PCIe 4.0 x4)
+   - Verify BIOS supports virtualization (VT-d/AMD-Vi for PCIe passthrough)
+   - Check: does ludo-mini have a second drive for Proxmox OS? If not, we need to partition the library SSD for Proxmox boot + library LVM, which complicates things significantly
+
+3. **Install Proxmox on ludo-mini:**
+   - Download Proxmox VE 9.x ISO
+   - Install to ludo-mini's system drive (NOT the library SSD)
+   - Configure networking: bridge for Proxmox, IP on 192.168.178.x
+   - Test: web UI accessible
+
+4. **Prepare NFS server on ludo-mini Proxmox:**
+   - Create NFS-export LXC (or serve from host — simpler for now)
+   - Prepare `/etc/exports`: `192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)`
+   - Same squash params as current nfs-export LXC 102
+
+### Phase 2 — Physical SSD move (planned downtime)
+
+1. **Graceful shutdown on hubris:**
+   - Stop all library-mounting LXCs (101, 103, 104, 105, 114, 119, 120, 122, 126)
+   - Unmount `/mnt/library` on hubris host
+   - Edit `/etc/fstab` to comment out the library mount line
+   - Power off hubris
+
+2. **Physical drive swap:**
+   - Remove Samsung 990 EVO Plus (library SSD) from hubris
+   - Install into ludo-mini M.2 slot
+   - Power on ludo-mini
+
+3. **Bring library online on ludo-mini:**
+   - Detect the new NVMe device
+   - If it's the whole device with LVM, activate the VG:
+     ```
+     vgscan && vgchange -ay library
+     mount /dev/mapper/library-library /mnt/library
+     ```
+   - Add to `/etc/fstab` for auto-mount
+   - Verify content: `ls /mnt/library` — same tree as before
+
+4. **Start NFS export on ludo-mini:**
+   - `exportfs -ra`
+   - Verify: `showmount -e <ludo-mini-ip>`
+
+### Phase 3 — Reconnect hubris LXCs
+
+1. **Power on hubris** (without library SSD — it'll boot fine, just won't mount library)
+
+2. **Mount NFS on hubris host:**
+   - Install `nfs-common` if not present
+   - Add to `/etc/fstab`:
+     ```
+     192.168.178.181:/mnt/library /mnt/library nfs rw,vers=4,soft,timeo=30,retrans=3 0 0
+     ```
+     Use `soft` to prevent hangs if ludo-mini is down; `hard` with `intr` is safer for data integrity but can block processes.
+   - `mount /mnt/library`
+   - **Verify permissions:** `ls -la /mnt/library` — should show `www-data:media` ownership for shared subtrees (same uid 33, gid 10000). The NFS all_squash guarantees this.
+
+3. **Start LXCs:**
+   - Start the LXCs that stayed on hubris (103, 104, 105, 114, 119, 126)
+   - Their bind-mounts should work — `/mnt/library` is populated via NFS
+   - Verify each service: web UIs, git clone, document access
+
+4. **Update DNS/Caddy** for services that moved:
+   - If jellyfin, arr services moved to ludo-mini, update Caddyfile to point to ludo-mini IPs
+   - Update DNS entries if needed
+
+### Phase 4 — Migrate services to ludo-mini
+
+1. **Create LXCs/VM on ludo-mini Proxmox:**
+   - 101 jellyfin — privileged LXC, mount `/mnt/library`, add `media` group
+   - 120 mule-images — privileged LXC, mount `/mnt/library` + `/dev/dri` passthrough
+   - 122 arriman — privileged LXC, mount `/mnt/library`
+
+2. **Migrate configs:**
+   - Copy LXC configs from hubris (`/etc/pve/lxc/<id>.conf`) as templates
+   - Adjust network (IPs on household subnet or Proxmox bridge)
+   - Restore app data from backups or copy over NFS
+
+3. **Gaming VM:**
+   - Create VM with GPU passthrough
+   - Pass through the dGPU for gaming performance
+   - Install Sunshine + game libraries
+   - Storage: VM disk on Proxmox storage, games on library SSD
+
+4. **Update reverse proxy:**
+   - Caddy on hubris (121): update backend IPs for jellyfin, jellyseerr, qbit, sab, mule-images → point to ludo-mini
+   - Test: `media.hubris.network` serves from ludo-mini jellyfin
+
+### Phase 5 — Cleanup
+
+1. **Decommission nfs-export LXC 102** on hubris (no longer needed)
+2. **Update zimaos (100)** — change NFS mount from `192.168.8.200` → `192.168.178.181`
+3. **Remove old LXCs** from hubris (101, 120, 122) after confirming migration works
+4. **Update inventory.yaml:**
+   - ludo-mini: `kind: proxmox-host`, add mounts/storage, add LXCs
+   - Move services from hubris to ludo-mini
+   - Remove nfs-export
+5. **Update DNS:** `nfs-export.hubris.network` → ludo-mini IP (or remove)
+6. **Run `homelab sync`** to propagate changes
+
+## Open questions / unknowns
+
+1. **Does ludo-mini have a second drive for Proxmox OS?** If not, we'd need to repartition the library SSD — carve out ~100 GB for Proxmox, then the rest for library LVM. This is risky (data loss if partitioning goes wrong) and requires a full backup first. **Alternative:** Buy a small SSD for ludo-mini's OS.
+
+2. **What GPU does ludo-mini have?** Proxmox GPU passthrough requires IOMMU support and a GPU that doesn't have the reset bug. Need to check the exact GPU model.
+
+3. **NFS performance for git (gitea)?** Git operations over NFS can be problematic (locking, stat() storms). Gitea bare repos at `/mnt/library/repos/*.git` might need testing. Worst case: move gitea's repo storage to local disk and keep `/mnt/library` for large file/LFS storage only.
+
+4. **Media permission drift.** NFS `all_squash,anonuid=33,anongid=10000` ensures all writes from hubris LXCs (over NFS) and ludo-mini LXCs (local) land as `www-data:media`. This is the same squash currently used by nfs-export (102). Should be fine.
+
+5. **ludo-mini network — add 192.168.8.x address?** Adding a secondary IP on ludo-mini's interface directly on the homelab subnet avoids routing through hubris for NFS traffic. Cleaner, but requires Proxmox bridge setup. Worth doing during Proxmox install.
+
+6. **Sunshine migration.** Currently runs on ludo-mini bare metal. After Proxmox install, it needs to run in a VM. What happens to existing Sunshine configs, game libraries, save files? Need to preserve these during the Proxmox install.
+
+7. **Backup before moving.** The library SSD holds 3.7 TB of irreplaceable data (documents, photos, repos). Restic backups are currently disabled. **Before physically moving the drive, verify the data is readable and consider doing one backup** — or at minimum, `rsync` critical directories.
+
+## Files affected
+
+| File | Change |
+|------|--------|
+| `/opt/homelab-context/inventory.yaml` | ludo-mini: workstation → proxmox-host; add LXCs, mounts; remove nfs-export; move service backends |
+| `/opt/homelab-context/hosts/hubris.md` | Remove library storage, add NFS mount note |
+| `/opt/homelab-context/hosts/ludo-mini.yaml` | Complete rewrite — Proxmox host, storage, tenants |
+| `/opt/homelab-context/containers/102-nfs-export.md` | Mark decommissioned |
+| `/opt/homelab-context/containers/index.md` | Move 101, 120, 122 to ludo-mini; remove 102 |
+| `/opt/homelab-context/infrastructure/dns.md` | Update nfs-export entry |
+| `/opt/homelab-context/infrastructure/media-permissions.md` | Note NFS squash from ludo-mini, not hubris |
+| hubris `/etc/fstab` | Replace ext4 mount with NFS mount |
+| ludo-mini `/etc/fstab` | Add library ext4 mount |
+| ludo-mini `/etc/exports` | Add NFS export config |
+| caddy (LXC 121) Caddyfile | Backend IPs for moved services |
+| DNS (LXC 107 Technitium) | Update entries for moved services |
+
+## Validation checklist
+
+- [ ] ludo-mini Proxmox web UI accessible
+- [ ] Library SSD detected and mountable on ludo-mini
+- [ ] NFS export from ludo-mini: `showmount -e <ip>` shows `/mnt/library`
+- [ ] hubris host NFS mount: `df -h /mnt/library` shows NFS, not ext4
+- [ ] hubris LXCs start and bind-mount /mnt/library (content visible)
+- [ ] gitea: `git clone` over SSH works, repos readable
+- [ ] paperless: document ingestion works, OCR processing
+- [ ] nextcloud: file sync, WebDAV
+- [ ] jellyfin: media plays from ludo-mini, transcoding works
+- [ ] arriman: downloads write to library, jellyfin picks up new media
+- [ ] mule-images: photo import and processing
+- [ ] zimaos: NFS mount from ludo-mini works, Files UI shows library
+- [ ] Sunshine: game streaming from ludo-mini VM works
+- [ ] All `*.hubris.network` services resolve and load through Caddy
--- a/.hermes/plans/2026-06-03_150000-homelab-structure-revision.md
+++ b/.hermes/plans/2026-06-03_150000-homelab-structure-revision.md
@@ -0,0 +1,271 @@
+# Homelab structure revision & improvement plan
+
+## Goal
+
+Identify structural issues in the current hubris homelab topology and propose an
+actionable improvement roadmap — DNS consolidation, monitoring gaps, backup
+recovery, mesh completion, resource rightsizing, and operational hygiene.
+
+---
+
+## Current state summary
+
+| Dimension | Status |
+|-----------|--------|
+| Hypervisor | hubris (single-node Proxmox) — also does subnet routing (192.168.8.0/24) |
+| LXCs | 14 active, 1 retired (124), 1 new (107 dns) |
+| VMs | HAOS (108), ZimaOS (100) |
+| Workstations | mac-mini (macOS), republic-laptop, ludo-mini |
+| VPS | 1 IONOS box — netbird mgmt+signal+relay, traefik, authentik, coturn |
+| Switch | SODOLA 5-Port 2.5Gbit (L2, flat) |
+| Networking | 192.168.8.0/24 internal, fr!tz box main LAN via vmbr0→vmbr1 routing |
+| Mesh | Netbird + Tailscale (migrating), ~mixed state |
+| DNS | 3 sources: Technitium (CT 107), NetBird managed zone, public IONOS |
+| Backups | DISABLED since 2026-04-22 |
+| Monitoring | claudio-monitor (host health only, no apt/docker checks yet) |
+| Agent enrollment | 3/19 hosts enrolled (hubris, apps, republic-laptop) |
+
+---
+
+## Issues identified
+
+### 1. Three overlapping DNS sources (highest risk)
+
+**Problem:** Technitium on CT 107, NetBird managed DNS zone, and the public
+IONOS wildcard all answer `*.hubris.network` queries. The dns.md doc still
+references the old dnsmasq on LXC 124 (though the change log says it moved).
+NetBird's managed DNS bypasses Technitium entirely for some app names — there
+is no single source of truth for DNS.
+
+**Risk:** Mismatched answers → services unreachable → "works on some clients
+but not others" debugging sessions. Already cost time when `auth.hubris.network`
+re-pointed to the VPS.
+
+**Proposal:**
+- Phase out NetBird managed DNS zone for `hubris.network` — Technitium is the
+  authoritative answerer for mesh & LAN clients
+- Set Technitium as the sole DNS for all LXCs (remove router DNS / Tailscale
+  MagicDNS fallbacks)
+- Document the full authoritative chain: Technitium → upstream forwarders → public
+- Track Technitium config in git (dtoro/technitium-config or equivalent)
+
+### 2. Backups disabled with no alternative (data loss risk)
+
+**Problem:** The only backup was restic to an external USB that caused host
+crashes. It was disabled 2026-04-22 as an A/B test — host stability was
+confirmed by the SODOLA migration (no more Slate AX double-NAT hangs). The
+drive is still removed.
+
+**Risk:** `/mnt/library` (~429 GB of irreplaceable data: photos, documents,
+git repos, gitea data) has no off-host copy. Single-disk failure = total loss.
+
+**Proposal:**
+- Re-evaluate the USB drive stability with the new SODOLA switch topology
+  (direct rear USB 3.0 port, no hub chain)
+- OR adopt a cloud-backed strategy: `rclone`-to-Hetzner Storage Box or
+  Backblaze B2 for the irreplaceable subset (docs, photos, gitea data)
+- OR use hubris's own `zfs send` to a second host/disk if ZFS is feasible
+- Minimum viable: at minimum restore gitea backups + sops-encrypted secrets
+  via an off-site cron (cheap B2 bucket)
+
+### 3. Mesh migration still incomplete
+
+**Problem:** Most LXCs still use Tailscale. This forces per-LXC DNS workarounds
+(/etc/hosts overrides, local dnsmasq) and runs two VPN stacks in parallel.
+Mesh migration doc (mesh.md) is comprehensive but execution stalled.
+
+**Proposal:**
+- Batch-migrate all LXCs from Tailscale to Netbird in one maintenance window
+- Remove Tailscale from the PVE host
+- Ensure all LXCs resolve `*.hubris.network` via Technitium (no more /etc/hosts
+  overrides)
+- Document Netbird client on each LXC (netbird version, setup key rotation)
+
+### 4. LXC resource imbalance & disk pressure
+
+**Problem:**
+| LXC | Cores | RAM | Rootfs | Disk usage |
+|-----|-------|-----|--------|------------|
+| mule-images (120) | 6 | 12 GiB | 60 GiB | photo AI, reasonable |
+| arriman (122) | 4 | 8 GiB | 24 GiB | media stack, OK |
+| nextcloud (114) | 4 | 6 GiB | 25 GiB | file sync, OK |
+| apps (105) | 2 | 4 GiB | 30 GiB | 6+ services, tight |
+| paperless (103) | 2 | 3 GiB | 8 GiB | 86.9% disk |
+| elementsynapse (118) | 1 | 2 GiB | 8 GiB | 86.8% disk |
+| gitea (104) | 1 | 1 GiB | 8 GiB | git server, adequate |
+| caddy (121) | 1 | 512 MiB | 6 GiB | fine for reverse proxy |
+
+Paperless (103) and elementsynapse (118) are at critical disk levels. apps (105)
+is undersized for 6+ services.
+
+**Proposal:**
+- Resize rootfs on paperless (8→16 GiB) and elementsynapse (8→16 GiB)
+- Bump apps (105) to 4 cores / 6 GiB RAM / 40 GiB rootfs
+- Enable claudio-monitor's disk check to alert before next crisis
+
+### 5. VPS is a single point of failure
+
+**Problem:** One IONOS VM runs netbird management (control plane), traefik
+(public ingress), authentik (identity), and coturn (TURN relay). If it goes
+down: no remote mesh, no public services, no auth.
+
+**Proposal:**
+- Document a VPS recovery runbook (how to restore from a known-working backup)
+- Consider splitting authentik into a separate host or at minimum having a
+  standby configuration
+- Not a high priority (the VPS has been stable) but worth documenting the
+  blast radius and recovery path
+
+### 6. No centralized logging
+
+**Problem:** Each LXC has independent journald. Cross-service debugging
+involves hopping between `pct exec <id> -- journalctl -u <service>`. There is
+no aggregation or retention.
+
+**Proposal:**
+- Deploy a lightweight log shipper (Loki + promtail, or vector.dev) on each LXC
+- Ship logs to a central Loki instance on apps (105) or a new small LXC
+- Grafana dashboard optional — even a simple `logcli` query saves time
+
+### 7. Agent enrollment incomplete
+
+**Problem:** Only hubris, apps, and republic-laptop are enrolled in the
+homelab-context system (age keys, sync timers, MCP access). mac-mini,
+ludo-mini, claudio-bot, and all other LXCs are not.
+
+**Proposal:**
+- Batch-enroll remaining LXCs (jellyfin, paperless, gitea, nextcloud, etc.)
+- Enroll mac-mini (macOS — exercises the launchd timer path)
+- Enroll ludo-mini (needs SSH user config in inventory first)
+- Wire claudio-bot into inventory-aware queries
+
+### 8. Configuration drift on untracked configs
+
+**Problem:** Technitium config, dnsmasq (legacy), and several service-specific
+configs are not git-tracked.
+
+**Proposal:**
+- Track Technitium zone backup + compose config in a git repo
+- Apply the same pattern as caddy-conf: dtoro/technitium-conf with auto-deploy
+
+### 9. No capacity planning / resource monitoring
+
+**Problem:** No trend data on CPU, RAM, or disk growth. The 86% disk alerts
+were discovered reactively. Rootfs resize is painful (requires Proxmox stop +
+resize + growfs inside).
+
+**Proposal:**
+- Enable the missing claudio-monitor checks (disk growth trend, apt upgradable
+  counts, docker image drift)
+- Set up a simple Prometheus + node_exporter on hubris or use the PVE API
+  directly
+- At minimum, surface disk usage in the existing homelab-mcp management tools
+
+### 10. No standard deploy / orchestration for bare-metal LXCs
+
+**Problem:** Some services are bare-metal CLI apps (sophia, claudio-bot),
+some are Docker on apps (105), some are Portainer-managed. No consistent
+deploy pattern means every new service reinvents the deployment.
+
+**Proposal:**
+- Don't over-engineer this — the current pragmatism works
+- Just document the decision tree:
+  - Needs `/mnt/library` mount + heavy I/O → dedicated LXC
+  - Small stateless web service → Docker on apps (105)
+  - Media stack → dedicated LXC (arriman, jellyfin)
+  - Everything else → judge by complexity
+
+---
+
+## Phased implementation plan
+
+### Phase 1 — Critical fixes (this week)
+1. Resize rootfs on paperless (103) 8→16 GiB, elementsynapse (118) 8→16 GiB
+2. Enable claudio-monitor disk check + disk-growth alerting
+3. Pick one backup strategy and implement minimum viable (e.g. nightly
+   gitea dump + sops-encrypted secrets to B2 via rclone)
+4. Verify Technitium is the sole DNS for all LXCs (remove NetBird managed zone
+   for hubris.network)
+
+### Phase 2 — Mesh consolidation (next week)
+5. Batch-migrate remaining LXCs from Tailscale to Netbird
+6. Remove Tailscale from PVE host
+7. Remove all per-LXC /etc/hosts DNS overrides
+8. Update DNS documentation to reflect Technitium as single source
+
+### Phase 3 — Agent enrollment & logging (next 2 weeks)
+9. Enroll all LXCs in homelab-context (age keys, sync timers)
+10. Enroll mac-mini (macOS launchd path — exercises untested code path)
+11. Enroll ludo-mini
+12. Deploy log shipper (Loki + promtail) on apps (105) + all LXCs
+
+### Phase 4 — Resource & monitoring hardening (next month)
+13. Resize apps (105) rootfs, bump RAM
+14. Deploy Prometheus + node_exporter or equivalent for trend data
+15. Track Technitium config in git with auto-deploy
+16. Write VPS recovery runbook
+
+### Phase 5 — Drive re-evaluation (optional, behind host-stability gate)
+17. Re-attach USB backup drive with the new SODOLA topology (direct port)
+18. If stable for 7 days, re-enable restic backup schedule (chunked)
+19. If not stable, finalize cloud backup as permanent strategy
+
+---
+
+## Files likely to change
+
+| Path | Change |
+|------|--------|
+| `/opt/homelab-context/inventory.yaml` | LXCs enrolled, resource updates |
+| `/opt/homelab-context/secrets/*.yaml` | New recipients for enrolled LXCs |
+| `/opt/homelab-context/infrastructure/dns.md` | Reflect Technitium as sole source |
+| `/opt/homelab-context/infrastructure/mesh.md` | Remove Tailscale references post-migration |
+| `/opt/homelab-context/infrastructure/backups.md` | New strategy |
+| `/opt/homelab-context/infrastructure/monitoring.md` | Enable missing checks |
+| `/opt/homelab-context/containers/103-paperless.md` | Rootfs resize |
+| `/opt/homelab-context/containers/118-elementsynapse.md` | Rootfs resize |
+| `/opt/homelab-context/containers/105-apps.md` | Resource bump, logging addition |
+| `/opt/homelab-context/containers/index.md` | Updated resource table |
+| `.sops.yaml` | New age pubkeys for enrolled LXCs |
+
+## Verification
+
+Each phase ends with a verification milestone:
+- Phase 1: `claudio-monitor` triggers on paperless disk → confirmed alert. Backup
+  of gitea data lands in B2 (or equivalent). DNS query from any LXC returns
+  Technitium answer.
+- Phase 2: `netbird status` shows all LXCs connected. Tailscale not running on
+  PVE host. `curl auth.hubris.network` from any LXC resolves correctly without
+  /etc/hosts.
+- Phase 3: Every LXC has `/opt/homelab-context/` + `/etc/age/key.txt`. MCP
+  tools return valid host info for all enrolled LXC names. `journalctl` shows
+  promtail shipping to Loki.
+- Phase 4: `claudio-monitor` shows disk growth trend. apps (105) can run all
+  6+ services without OOM.
+
+## Risks & tradeoffs
+
+- **Netbird migration window:** All LXCs will briefly lose mesh connectivity
+  during the Tailscale→Netbird cutover. Schedule in off-hours.
+- **Backup cost:** B2/e2 costs ~$5/month for ~500 GB. The USB drive was free
+  but unstable — trade money for reliability.
+- **DNS consolidation:** Removing the NetBird managed DNS zone means any
+  NetBird-specific names stop resolving for hubris.network — verify nothing
+  depends on that path.
+- **Loki on apps (105):** Adds another container to an already-loaded host.
+  May need to bump resources before deploying.
+- **Agent enrollment on every LXC:** Each enrollment creates an age keypair
+  and commits a pubkey to inventory. Process is scriptable via `homelab client
+  add` but still takes ~2 min per host for verification.
+
+## Open questions
+
+1. Is the USB backup drive still physically attached to hubris? If not, the
+   simplest "re-enable" path requires physically re-attaching it.
+2. Authentik is now on the VPS — is LXC 124 (old Authentik) still running or
+   was it fully decommissioned? The dns.md changelog says "shut down" but
+   index.md lists it as "running".
+3. What's the actual disk layout on hubris? nvme0n1, ZFS pool, mount structure
+   — needed to plan rootfs resizes safely.
+4. Does the user want to keep Tailscale on any host for a specific reason, or
+   is full Netbird migration the clear goal?
--- a/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md
+++ b/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md
@@ -0,0 +1,70 @@
+# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
+
+## Summary
+
+[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
+
+## Timeline
+
+### Pre-existing state
+- ludo-mini: wired Ethernet (eno1, 2.5 Gbps), 192.168.178.181/24
+- mac-mini: WiFi (en1, 802.11ac, 234 Mbps Tx rate, 1 stream, -60 dBm RSSI), 192.168.178.99/24
+- Both on the same subnet via a consumer router at 192.168.178.1
+- Sunshine configured: bitrate=80 Mbps, encoder=amf (AMD GPU), fec_percentage=5, hevc_mode=2
+- Moonlight configured: bitrate=57 Mbps, fps=60, localaddr=192.168.178.181 (correct)
+
+### 2026-06-03 — investigation
+**Network tests (iperf3 between mac-mini ↔ ludo-mini):**
+- TCP throughput: 49–86 Mbps, average ~65 Mbps — highly variable
+- UDP throughput: 1-second complete dropout during a 5-second test; 0% loss otherwise
+- Ping: 4.7 ms avg, but **28 ms stddev**, max **138 ms** — WiFi-level jitter
+- Netbird P2P connection also showed 7 ms latency over the tunnel (unnecessary given same-subnet direct connection)
+
+**Additional findings:**
+- mac-mini has a built-in Ethernet port (en0) but it is inactive — no cable connected
+- mac-mini also has Thunderbolt Ethernet adapters (en5, en6, en7) all inactive
+- ludo-mini's en0 shows `speed 2500` (2.5 GbE)
+- Netbird routes `192.168.8.0/24` via `utun100`, but 192.168.178.0/24 traffic stays on `en1`
+
+## Root cause
+
+mac-mini is on WiFi. WiFi introduces:
+1. **Throughput variance** — 49–86 Mbps TCP fluctuation
+2. **Dropouts** — 1-second UDP blackouts from radio retransmissions
+3. **Jitter** — 28 ms stddev with spikes to 138 ms
+
+Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
+
+## Mitigations applied
+
+### mac-mini — Moonlight (via `defaults write`)
+
+| Setting | Before | After | Rationale |
+|---------|--------|-------|-----------|
+| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
+| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
+| fps | 30 | 60 | Restored to game-appropriate value |
+
+### ludo-mini — Sunshine (via SSH)
+
+| Setting | Before | After | Rationale |
+|---------|--------|-------|-----------|
+| bitrate | 80 Mbps | 40 Mbps | Match Moonlight's reduced bitrate ceiling |
+| fec_percentage | 5 | 2 | Less forward-error-correction overhead over WiFi |
+| packetsize | (default) | 1024 | Smaller packets = less WiFi retransmit cost |
+
+Sunshine restarted after changes; confirmed active.
+
+## Open questions
+
+1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.
+2. **WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
+3. **Netbird route** — `192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
+
+## Changelog
+
+### 2026-06-03 — Initial investigation and mitigations
+- Diagnosed WiFi jitter as root cause of unstable Sunshine/Moonlight streaming
+- Applied Moonlight bitrate reduction (57→25 Mbps), frame pacing, fps fix
+- Applied Sunshine bitrate reduction (80→40 Mbps), fec_percentage (5→2), packetsize (1024)
+- Documented findings and open questions
--- a/investigations/index.md
+++ b/investigations/index.md
@@ -7,6 +7,7 @@ Time-stamped incident reports and experiments. One entry per incident; the entry
 | Date         | Title                                                              | Status        |
 | ------------ | ------------------------------------------------------------------ | ------------- |
 | 2026-05-31   | [Authentik migrated from LXC 124 to the VPS](2026-05-31-authentik-vps-migration.md) | Resolved; LXC 124 Authentik in ~2-week dual-run before decommission |
+| 2026-06-03   | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
 | 2026-04-21   | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) |

 ## Conventions