DNS Phase 4 COMPLETE: single zone — managed zone removed

After upgrading the Mac client 0.68.3 -> 0.71.3 (matching mgmt) and clearing
its NetBird resolver cache (netbird service restart), the managed-zone
deletion works: with 0 managed-zone records, the Mac resolves all
hubris.network names by forwarding to Technitium (192.168.8.2). iPhone
confirmed on cellular (no LAN path -> proves mesh-forward).

The first deletion "failure" was a misdiagnosis: the old 0.68.3 resolver
cache held stale answers and wouldn't clear on down/up (needs daemon
restart); the Mac's dual LAN+mesh paths muddied it. A direct
dig @100.122.255.254 of an unsynced name had shown forwarding working.

Done:
- Deleted all 23 NetBird managed-zone A-records.
- Removed the */10 dns-sync cron. Kept /opt/dns-sync/sync.py + token +
  pre-deletion backup as an emergency-restore tool only.

End state: Technitium is the single DNS source. Mesh peers forward to it
(Core route -> 192.168.8.0/24); LAN/household query it directly. No replica,
no sync. Requires mesh clients on 0.71.x+.

Docs: dns.md + 107-dns.md updated to single-source; subdomain recipe no
longer references the sync.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.gitignore

@@ -0,0 +1 @@
+.DS_Store

.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md

@@ -0,0 +1,277 @@
+# Plan: Migrate library SSD to ludo-mini + Proxmox gaming/media server
+
+## Goal
+
+Split the homelab into two Proxmox hosts:
+
+| Host | Role | Storage |
+|------|------|---------|
+| **hubris** | Core services (reverse-proxy, SSO, Matrix, git, documents, HA) | SSD 1 — boot + LXC rootfs (unchanged) |
+| **ludo-mini** | Gaming server + media/library services | SSD 2 — Samsung 990 EVO Plus 4 TB (moved from hubris) |
+
+The library SSD physically moves from hubris to ludo-mini. hubris LXCs that still need `/mnt/library` access it over NFS from ludo-mini.
+
+## Current state
+
+### hubris hardware
+- GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS, 12 vCPU, ~28 GiB RAM
+- 2× Samsung 990 EVO Plus NVMe:
+  - nvme0: `local` (95G) + `local-lvm` (856G) — boot, ISOs, LXC rootfs
+  - nvme1: `library` LVM (3.7T) — `/mnt/library` ext4 via `/dev/mapper/library-library`
+
+### LXCs binding `/mnt/library` (host-level bind-mount)
+| ID | Name | Role | I/O profile |
+|----|------|------|-------------|
+| 101 | jellyfin | Media streaming | Read-heavy, sequential |
+| 103 | paperless | Document archive | Mixed, OCR writes |
+| 104 | gitea | Git server | Mixed, lots of small files |
+| 105 | apps | Docker (booklore, audiobookshelf, artifacto, MCP) | Mixed, depends on container |
+| 114 | nextcloud | File sync | Mixed, WebDAV |
+| 119 | sophia | Workshop | Low I/O |
+| 120 | mule-images | Photo management | Write-heavy (processing), iGPU |
+| 122 | arriman | *arr stack + downloads | Write-heavy (downloads) |
+| 126 | plato | App (sub-mount: `/mnt/library/documents/plato`) | Light |
+
+### NFS export chain (for VM 100 zimaos)
+```
+/mnt/library (ext4, host) → bind-mount → LXC 102 (nfs-export) → NFSv4 → VM 100 (zimaos)
+```
+
+### ludo-mini current
+- Linux workstation, wired Ethernet 2.5 Gbps, `192.168.178.181` (household LAN)
+- Runs Sunshine for game streaming
+- No Proxmox, no LVM config
+- Connected to SODOLA switch (same switch as hubris eno1)
+
+### Network topology
+```
+Fritz!Box 7590 (192.168.178.1)
+  └── SODOLA 2.5G switch
+       ├── hubris eno1 → vmbr1 (192.168.178.10)
+       │     └── routes to vmbr0 (192.168.8.0/24) — all LXCs
+       └── ludo-mini (192.168.178.181)
+```
+
+hubris routes between `192.168.8.0/24` (vmbr0) and `192.168.178.0/24` (vmbr1). So LXCs can reach ludo-mini via hubris as a router.
+
+## Key decisions
+
+### 1. Service split — what moves, what stays
+
+**Move to ludo-mini** (high I/O, benefits from data locality + GPU):
+- 101 jellyfin — media streaming, GPU transcoding
+- 120 mule-images — photo processing, iGPU passthrough
+- 122 arriman — *arr stack, downloads write to library
+
+**Stay on hubris, NFS-mount library from ludo-mini:**
+- 103 paperless — documents, moderate I/O
+- 104 gitea — git repos (small files, some I/O sensitivity but acceptable over NFS)
+- 105 apps — Docker apps, mixed workloads
+- 114 nextcloud — file sync
+- 119 sophia — workshop, light use
+- 126 plato — app, light use
+- 100 zimaos — NAS frontend, already NFS-mounted
+
+### 2. NFS architecture
+
+Instead of changing every LXC's mount config, keep the bind-mount pattern on hubris:
+
+```
+ludo-mini: /mnt/library (ext4, local NVMe)
+    │
+    └── NFSv4 export to 192.168.8.0/24
+              │
+              └── hubris host: NFS-mount at /mnt/library
+                        │
+                        └── LXCs: bind-mount /mnt/library (unchanged!)
+```
+
+This is transparent to all hubris LXCs — no container config changes needed. Only the hubris host changes from ext4 local mount to NFS mount. The LXC bind-mounts "just work" because `/mnt/library` is still at the same path on the host.
+
+### 3. Network — ludo-mini reachability from hubris LXCs
+
+LXCs on `192.168.8.0/24` reach ludo-mini (`192.168.178.181`) through hubris routing:
+- `vmbr0` → hubris kernel routing → `vmbr1` → SODOLA → ludo-mini
+- Already works (IP forwarding enabled on hubris)
+
+**Alternative (cleaner):** Add a secondary IP `192.168.8.x` on ludo-mini's physical interface so it's directly on the homelab subnet. This avoids the router hop and keeps NFS traffic off kernel forwarding path. Worth considering but not required.
+
+### 4. Gaming on ludo-mini with Proxmox
+
+ludo-mini runs Sunshine (game streaming). Under Proxmox:
+- **Option A:** Gaming VM with GPU passthrough — Sunshine + games in a VM, full GPU access
+- **Option B:** LXC with GPU device passthrough (`/dev/dri`) — lighter, shares kernel
+- **Option C:** Keep Sunshine on the Proxmox host itself (not recommended, but simplest)
+
+Option A is the cleanest for isolation. Games need a full desktop environment and GPU drivers; a VM with GPU passthrough gives them that.
+
+### 5. What about nfs-export (LXC 102)?
+
+Currently exports `/mnt/library` to zimaos. After migration:
+- If zimaos stays on hubris and accesses library via host NFS → bind-mount → LXC 102, that's triple-hop (ludo-mini → NFS → hubris → bind-mount → LXC 102 → NFS → zimaos). Terrible.
+- Better: zimaos NFS-mounts directly from ludo-mini.
+- So LXC 102 gets decommissioned (or repurposed).
+- zimaos gets a new NFS mount pointing directly at ludo-mini.
+
+## Migration phases
+
+### Phase 1 — Preparation (no downtime)
+
+1. **Document current state** on hubris:
+   - `pct list` — full container inventory
+   - `pct config <id>` for every library-mounting LXC
+   - `cat /etc/fstab` — capture the library mount line
+   - `df -h /mnt/library` — confirm space usage
+   - `lsblk -f` — UUID, filesystem
+   - Identify the exact NVMe device (`nvme1n1`)
+
+2. **Pre-flight on ludo-mini:**
+   - Confirm hardware: CPU, RAM, available M.2 slots, GPU model
+   - Confirm it can take the Samsung 990 EVO Plus (M.2 NVMe, PCIe 4.0 x4)
+   - Verify BIOS supports virtualization (VT-d/AMD-Vi for PCIe passthrough)
+   - Check: does ludo-mini have a second drive for Proxmox OS? If not, we need to partition the library SSD for Proxmox boot + library LVM, which complicates things significantly
+
+3. **Install Proxmox on ludo-mini:**
+   - Download Proxmox VE 9.x ISO
+   - Install to ludo-mini's system drive (NOT the library SSD)
+   - Configure networking: bridge for Proxmox, IP on 192.168.178.x
+   - Test: web UI accessible
+
+4. **Prepare NFS server on ludo-mini Proxmox:**
+   - Create NFS-export LXC (or serve from host — simpler for now)
+   - Prepare `/etc/exports`: `192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)`
+   - Same squash params as current nfs-export LXC 102
+
+### Phase 2 — Physical SSD move (planned downtime)
+
+1. **Graceful shutdown on hubris:**
+   - Stop all library-mounting LXCs (101, 103, 104, 105, 114, 119, 120, 122, 126)
+   - Unmount `/mnt/library` on hubris host
+   - Edit `/etc/fstab` to comment out the library mount line
+   - Power off hubris
+
+2. **Physical drive swap:**
+   - Remove Samsung 990 EVO Plus (library SSD) from hubris
+   - Install into ludo-mini M.2 slot
+   - Power on ludo-mini
+
+3. **Bring library online on ludo-mini:**
+   - Detect the new NVMe device
+   - If it's the whole device with LVM, activate the VG:
+     ```
+     vgscan && vgchange -ay library
+     mount /dev/mapper/library-library /mnt/library
+     ```
+   - Add to `/etc/fstab` for auto-mount
+   - Verify content: `ls /mnt/library` — same tree as before
+
+4. **Start NFS export on ludo-mini:**
+   - `exportfs -ra`
+   - Verify: `showmount -e <ludo-mini-ip>`
+
+### Phase 3 — Reconnect hubris LXCs
+
+1. **Power on hubris** (without library SSD — it'll boot fine, just won't mount library)
+
+2. **Mount NFS on hubris host:**
+   - Install `nfs-common` if not present
+   - Add to `/etc/fstab`:
+     ```
+     192.168.178.181:/mnt/library /mnt/library nfs rw,vers=4,soft,timeo=30,retrans=3 0 0
+     ```
+     Use `soft` to prevent hangs if ludo-mini is down; `hard` with `intr` is safer for data integrity but can block processes.
+   - `mount /mnt/library`
+   - **Verify permissions:** `ls -la /mnt/library` — should show `www-data:media` ownership for shared subtrees (same uid 33, gid 10000). The NFS all_squash guarantees this.
+
+3. **Start LXCs:**
+   - Start the LXCs that stayed on hubris (103, 104, 105, 114, 119, 126)
+   - Their bind-mounts should work — `/mnt/library` is populated via NFS
+   - Verify each service: web UIs, git clone, document access
+
+4. **Update DNS/Caddy** for services that moved:
+   - If jellyfin, arr services moved to ludo-mini, update Caddyfile to point to ludo-mini IPs
+   - Update DNS entries if needed
+
+### Phase 4 — Migrate services to ludo-mini
+
+1. **Create LXCs/VM on ludo-mini Proxmox:**
+   - 101 jellyfin — privileged LXC, mount `/mnt/library`, add `media` group
+   - 120 mule-images — privileged LXC, mount `/mnt/library` + `/dev/dri` passthrough
+   - 122 arriman — privileged LXC, mount `/mnt/library`
+
+2. **Migrate configs:**
+   - Copy LXC configs from hubris (`/etc/pve/lxc/<id>.conf`) as templates
+   - Adjust network (IPs on household subnet or Proxmox bridge)
+   - Restore app data from backups or copy over NFS
+
+3. **Gaming VM:**
+   - Create VM with GPU passthrough
+   - Pass through the dGPU for gaming performance
+   - Install Sunshine + game libraries
+   - Storage: VM disk on Proxmox storage, games on library SSD
+
+4. **Update reverse proxy:**
+   - Caddy on hubris (121): update backend IPs for jellyfin, jellyseerr, qbit, sab, mule-images → point to ludo-mini
+   - Test: `media.hubris.network` serves from ludo-mini jellyfin
+
+### Phase 5 — Cleanup
+
+1. **Decommission nfs-export LXC 102** on hubris (no longer needed)
+2. **Update zimaos (100)** — change NFS mount from `192.168.8.200` → `192.168.178.181`
+3. **Remove old LXCs** from hubris (101, 120, 122) after confirming migration works
+4. **Update inventory.yaml:**
+   - ludo-mini: `kind: proxmox-host`, add mounts/storage, add LXCs
+   - Move services from hubris to ludo-mini
+   - Remove nfs-export
+5. **Update DNS:** `nfs-export.hubris.network` → ludo-mini IP (or remove)
+6. **Run `homelab sync`** to propagate changes
+
+## Open questions / unknowns
+
+1. **Does ludo-mini have a second drive for Proxmox OS?** If not, we'd need to repartition the library SSD — carve out ~100 GB for Proxmox, then the rest for library LVM. This is risky (data loss if partitioning goes wrong) and requires a full backup first. **Alternative:** Buy a small SSD for ludo-mini's OS.
+
+2. **What GPU does ludo-mini have?** Proxmox GPU passthrough requires IOMMU support and a GPU that doesn't have the reset bug. Need to check the exact GPU model.
+
+3. **NFS performance for git (gitea)?** Git operations over NFS can be problematic (locking, stat() storms). Gitea bare repos at `/mnt/library/repos/*.git` might need testing. Worst case: move gitea's repo storage to local disk and keep `/mnt/library` for large file/LFS storage only.
+
+4. **Media permission drift.** NFS `all_squash,anonuid=33,anongid=10000` ensures all writes from hubris LXCs (over NFS) and ludo-mini LXCs (local) land as `www-data:media`. This is the same squash currently used by nfs-export (102). Should be fine.
+
+5. **ludo-mini network — add 192.168.8.x address?** Adding a secondary IP on ludo-mini's interface directly on the homelab subnet avoids routing through hubris for NFS traffic. Cleaner, but requires Proxmox bridge setup. Worth doing during Proxmox install.
+
+6. **Sunshine migration.** Currently runs on ludo-mini bare metal. After Proxmox install, it needs to run in a VM. What happens to existing Sunshine configs, game libraries, save files? Need to preserve these during the Proxmox install.
+
+7. **Backup before moving.** The library SSD holds 3.7 TB of irreplaceable data (documents, photos, repos). Restic backups are currently disabled. **Before physically moving the drive, verify the data is readable and consider doing one backup** — or at minimum, `rsync` critical directories.
+
+## Files affected
+
+| File | Change |
+|------|--------|
+| `/opt/homelab-context/inventory.yaml` | ludo-mini: workstation → proxmox-host; add LXCs, mounts; remove nfs-export; move service backends |
+| `/opt/homelab-context/hosts/hubris.md` | Remove library storage, add NFS mount note |
+| `/opt/homelab-context/hosts/ludo-mini.yaml` | Complete rewrite — Proxmox host, storage, tenants |
+| `/opt/homelab-context/containers/102-nfs-export.md` | Mark decommissioned |
+| `/opt/homelab-context/containers/index.md` | Move 101, 120, 122 to ludo-mini; remove 102 |
+| `/opt/homelab-context/infrastructure/dns.md` | Update nfs-export entry |
+| `/opt/homelab-context/infrastructure/media-permissions.md` | Note NFS squash from ludo-mini, not hubris |
+| hubris `/etc/fstab` | Replace ext4 mount with NFS mount |
+| ludo-mini `/etc/fstab` | Add library ext4 mount |
+| ludo-mini `/etc/exports` | Add NFS export config |
+| caddy (LXC 121) Caddyfile | Backend IPs for moved services |
+| DNS (LXC 107 Technitium) | Update entries for moved services |
+
+## Validation checklist
+
+- [ ] ludo-mini Proxmox web UI accessible
+- [ ] Library SSD detected and mountable on ludo-mini
+- [ ] NFS export from ludo-mini: `showmount -e <ip>` shows `/mnt/library`
+- [ ] hubris host NFS mount: `df -h /mnt/library` shows NFS, not ext4
+- [ ] hubris LXCs start and bind-mount /mnt/library (content visible)
+- [ ] gitea: `git clone` over SSH works, repos readable
+- [ ] paperless: document ingestion works, OCR processing
+- [ ] nextcloud: file sync, WebDAV
+- [ ] jellyfin: media plays from ludo-mini, transcoding works
+- [ ] arriman: downloads write to library, jellyfin picks up new media
+- [ ] mule-images: photo import and processing
+- [ ] zimaos: NFS mount from ludo-mini works, Files UI shows library
+- [ ] Sunshine: game streaming from ludo-mini VM works
+- [ ] All `*.hubris.network` services resolve and load through Caddy

.hermes/plans/2026-06-03_150000-homelab-structure-revision.md

@@ -0,0 +1,271 @@
+# Homelab structure revision & improvement plan
+
+## Goal
+
+Identify structural issues in the current hubris homelab topology and propose an
+actionable improvement roadmap — DNS consolidation, monitoring gaps, backup
+recovery, mesh completion, resource rightsizing, and operational hygiene.
+
+---
+
+## Current state summary
+
+| Dimension | Status |
+|-----------|--------|
+| Hypervisor | hubris (single-node Proxmox) — also does subnet routing (192.168.8.0/24) |
+| LXCs | 14 active, 1 retired (124), 1 new (107 dns) |
+| VMs | HAOS (108), ZimaOS (100) |
+| Workstations | mac-mini (macOS), republic-laptop, ludo-mini |
+| VPS | 1 IONOS box — netbird mgmt+signal+relay, traefik, authentik, coturn |
+| Switch | SODOLA 5-Port 2.5Gbit (L2, flat) |
+| Networking | 192.168.8.0/24 internal, fr!tz box main LAN via vmbr0→vmbr1 routing |
+| Mesh | Netbird + Tailscale (migrating), ~mixed state |
+| DNS | 3 sources: Technitium (CT 107), NetBird managed zone, public IONOS |
+| Backups | DISABLED since 2026-04-22 |
+| Monitoring | claudio-monitor (host health only, no apt/docker checks yet) |
+| Agent enrollment | 3/19 hosts enrolled (hubris, apps, republic-laptop) |
+
+---
+
+## Issues identified
+
+### 1. Three overlapping DNS sources (highest risk)
+
+**Problem:** Technitium on CT 107, NetBird managed DNS zone, and the public
+IONOS wildcard all answer `*.hubris.network` queries. The dns.md doc still
+references the old dnsmasq on LXC 124 (though the change log says it moved).
+NetBird's managed DNS bypasses Technitium entirely for some app names — there
+is no single source of truth for DNS.
+
+**Risk:** Mismatched answers → services unreachable → "works on some clients
+but not others" debugging sessions. Already cost time when `auth.hubris.network`
+re-pointed to the VPS.
+
+**Proposal:**
+- Phase out NetBird managed DNS zone for `hubris.network` — Technitium is the
+  authoritative answerer for mesh & LAN clients
+- Set Technitium as the sole DNS for all LXCs (remove router DNS / Tailscale
+  MagicDNS fallbacks)
+- Document the full authoritative chain: Technitium → upstream forwarders → public
+- Track Technitium config in git (dtoro/technitium-config or equivalent)
+
+### 2. Backups disabled with no alternative (data loss risk)
+
+**Problem:** The only backup was restic to an external USB that caused host
+crashes. It was disabled 2026-04-22 as an A/B test — host stability was
+confirmed by the SODOLA migration (no more Slate AX double-NAT hangs). The
+drive is still removed.
+
+**Risk:** `/mnt/library` (~429 GB of irreplaceable data: photos, documents,
+git repos, gitea data) has no off-host copy. Single-disk failure = total loss.
+
+**Proposal:**
+- Re-evaluate the USB drive stability with the new SODOLA switch topology
+  (direct rear USB 3.0 port, no hub chain)
+- OR adopt a cloud-backed strategy: `rclone`-to-Hetzner Storage Box or
+  Backblaze B2 for the irreplaceable subset (docs, photos, gitea data)
+- OR use hubris's own `zfs send` to a second host/disk if ZFS is feasible
+- Minimum viable: at minimum restore gitea backups + sops-encrypted secrets
+  via an off-site cron (cheap B2 bucket)
+
+### 3. Mesh migration still incomplete
+
+**Problem:** Most LXCs still use Tailscale. This forces per-LXC DNS workarounds
+(/etc/hosts overrides, local dnsmasq) and runs two VPN stacks in parallel.
+Mesh migration doc (mesh.md) is comprehensive but execution stalled.
+
+**Proposal:**
+- Batch-migrate all LXCs from Tailscale to Netbird in one maintenance window
+- Remove Tailscale from the PVE host
+- Ensure all LXCs resolve `*.hubris.network` via Technitium (no more /etc/hosts
+  overrides)
+- Document Netbird client on each LXC (netbird version, setup key rotation)
+
+### 4. LXC resource imbalance & disk pressure
+
+**Problem:**
+| LXC | Cores | RAM | Rootfs | Disk usage |
+|-----|-------|-----|--------|------------|
+| mule-images (120) | 6 | 12 GiB | 60 GiB | photo AI, reasonable |
+| arriman (122) | 4 | 8 GiB | 24 GiB | media stack, OK |
+| nextcloud (114) | 4 | 6 GiB | 25 GiB | file sync, OK |
+| apps (105) | 2 | 4 GiB | 30 GiB | 6+ services, tight |
+| paperless (103) | 2 | 3 GiB | 8 GiB | 86.9% disk |
+| elementsynapse (118) | 1 | 2 GiB | 8 GiB | 86.8% disk |
+| gitea (104) | 1 | 1 GiB | 8 GiB | git server, adequate |
+| caddy (121) | 1 | 512 MiB | 6 GiB | fine for reverse proxy |
+
+Paperless (103) and elementsynapse (118) are at critical disk levels. apps (105)
+is undersized for 6+ services.
+
+**Proposal:**
+- Resize rootfs on paperless (8→16 GiB) and elementsynapse (8→16 GiB)
+- Bump apps (105) to 4 cores / 6 GiB RAM / 40 GiB rootfs
+- Enable claudio-monitor's disk check to alert before next crisis
+
+### 5. VPS is a single point of failure
+
+**Problem:** One IONOS VM runs netbird management (control plane), traefik
+(public ingress), authentik (identity), and coturn (TURN relay). If it goes
+down: no remote mesh, no public services, no auth.
+
+**Proposal:**
+- Document a VPS recovery runbook (how to restore from a known-working backup)
+- Consider splitting authentik into a separate host or at minimum having a
+  standby configuration
+- Not a high priority (the VPS has been stable) but worth documenting the
+  blast radius and recovery path
+
+### 6. No centralized logging
+
+**Problem:** Each LXC has independent journald. Cross-service debugging
+involves hopping between `pct exec <id> -- journalctl -u <service>`. There is
+no aggregation or retention.
+
+**Proposal:**
+- Deploy a lightweight log shipper (Loki + promtail, or vector.dev) on each LXC
+- Ship logs to a central Loki instance on apps (105) or a new small LXC
+- Grafana dashboard optional — even a simple `logcli` query saves time
+
+### 7. Agent enrollment incomplete
+
+**Problem:** Only hubris, apps, and republic-laptop are enrolled in the
+homelab-context system (age keys, sync timers, MCP access). mac-mini,
+ludo-mini, claudio-bot, and all other LXCs are not.
+
+**Proposal:**
+- Batch-enroll remaining LXCs (jellyfin, paperless, gitea, nextcloud, etc.)
+- Enroll mac-mini (macOS — exercises the launchd timer path)
+- Enroll ludo-mini (needs SSH user config in inventory first)
+- Wire claudio-bot into inventory-aware queries
+
+### 8. Configuration drift on untracked configs
+
+**Problem:** Technitium config, dnsmasq (legacy), and several service-specific
+configs are not git-tracked.
+
+**Proposal:**
+- Track Technitium zone backup + compose config in a git repo
+- Apply the same pattern as caddy-conf: dtoro/technitium-conf with auto-deploy
+
+### 9. No capacity planning / resource monitoring
+
+**Problem:** No trend data on CPU, RAM, or disk growth. The 86% disk alerts
+were discovered reactively. Rootfs resize is painful (requires Proxmox stop +
+resize + growfs inside).
+
+**Proposal:**
+- Enable the missing claudio-monitor checks (disk growth trend, apt upgradable
+  counts, docker image drift)
+- Set up a simple Prometheus + node_exporter on hubris or use the PVE API
+  directly
+- At minimum, surface disk usage in the existing homelab-mcp management tools
+
+### 10. No standard deploy / orchestration for bare-metal LXCs
+
+**Problem:** Some services are bare-metal CLI apps (sophia, claudio-bot),
+some are Docker on apps (105), some are Portainer-managed. No consistent
+deploy pattern means every new service reinvents the deployment.
+
+**Proposal:**
+- Don't over-engineer this — the current pragmatism works
+- Just document the decision tree:
+  - Needs `/mnt/library` mount + heavy I/O → dedicated LXC
+  - Small stateless web service → Docker on apps (105)
+  - Media stack → dedicated LXC (arriman, jellyfin)
+  - Everything else → judge by complexity
+
+---
+
+## Phased implementation plan
+
+### Phase 1 — Critical fixes (this week)
+1. Resize rootfs on paperless (103) 8→16 GiB, elementsynapse (118) 8→16 GiB
+2. Enable claudio-monitor disk check + disk-growth alerting
+3. Pick one backup strategy and implement minimum viable (e.g. nightly
+   gitea dump + sops-encrypted secrets to B2 via rclone)
+4. Verify Technitium is the sole DNS for all LXCs (remove NetBird managed zone
+   for hubris.network)
+
+### Phase 2 — Mesh consolidation (next week)
+5. Batch-migrate remaining LXCs from Tailscale to Netbird
+6. Remove Tailscale from PVE host
+7. Remove all per-LXC /etc/hosts DNS overrides
+8. Update DNS documentation to reflect Technitium as single source
+
+### Phase 3 — Agent enrollment & logging (next 2 weeks)
+9. Enroll all LXCs in homelab-context (age keys, sync timers)
+10. Enroll mac-mini (macOS launchd path — exercises untested code path)
+11. Enroll ludo-mini
+12. Deploy log shipper (Loki + promtail) on apps (105) + all LXCs
+
+### Phase 4 — Resource & monitoring hardening (next month)
+13. Resize apps (105) rootfs, bump RAM
+14. Deploy Prometheus + node_exporter or equivalent for trend data
+15. Track Technitium config in git with auto-deploy
+16. Write VPS recovery runbook
+
+### Phase 5 — Drive re-evaluation (optional, behind host-stability gate)
+17. Re-attach USB backup drive with the new SODOLA topology (direct port)
+18. If stable for 7 days, re-enable restic backup schedule (chunked)
+19. If not stable, finalize cloud backup as permanent strategy
+
+---
+
+## Files likely to change
+
+| Path | Change |
+|------|--------|
+| `/opt/homelab-context/inventory.yaml` | LXCs enrolled, resource updates |
+| `/opt/homelab-context/secrets/*.yaml` | New recipients for enrolled LXCs |
+| `/opt/homelab-context/infrastructure/dns.md` | Reflect Technitium as sole source |
+| `/opt/homelab-context/infrastructure/mesh.md` | Remove Tailscale references post-migration |
+| `/opt/homelab-context/infrastructure/backups.md` | New strategy |
+| `/opt/homelab-context/infrastructure/monitoring.md` | Enable missing checks |
+| `/opt/homelab-context/containers/103-paperless.md` | Rootfs resize |
+| `/opt/homelab-context/containers/118-elementsynapse.md` | Rootfs resize |
+| `/opt/homelab-context/containers/105-apps.md` | Resource bump, logging addition |
+| `/opt/homelab-context/containers/index.md` | Updated resource table |
+| `.sops.yaml` | New age pubkeys for enrolled LXCs |
+
+## Verification
+
+Each phase ends with a verification milestone:
+- Phase 1: `claudio-monitor` triggers on paperless disk → confirmed alert. Backup
+  of gitea data lands in B2 (or equivalent). DNS query from any LXC returns
+  Technitium answer.
+- Phase 2: `netbird status` shows all LXCs connected. Tailscale not running on
+  PVE host. `curl auth.hubris.network` from any LXC resolves correctly without
+  /etc/hosts.
+- Phase 3: Every LXC has `/opt/homelab-context/` + `/etc/age/key.txt`. MCP
+  tools return valid host info for all enrolled LXC names. `journalctl` shows
+  promtail shipping to Loki.
+- Phase 4: `claudio-monitor` shows disk growth trend. apps (105) can run all
+  6+ services without OOM.
+
+## Risks & tradeoffs
+
+- **Netbird migration window:** All LXCs will briefly lose mesh connectivity
+  during the Tailscale→Netbird cutover. Schedule in off-hours.
+- **Backup cost:** B2/e2 costs ~$5/month for ~500 GB. The USB drive was free
+  but unstable — trade money for reliability.
+- **DNS consolidation:** Removing the NetBird managed DNS zone means any
+  NetBird-specific names stop resolving for hubris.network — verify nothing
+  depends on that path.
+- **Loki on apps (105):** Adds another container to an already-loaded host.
+  May need to bump resources before deploying.
+- **Agent enrollment on every LXC:** Each enrollment creates an age keypair
+  and commits a pubkey to inventory. Process is scriptable via `homelab client
+  add` but still takes ~2 min per host for verification.
+
+## Open questions
+
+1. Is the USB backup drive still physically attached to hubris? If not, the
+   simplest "re-enable" path requires physically re-attaching it.
+2. Authentik is now on the VPS — is LXC 124 (old Authentik) still running or
+   was it fully decommissioned? The dns.md changelog says "shut down" but
+   index.md lists it as "running".
+3. What's the actual disk layout on hubris? nvme0n1, ZFS pool, mount structure
+   — needed to plan rootfs resizes safely.
+4. Does the user want to keep Tailscale on any host for a specific reason, or
+   is full Netbird migration the clear goal?

.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md

@@ -0,0 +1,246 @@
+# Plan: Narrow Technitium DHCP Pool to Avoid Static-IP Conflicts
+
+> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
+
+**Goal:** Eliminate the IP conflict risk created by the Technitium DHCP pool (`.100–.240`) overlapping with all static LXC/VM IPs (`.101–.239`).
+
+**Architecture:** Shrink the DHCP pool range on Technitium so it only covers IPs that no static host uses. No LXC/VM IPs change. Single server-side change (Technitium API), plus documentation updates.
+
+**Tech Stack:** Technitium DNS API (`/api/dhcp/scopes/set`), bash/curl, homelab-context repo for docs.
+
+---
+
+## Problem statement
+
+The Technitium DHCP server on [CT 107](containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
+
+| Host | IP | Inside pool? |
+|---|---|---|
+| hubris (Proxmox) | .77 | No — below `.100` |
+| haos (VM 108) | .101 | YES |
+| gitea (104) | .121 | YES |
+| paperless (103) | .130 | YES |
+| arriman (122) | .132 | YES |
+| mule-images (120) | .136 | YES |
+| sophia (119) | .157 | YES |
+| mac-mini | .174 | YES |
+| caddy (121) | .175 | YES |
+| authentik (124) | .180 | YES |
+| plato (126) | .190 | YES |
+| zimaos (VM 100) | .195 | YES |
+| nfs-export (102) | .200 | YES |
+| apps (105) | .205 | YES |
+| jellyfin (101) | .206 | YES |
+| nextcloud (114) | .224 | YES |
+| claudio-bot (123) | .230 | YES |
+| elementsynapse (118) | .239 | YES |
+
+The docs claim "Static-IP LXCs (below `.100`) are unaffected" — this is **false**. Static IPs span `.101–.239`, the DHCP pool spans `.100–.240`. They overlap almost entirely.
+
+If the DHCP server hands out `.121/.136/.224` (or any of the above) to a new dynamic client before the static LXC claims it on boot, the static service will fail to bind and the service goes dark.
+
+---
+
+## Proposed approach: Shrink the pool
+
+**Move the DHCP pool start from `.100` to `.241`**, resulting in:
+- **New pool:** `192.168.8.241 – 192.168.8.254` (14 dynamic IPs)
+- **Reserved:** `.100–.240` stays for static hosts, `.2` for Technitium, `.1` for gateway
+- **Zero changes to any LXC, VM, Caddy, or Proxmox config.**
+
+Why `.241–.254`:
+- Highest static IP is `.239` (elementsynapse) — `.241` gives a 1-IP gap
+- `.255` is the broadcast address (unusable)
+- 14 IPs is plenty for truly dynamic clients (new transient containers, test VMs)
+- If more are ever needed, the pool can easily be widened back down
+
+---
+
+## Tasks
+
+### Task 1: Verify current Technitium DHCP scope from the API
+
+**Objective:** Confirm the active pool range matches what's documented.
+
+**Step 1: Log in to Technitium API and get a token**
+
+```bash
+TOKEN=$(curl -sk -X POST http://192.168.8.2:5380/api/user/login \
+  -H "Content-Type: application/json" \
+  -d '{"user":"admin","pass":"'$(cat /opt/technitium/admin_password.txt)'","includeInfo":false}' \
+  | jq -r '.token')
+echo "Token: ${TOKEN:0:10}..."
+```
+
+**Step 2: Fetch current DHCP scopes**
+
+```bash
+curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq .
+```
+
+**Expected:** One scope named `homelab` with `startingAddress: "192.168.8.100"` and `endingAddress: "192.168.8.240"`.
+
+**Verification:** If the scope is NOT `.100–.240`, note the actual range and adjust the plan.
+
+---
+
+### Task 2: Update the DHCP scope to `.241–.254`
+
+**Objective:** Shrink the pool so it no longer overlaps static IPs.
+
+**Step 1: Update the scope via API**
+
+```bash
+curl -sk -X POST "http://192.168.8.2:5380/api/dhcp/scopes/set?token=$TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "homelab",
+    "startingAddress": "192.168.8.241",
+    "endingAddress": "192.168.8.254",
+    "subnetMask": "255.255.255.0",
+    "gatewayAddress": "192.168.8.1",
+    "dnsServerAddresses": ["192.168.8.2"],
+    "leaseTime": 86400
+  }'
+```
+
+**Step 2: Verify the change took effect**
+
+```bash
+curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq '.response.scopes[0] | {startingAddress, endingAddress}'
+```
+
+**Expected:**
+```json
+{
+  "startingAddress": "192.168.8.241",
+  "endingAddress": "192.168.8.254"
+}
+```
+
+**Pitfall:** If the API returns `{"status":"error"}`, the scope name or parameter format may differ. Inspect the response body. Technitium's API might use `rangeStart`/`rangeEnd` instead of `startingAddress`/`endingAddress`. Adjust if needed (check the full scope object from Task 1 step 2 for exact key names).
+
+---
+
+### Task 3: Check for active DHCP leases in the old pool that would be stranded
+
+**Objective:** Ensure no DHCP client is currently holding an IP in `.100–.240` that it will lose when its lease expires.
+
+**Step 1: List active DHCP leases**
+
+```bash
+curl -sk "http://192.168.8.2:5380/api/dhcp/leases/list?token=$TOKEN" | jq '.response.leases[] | {ip: .ipAddress, client: .clientHostname, mac: .hardwareAddress, expires: .leaseExpires}'
+```
+
+**Step 2: Interpret results**
+
+- If the only leases are from static LXCs that configured themselves before the DHCP move (e.g., old leases from before the 2026-06-02 static-IP migration), these leases are stale and harmless.
+- If a *dynamic* client (e.g., a test laptop, transient VM) holds `.195` or similar, note it — it will lose its IP on next renew and should be moved to a static assignment or into the `.241+` pool.
+- **ZimaOS (VM 100) at `.195` is a DHCP lease, not static** — this is the one host that needs attention. Either:
+  - Set a static IP inside ZimaOS (preferred), or
+  - Add a DHCP reservation for MAC in Technitium to pin `.195`
+
+**Verification:** No "surprise" dynamic clients that would break on lease expiry.
+
+---
+
+### Task 4: Fix ZimaOS IP stability (if needed)
+
+**Objective:** Ensure ZimaOS at `.195` won't float or break when the pool shrinks.
+
+**If ZimaOS already has a static IP configured inside the VM:** Nothing to do.
+
+**If ZimaOS is DHCP-only (likely — doc says "DHCP lease, not a reservation"):**
+
+Option A (preferred): Set a static IP inside ZimaOS via its web UI at `http://192.168.8.195` → Settings → Network → Static IP → `192.168.8.195/24`, gateway `192.168.8.1`, DNS `192.168.8.2`.
+
+Option B: Add a DHCP reservation in Technitium for ZimaOS's MAC address:
+```bash
+ZIMAMAC=$(ssh root@hubris "qm config 100 | grep net0 | grep -oE '([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}'")
+curl -sk -X POST "http://192.168.8.2:5380/api/dhcp/reservations/add?token=$TOKEN" \
+  -H "Content-Type: application/json" \
+  -d "{\"hardwareAddress\":\"$ZIMAMAC\",\"ipAddress\":\"192.168.8.195\"}"
+```
+
+**Pitfall:** The `/api/dhcp/reservations/add` endpoint signature is unverified — confirm the exact endpoint name from Technitium's API docs or the web UI before running it. The web console at `http://192.168.8.2:5380` → DHCP → Reservations can be used as a manual fallback.
+
+---
+
+### Task 5: Update documentation in homelab-context
+
+**Objective:** Fix the now-wrong claims about static IPs being "below .100".
+
+**Files to edit:**
+
+1. **`infrastructure/network.md`** — Line 53
+   - Old: `Most homelab LXCs use static IPs below \`.100\`. DHCP only covers new/transient containers.`
+   - New: `Static IPs span \`.101–.239\` (all LXCs + VMs + workstations). DHCP pool narrowed to \`.241–.254\` to avoid overlap.`
+
+2. **`containers/107-dns.md`** — Lines 37, 42, 55
+   - Line 37: Update pool range: `192.168.8.241 – 192.168.8.254`
+   - Line 42: `Static-IP LXCs (below \`.100\`)` → `Static-IP LXCs (\`.101–.239\`) are excluded from the pool.`
+   - Line 55: Add changelog entry for the pool shrink
+
+3. **`containers/107-dns.md`** — Add changelog entry:
+   ```markdown
+   ### 2026-06-03 — DHCP pool narrowed to `.241–.254` to exclude static IPs
+   Previous pool `.100–.240` overlapped with all static LXCs/VMs (\`.101–.239\`), creating IP conflict risk. Shrunk pool to `.241–.254`. No services re-IP'd. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
+   ```
+
+4. **`infrastructure/network.md`** — Line 51: Update pool range in the DHCP table row.
+
+5. **`plans/2026-06-01-slate-ax-to-sodola-migration.md`** — Line 60: Optionally update the pool range in the config table (or add a post-migration note). This is the historical migration plan, so a footnote rather than an edit may be better.
+
+**Commit:**
+```bash
+cd /opt/homelab-context
+git add infrastructure/network.md containers/107-dns.md plans/
+git commit -m "docs: DHCP pool narrowed to .241-.254 to exclude static IPs"
+git push
+```
+
+---
+
+### Task 6: Verify no regressions
+
+**Objective:** Smoke-test that DNS and key services still work after the scope change.
+
+```bash
+# 1. DNS resolution via Technitium
+dig @192.168.8.2 +short git.hubris.network
+# Expected: 192.168.8.175
+
+# 2. Caddy reverse-proxy chain
+curl -sI https://git.hubris.network | head -1
+# Expected: HTTP/2 200
+
+# 3. All app names resolve
+for name in git cloud media paperless photos matrix auth plato artifacto; do
+  result=$(dig @192.168.8.2 +short ${name}.hubris.network)
+  printf "%-20s → %s\n" "${name}.hubris.network" "$result"
+done
+
+# 4. Technitium DHCP scope is correct
+curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq '.response.scopes[0] | {startingAddress, endingAddress}'
+```
+
+---
+
+## Risk assessment
+
+| Risk | Likelihood | Impact | Mitigation |
+|---|---|---|---|
+| API call fails (wrong field names) | Medium | Low | Inspect live scope object first (Task 1); adjust payload |
+| ZimaOS loses IP on next boot | Low | Medium | Task 4 makes ZimaOS static or reserved |
+| Active DHCP client in `.100–.240` gets stranded | Low | Low | Task 3 surfaces this; client just requests a new IP from `.241+` |
+| Technitium admin password file missing | Low | Medium | `/opt/technitium/admin_password.txt` was created during setup; verify existence |
+
+## Open questions
+
+1. Is zimaos (VM 100) currently DHCP or static? The doc says DHCP lease, but it's listed as `lan_ip: 192.168.8.195` in inventory. If it's actually DHCP, it's the one host that needs a static assignment before the pool shrinks.
+2. Are there any transient DHCP clients (test laptops, phones) on the homelab subnet that hold `.100–.240` addresses? Check leases before cutting over.
+3. Should we widen the pool slightly (e.g., `.230–.254`) for more headroom? Currently 14 IPs. If 3+ transient devices are expected, `.230–.254` = 25 IPs — still safe since the highest static is `.239` and `.230–.239` could be excluded.
+
+## Execution preference
+
+All changes are on the Technitium API + homelab-context repo. No LXC/VM restarts needed. The pool shrink takes effect immediately for NEW DHCP requests; existing leases in the old range continue until expiry (24h max).

.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md

@@ -0,0 +1,129 @@
+# Plan: Prevent DHCP IP drift from breaking Caddy backends
+
+**Date:** 2026-06-05  
+**Slug:** prevent-dhcp-ip-drift
+
+---
+
+## Goal
+
+Eliminate the root cause of services becoming unreachable when DHCP lease renewals change backend IPs that Caddy's `reverse_proxy` directives hardcode.
+
+**Triggering incident:** Paperless (LXC 103) and HAOS (VM 108) had DHCP-assigned IPs change from `.130→.243` and `.101→.241` respectively. Caddyfile still pointed at the old IPs → services unreachable from iPhone on Netbird.
+
+## Current context
+
+### DHCP vs static IP inventory
+
+| Machine | Type | PVE ID | Current IP | Allocation method | Status |
+|---------|------|--------|-----------|-------------------|--------|
+| **jellyfin** | LXC | 101 | 192.168.8.206 | Static (`ip=.../24`) | ✅ |
+| **paperless** | LXC | 103 | 192.168.8.243 | **DHCP** (`ip=dhcp`) | ❌ broken, hotfixed to .243 |
+| **gitea** | LXC | 104 | 192.168.8.121 | Static | ✅ |
+| **apps** | LXC | 105 | 192.168.8.205 | Static | ✅ |
+| **nextcloud** | LXC | 114 | 192.168.8.224 | Static | ✅ |
+| **elementsynapse** | LXC | 118 | 192.168.8.239 | Static | ✅ |
+| **mule-images** | LXC | 120 | 192.168.8.136 | Static | ✅ |
+| **caddy** | LXC | 121 | 192.168.8.175 | Static | ✅ |
+| **arriman** | LXC | 122 | 192.168.8.132 | Static | ✅ |
+| **sophia** | LXC | 119 | 192.168.8.157 | Static? | ? (not in 2026-06-02 list) |
+| **nfs-export** | LXC | 102 | 192.168.8.200 | Static? | ? |
+| **plato** | LXC | 126 | 192.168.8.190 | Static? | ? |
+| **HAOS** | VM | 108 | 192.168.8.241 | **DHCP** (VM — OS-managed) | ❌ broken, hotfixed to .241 |
+| **zimaos** | VM | 100 | 192.168.8.195 | DHCP (known stale lease, see note) | ⚠️ open issue |
+| **authentik** | — | — | — | migrated to VPS (external) | N/A |
+
+### Infrastructure facts
+
+- **DHCP server:** Technitium on CT 107 (192.168.8.2), pool `.241–.254`
+- **Static IP range:** `.101–.239`
+- **DNS:** Split-horizon on Technitium — `*.hubris.network → 192.168.8.175` (Caddy itself)
+- **Caddyfile:** Has 29 `reverse_proxy` directives, all using **hardcoded IP:port**
+- **Caddy reload:** Auto-deployed via webhook on git push to `dtoro/caddy-conf`
+- **Documentation:** `inventory.yaml` updated, `hosts/*.yaml` stale-before-regenerate
+
+### Why it happened
+
+1. Paperless LXC 103 was **missed** during the 2026-06-02 static-IP migration (Proxmox config still shows `ip=dhcp`)
+2. HAOS VM 108 uses DHCP internally (HAOS manages its own network — can't just `pct set`)
+3. Both got new IPs from the Technitium `.241–.254` pool after a lease renewal
+4. Caddyfile still pointed at the old static-range IPs → connection refused
+
+---
+
+## Proposed approach
+
+Three-layer solution:
+
+### Layer 1: Fix the immediate offenders (static assignment)
+
+**Paperless LXC 103:**
+- `pct set 103 --net0 name=eth0,bridge=vmbr0,gw=192.168.8.1,hwaddr=BC:24:11:0A:8D:C2,ip=192.168.8.130/24,ip6=auto,type=veth`
+- Inside the LXC, update `/etc/network/interfaces` to match
+- Reboot/restart networking
+
+**HAOS VM 108:**
+- Set a **DHCP reservation** in Technitium for the VM's MAC address, pinning it to `192.168.8.101`
+- This avoids needing to reconfigure HAOS internally (which is tedious)
+- Alternatively: use `ha network update` from the HAOS console to set a static IP
+
+### Layer 2: Audit and fix all remaining DHCP hosts
+
+Check every LXC/VM hosted on hubris:
+```bash
+for ct in $(pct list | awk 'NR>1{print $1}'); do
+  echo "=== CT $ct ==="
+  pct config $ct | grep "^net0"
+done
+```
+Any with `ip=dhcp` that Caddy reverse-proxies to → convert to static.
+
+**Known candidates to check:**
+- CT 102 (nfs-export) — `.200` but not in Caddy. May not need static.
+- CT 119 (sophia) — `.157` — is this static or DHCP? Not sure.
+- CT 126 (plato) — `.190` — same question.
+- VM 100 (zimaos) — `.195` but known to have a stale lease (see 2026-06-03 changelog)
+
+### Layer 3: Add validation
+
+Create a script that runs periodically (cron or homelab cronjob):
+
+**`/opt/homelab-context/scripts/check-caddy-backends.sh`:**
+1. Parse `/etc/caddy/Caddyfile` on CT 121 to extract all `reverse_proxy IP:port` targets
+2. For each `IP:port`, attempt a TCP connect (timeout 3s)
+3. Report any that fail
+
+Could also run as a homelab cron job that notifies dtoro on Matrix if a backend is unreachable.
+
+This catches any future drift proactively (before a user reports it).
+
+### Files likely to change
+
+| File | Change |
+|------|--------|
+| `inventory.yaml` | May update paperless/HAOS IPs if we choose different static IPs |
+| LXC 103 Proxmux config (via `pct set`) | Set static IP |
+| Technitium DHCP reservations | Add HAOS reservation |
+| `/etc/caddy/Caddyfile` on CT 121 | Already fixed — only changes again if we re-assign paperless IP to `.130` |
+| `scripts/check-caddy-backends.sh` | New validation script (new file in homelab-context) |
+
+### Risks / Tradeoffs
+
+- **Pinning paperless to `.130`** — if the LXC was reinstalled since then, `.130` may already be in use. Verify first with `arp-scan` or `nmap`.
+- **HAOS static IP via Technitium reservation** vs **inside HAOS**: Technitium reservation is simpler (no HA config changes), but if HAOS's DHCP lease expires and the Technitium server is down, the reservation won't help. A static IP inside HAOS is more robust but requires poking the HA console.
+- **Validation script false positives** — a service might be legitimately down for maintenance. The script should be a warning, not an alert.
+- **Caddy reload** — each Caddyfile edit triggers an auto-reload via webhook. If the backend is down during reload, Caddy itself stays up (it's just a reverse_proxy target).
+
+### Verification
+
+1. After setting paperless static: `ssh root@192.168.8.175 "curl -s -o /dev/null -w '%{http_code}' http://192.168.8.130:8000"` → 302
+2. After Technitium HAOS reservation: `curl -s -o /dev/null -w '%{http_code}' http://192.168.8.101:8123` → 200
+3. Run validation script → all targets reachable
+4. Confirm from iPhone: both `paperless.hubris.network` and `home.hubris.network` load
+
+### Open questions
+
+1. Should paperless go back to `.130` (its original), or stay at `.243` (current)? Going back to `.130` means updating the Caddyfile again, but keeps the static range allocation consistent.
+2. HAOS: Technitium reservation or HAOS-internal static config? Reservation is easier; HAOS-internal is more robust.
+3. Should the Caddyfile validation script run as a homelab cron job, or as a cron on the caddy LXC itself?
+4. ZimaOS (VM 100) — should we also pin its IP while we're at it?

.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md

@@ -0,0 +1,130 @@
+# Plan: Fix Frequent Authentik Login Prompts
+
+## Goal
+
+Stop requiring repeated login to Authentik (several times per day) by fixing session and cookie expiry settings so the user stays logged in for longer periods (e.g., 7–30 days, or until explicit logout).
+
+## Current Context
+
+Authentik runs on the VPS (`82.165.190.79`) in Docker Compose. Traffic flows:
+
+```
+Browser → Caddy (LXC 121) → VPS Traefik → Authentik
+```
+
+Caddy's `forward_auth` uses the `(authentik)` snippet which proxies to `auth.hubris.network/outpost.goauthentik.io/auth/caddy`. The Authentik server version is **2026.5.2**.
+
+## Root Cause Found
+
+### Primary: `SESSION_EXPIRE_AT_BROWSER_CLOSE = True`
+
+The Authentik Django session (`authentik_session` cookie) is configured to **expire on browser close**. Every time the user closes and reopens their browser, the session cookie is cleared. The next visit to a service that requires OAuth2 authorization (Gitea, Jellyfin, etc.) will redirect to the Authentik login page.
+
+### Secondary: `SESSION_COOKIE_AGE = 86400` (24 hours)
+
+Even with the browser left open continuously, the session expires after 24 hours. Combined with `SESSION_SAVE_EVERY_REQUEST = False`, activity does NOT extend the session.
+
+### Session configuration (from Docker Python environment):
+
+| Setting | Current Value | Default in Django |
+|---------|---------------|-------------------|
+| `SESSION_EXPIRE_AT_BROWSER_CLOSE` | `True` | `False` |
+| `SESSION_COOKIE_AGE` | `86400` (24h) | `1209600` (14d) |
+| `SESSION_SAVE_EVERY_REQUEST` | `False` | `False` |
+| `SESSION_COOKIE_SAMESITE` | `Lax` | `Lax` |
+
+### What ISN'T the problem:
+
+- **Proxy cookie validity** — `hubris-forward-auth` has `access_token_validity = hours=24`, which is reasonable for the forward-auth token.
+- **Server-side session duration** — The `user_login` stage has `session_duration = seconds=0` (indefinite).
+- **Refresh tokens** — All OAuth2 providers have `refresh_token_validity = days=30`, which is fine.
+- **Caddy configuration** — The forward-auth chain is correctly set up.
+- **Outpost health** — All containers healthy, up for 6 days.
+
+## Proposed Approach
+
+Change two Django session settings via Authentik environment variables:
+
+1. **`AUTHENTIK_SESSION_COOKIE_AGE` = 604800** (7 days) — extends session cookie lifetime from 24h to 7 days
+2. **`AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` = false** — prevents session cookie from being cleared on browser close
+
+This keeps users logged in for up to 7 days with normal browser use (close/reopen, daily usage). The session still expires after 7 days of inactivity (`SESSION_SAVE_EVERY_REQUEST` stays False).
+
+## Step-by-step Plan
+
+### Step 1: Add environment variables to Docker compose
+
+Edit `/opt/docker-compose.yml` on the VPS to add these env vars to the `authentik-server` service:
+
+```yaml
+authentik-server:
+  environment:
+    # ... existing vars ...
+    AUTHENTIK_SESSION_COOKIE_AGE: "604800"      # 7 days (was 86400 / 24h)
+    AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE: "false"  # was true
+```
+
+Note: the Authentik config system uses `__` (double underscore) for nesting. The env vars map to the Django settings via the config YAML path. The correct Authentik env var for `SESSION_COOKIE_AGE` would be `AUTHENTIK_SESSION__COOKIE_AGE` if it goes through the config system, or just `SESSION_COOKIE_AGE` if it's passed directly. Need to verify the exact variable name Authentik expects.
+
+### Step 2: Verify variable naming
+
+Check the Authentik config YAML (`/authentik/lib/default.yml` inside the container) to confirm the exact env var name mapping. Authentik uses a custom config layer that maps env vars to settings.
+
+**Alternative if env vars don't work:** Some Authentik settings need to be set via the admin UI (under System Settings or Tenant settings). The Django session settings might need to be configured differently in this version.
+
+### Step 3: Restart Authentik server
+
+```bash
+ssh root@82.165.190.79
+docker compose -f /opt/docker-compose.yml restart authentik-server
+```
+
+### Step 4: Verify the fix
+
+```bash
+# Check session settings took effect
+ssh root@82.165.190.79 'docker exec -i authentik-server python3 << "PYEOF"
+import os
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
+import django
+django.setup()
+from django.conf import settings
+print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)
+print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)
+PYEOF'
+```
+
+### Step 5: Functional test
+
+1. Login to Authentik at `auth.hubris.network`
+2. Close the browser completely
+3. Re-open browser, navigate to a forward-auth-gated service (e.g., paperless.hubris.network)
+4. Verify you're NOT redirected to login
+5. Verify OAuth2 services (Gitea) also maintain the session
+
+## Files Likely to Change
+
+| File | Change |
+|------|--------|
+| `/opt/docker-compose.yml` | Add `AUTHENTIK_SESSION_COOKIE_AGE` and `AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` env vars |
+
+## Tests / Validation
+
+1. **Config verification** — Run Python snippet inside container to confirm Django settings changed
+2. **Browser test** — Close/reopen browser, verify session persists (Step 5 above)
+3. **24-hour test** — Check session is still alive after 24h of normal use
+
+## Risks, Tradeoffs, and Open Questions
+
+| Risk | Mitigation |
+|------|------------|
+| Env var names don't match Authentik's config schema | First verify in the container's `default.yml` config file |
+| 7-day persistent cookie is a security concern (stolen cookie = 7 days of access) | This is the same risk as any "Remember Me" feature on any web app. The tradeoff is convenience vs. security. |
+| The proxy cookie (`authentik_proxy_*`) may still have its own 24h limit | That's managed separately via the OAuth2 provider's `access_token_validity` setting. If we also want to extend that, we can update `hubris-forward-auth` provider's `access_token_validity` from `hours=24` to `days=7`. |
+| `SESSION_COOKIE_SECURE = False` | Should be `True` since Authentik is served behind HTTPS. However, the forward-auth subrequest from Caddy to the outpost is HTTP internally (`http://127.0.0.1:8099`), so `False` may be intentional for the outpost check. |
+
+## Open Questions
+
+1. **What environment variable name does Authentik use for Django session settings?** Need to check `default.yml`. The config layer may use `AUTHENTIK_SESSION__COOKIE_AGE` (double underscore) or the raw Django setting name.
+2. **Should we also extend the proxy token validity?** The `hubris-forward-auth` provider has `access_token_validity = hours=24`. If we want users to not need re-login for more than 24h, we should also bump this to match the session cookie age.
+3. **Which specific service triggers the most login prompts?** The forward-auth (Caddy-gated) services use proxy cookies. OAuth2 services (Gitea, Jellyfin) use the Django session. Understanding which one the user is hitting most could narrow the fix scope.

.hermes/plans/2026-06-06_234500-caddyfile-truncation-permanent-fix.md

@@ -0,0 +1,149 @@
+# Plan: Fix Caddyfile truncation + prevent recurring outages
+
+**Date:** 2026-06-06
+**Slug:** caddyfile-truncation-permanent-fix
+
+---
+
+## Goal
+
+Restore all `*.hubris.network` services that went offline when the Caddyfile on LXC 121 was truncated to only 3 photo-related site blocks, and implement automated safeguards to prevent this class of outage from recurring.
+
+## Root cause
+
+The Caddyfile at `/etc/caddy/Caddyfile` on LXC 121 was manually edited locally (not via the `dtoro/caddy-conf` git repo), overwriting ~260 lines (30+ site blocks + forward-auth infrastructure) with only 43 lines covering `photos.hubris.network`, `prism.hubris.network`, and a manually-added `photos2.hubris.network`.
+
+**Evidence:**
+- `git diff HEAD -- Caddyfile` shows `+3 / -159` lines diff — all other blocks deleted
+- Git reflog shows HEAD at `32575ce` (`fix: sab... port 8081→8082`), but working tree diverges
+- Deploy webhook log: Jun 06 12:39 — `deploy failed: git pull` (dirty tree blocks merge)
+- Backup file `Caddyfile.bak.1780263919` (225 lines) confirms the full original was intact before truncation
+- `origin/master` at `1b977aa` is the authoritative source — 260 lines, all blocks present
+
+**Why "third time this week":**
+| Incident | Date | Cause |
+|---|---|---|
+| 1 | Jun 02 | DHCP IP drift — paperless (130→243), HAOS (101→241) |
+| 2 | Jun 05 | More DHCP drift — apps (205), mule-images (136 overridden by dhclient) |
+| 3 | Jun 06 | **Caddyfile truncated** — unrelated to IPs, much worse |
+
+The Caddyfile truncation is the most severe: it took down **all LAN services** except `photos.hubris.network` and `auth.hubris.network` (VPS-hosted).
+
+## Immediate fix
+
+### Step 1: Restore Caddyfile from origin/master and reload
+
+On LXC 121:
+
+```bash
+cd /etc/caddy
+# Stash any local changes
+git stash
+# Reset to origin/master
+git checkout --force origin/master -- Caddyfile
+# Caddyfile now has all 30+ sites
+caddy validate --config /etc/caddy/Caddyfile
+systemctl reload caddy
+```
+
+This restores all service blocks including: media, git, paperless, books, home, cloud, matrix, proxmox, docker, jellyseerr, qbit, sab, blog, auth, artifacto, plato, zimaos, mcp, secrets, sso + authentik forward-auth infrastructure.
+
+### Step 2: Add `photos2.hubris.network` via git (if still needed)
+
+The `photos2.hubris.network` block was manually added locally and is NOT in origin/master. If the user wants to keep it, submit a PR/commit to the `dtoro/caddy-conf` repo.
+
+### Step 3: Verify
+
+- From any LAN/mesh client: `curl -sk https://media.hubris.network/` → 200
+- Run `bash /opt/homelab-context/scripts/check-caddy-backends.sh` from hubris → all targets reachable
+- Flush mac-mini DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
+
+## Permanent safeguards
+
+### Layer 1: Caddyfile integrity check (deploy hook)
+
+Add a site-count validation to the deploy script (`/etc/caddy/scripts/deploy.sh`):
+
+```bash
+# Count site blocks (lines matching *.hubris.network {)
+SITE_COUNT=$(grep -c '^[a-z].*hubris.network {' Caddyfile)
+if [ "$SITE_COUNT" -lt 20 ]; then
+  echo "[deploy] ERROR: Only $SITE_COUNT sites found (expected 20+). Refusing to reload."
+  exit 1
+fi
+```
+
+This catches any future truncation before `caddy reload` runs.
+
+### Layer 2: Caddyfile backup on deploy
+
+Add to deploy script before git pull:
+
+```bash
+cp Caddyfile "Caddyfile.bak.$(date +%s)"
+```
+
+Keep last 3 backups, auto-rotate.
+
+### Layer 3: Dirty-tree handling in deploy webhook
+
+The deploy webhook currently hard-fails when the working tree is dirty. Change the receiver script to handle this gracefully:
+
+```bash
+cd /etc/caddy
+# If dirty, stash local changes
+if ! git diff --quiet; then
+  echo "[deploy] Working tree dirty — stashing"
+  git stash push -m "auto-stash by deploy webhook $(date)"
+fi
+git pull --ff-only
+```
+
+This prevents the webhook from blocking on future local edits.
+
+### Layer 4: Scheduled Caddyfile health check
+
+Add a homelab cron job that runs `check-caddy-backends.sh` every 10 minutes and notifies if any Caddy backend is unreachable.
+
+```yaml
+# In homelab context: cronjob
+schedule: "*/10 * * * *"
+script: /opt/homelab-context/scripts/check-caddy-backends.sh
+```
+
+### Layer 5: DNS sync cron (fix already-deployed sync)
+
+The `dns-sync.py` on LXC 107 at `/opt/dns-sync/sync.py` is installed but has **no crontab** — the sync never runs automatically. The NetBird managed DNS zone has drifted from Technitium. Add a systemd timer or crontab:
+
+```bash
+echo "*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1" > /etc/cron.d/dns-sync
+```
+
+## Files likely to change
+
+| File | Change |
+|------|--------|
+| `/etc/caddy/Caddyfile` on LXC 121 | Restore from origin/master |
+| `/etc/caddy/scripts/deploy.sh` on LXC 121 | Add site-count validation + backup + dirty-tree handling |
+| `caddy-conf` git repo | PR with deploy.sh improvements + photos2 (if wanted) |
+| `cronjob` in Hermes | Schedule `check-caddy-backends.sh` |
+| `/etc/cron.d/dns-sync` on LXC 107 | New — add dns-sync cron |
+
+## Verification
+
+1. All `*.hubris.network` URLs load from mac-mini: `media`, `git`, `paperless`, `cloud`, `home`, `proxmox`, etc.
+2. `check-caddy-backends.sh` exits 0 on hubris
+3. `systemctl status caddy` shows active on LXC 121
+4. `dns-sync` runs and writes to `/var/log/dns-sync.log`
+
+## Risks / Tradeoffs
+
+- **Restoring from origin/master overwrites photos2.hubris.network** — recreate it via proper git commit
+- **Caddy staging ACME certs for prism/photos2**: The `tls dns ionos` directive uses staging env (`acme-staging-v02.api.letsencrypt.org`), which fails DNS propagation check (VPS port 53 unreachable from LXC). Once restored, these two subdomains will have the same issue. Move them to production IONOS DNS-01 by removing the staging CA directive or setting the correct `acme_issuer` in Caddyfile.
+- **Dirty-tree stash could lose edits** — mitigated by `git stash push --message` + backup file creation before stash
+
+## Open questions
+
+1. Keep `photos2.hubris.network`? If yes, add via proper git push.
+2. `prism.hubris.network` and `photos2` certs fail on staging ACME — set production `acme_issuer` in Caddyfile?
+3. Should `check-caddy-backends.sh` run as a homelab cron job or as a regular cron on LXC 121?

.sops.yaml

@@ -18,12 +18,23 @@
 creation_rules:
   - path_regex: ^secrets/hello\.yaml$
     # The "hello" secret is encrypted to every enrolled client so the bootstrap
-    # decrypt test works for everyone. Populated by the operator with the
-    # age_pubkey from each hosts/<name>.yaml after Phase 3a generates keys.
+    # decrypt test works for everyone. Add each new client's age_pubkey when
+    # they enrol; re-key with `sops updatekeys -y secrets/hello.yaml`.
     age: >-
-      # placeholder — operator fills in after first issuance run.
-      # comma-separated age public keys, e.g.:
-      #   age1xxx,age1yyy,age1zzz
+      age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
+      age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
+      age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
+      age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+
+  - path_regex: ^secrets/gitea-pat\.yaml$
+    # Write-scoped Gitea PAT (dtoro user). Same recipient list as hello.yaml
+    # since every enrolled client should be able to push (homelab client
+    # add/remove, wiki edits, etc.).
+    age: >-
+      age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
+      age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
+      age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
+      age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
 
   - path_regex: ^secrets/gitea-tokens\.yaml$
     # Workstations only.
@@ -33,4 +44,47 @@ creation_rules:
   - path_regex: ^secrets/webhook-hmacs\.yaml$
     # LXCs that run a webhook receiver.
     age: >-
-      # placeholder — fill with age_pubkey of: apps, caddy, claudio-bot, claudio-monitor host
+      # placeholder — fill with age_pubkey of: apps, caddy
+
+  - path_regex: ^secrets/turn-shared-secret\.yaml$
+    # coturn TURN long-term-credential password. Consumed by hubris (which
+    # renders /etc/turnserver.conf + /opt/management.json on the VPS via
+    # `homelab render-vps-configs`). Other recipients are convenience for
+    # operator debugging — only hubris's pubkey is strictly required.
+    age: >-
+      age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
+      age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
+      age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
+      age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+
+  - path_regex: ^secrets/netbird-authentik-oidc\.yaml$
+    # Authentik OIDC client secret for the netbird-dashboard provider.
+    # Consumed by hubris to render /opt/management.json on the VPS
+    # (PKCEAuthorizationFlow.ProviderConfig.ClientSecret).
+    age: >-
+      age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
+      age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
+      age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
+      age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+
+  - path_regex: ^secrets/netbird-pat\.yaml$
+    # NetBird API Personal Access Token. Consumed by the dns-sync job on the
+    # `dns` LXC (107) to reconcile Technitium -> NetBird managed DNS zone.
+    # (When 107 is enrolled, add its age_pubkey here and updatekeys.)
+    age: >-
+      age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
+      age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0,
+      age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
+      age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+
+  - path_regex: ^secrets/openrouter-api-key\.yaml$
+    # OpenRouter API key consumed by the `hermes` wrapper (bin/hermes) when
+    # spawning a Goose session. Recipients are any host that should run a
+    # Nous-Hermes agent. Add a host's age_pubkey here, then
+    # `sops updatekeys -y secrets/openrouter-api-key.yaml`.
+    # See operations/hermes-agent.md.
+    age: >-
+      age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6,
+      age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6,
+      age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+# webhook noop 2026-05-20T18:16:57+02:00

AGENTS.md

@@ -19,8 +19,8 @@ the operator to run `homelab client add <hostname>` from an existing client.
 - `/opt/homelab-context/inventory.yaml` — every host, LXC, VM, and workstation
   with their mesh addresses, roles, and service mappings. Treat this file as
   authoritative; anything you read in narrative pages should agree with it.
-- `/opt/homelab-context/infrastructure/mesh.md` — Tailscale → Netbird state.
-  Both meshes are accepted today; Netbird is preferred for new traffic.
+- `/opt/homelab-context/infrastructure/mesh.md` — Netbird mesh state.
+  Tailscale was decommissioned 2026-06-21; the fleet is Netbird-only.
 - `/opt/homelab-context/infrastructure/dns.md` — split-horizon DNS via
   dnsmasq on LXC 124. `*.hubris.network` resolves to 192.168.x.x on the LAN
   and to mesh addresses off-LAN.
@@ -83,7 +83,28 @@ Grep is fine for browsing or when MCP is unreachable.
 - **Wiki updates**: same-session rule applies to any meaningful state change
   this client makes.
 
-## 6. When in doubt
+## 6. Auto-setup mechanism
+
+The homelab-context repo ships tooling that gets automatically installed
+on every client after `git pull`. This is handled by `tools/post-pull.sh`
+(replaces the raw git pull in the sync timer) which runs any script matching
+`tools/*.setup.sh` after pull.
+
+Currently auto-setup:
+- **Caveman + templates** (`tools/setup-caveman.sh`): Installs Caveman npm
+  package, wrapper scripts, and compact output templates for token-efficient
+  CLI output. Wrapper at `~/bin/caveman_wrapper.sh`.
+- **Hermes agent persona** (`tools/setup-hermes-soul.sh`): Provisions
+  `~/.hermes/SOUL.md` from `HERMES.md` on Hermes agents. This ensures every
+  Hermes agent follows the canonical homelab persona (token efficiency, source
+  of truth hierarchy). No-op on non-Hermes agents.
+
+To add a new auto-setup, create `tools/<name>.setup.sh` in the repo,
+commit and push. All enrolled clients pick it up within 5 minutes.
+
+To trigger sync manually: `sudo homelab sync` or wait for the 5-min timer.
+
+## 7. When in doubt
 
 Run `homelab mcp search_docs <query>` or `homelab mcp get_host <name>`.
 The clone is the fallback; MCP is the index.

CONTRIBUTING.md

@@ -67,6 +67,30 @@ Links to nodes that host or depend on this.
 ## Changelog
 ```
 
+### Plan (`plans/YYYY-MM-DD-slug.md`)
+
+```markdown
+# YYYY-MM-DD — <title>
+
+## Goal
+What this change achieves and why.
+
+## Current topology / state
+Diagram or description of what exists now.
+
+## Target topology / state
+What it looks like after.
+
+## Pre-flight checklist
+
+## Step-by-step procedure
+
+## Verification
+
+## Post-migration
+Changelog entries to write, index status to update.
+```
+
 ### Investigation (`investigations/YYYY-MM-DD-slug.md`)
 
 ```markdown
@@ -89,6 +113,7 @@ Links to nodes that host or depend on this.
 - Every container page links to every cross-cutting page it participates in.
 - Every cross-cutting page lists the nodes that participate.
 - Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
+- Every plan links to the infrastructure pages it affects. When done, update the plan's status in `plans/index.md` and write changelog entries on affected node pages.
 
 ## Changelog hygiene
 

HERMES.md

@@ -0,0 +1,83 @@
+# HERMES.md — Agent persona for homelab clients
+
+This file is the canonical agent persona for **all** AI agents running on
+machines in the **hubris** homelab. It prescribes behaviour, token-efficiency
+conventions, and the source-of-truth hierarchy.
+
+## Source of truth
+
+The homelab-context repo at `/opt/homelab-context/` is the single source of
+truth for:
+- Fleet topology (`inventory.yaml`, `hosts/*.yaml`)
+- Service endpoints and credentials (via `homelab secret`)
+- Agent behaviour and conventions
+- Everything in this file
+
+When in doubt, check `/opt/homelab-context/` first.
+
+## Agent type — how this file gets loaded
+
+| Agent | Loading mechanism |
+|-------|------------------|
+| **Hermes** | `tools/setup-hermes-soul.sh` (auto-setup) → provisions `~/.hermes/SOUL.md` from this file |
+| **Goose** | `.goosehints` symlink at `~/.config/goose/.goosehints` → `/opt/homelab-context/HERMES.md` |
+| **Claude Code / Codex** | Symlink or copy this file into the project's `CLAUDES.md` / `.claude` instructions |
+
+**Do not edit SOUL.md or .goosehints directly.** Edit this file in the
+homelab-context repo instead. Changes propagate to all clients on the next
+sync (`sudo homelab sync`).
+
+---
+
+## Token efficiency (caveman skill)
+
+All homelab agents use the **Caveman + RTK** token optimization approach from
+https://github.com/adityahimaone/hermes-agent-rtk-caveman.
+
+### Before running any CLI command, ask:
+
+1. **Is there a caveman wrapper equivalent?** Use the wrapper for token-efficient
+   output. Available wrappers (installed at `~/bin/caveman_wrapper.sh`):
+   - `~/bin/caveman_wrapper.sh git-status` — compact git status
+   - `~/bin/caveman_wrapper.sh git-log [n]` — compact git log
+   - `~/bin/caveman_wrapper.sh lint [target]` — compact lint results
+   - `~/bin/caveman_wrapper.sh test-results [cmd]` — compact test results
+
+2. **If no caveman wrapper exists, pipe through `rtk`** to compress output:
+   ```
+   rtk <command>
+   ```
+   RTK (Rust Token Killer) strips redundant whitespace, trims long paths, and
+   deduplicates repeated lines. This reduces token usage by 60-90% on CLI
+   operations.
+
+3. **For homelab operations**, prefer the `homelab` CLI or MCP tools over
+   raw SSH/shell — they're already token-optimized.
+
+### Templates
+
+Caveman templates live at `~/templates/`:
+- `git_status.txt` — compact git status format
+- `git_log.txt` — compact git log format
+- `lint_results.txt` — compact ESLint format
+- `test_results.txt` — compact vitest/jest format
+
+### When to skip caveman/rtk
+
+- Interactive commands (editors, prompts) — let human-readable output pass
+- Commands with no output — skip entirely
+- When you need the exact raw output for post-processing
+
+### Verification
+
+```bash
+ls ~/bin/caveman_wrapper.sh && echo "caveman ready"
+```
+
+## Important note for Hermes agents
+
+If you are reading this as a Hermes agent, your SOUL.md was auto-provisioned
+by `tools/setup-hermes-soul.sh`. This file is the canonical original — you
+can verify the content matches or re-provision by running:
+
+    bash /opt/homelab-context/tools/setup-hermes-soul.sh

README.md

@@ -29,26 +29,27 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links:
 | 120 | [mule-images](containers/120-mule-images.md)       | 192.168.8.136 | Mule-image / mulita photos                    |
 | 121 | [caddy](containers/121-caddy.md)                   | 192.168.8.175 | Reverse proxy                                 |
 | 122 | [arriman](containers/122-arriman.md)               | 192.168.8.132 | Docker host (\*arr stack)                     |
-| 123 | [claudio-bot](containers/123-claudio-bot.md)       | 192.168.8.230 | Matrix control plane                          |
 | 124 | [authentik](containers/124-authentik.md)           | 192.168.8.180 | SSO + split-horizon DNS                       |
 | 126 | [plato](containers/126-plato.md)                   | 192.168.8.190 | Plato (notes/discovery workspace)             |
 
 ### Cross-cutting infrastructure
 - [DNS — split-horizon](infrastructure/dns.md)
 - [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
-- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
-- [Monitoring — claudio-monitor](infrastructure/monitoring.md)
+- [Mesh — Netbird overlay](infrastructure/mesh.md)
+- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md)
 - [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
 - [SSH access](infrastructure/ssh-access.md)
 - [Backups — restic on external drive (disabled)](infrastructure/backups.md)
 - [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md)
 - [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md)
+- [Homelab context distribution](infrastructure/homelab-context.md) — cross-client `/opt/homelab-context` + MCP + secrets-issuance
 
 ### Investigations
 Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md).
 
 ### Operations
 - [Command cheatsheet](operations/commands.md)
+- [Agent enrollment](operations/agent-enrollment.md) — bootstrap a new client (workstation, LXC, VM) into the homelab context system
 
 ## Conventions
 
@@ -59,7 +60,7 @@ Time-stamped incident notes / experiments in [`investigations/`](investigations/
   ```
 - **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
 - **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
-- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image, claudio-bot) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
+- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
 - **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
 
 ## Maintaining this wiki

bin/hermes

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+#
+# hermes — launch a Goose session pre-wired with the homelab persona,
+# OpenRouter (Nous Hermes) provider, and the homelab MCP server.
+#
+# See operations/hermes-agent.md for the full onboarding flow.
+
+set -euo pipefail
+
+die() { echo "hermes: $*" >&2; exit 1; }
+
+command -v goose >/dev/null \
+    || die "goose binary not found — re-run bootstrap.sh with --with-hermes"
+command -v homelab >/dev/null \
+    || die "homelab CLI not found — is this client bootstrapped?"
+
+# Decrypt OpenRouter API key.
+# `homelab secret` re-execs via sudo for non-root users (age key is 0600 root).
+SECRET_YAML=$(homelab secret openrouter-api-key 2>&1) || \
+    die "could not decrypt secrets/openrouter-api-key.yaml — this host probably
+isn't a recipient yet. See operations/hermes-agent.md ('Granting the OpenRouter
+key to a new host'). sops output:
+${SECRET_YAML}"
+
+API_KEY=$(printf '%s' "$SECRET_YAML" | python3 -c \
+    'import sys, yaml; print(yaml.safe_load(sys.stdin)["api_key"])')
+
+case "$API_KEY" in
+    PLACEHOLDER_*|"")
+        die "openrouter-api-key.yaml still contains the placeholder; operator
+must run \`sops secrets/openrouter-api-key.yaml\` on hubris to insert a real
+\`sk-or-...\` key and push the change." ;;
+esac
+
+export OPENROUTER_API_KEY="$API_KEY"
+exec goose session "$@"

bootstrap.sh

@@ -5,6 +5,7 @@
 #   curl -fsSL https://git.hubris.network/dtoro/Homelab-Docs/raw/main/bootstrap.sh \
 #     | sudo bash
 #   curl ... | sudo bash -s -- --with-mcp           # also wire Claude's .mcp.json
+#   curl ... | sudo bash -s -- --with-hermes        # also install Goose + Hermes wrapper
 #   curl ... | sudo bash -s -- --dry-run            # show what would happen
 #   curl ... | sudo bash -s -- --no-secrets         # skip age-key issuance
 #
@@ -20,20 +21,28 @@ set -euo pipefail
 # -------- defaults --------
 REPO_HTTPS="${HOMELAB_REPO_URL:-https://git.hubris.network/dtoro/Homelab-Docs.git}"
 CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab-context}"
-ISSUANCE_URL_NETBIRD="${HOMELAB_ISSUANCE_NETBIRD:-http://apps.netbird.selfhosted:9820/issue}"
-ISSUANCE_URL_TAILSCALE="${HOMELAB_ISSUANCE_TAILSCALE:-http://apps.ts:9820/issue}"
-MCP_URL="${HOMELAB_MCP_URL:-http://apps.netbird.selfhosted:9810/sse}"
+ISSUANCE_URL_NETBIRD="${HOMELAB_ISSUANCE_NETBIRD:-https://secrets.hubris.network/issue}"
+ISSUANCE_URL_TAILSCALE="${HOMELAB_ISSUANCE_TAILSCALE:-https://secrets.hubris.network/issue}"
+MCP_URL="${HOMELAB_MCP_URL:-https://mcp.hubris.network/mcp}"
+HERMES_MCP_URI="${HOMELAB_HERMES_MCP_URI:-https://mcp.hubris.network/mcp}"
+HERMES_MODEL="${HOMELAB_HERMES_MODEL:-nousresearch/hermes-4-405b}"
 
 WITH_MCP=0
+WITH_HERMES=0
 DRY_RUN=0
 NO_SECRETS=0
+GITEA_TOKEN="${HOMELAB_GITEA_TOKEN:-}"
+GITEA_USER="${HOMELAB_GITEA_USER:-dtoro}"
 
 # -------- flag parsing --------
 while [ $# -gt 0 ]; do
     case "$1" in
         --with-mcp)    WITH_MCP=1; shift ;;
+        --with-hermes) WITH_HERMES=1; shift ;;
         --dry-run)     DRY_RUN=1; shift ;;
         --no-secrets)  NO_SECRETS=1; shift ;;
+        --gitea-token) GITEA_TOKEN="$2"; shift 2 ;;
+        --gitea-user)  GITEA_USER="$2"; shift 2 ;;
         --help|-h)
             sed -n '2,11p' "$0" | sed 's/^# *//'
             exit 0
@@ -43,6 +52,29 @@ while [ $# -gt 0 ]; do
     esac
 done
 
+# If a gitea token is provided, write it to the standard credential store
+# BEFORE the clone happens. The HTTPS REPO_HTTPS will then pick it up via
+# git's credential helper.
+configure_gitea_creds() {
+    if [ -z "$GITEA_TOKEN" ]; then return 0; fi
+    local creds_dir=/etc/homelab-context
+    local creds_file=$creds_dir/git-credentials
+    mkdir -p "$creds_dir"
+    chmod 700 "$creds_dir"
+    # Format the credential URL: <scheme>://user:token@host (scheme must match
+    # the actual REPO_HTTPS — git's credential helper does exact prefix match).
+    local proto host
+    proto=$(echo "$REPO_HTTPS" | sed -E 's|^(https?)://.*|\1|')
+    host=$(echo "$REPO_HTTPS" | sed -E 's|^https?://([^/]+).*|\1|')
+    printf '%s://%s:%s@%s\n' "$proto" "$GITEA_USER" "$GITEA_TOKEN" "$host" > "$creds_file"
+    chmod 600 "$creds_file"
+    # Point git at this store system-wide (/etc/gitconfig) so the systemd
+    # sync timer's git — which runs as root without HOME set — finds it.
+    # --global writes to /root/.gitconfig which the timer doesn't read.
+    git config --system credential.helper "store --file=$creds_file"
+    echo "[bootstrap] wrote gitea credentials to $creds_file"
+}
+
 run() {
     if [ "$DRY_RUN" -eq 1 ]; then
         printf '+ %s\n' "$*"
@@ -78,34 +110,143 @@ echo "[bootstrap] hostname: $HNAME"
 
 # Check dependencies.
 missing=()
-for cmd in git; do command -v "$cmd" >/dev/null || missing+=("$cmd"); done
+for cmd in git python3; do command -v "$cmd" >/dev/null || missing+=("$cmd"); done
+# The homelab CLI needs PyYAML.
+if ! python3 -c "import yaml" >/dev/null 2>&1; then
+    missing+=("python3-yaml")
+fi
 if [ "$NO_SECRETS" -eq 0 ]; then
     for cmd in age sops; do command -v "$cmd" >/dev/null || missing+=("$cmd"); done
 fi
 if [ "${#missing[@]}" -gt 0 ]; then
-    echo "missing required tools: ${missing[*]}" >&2
-    if [ "$OS" = "Darwin" ]; then
-        echo "  brew install ${missing[*]}"
+    if [ "$DRY_RUN" -eq 1 ]; then
+        echo "+ would install missing tools: ${missing[*]}"
     else
-        echo "  apt install -y ${missing[*]}  (or platform equivalent)"
-    fi
+        echo "[bootstrap] installing missing tools: ${missing[*]}"
+        if [ "$OS" = "Darwin" ]; then
+            brew_list=()
+            for m in "${missing[@]}"; do
+                case "$m" in
+                    python3-yaml) python3 -m pip install --break-system-packages pyyaml >/dev/null 2>&1 \
+                                  || python3 -m pip install pyyaml ;;
+                    *) brew_list+=("$m") ;;
+                esac
+            done
+            [ "${#brew_list[@]}" -gt 0 ] && brew install "${brew_list[@]}"
+        elif command -v dnf >/dev/null 2>&1; then
+            dnf_list=()
+            for m in "${missing[@]}"; do
+                case "$m" in
+                    python3-yaml) dnf_list+=("python3-pyyaml") ;;
+                    *)            dnf_list+=("$m") ;;
+                esac
+            done
+            dnf install -y "${dnf_list[@]}"
+        elif command -v apt-get >/dev/null 2>&1; then
+            DEBIAN_FRONTEND=noninteractive apt-get update
+            DEBIAN_FRONTEND=noninteractive apt-get install -y "${missing[@]}"
+        else
+            echo "[bootstrap] no supported package manager for: ${missing[*]}" >&2
+            echo "[bootstrap] install with your package manager + re-run" >&2
             exit 1
         fi
+        # Re-verify (especially python yaml — the rename is the most common gotcha).
+        for cmd in git python3; do
+            command -v "$cmd" >/dev/null || { echo "[bootstrap] post-install $cmd still missing" >&2; exit 1; }
+        done
+        python3 -c "import yaml" 2>/dev/null \
+            || { echo "[bootstrap] post-install python3-yaml/pyyaml still missing" >&2; exit 1; }
+        if [ "$NO_SECRETS" -eq 0 ]; then
+            for cmd in age sops; do
+                command -v "$cmd" >/dev/null || { echo "[bootstrap] post-install $cmd still missing" >&2; exit 1; }
+            done
+        fi
+    fi
+fi
 
-# Mesh check — accept either Netbird OR Tailscale.
+# -------- ensure netbird is installed + connected (workstation/VM hosts) --------
+# Skipped on --no-secrets (LXCs that route via the LAN already) and --dry-run.
+# Installs netbird if missing, then drives `netbird up` against the homelab
+# management server. The operator clicks the printed device-code URL once.
+if [ "$NO_SECRETS" -eq 0 ] && [ "$DRY_RUN" -eq 0 ]; then
+    if ! command -v netbird >/dev/null 2>&1 && ! command -v tailscale >/dev/null 2>&1; then
+        echo "[bootstrap] no mesh CLI found; installing netbird..."
+        if [ "$OS" = "Darwin" ]; then
+            brew install --cask netbird || { echo "[bootstrap] brew install netbird failed" >&2; exit 1; }
+        elif command -v dnf >/dev/null 2>&1; then
+            cat > /etc/yum.repos.d/netbird.repo <<'NBREPO'
+[netbird]
+name=NetBird stable
+baseurl=https://pkgs.netbird.io/yum/$basearch
+enabled=1
+gpgcheck=0
+NBREPO
+            dnf install -y netbird netbird-ui || { echo "[bootstrap] dnf install netbird failed" >&2; exit 1; }
+        elif command -v apt-get >/dev/null 2>&1; then
+            install -d -m 0755 /usr/share/keyrings
+            curl -fsSL https://pkgs.netbird.io/debian/public.key \
+                | gpg --dearmor -o /usr/share/keyrings/netbird-archive-keyring.gpg
+            echo "deb [signed-by=/usr/share/keyrings/netbird-archive-keyring.gpg] https://pkgs.netbird.io/debian stable main" \
+                > /etc/apt/sources.list.d/netbird.list
+            apt-get update
+            DEBIAN_FRONTEND=noninteractive apt-get install -y netbird \
+                || { echo "[bootstrap] apt install netbird failed" >&2; exit 1; }
+        else
+            echo "[bootstrap] can't auto-install netbird on this OS; install manually + re-run" >&2
+            exit 1
+        fi
+    fi
+    # Bring netbird up if not already connected.
+    if command -v netbird >/dev/null && ! netbird status 2>/dev/null | grep -q "Management: Connected"; then
+        cat <<MSG
+
+[bootstrap] netbird is not connected to https://netbird.hubris.network.
+[bootstrap] running 'netbird up' — a device-code URL will print below.
+[bootstrap] OPEN THAT URL in a browser and approve the device when prompted.
+[bootstrap] You may need to log in to https://auth.hubris.network first.
+
+MSG
+        # --ssh-jwt-cache-ttl=86400 keeps the SSO valid for 24h of subsequent ssh
+        # ops into mesh peers; saves repeated browser clicks during this bootstrap.
+        netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400 \
+            || { echo "[bootstrap] 'netbird up' failed (see error above)" >&2; exit 1; }
+        # `netbird up` returns once the device-code SSO completes; give the
+        # mgmt connection ~30s to settle before continuing.
+        for _ in $(seq 1 10); do
+            netbird status 2>/dev/null | grep -q "Management: Connected" && break
+            sleep 3
+        done
+        if ! netbird status 2>/dev/null | grep -q "Management: Connected"; then
+            echo "[bootstrap] netbird daemon not reporting Management: Connected after 30s" >&2
+            echo "[bootstrap] try: 'netbird status -d' and 'sudo journalctl -u netbird -n 30'" >&2
+            exit 1
+        fi
+        echo "[bootstrap] netbird connected."
+    fi
+fi
+
+# Mesh check — accept Netbird, Tailscale, or LAN reachability of the issuance
+# endpoint. LAN is fine for LXCs that don't run a mesh CLI but sit in the
+# trusted subnet (192.168.8.0/24) included in MESH_SUBNETS.
 MESH_CONNECTED=""
 if command -v netbird >/dev/null && netbird status 2>/dev/null | grep -q "Management: Connected"; then
     MESH_CONNECTED="netbird"
 elif command -v tailscale >/dev/null && tailscale status >/dev/null 2>&1; then
     MESH_CONNECTED="tailscale"
+elif curl -sf -o /dev/null --max-time 3 "${ISSUANCE_URL_NETBIRD%/issue}/health" 2>/dev/null; then
+    MESH_CONNECTED="lan"
 fi
 if [ -z "$MESH_CONNECTED" ] && [ "$NO_SECRETS" -eq 0 ]; then
-    echo "neither netbird nor tailscale is connected; cannot bootstrap." >&2
-    echo "either bring up the mesh first, or pass --no-secrets to skip issuance." >&2
+    echo "no reachable issuance endpoint (no netbird/tailscale connected and" >&2
+    echo "$ISSUANCE_URL_NETBIRD did not respond to /health)." >&2
+    echo "either bring up the mesh first, fix DNS for *.hubris.network, or pass --no-secrets." >&2
     exit 1
 fi
 echo "[bootstrap] mesh: ${MESH_CONNECTED:-none (skipped, --no-secrets)}"
 
+# -------- gitea creds (if provided) --------
+configure_gitea_creds
+
 # -------- clone --------
 if [ -d "$CLONE_DIR/.git" ]; then
     existing_remote="$(git -C "$CLONE_DIR" remote get-url origin 2>/dev/null || true)"
@@ -190,8 +331,9 @@ echo "[bootstrap] installing sync mechanism for $OS"
 run "bash '$CLONE_DIR/scripts/sync/install.sh'"
 
 # -------- install homelab CLI --------
-echo "[bootstrap] installing homelab CLI to /usr/local/bin/homelab"
-run "install -m 0755 '$CLONE_DIR/bin/homelab' /usr/local/bin/homelab"
+# Symlink rather than copy so the 5-min sync auto-updates the CLI.
+echo "[bootstrap] linking homelab CLI to /usr/local/bin/homelab"
+run "ln -sfn '$CLONE_DIR/bin/homelab' /usr/local/bin/homelab"
 
 # -------- AGENTS.md symlink --------
 case "$OS" in
@@ -205,6 +347,26 @@ esac
 run "ln -sfn '$CLONE_DIR/AGENTS.md' '$AGENTS_LINK'"
 echo "[bootstrap] linked AGENTS.md → $AGENTS_LINK"
 
+# -------- auto-upgrade to write-scoped Gitea PAT --------
+# After enrollment, if this client is already a recipient on
+# secrets/gitea-pat.yaml (i.e. the operator has run
+# `homelab client add --finalize-pubkey` from another client), swap the
+# read-only bootstrap PAT for the write-scoped one. Best-effort: fails
+# silently if the client isn't yet a recipient — the operator just re-runs
+# bootstrap or `homelab refresh-creds` later.
+if [ "$NO_SECRETS" -eq 0 ] && [ "$DRY_RUN" -eq 0 ] \
+   && command -v sops >/dev/null 2>&1 \
+   && [ -f "$CLONE_DIR/secrets/gitea-pat.yaml" ]; then
+    if /usr/local/bin/homelab refresh-creds >/tmp/homelab-refresh-creds.log 2>&1; then
+        echo "[bootstrap] refresh-creds: write-scoped Gitea PAT installed"
+    else
+        echo "[bootstrap] refresh-creds: skipped (this client isn't yet a recipient"
+        echo "            on secrets/gitea-pat.yaml — run 'homelab client add"
+        echo "            $HNAME --finalize-pubkey <age...>' from an existing client,"
+        echo "            then re-run bootstrap or 'homelab refresh-creds')"
+    fi
+fi
+
 # -------- MCP wiring --------
 if [ "$WITH_MCP" -eq 1 ]; then
     # Pick the right user's home — when invoked via sudo, SUDO_USER is set.
@@ -246,6 +408,211 @@ PYEOF
     fi
 fi
 
+# -------- Hermes (Goose + Nous Hermes) wiring --------
+# Installs the Goose CLI binary system-wide, symlinks the `hermes` wrapper
+# and HERMES.md persona, and drops a Goose config that pins the OpenRouter
+# provider, the Nous Hermes model, and the homelab MCP extension.
+# See operations/hermes-agent.md.
+if [ "$WITH_HERMES" -eq 1 ]; then
+    # Resolve the operator's home (SUDO_USER under `sudo bash`).
+    if [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != "root" ]; then
+        H_USER="$SUDO_USER"
+        H_HOME=$(eval echo "~$SUDO_USER")
+    else
+        H_USER="root"
+        H_HOME="$HOME"
+    fi
+
+    # 1. Install Goose binary at /usr/local/bin/goose (idempotent).
+    if ! command -v goose >/dev/null 2>&1; then
+        echo "[bootstrap] installing Goose CLI"
+        if [ "$DRY_RUN" -eq 1 ]; then
+            echo "+ would run upstream goose installer and symlink to /usr/local/bin/goose"
+        else
+            # Upstream installer drops the binary at ~/.local/bin/goose for the
+            # invoking user. We run it as $H_USER then symlink system-wide.
+            sudo -u "$H_USER" \
+                env CONFIGURE=false \
+                bash -c 'curl -fsSL https://github.com/aaif-goose/goose/releases/download/stable/download_cli.sh | bash'
+            if [ -x "$H_HOME/.local/bin/goose" ]; then
+                ln -sfn "$H_HOME/.local/bin/goose" /usr/local/bin/goose
+            else
+                echo "[bootstrap] WARNING: goose binary not found at $H_HOME/.local/bin/goose after install" >&2
+            fi
+        fi
+    else
+        echo "[bootstrap] goose already installed: $(command -v goose)"
+    fi
+
+    # 2. Symlink hermes wrapper.
+    echo "[bootstrap] linking hermes CLI to /usr/local/bin/hermes"
+    run "ln -sfn '$CLONE_DIR/bin/hermes' /usr/local/bin/hermes"
+
+    # 3. Symlink HERMES.md persona. The hermes wrapper does not need it — the
+    # Goose config below references the canonical clone path — but operators
+    # frequently `cat /root/HERMES.md` to inspect the persona, mirroring the
+    # AGENTS.md convention above.
+    case "$OS" in
+        Linux)  HERMES_LINK=/root/HERMES.md ;;
+        Darwin) HERMES_LINK=/etc/HERMES.md ;;
+    esac
+    run "ln -sfn '$CLONE_DIR/HERMES.md' '$HERMES_LINK'"
+    echo "[bootstrap] linked HERMES.md → $HERMES_LINK"
+
+    # 4. Drop the Goose config. Idempotent YAML merge — preserves any keys the
+    # operator added by hand, overwrites only the keys we manage.
+    GOOSE_DIR="$H_HOME/.config/goose"
+    GOOSE_CONFIG="$GOOSE_DIR/config.yaml"
+    GOOSEHINTS="$GOOSE_DIR/.goosehints"
+    run "mkdir -p '$GOOSE_DIR'"
+
+    PY_GOOSE_MERGE=$(cat <<PYEOF
+import os, sys
+try:
+    import yaml
+except ImportError:
+    print("PyYAML required", file=sys.stderr); sys.exit(2)
+
+path = "$GOOSE_CONFIG"
+mcp_uri = "$HERMES_MCP_URI"
+model = "$HERMES_MODEL"
+
+cfg = {}
+if os.path.exists(path):
+    with open(path) as f:
+        try:
+            cfg = yaml.safe_load(f) or {}
+        except Exception:
+            cfg = {}
+
+cfg["GOOSE_PROVIDER"] = "openrouter"
+cfg["GOOSE_MODEL"] = model
+cfg.setdefault("GOOSE_MODE", "smart_approve")
+cfg.setdefault("extensions", {})
+cfg["extensions"]["developer"] = {
+    "bundled": True, "enabled": True, "name": "developer",
+    "timeout": 300, "type": "builtin",
+}
+cfg["extensions"]["homelab"] = {
+    "enabled": True, "name": "homelab",
+    "description": "Read-only homelab context tools (FastMCP).",
+    "type": "streamable_http", "uri": mcp_uri, "timeout": 60,
+}
+with open(path, "w") as f:
+    yaml.safe_dump(cfg, f, sort_keys=False)
+print("[bootstrap] merged Goose config at", path)
+PYEOF
+)
+    if [ "$DRY_RUN" -eq 1 ]; then
+        echo "+ would merge Goose config at $GOOSE_CONFIG"
+    else
+        python3 -c "$PY_GOOSE_MERGE"
+        chown -R "$H_USER" "$GOOSE_DIR"
+    fi
+
+    # 5. Symlink HERMES.md as the global .goosehints — Goose injects it into
+    # the system prompt on every session start.
+    run "ln -sfn '$CLONE_DIR/HERMES.md' '$GOOSEHINTS'"
+    if [ "$DRY_RUN" -eq 0 ]; then
+        chown -h "$H_USER" "$GOOSEHINTS" 2>/dev/null || true
+    fi
+fi
+
+# -------- netbird tuning (skip per-session SSO for ssh into mesh peers) --------
+# Apply the SSH JWT cache TTL so `ssh ... .netbird.selfhosted` doesn't trigger
+# device-code SSO on every connection. Flag added in netbird 0.71.x
+# (netbirdio/netbird#4015). It belongs on `netbird up` — putting it on the
+# daemon's ExecStart crashes the daemon with "unknown flag". After this runs the
+# FIRST ssh still prompts SSO once; subsequent sessions within 24h skip it.
+if [ "$MESH_CONNECTED" = "netbird" ]; then
+    if [ "$DRY_RUN" -eq 1 ]; then
+        echo "+ would: netbird down && netbird up --ssh-jwt-cache-ttl=86400"
+    elif netbird up --help 2>&1 | grep -q -- "--ssh-jwt-cache-ttl"; then
+        echo "[bootstrap] netbird: enabling ssh-jwt-cache-ttl=86400 (one SSO per 24h)"
+        # `netbird up` short-circuits with "Already connected" — need down first.
+        netbird down >/dev/null 2>&1 || true
+        if ! netbird up --ssh-jwt-cache-ttl=86400; then
+            echo "[bootstrap] WARNING: netbird up with --ssh-jwt-cache-ttl failed; rerun manually:"
+            echo "[bootstrap]   netbird down && netbird up --ssh-jwt-cache-ttl=86400"
+        fi
+    else
+        echo "[bootstrap] netbird: --ssh-jwt-cache-ttl flag not supported (need >=0.71.x); skipping"
+    fi
+fi
+
+# -------- ssh ControlMaster for netbird peers (workstations) --------
+# Drop a Host block into the enrolling user's ~/.ssh/config so that ssh to
+# `*.netbird.selfhosted` multiplexes over a single connection. After one SSO
+# device-code completion, subsequent ssh / scp / `pct exec` invocations
+# (within ControlPersist=2h) reuse the master socket with no re-auth — the
+# real workaround for netbird's flaky SSH JWT cache. Skip on LXCs (no
+# outbound ssh expected from them).
+HKIND="$(python3 -c "import yaml; print(yaml.safe_load(open('$HOST_YAML')).get('kind',''))" 2>/dev/null || true)"
+if [ "$MESH_CONNECTED" = "netbird" ] && [ "$HKIND" != "lxc" ]; then
+    if [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != "root" ]; then
+        SSH_USER_HOME=$(eval echo "~$SUDO_USER")
+        SSH_OWNER="$SUDO_USER"
+    else
+        SSH_USER_HOME="$HOME"
+        SSH_OWNER=""
+    fi
+    SSH_CFG="$SSH_USER_HOME/.ssh/config"
+    SSH_CM_DIR="$SSH_USER_HOME/.ssh/cm"
+    SENTINEL="# homelab-bootstrap: ssh ControlMaster for netbird mesh"
+    if [ "$DRY_RUN" -eq 1 ]; then
+        echo "+ would write Host *.netbird.selfhosted ControlMaster block into $SSH_CFG"
+    elif [ -f "$SSH_CFG" ] && grep -qF "$SENTINEL" "$SSH_CFG"; then
+        echo "[bootstrap] ssh ControlMaster block already present in $SSH_CFG (skip)"
+    else
+        mkdir -p "$SSH_USER_HOME/.ssh" "$SSH_CM_DIR"
+        chmod 700 "$SSH_USER_HOME/.ssh" "$SSH_CM_DIR"
+        cat >> "$SSH_CFG" <<'SSHEOF'
+
+# homelab-bootstrap: ssh ControlMaster for netbird mesh
+# One SSO covers many back-to-back ssh/scp/pct ops within ControlPersist.
+Host *.netbird.selfhosted
+    ControlMaster auto
+    ControlPath ~/.ssh/cm/%C
+    ControlPersist 2h
+SSHEOF
+        chmod 600 "$SSH_CFG"
+        if [ -n "$SSH_OWNER" ]; then
+            chown -R "$SSH_OWNER":"$SSH_OWNER" "$SSH_USER_HOME/.ssh"
+        fi
+        echo "[bootstrap] ssh: installed ControlMaster block into $SSH_CFG"
+    fi
+fi
+
+# -------- mcp CLI install (workstations) --------
+# `homelab mcp <tool>` shells out to the `mcp` python CLI. Install it via
+# pipx for the enrolling user. Skip on LXCs / VMs.
+if [ "$HKIND" != "lxc" ] && [ "$HKIND" != "vm" ]; then
+    if [ "$DRY_RUN" -eq 1 ]; then
+        echo "+ would install 'mcp[cli]' via pipx for the enrolling user"
+    elif command -v mcp >/dev/null 2>&1; then
+        echo "[bootstrap] mcp CLI already on PATH (skip)"
+    else
+        # Make sure pipx is available; OS-specific install.
+        if ! command -v pipx >/dev/null 2>&1; then
+            if [ "$OS" = "Darwin" ] && command -v brew >/dev/null 2>&1; then
+                sudo -u "${SUDO_USER:-$USER}" brew install pipx 2>&1 | tail -2 || true
+            elif command -v dnf >/dev/null 2>&1; then
+                dnf install -y pipx 2>&1 | tail -2 || true
+            elif command -v apt-get >/dev/null 2>&1; then
+                DEBIAN_FRONTEND=noninteractive apt-get install -y pipx 2>&1 | tail -2 || true
+            fi
+        fi
+        if command -v pipx >/dev/null 2>&1; then
+            INVOKING_USER="${SUDO_USER:-$USER}"
+            sudo -u "$INVOKING_USER" -- bash -lc "pipx install 'mcp[cli]'" 2>&1 | tail -3 || true
+            sudo -u "$INVOKING_USER" -- bash -lc "pipx ensurepath" >/dev/null 2>&1 || true
+            echo "[bootstrap] mcp CLI installed for $INVOKING_USER via pipx"
+        else
+            echo "[bootstrap] WARNING: pipx unavailable; install manually: pipx install 'mcp[cli]'" >&2
+        fi
+    fi
+fi
+
 # -------- done --------
 cat <<EOF
 

containers/103-paperless.md

@@ -24,7 +24,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo
 - Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
 
 ## Known issues
-- Disk usage was 86.9% at last claudio-monitor reading on 2026-04-21. Monitor or grow rootfs.
+- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
 
 ## Related
 - [Authentik](124-authentik.md)

containers/105-apps.md

@@ -1,6 +1,6 @@
 # 105 — `apps`
 
-Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog).
+Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20.
 
 ## At a glance
 - **Hostname:** `apps`
@@ -19,6 +19,8 @@ Docker host for everything that doesn't justify its own LXC. Currently runs Arti
 | `artifacto.hubris.network`        | Artifacto        | `:3100`      | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
 | `blog.hubris.network`             | WriteFreely      | `:8080`      | Native OIDC via `[oauth.generic]`. |
 | `git.hubris.network/_plantuml/*`  | PlantUML server  | `:8079`      | Same-origin route from [gitea (104)](104-gitea.md). |
+| `mcp.hubris.network`              | homelab-mcp      | `:9810`      | MCP server over SSE. No SSO — agent-facing, gated at source-IP layer. See [homelab-context](../infrastructure/homelab-context.md). |
+| `secrets.hubris.network`          | secrets-issuance | `:9820`      | Per-client age-key issuance. Mesh+LAN source-IP gated (`MESH_SUBNETS`). |
 
 ## Stacks
 
@@ -49,6 +51,52 @@ Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` (
 
 > ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`. Running `docker compose up -d <svc>` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning.
 
+### homelab-mcp (`/opt/homelab-mcp/`)
+FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
+`/opt/homelab-context/` (this LXC is itself an enrolled
+[homelab-context](../infrastructure/homelab-context.md) client). Source
+is `dtoro/Homelab-Docs/mcp/server.py`; service unit
+`homelab-mcp.service` binds `0.0.0.0:9810`. DNS-rebinding protection
+disabled at the FastMCP layer because mesh+LAN gating is the actual
+trust boundary.
+
+- Endpoint: `https://mcp.hubris.network/mcp` (Caddy → `:9810`). StreamableHTTP transport (POST `/mcp`).
+- 14 tools registered: `get_host`, `list_services`, `find_service`,
+  `get_topology`, `search_docs`, `get_page`, `get_changelog`, `whoami`,
+  `list_my_secrets` (context); `get_service_status`, `tail_log`,
+  `list_lxcs`, `get_lxc_state`, `ping_service` (read-only management).
+- Management tools shell out via SSH using a dedicated `mcp-reader` key
+  (planned; not provisioned yet — management tools fail until then).
+
+#### Auto-deploy pipeline (homelab-mcp)
+Receiver at `/opt/homelab-mcp/mcp/deploy/webhook/` (in-tree alongside
+the server code). Listens on `0.0.0.0:9811`, secret in
+`/etc/homelab-mcp-deploy/secret`, systemd unit
+`homelab-mcp-deploy.service`. Gitea webhook id 10 on
+`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
+restarts on push.
+
+### secrets-issuance (`/opt/secrets-issuance/`)
+Tiny HTTP service that issues per-client age keypairs the first time
+each client calls `/issue`. Idempotent: subsequent calls return the
+same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
+`secrets-issuance.service`. State (per-client keys + denylist) lives at
+`/var/lib/secrets-issuance/` (NOT in the repo).
+
+- Endpoint: `https://secrets.hubris.network/issue` (Caddy → `:9820`).
+- `/revoke` is admin-token-gated by `/etc/secrets-issuance/admin-token`;
+  shreds the local key file and adds the hostname to the denylist.
+  Called by `homelab client remove`.
+- Trust subnets today: `100.122.0.0/16` (Netbird), `192.168.8.0/24` (LAN).
+  (The legacy Tailscale CGNAT range `100.64.0.0/10` was removed from
+  `secrets-issuance`'s `MESH_SUBNETS` on 2026-06-21 when Tailscale was
+  decommissioned; service restarted.) Tighten if the LAN gets untrusted devices.
+
+#### Auto-deploy pipeline (secrets-issuance)
+Receiver at `/opt/secrets-issuance/secrets-issuance/deploy/webhook/`,
+port `9821`, secret in `/etc/secrets-issuance-deploy/secret`, unit
+`secrets-issuance-deploy.service`. Gitea webhook id 11.
+
 ### WriteFreely (`/opt/writefreely-docker/`)
 Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts.
 
@@ -66,6 +114,16 @@ Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.h
 
 ## Changelog
 
+### 2026-05-20 — homelab-mcp + secrets-issuance live
+Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
+`homelab-mcp.service` on `:9810` (MCP read+management surface) and
+`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
+Caddy fronts both with Let's Encrypt; new vhosts on
+[caddy](121-caddy.md), split-horizon DNS entries on
+[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire
+auto-deploy. LXC is itself an enrolled context client
+(`/opt/homelab-context/`).
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 

containers/106-auth-outpost.md

@@ -0,0 +1,56 @@
+# 106 — `auth-outpost`
+
+Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
+
+## At a glance
+- **Hostname:** `auth-outpost`
+- **IP:** `192.168.8.6` (static)
+- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
+- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
+- **Mounts:** none
+- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md)
+
+## Role
+
+Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../investigations/2026-05-31-authentik-vps-migration.md).
+
+## Service / port map
+| Service | Listen | Notes |
+| --- | --- | --- |
+| authentik proxy outpost | `192.168.8.6:9000` | `/outpost.goauthentik.io/*` (ping, auth/caddy) |
+
+## Config paths
+- `/opt/authentik-outpost/docker-compose.yml` — `ghcr.io/goauthentik/proxy:2026.5.2`, `AUTHENTIK_HOST=https://auth.hubris.network`, `AUTHENTIK_INSECURE=false`, port bound `192.168.8.6:9000`.
+- `/opt/authentik-outpost/.env` (mode 600, **untracked**) — `AUTHENTIK_TOKEN` for outpost `hubris-lan-outpost` (sops-encrypt into `secrets/` — TODO Phase 5).
+
+## The outpost object (VPS Authentik)
+- Outpost **`hubris-lan-outpost`** (type proxy), providers: `hubris-forward-auth` (domain, Paperless + domain-level apps), `Provider for Torrent` (qBittorrent), `Provider for Artifacto` (single).
+- Health: VPS admin → Applications → Outposts → last-seen current; `curl http://192.168.8.6:9000/outpost.goauthentik.io/ping` → `204`; container log `Successfully connected websocket`.
+
+## Caddy wiring
+[Caddy (121)](121-caddy.md) `(authentik)` snippet `reverse_proxy`/`forward_auth` → `http://192.168.8.6:9000` (was `192.168.8.180:9000` on LXC 124). Tracked in `dtoro/caddy-conf`.
+
+## `sso.hubris.network` — the callback domain (critical)
+Domain-level (`forward_domain`) providers redirect the browser to `{external_host}/outpost.goauthentik.io/callback` after login. On LXC 124 that `external_host` was `auth.hubris.network`, which resolved (LAN) to Caddy → the 124 outpost — same box, so it worked. Post-migration `auth.hubris.network` points to the **VPS core**, so the callback hit the wrong outpost → **400 / redirect-uri mismatch**.
+
+Fix: the LAN outpost gets its **own** domain.
+- DNS: `sso.hubris.network → 192.168.8.175` (Caddy). *(in dnsmasq today — must be carried into Technitium in DNS Phase 2.)*
+- Caddy: site `sso.hubris.network { tls dns ionos; reverse_proxy 192.168.8.6:9000 }` (in `caddy-conf`).
+- Authentik: `hubris-forward-auth` and `Provider for Torrent` have `external_host=https://sso.hubris.network`; their `redirect_uris` must match (`set_oauth_defaults()` regenerates them from `external_host` — changing `external_host` alone does NOT update them).
+- `Provider for Artifacto` is `forward_single` on its own domain, so it was unaffected.
+
+**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
+
+## Related
+- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only)
+- [Caddy (121)](121-caddy.md) — forward-auth consumer
+- [Ingress (VPS traefik)](../infrastructure/ingress.md)
+- [Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md)
+
+## Changelog
+
+### 2026-06-06 — Authentik session lifetime extended to 30 days
+VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../investigations/2026-06-06-authentik-session-lifetime.md).
+
+### 2026-06-01 — created; forward-auth cut over from LXC 124
+New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).

containers/107-dns.md

@@ -0,0 +1,66 @@
+# 107 — `dns`
+
+Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job.
+
+## At a glance
+- **Hostname:** `dns`
+- **IP:** `192.168.8.2` (static — stable, decoupled from any app)
+- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
+- **Resources:** 1 core / 1 GiB / 8 GiB rootfs
+- **Created:** 2026-06-01, Debian 13. Its own resolver is `1.1.1.1` (no circular dependency on the DNS it serves).
+
+## Role
+Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recursive forwarding (`1.1.1.1`, `8.8.8.8`) for everything else. Technitium runs in Docker (`technitium/dns-server:latest`, `network_mode: host`), web console on `:5380`.
+
+## The `hubris.network` zone
+- Specific A overrides: app names → `192.168.8.175` (Caddy), `nfs-export → 192.168.8.200`, `auth/sso/...` as needed.
+- `auth.hubris.network → 82.165.190.79` (VPS Authentik), `sso.hubris.network → 192.168.8.175` (LAN forward-auth outpost).
+- **Wildcard** `*.hubris.network → 82.165.190.79` — mirrors the public IONOS wildcard so undefined names (e.g. `netbird`) resolve to the VPS, matching public behaviour.
+- **MX / SPF-TXT / CAA replicated from public** so an authoritative zone doesn't shadow `hubris.network` email/cert records.
+
+## Config / access
+- `/opt/technitium/docker-compose.yml`; admin password in `/opt/technitium/admin_password.txt` (mode 600 — sops-encrypt in Phase 5).
+- Console: `http://192.168.8.2:5380` (user `admin`).
+- API: `http://192.168.8.2:5380/api/...` (token via `/api/user/login`). Zone was built via the API.
+
+## Who points here
+- **NetBird mesh peers:** resolve `hubris.network` by **forwarding to Technitium** via the `home-lab-dns` nameserver group (`→ 192.168.8.2`, domain `hubris.network`, applied to all peers). The **NetBird managed DNS zone was removed 2026-06-21 (Phase 4)** — Technitium is now the single DNS source for the mesh too. This works because: (a) roaming peers (`Core`) have the `192.168.8.0/24` route to reach `192.168.8.2` (added 2026-06-21), and (b) clients run NetBird **0.71.x** — on the old 0.68.3 client, forwarding reported `Available` but didn't serve queries, and the resolver cache (`100.122.255.254`) wouldn't clear on `down/up`; a `netbird service restart` (or app toggle) clears it. See [dns.md changelog 2026-06-21](../infrastructure/dns.md).
+- **Homelab DHCP clients:** Technitium's own DHCP scope hands out `192.168.8.2` as the DNS server for `192.168.8.x` leases (see DHCP section below).
+- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS, **but** the Fritz!Box now *forwards* upstream to Technitium — DNSv4 server set to `192.168.8.2` (Internet → Filter → DNS Server, 2026-06-17). So household clients get split-horizon `*.hubris.network` answers via Fritz!Box→Technitium, with **no NetBird dependency**. (This is the change that decoupled the on-prem tier from the mesh — see [dns.md](../infrastructure/dns.md) 2026-06-17.)
+
+## dns-sync (RETIRED 2026-06-21 — Phase 4 complete)
+**The managed-zone sync is no longer scheduled.** `/opt/dns-sync/sync.py` reconciled this zone's named A-records → the NetBird managed DNS zone; the `*/10` cron (`/etc/cron.d/dns-sync`) was **removed 2026-06-21** when the managed zone was retired. Technitium is now the **single** DNS source — mesh peers forward to it (see "Who points here" above), LAN/household clients query it directly.
+
+The script + token + a pre-deletion record backup remain at `/opt/dns-sync/` **as an emergency-restore tool only**: running `python3 /opt/dns-sync/sync.py` once re-creates the managed zone from Technitium (used during the Phase 4 rollback). Do not re-add the cron unless reverting Phase 4. Tracked: [scripts/dns-sync.py](../scripts/dns-sync.py).
+
+## DHCP
+
+Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
+- **Scope:** `homelab` — `192.168.8.241 – 192.168.8.254`
+- **Gateway:** `192.168.8.1` (Proxmox `vmbr0` alias)
+- **DNS:** `192.168.8.2` (self)
+- **Lease time:** 24 h
+
+Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk.
+
+## Related
+- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq
+- [DNS split-horizon](../infrastructure/dns.md)
+- [Mesh](../infrastructure/mesh.md)
+
+## Changelog
+
+### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
+Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
+
+### 2026-06-03 — DHCP pool narrowed to `.241–.254`
+Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
+
+### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
+This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
+
+### 2026-06-02 — DHCP server enabled; replaces Slate AX DHCP
+Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
+
+### 2026-06-01 — created; replaced dnsmasq on 124
+Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md).

containers/118-elementsynapse.md

@@ -1,6 +1,6 @@
 # 118 — `elementsynapse`
 
-Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
+Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
 
 ## At a glance
 - **Hostname:** `elementsynapse`
@@ -33,16 +33,29 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
 - `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
 
 ## Known issues
-- ~~Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
+- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
 
 ## Related
-- [claudio-bot (123)](123-claudio-bot.md) — connects directly to `192.168.8.239:8008` (avoids hairpin-NAT TLS issue on the public URL)
+- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
 - [Caddy](121-caddy.md)
 - [DNS](../infrastructure/dns.md)
 - [Monitoring](../infrastructure/monitoring.md)
 
 ## Changelog
 
+### 2026-06-06 — DHCP drift fixed: internal `/etc/network/interfaces` was `dhcp` despite Proxmox static config
+**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
+
+**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
+
+**Fix:**
+- Replaced `iface eth0 inet dhcp` with `iface eth0 inet static` + `address 192.168.8.239/24` + `gateway 192.168.8.1`
+- `ifdown eth0 && ifup eth0` applied the static IP
+- Killed lingering dhclient process
+- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
+
+**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
+
 ### 2026-05-15 — phantom-notification cleanup for `@admin`
 After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
 1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.

containers/120-mule-images.md

@@ -1,68 +1,46 @@
 # 120 — `mule-images`
 
-Hosts `mule-image` / "mulita" — the photos app at `photos.hubris.network`. Auto-deploys from `dtoro/mule-image` on `git push origin main`.
+Hosts `mule-image` — the photos app at `photos.hubris.network`. PhotoPrism + Go sidecar + SvelteKit, replacing the legacy FastAPI/Celery stack as of 2026-05-22 (see Changelog). Auto-deploys from `dtoro/mule-image` on `git push origin main`.
 
 ## At a glance
 - **Hostname:** `mule-images`
 - **IP:** `192.168.8.136`
 - **Privilege:** privileged
-- **Resources:** 4 cores / 8 GiB RAM / 60 GiB rootfs
-- **Mounts:** `/mnt/library` ↔ `/mnt/library`
-- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000` (frontend)
+- **Resources:** 6 cores / 12 GiB RAM / 60 GiB rootfs / 2 GiB swap
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`; `/dev/dri/{card0,renderD128}` passed through for VA-API video accel on the AMD Phoenix1 iGPU.
+- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → path-routed to PhotoPrism `:2342` / sidecar `:8000` / nginx-static `:3000`.
 
 ## Stack (`/opt/mule-image`)
 
-`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose at `/opt/mule-image/docker-compose.yml`. Services:
+`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose stack: base `docker-compose.yml` + LAN-port-rebind `docker-compose.override.yml` (untracked) + VA-API `docker-compose.gpu.yml`. The SvelteKit frontend is built on the host and served as a static bundle by nginx — no `vite dev` in prod.
 
-| Service           | Port   | Notes |
-| ----------------- | ------ | ----- |
-| frontend          | 3000   | Reverse-proxied by Caddy |
-| backend           | 8001   | FastAPI |
-| worker-light      | —      | Sole celery worker — runs default/high/low queues + beat scheduler |
-| db                | (pg)   | postgres:16 (was pgvector/pg16 until 2026-05-14) |
-| redis             | (rd)   | queue |
+| Service / process        | Port (LAN)            | Notes |
+| ------------------------ | --------------------- | ----- |
+| pp-mariadb               | (internal 3306)       | MariaDB 11; holds PhotoPrism's `photoprism` DB + sidecar's `mule_sidecar.*` |
+| pp-app                   | `0.0.0.0:2342`        | PhotoPrism `:latest`; `PHOTOPRISM_FFMPEG_ENCODER=h264_vaapi` |
+| pp-sidecar               | `0.0.0.0:8000`        | Go service (file rename / folder ops / heap convert / dup scan / per-photo marks); also reconciles `USER_BASEPATHS` into PhotoPrism's `auth_users` every 60 s |
+| nginx (host process)     | `0.0.0.0:3000`        | Serves the SvelteKit static bundle from `web/build/` |
 
-**No vision / ML services.** The OpenCLIP ONNX classifier and its dedicated `worker-vision` were ripped out on 2026-05-14 — see Changelog.
+**Per-user scoping** is driven by `USER_BASEPATHS=admin:admin/files, muli:muli/files` in `.env`. Sidecar applies it to PhotoPrism's `auth_users` table on boot + every 60s, `mkdir -p`s each target so PhotoPrism's ACL filter has somewhere to point.
 
-`worker-watcher` was retired 2026-05-11 — file events come from NC webhook_listeners instead of watchfiles inotify. See "Nextcloud integration" below.
+`.env` is **untracked** — `git reset --hard` won't touch it, but `git clean -fdx` would. Holds PP_*, MariaDB passwords, SIDECAR_DB_PASSWORD, OIDC_* (existing `mule-image` Authentik app, redirect URI now `/api/v1/oidc/redirect`), `USER_BASEPATHS`, and `PP_FFMPEG_ENCODER=vaapi`.
 
-`.env` is **untracked** — `git checkout .env` will wipe it. Holds:
-- `PHOTO_DIRS=/mnt/library/images/`
-- `NEXTCLOUD_USERS_HOST_PATH=/mnt/library/homecloud`
-- `NEXTCLOUD_BASE_URL=https://cloud.hubris.network`
-- OIDC client secret + scopes
-- `SECRET_KEY` (generated)
-- `NEXTCLOUD_WEBHOOK_SECRET` (since 2026-05-11; bearer token NC presents on every webhook call to `/api/v1/internal/nc-webhook`)
+`docker-compose.override.yml` is also untracked — it `!override`s the upstream `127.0.0.1:port` mappings to `0.0.0.0:port` so cross-host Caddy on [LXC 121](121-caddy.md) can reach pp-app + sidecar.
 
-## Nextcloud integration (since 2026-05-11)
+## Library access
 
-Phase 1 + 2 of dropping the duplicated indexing work between mule and Nextcloud — see `~/.claude/projects/-root/memory/project_mule_nc_integration.md` for the full detail.
+PhotoPrism reads the library **directly off the bind-mounted filesystem** — no Nextcloud webhook integration in the new stack. The base path is `/mnt/library/homecloud` and per-user scoping comes from `USER_BASEPATHS` (see Stack section above).
 
-**Thumbnails (Phase 1, commit 576b0c2):** `GET /api/v1/photos/{id}/thumb/{size}` proxies `https://cloud.hubris.network/index.php/core/preview` keyed by `photos.nextcloud_fileid` (alembic 0018), authenticated with the photo owner's encrypted NC app password. NC handles JPEG/HEIC/TIFF/RAW/DNG/Movie previews natively. `X-Mule-Thumb-Source: nextcloud|disk` on every response. Worker now only writes the `medium` size to `/data/thumbs/{user}/{photo}/medium.webp` (the vision worker still loads it from disk); `small` + `large` are NC-proxied. Backfill script: `backend/scripts/backfill_nextcloud_fileid.py` — 17,591/17,591 photos filled.
+- `dtoro` is mapped to NC user `admin` → `/mnt/library/homecloud/admin/files/`
+- `muli` is mapped to NC user `muli` → `/mnt/library/homecloud/muli/files/`
 
-**File events (Phase 2, commits 362fbc6 + f657e2c):** mule no longer runs `watchfiles`. NC's `webhook_listeners` app (already installed, registered via `backend/scripts/register_nc_webhooks.py`) POSTs every `NodeCreated|NodeWritten|NodeDeleted|NodeRenamed` event to `http://192.168.8.136:8001/api/v1/internal/nc-webhook` with `Authorization: Bearer $NEXTCLOUD_WEBHOOK_SECRET`. The endpoint dispatches the same `scan_folder` / `handle_file_deletion` machinery the watcher used. **NC dispatches webhooks asynchronously via its background-job queue** — `crontab -u www-data` on LXC 114 runs `cron.php` every minute (`*/1`), so end-to-end latency is ≲60 s. Re-running the registrar script is idempotent (it deletes existing webhooks pointing at the same URL first).
+PhotoPrism's container user is `33:10000` (`www-data:media`), matching the host ownership of the NC tree. The legacy `oc_webhook_listeners` rows + `NEXTCLOUD_WEBHOOK_SECRET` are gone — when NC writes via WebDAV (still its own primary surface), PhotoPrism picks up the new file on its next index pass.
 
-## Nextcloud-rooted libraries (since 2026-04-26)
+## Authentication
 
-Photo libraries live under each user's Nextcloud `files/` tree, NOT in `/mnt/library/images/*`.
-- `/mnt/library/homecloud` is bind-mounted into `backend`, `worker-light`, `worker-watcher`, `worker-vision` as `/nextcloud-users`. Each NC user is `/nextcloud-users/<nc_user>/files/`.
-- Reads use that bind directly.
-- Mutations (upload, delete, rename, move) dispatch through `services/nextcloud_dav.py` (HTTP Basic auth, per-user app password Fernet-encrypted in `users.nextcloud_app_password_enc`) so Nextcloud's `oc_filecache`, trashbin, comments, and desktop-sync clients stay coherent.
-- Photo copy + cross-system moves return 501 with a "use Nextcloud's web UI" hint — defer until needed.
-- `users.nextcloud_username` overrides the default OIDC `preferred_username`. **`dtoro` (mule-image) maps to `admin` (Nextcloud)** — don't assume username equality.
-- Surviving SourceRoots in DB: `Photos` → `/nextcloud-users/admin/files/Photos`; `Memories` → `/nextcloud-users/admin/files/Memories` (both owned by `dtoro`). User `muli` has `nextcloud_username=muli` backfilled but no SourceRoot yet.
-- Pre-migration DB dump: `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump` (11 MB) on the host.
+OIDC via Authentik. App slug `mule-image`, redirect URI `https://photos.hubris.network/api/v1/oidc/redirect` (PhotoPrism's auto-derived path; PhotoPrism builds it from `PHOTOPRISM_SITE_URL`). `OIDC_REGISTER=true` auto-creates a PhotoPrism user at role `user` on first SSO login. The sidecar's basepath reconciler then assigns their scoped folder.
 
-## Authentication (since 2026-04-22)
-
-Native OIDC via Authentik. Code in `backend/app/auth_oidc.py`, routes `/api/v1/auth/oidc/{login,callback}`. Authentik side:
-- OAuth2/OIDC Provider, client ID `fCuHew48ONTskDjUKnMTZjFbVXuHwvQqTScQRNQ1`
-- App slug `mule-image`
-- Redirect URI: `https://photos.hubris.network/api/v1/auth/oidc/callback`
-
-Backend container needs `extra_hosts: auth.hubris.network:192.168.8.175` via `docker-compose.override.yml` (gitignored). Otherwise Authlib's metadata fetch fails with `SSL: CERTIFICATE_VERIFY_FAILED: self-signed certificate` (it ends up at a random public host because LXC DNS resolves the public IONOS A record).
-
-Caddyfile stays plain `reverse_proxy 192.168.8.136:3000` — no forward-auth, no `/api/*` bypass needed.
+The `mule-image` Authentik app's redirect URI was migrated from the legacy FastAPI `/api/v1/auth/oidc/callback` path on 2026-05-22 — same client ID/secret were reused. The separate `mule-photos-new` Authentik app was deleted in the same operation.
 
 ## Auto-deploy
 
@@ -70,11 +48,10 @@ Push to `dtoro/mule-image` `main` → gitea webhook → `http://192.168.8.136:97
 - Validates HMAC against `/etc/mule-deploy/secret`
 - Filters to `refs/heads/main`
 - Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
-- `git pull --ff-only` + `docker compose up -d --build` + `docker image prune -f`
+- `git fetch && git reset --hard origin/main`, force-recreates **only the sidecar** image (so PhotoPrism's `Session` HMAC key in `pp/storage/config/hub.yml` doesn't rotate and invalidate every in-flight OIDC state cookie), reconciles `pp-app` + `mariadb` in place, `cd web && npm ci && npm run build`, `systemctl reload nginx`.
+- Conditionally layers `docker-compose.gpu.yml` when `/dev/dri/renderD128` exists, and always layers `docker-compose.override.yml` (the LAN-port rebind) when present.
 
-Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`, unit at `/etc/systemd/system/mule-deploy-webhook.service`. Same shape as the Caddy + Artifacto pipelines. Gitea webhook id 6.
-
-`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
+Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`. Gitea webhook id 6. `app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
 
 Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`.
 Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`.
@@ -91,6 +68,53 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials
 
 ## Changelog
 
+### 2026-05-22 — Cutover to PhotoPrism stack (Mulimage 2.0)
+
+The `new` branch of `dtoro/mule-image` was merged into `main` as commit
+`70dc1b6`. The merge replaces the legacy FastAPI + PostgreSQL + Celery
++ React stack with PhotoPrism + Go sidecar + MariaDB + SvelteKit, lifted
+in from the M0 evaluation on (now-destroyed) [LXC 127](#).
+
+Cutover on 120 (this session):
+- Bumped to 6 cores / 2 GB swap. `/dev/dri/{card0,renderD128}` already
+  passed through; added an opt-in `docker-compose.gpu.yml` overlay that
+  the deploy script layers in when the device is present. PhotoPrism now
+  uses `h264_vaapi` instead of `libx264`.
+- Installed host nginx + a small `photos.conf` vhost serving the SvelteKit
+  static bundle from `/opt/mule-image/web/build/` on `:3000` — no Vite
+  dev server in prod. SvelteKit's `adapter-static` produces a real SPA
+  bundle.
+- Replaced `/opt/mule-deploy/deploy.sh` with the 127-style multi-compose
+  deploy (force-recreates only sidecar to preserve PhotoPrism's Session
+  HMAC key; reconciles pp-app + mariadb in place; rebuilds web/ and
+  reloads nginx).
+- `/opt/mule-image/.env` was rewritten to the PhotoPrism shape. Reused
+  passwords from `/root/mule-photos-new-secrets.txt`. `USER_BASEPATHS=
+  admin:admin/files, muli:muli/files` exposes both libraries.
+- `docker-compose.override.yml` (untracked) `!override`s pp-app +
+  sidecar ports to `0.0.0.0` so cross-LXC caddy on [121](121-caddy.md)
+  can reach them.
+- Authentik: `mule-image` app's redirect URI updated to `/api/v1/oidc/redirect`;
+  the separate `mule-photos-new` app deleted.
+- Caddyfile (`dtoro/caddy-conf`): `photos.hubris.network` switched from
+  plain passthrough to path-matchers (PhotoPrism API + sidecar API +
+  nginx static catch-all), and `photos-new.hubris.network` was removed
+  entirely.
+- Cleanups: 4 Nextcloud `oc_webhook_listeners` rows for the legacy
+  `/api/v1/internal/nc-webhook` endpoint deleted; gitea webhook id 9
+  (the `refs/heads/new` deploy hook for 127) deleted; `192.168.8.181`
+  removed from gitea's `ALLOWED_HOST_LIST`; `photos-new.hubris.network`
+  removed from dnsmasq.
+- LXC 127 destroyed via `pct destroy 127` after the new stack passed
+  curl verification end-to-end (PP `/api/v1/status` returns
+  `{"status":"operational"}` through caddy; sidecar `/api/sidecar/healthz`
+  returns `{"ok":true,...}`).
+- Rollback safety net: legacy `mulita` postgres dump at
+  `/root/backups/mulita-legacy-final-20260521-212036.sql.gz` on hubris
+  (10 MB, 16,155 photo rows). The `mule-image_mule_db_data` docker
+  volume on 120 retains the on-disk postgres data for at least 24 h
+  before housekeeping removes it.
+
 ### 2026-05-14 — Drop AI/vision pipeline, plain Postgres, DB↔FS refresh
 
 **AI removal (commits `a27267f` + `423a73a`).** The OpenCLIP-on-ONNX

containers/121-caddy.md

@@ -28,7 +28,7 @@ The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the L
 - `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
 - `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
 - `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
-- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8081`
+- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
 
 > **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
 
@@ -65,6 +65,28 @@ Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scrip
 
 ## Changelog
 
+### 2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082
+`sab.hubris.network` now uses `import authentik` inside a `handle` block. SABnzbd own auth disabled, `local_ranges` set for transparent proxy. Port bumped from `:8081` to `:8082` (fix from 2026-06-04) now documented.
+
+### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
+**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
+
+**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
+
+**Fix:**
+- Restored Caddyfile from `origin/master` (`git checkout --force origin/master -- Caddyfile`)
+- `systemctl reload caddy`
+
+**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
+- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
+- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
+- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
+
+Also: [elementsynapse LXC 118](../containers/118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
+
+### 2026-06-02 — caddy.service unit missing; recreated
+After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation. 16 active sites at this date.
 

containers/122-arriman.md

@@ -22,19 +22,27 @@ Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr
 
 ## Service / port map
 
+All services route through gluetun's network namespace. Ports are exposed via
+the gluetun container:
+
 | Service       | Host:Container | Public hostname                      |
 | ------------- | -------------- | ------------------------------------ |
-| sonarr        | `8989:8989`    | direct only                          |
-| radarr        | `7878:7878`    | direct only                          |
-| lidarr        | `8686:8686`    | direct only                          |
-| prowlarr      | `9696:9696`    | direct only                          |
-| bazarr        | `6767:6767`    | direct only                          |
+| gluetun (VPN) | —              | —                                    |
+| sonarr        | `8989:8989`    | direct only (via gluetun)            |
+| radarr        | `7878:7878`    | direct only (via gluetun)            |
+| lidarr        | `8686:8686`    | direct only (via gluetun)            |
+| prowlarr      | `9696:9696`    | direct only (via gluetun)            |
+| bazarr        | `6767:6767`    | direct only (via gluetun)            |
 | jellyseerr    | `5056:5055`    | [`jellyseerr.hubris.network`](../infrastructure/dns.md) |
 | qbittorrent   | `8080:8080`    | [`qbit.hubris.network`](../infrastructure/dns.md)       |
-| sabnzbd       | `8081:8080`    | [`sab.hubris.network`](../infrastructure/dns.md)        |
+| sabnzbd       | `8082:8082` HTTP, `9090:9090` HTTPS | [`sab.hubris.network`](../infrastructure/dns.md)        |
 | flaresolverr  | `8191:8191`    | internal only                        |
+| homarr        | `7575:7575`    | internal only                        |
 
-Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `ezarr_default` using docker service names.
+Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `localhost:<port>`
+(services share gluetun's shared network namespace). External services reach them
+via `gluetun:<port>` (e.g. Sonarr → qBittorrent at `localhost:8080` or
+`gluetun:8080`).
 
 ## Categories (qBit + SAB + *arr)
 
@@ -48,18 +56,30 @@ Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `ezarr_defa
 
 Path mapping: host `/mnt/library/<cat>` ↔ container `/data/media/<cat>`. Downloads: host `/mnt/library/downloads/<torrents|usenet>/<cat>` ↔ container `/data/torrents/<cat>` and `/data/usenet/<cat>`.
 
-## Auth (qBit reverse-proxy + Authentik forward-auth)
+## Auth (reverse-proxy + Authentik forward-auth)
 
-qBit auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
-- `WebUI\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
-- `WebUI\ReverseProxySupportEnabled=true`
-- `WebUI\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
+### qBit
+
+Auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
+- `WebUI\\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
+- `WebUI\\ReverseProxySupportEnabled=true`
+- `WebUI\\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
 
 > **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`.
 
 Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy.
 
-SABnzbd `host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
+### SABnzbd
+
+Gated with Authentik forward-auth (applied 2026-06-13). Caddy `sab.hubris.network` block uses `import authentik` inside a `handle` block. SABnzbd's own web auth is disabled:
+
+- `html_login = 0` → no HTML login form
+- `username` / `password` cleared → CherryPy Basic Auth not activated
+- `local_ranges = 172.18.0.0/16, 192.168.8.0/24, 127.0.0.0/8` → proxied requests from Caddy (192.168.8.x) and Docker-proxy (172.18.x) pass without auth
+
+**API key** (`67ef5a45e4e04157994e977005a33878`) still works for internal service-to-service calls (Sonarr/Radarr/Lidarr via Docker internal networking — they talk to SAB at `localhost:8082`, not through Caddy).
+
+`host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
 
 ## Credentials
 
@@ -101,6 +121,32 @@ Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has
 
 ## Changelog
 
+### 2026-06-13 — SABnzbd gated with Authentik forward-auth
+SABnzbd now uses Authentik forward-auth (same `import authentik` Caddy pattern as qBit). SABnzbd's own web auth disabled: `html_login=0`, credentials cleared, `local_ranges` extended to cover Docker bridge + homelab LAN. API key still works for internal *arr service calls. See [Auth section](#auth-reverse-proxy--authentik-forward-auth) above.
+
+### 2026-06-04 — all arr services moved behind gluetun VPN; SAB port conflict fixed
+
+- All services (sonarr, radarr, lidarr, bazarr, prowlarr, jellyseerr, homarr,
+  flaresolverr) now use `network_mode: service:gluetun` — whole stack routes
+  through the VPN
+- Port mappings moved from individual services to gluetun container
+- **Fixed SABnzbd port conflict**: was crashing in a restart loop because
+  qBittorrent held port 8080 inside the shared gluetun namespace. Changed
+  SAB internal port to 8082 (config at `/config/sabnzbd-config/sabnzbd.ini`)
+- Caddy `sab.hubris.network` updated to point to `:8082`
+- Jellyseerr's `extra_hosts` (auth.hubris.network) moved to gluetun since
+  `extra_hosts` conflicts with `network_mode`
+
+### 2026-06-02 — ProtonVPN added (gluetun); LXC IP set static
+- Added `gluetun` container to compose as a WireGuard VPN sidecar (ProtonVPN, server AL#57, located in Tirana, Albania)
+- **qbittorrent** and **sabnzbd** now use `network_mode: service:gluetun` — all traffic routes through the VPN
+- Ports 8080 (qBit WebUI), 6881 tcp/udp (qBit BT), 8081 (SAB WebUI) exposed through gluetun
+- gluetun config at `gluetun-config/wireguard/wg0.conf` (read-only mount)
+- Healthcheck on gluetun; qBit/SAB wait for `service_healthy` before starting
+- LXC IP changed from DHCP to static (`192.168.8.132`) via `pct set` + `/etc/network/interfaces`
+- **After first start:** Sonarr/Radarr/Lidarr download client host needs updating from `qbittorrent` → `gluetun` (SAB similarly `sabnzbd` → `gluetun`)
+- **Also fixed:** 7 other DHCP LXCs (101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy) set to static IPs to prevent floating on reboot. See infrastructure/dns.md.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 

containers/123-claudio-bot.md

@@ -1,4 +1,9 @@
-# 123 — `claudio-bot`
+# 123 — `claudio-bot` (DEPRECATED — destroyed 2026-06-04)
+
+> **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini.
+> Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob.
+> Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea.
+> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details.
 
 Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications.
 
@@ -66,6 +71,12 @@ Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/dep
 
 ## Changelog
 
+### 2026-06-04 — LXC destroyed; replaced by Hermes Agent
+LXC 123 destroyed via `pct destroy 123 --purge`. Bot service stopped, systemd
+units disabled. `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived on
+Gitea. Monitoring replaced by Hermes `homelab-health-watchdog` cron job.
+`@claudio:avispero` Matrix account decommissioned.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 

containers/124-authentik.md

@@ -1,136 +0,0 @@
-# 124 — `authentik`
-
-Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box".
-
-## At a glance
-- **Hostname:** `authentik`
-- **IP:** `192.168.8.180` (statically configured — the only LXC with a static IP)
-- **Privilege:** privileged
-- **Resources:** 2 cores / 4 GiB RAM / 20 GiB rootfs
-- **Mounts:** none from `/mnt/library`
-- **Public hostname:** [`auth.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `:9000`
-- **Container DNS (in `/etc/pve/lxc/124.conf`):** `192.168.8.1 1.1.1.1` (router DNS plus a fallback added 2026-04-21 because router DNS flakes intermittently — Authentik is the resolver itself for the *rest* of the LAN, but its own LXC uses upstream).
-
-## Authentik stack (`/opt/authentik/`)
-
-Upstream `docker-compose.yml` + `.env`. Services: `postgresql` (16-alpine), `server`, `worker`. Authentik 2026.x dropped the Redis dependency.
-
-- `.env` mode 600, **untracked**, holds `AUTHENTIK_SECRET_KEY` and `PG_PASS`.
-- `AUTHENTIK_TAG=2026.2.2` — pinned. Don't let it drift to `:latest`. Telemetry / update-check / error-reporting disabled.
-- Ports: 9000 (http), 9443 (https) on the LXC.
-- Embedded outpost lives at `/outpost.goauthentik.io/*` on the Authentik host — the forward-auth endpoint Caddy points at.
-- Stack is **not** git-tracked yet. If/when wiring auto-deploy: mirror the `mule-image` pattern (webhook receiver outside the app repo at `/opt/authentik-deploy/`). Repo `dtoro/authentik-conf` is reserved but not created.
-
-## Forward-auth pattern (every gated app)
-
-- **One Proxy Provider per app.** Authentik enforces a UNIQUE constraint `application.provider_id`, so one Provider = one Application. "Domain-level" only means they share the cookie domain. Each provider in "Forward auth (domain level)" mode, External host `https://auth.hubris.network`, Cookie domain `hubris.network`. First one was `hubris-forward-auth` (Paperless).
-- **Authentication flow:** MUST be `default-authentication-flow` (NOT `default-source-authentication` — that's for IdP federation; gives `FlowNonApplicableException` + 404 on the authorize endpoint).
-- **Authorization flow:** `default-provider-authorization-implicit-consent` (or explicit).
-- **Application Launch URL** MUST be the full public URL `https://<sub>.hubris.network/` — outpost matches incoming `X-Forwarded-Host` against it.
-- Each Application MUST have at least one **policy/group/user binding** — zero bindings = outpost returns 404 on access.
-- **Restart Authentik after binding new apps to the outpost:**
-  ```
-  pct exec 124 -- docker compose -f /opt/authentik/docker-compose.yml restart server worker
-  ```
-
-Caddy snippet `(authentik)` lives at the top of `/etc/caddy/Caddyfile`. Points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid hairpin TLS round-trip stripping `X-Forwarded-Host`. Must explicitly set `header_up X-Forwarded-Host {host}` in the forward-auth block. Used by gated sites with `import authentik`.
-
-### Per-app username override pattern (Authentik)
-
-Used when the app's local user ID doesn't match the user's Authentik username (e.g., Nextcloud's `admin` ≠ Authentik's `dtoro`).
-
-1. On the Authentik user: add attribute `<app>_uid: <target_local_username>` (YAML, Directory → Users → Edit → Attributes).
-2. Customization → Property Mappings → Create → **Scope Mapping** (not SAML):
-   - Name: `<app>-uid-override`, Scope name: `<app>-uid`, Expression:
-     ```python
-     return {"nc_uid": user.attributes.get("<app>_uid", user.username)}
-     ```
-   - **Use a custom claim key** (e.g. `nc_uid`), not `preferred_username` — the default `profile` scope mapping emits `preferred_username` and will overwrite yours depending on ordering.
-3. Attach the new scope to the provider (Providers → app → Scopes).
-4. On the app side, point its OIDC UID-mapping setting at the custom claim.
-
-For Nextcloud:
-```
-occ user_oidc:provider <name> --mapping-uid=nc_uid
-occ user_oidc:provider <name> --scope="openid profile email <app>-uid"
-```
-
-### Bypass forward-auth for API paths (mobile apps)
-
-If the app has its own token-based API auth and a mobile client, API paths must bypass forward-auth — mobile apps can't follow the browser login redirect. Pattern in the Caddyfile site block:
-
-```
-paperless.hubris.network {
-    tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
-    @api path /api/*
-    handle @api {
-        reverse_proxy 192.168.8.130:8000
-    }
-    handle {
-        import authentik
-        reverse_proxy 192.168.8.130:8000
-    }
-}
-```
-
-API paths to bypass per app:
-- [Paperless](103-paperless.md): `/api/*` (Bearer)
-- [Sonarr / Radarr / Lidarr / etc.](122-arriman.md): `/api/*` (X-Api-Key)
-- [qBittorrent](122-arriman.md): `/api/*` (session cookie from `/api/v2/auth/login`)
-- [SABnzbd](122-arriman.md): `/api?*` (apikey query param) — match `/api*` for query-string APIs
-- Homarr: no mobile client
-- [Portainer](105-apps.md): mobile uses same session auth as web; no bypass typically needed
-
-### Backend trust of Authentik headers (skip the app's own login after SSO)
-
-- [**Paperless**](103-paperless.md): `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Restart `paperless-webserver paperless-task-queue paperless-scheduler paperless-consumer`. Django auto-creates matching users on first SSO login; promote to superuser via existing admin UI.
-- Apps without header-auth support: users log in twice (SSO + app login). Acceptable but degraded UX.
-
-## Per-app integration map
-
-| App                                       | Type             | Notes |
-| ----------------------------------------- | ---------------- | ----- |
-| [Paperless (103)](103-paperless.md)       | Forward-auth + REMOTE_USER | `/api/*` bypass |
-| [Nextcloud (114)](114-nextcloud.md)       | Native OIDC      | `nc_uid` override; local dnsmasq required (Guzzle bypasses `/etc/hosts`) |
-| [mulita (120)](120-mule-images.md)        | Native OIDC      | `extra_hosts` override in compose |
-| [Booklore (105)](105-apps.md)             | Native OIDC      | Redirect URI `/oauth2-callback`; `extra_hosts` |
-| [Portainer (105)](105-apps.md)            | Native OAuth2    | `portainer_uid` custom claim; `--trusted-origins` flag |
-| [WriteFreely (105)](105-apps.md)          | Native OIDC      | `[oauth.generic]` block; `extra_hosts` |
-| [qBittorrent (122)](122-arriman.md)       | Forward-auth via IP whitelist | Reverse-proxy support enabled in qBit |
-| [Artifacto (105)](105-apps.md)            | Forward-auth + gateway-secret auto-login | Public `/p/*` paths bypass |
-| [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. |
-
-## Netbird IdP integration — DEFERRED
-
-The `netbirdio/netbird-server` combined image has no config knobs for external OIDC. Verified in `combined/cmd/config.go` on main (v0.69.0): `AuthConfig` only exposes issuer + redirect URIs; `ToManagementConfig()` hardcodes `AuthAudience="netbird-dashboard"`, `UserIDClaim="sub"`, always calls `buildEmbeddedIdPConfig()`. Code comment: `"embedded IdP is always enabled in combined server"`.
-
-To wire Authentik into Netbird login/user-sync, must migrate to the legacy split stack (`netbirdio/management` + `signal` + `dashboard` + coturn) with the richer `management.json` schema. Sqlite management data should migrate but needs verification; peers stay connected via wireguard keys.
-
-**Pre-work already in place (keep for re-use):**
-- Provider `Provider for Netbird` + App `netbird`. Client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`. Redirect URIs for `https://netbird.hubris.network/{nb-auth,nb-silent-auth}` and `http://localhost:53000/`. Scopes: `openid profile email offline_access goauthentik.io/api`.
-- Service account `netbird-service` in `authentik Admins` group, non-expiring API token `netbird-service-api`.
-- Netbird mgmt host (`82.165.190.79`) is now a peer on its own mesh. See [mesh](../infrastructure/mesh.md).
-
-## DNS responsibility
-
-dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` + `127.0.0.1:53`, serving every `*.hubris.network` subdomain → `192.168.8.175`. **There is no wildcard** — every site needs an explicit `address=` entry. See [DNS split-horizon](../infrastructure/dns.md).
-
-## Related
-- [DNS split-horizon](../infrastructure/dns.md)
-- [Caddy (121)](121-caddy.md)
-- [Mesh migration](../infrastructure/mesh.md)
-- Every gated app under [containers/index](index.md)
-
-## Changelog
-
-### 2026-04-28 — wiki entry created
-Initial documentation.
-
-### 2026-04-22 — Phase 6 (Netbird IdP swap) deferred
-Combined netbird-server image can't take an external IdP. Pre-work in Authentik kept for later. Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
-
-### 2026-04-22 — Artifacto, mulita, WriteFreely, Portainer wired
-Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto.
-
-### 2026-04-21 — deployed; Phases 1–5 complete
-LXC 124 provisioned, stack at `/opt/authentik`, public URL via Caddy, Paperless + Booklore + Nextcloud + Home Assistant wired. dnsmasq for split-horizon DNS lives on the same LXC.

containers/127-mule-photos-new.md

@@ -144,8 +144,23 @@ curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \
   https://photos-new.hubris.network/api/v1/oidc/login -i | head -2   # 302 → auth.hubris.network
 ```
 
+> **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack
+> validated here was promoted into production on [LXC 120](120-mule-images.md)
+> via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This
+> page is retained for archaeology; everything below is historic. See the
+> 2026-05-22 entry in [120-mule-images.md](120-mule-images.md#changelog) for
+> the cutover detail.
+
 ## Changelog
 
+### 2026-05-22 — Destroyed
+
+`pct destroy 127` after curl verification of the new 120 stack passed
+end-to-end. dnsmasq `photos-new.hubris.network` line removed; gitea
+webhook id 9 + `192.168.8.181` ALLOWED_HOST_LIST entry removed; caddy
+`photos-new.hubris.network` site block dropped from `dtoro/caddy-conf`.
+Authentik `mule-photos-new` app + provider deleted.
+
 ### 2026-05-18 (pm) — OIDC state-cookie fix: deploy.sh no longer recreates pp-app
 
 User reported `failed to get state: securecookie: the value is not valid` after

containers/index.md

@@ -11,21 +11,21 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
 | 114 | [nextcloud](114-nextcloud.md)           | 192.168.8.224 | priv | 4 | 6 GiB  | 25 GiB | `/mnt/library`        | `cloud.hubris.network`                | running  |
 | 118 | [elementsynapse](118-elementsynapse.md) | 192.168.8.239 | unpriv | 1 | 2 GiB  | 8 GiB  | —                     | `matrix.hubris.network`               | running  |
 | 119 | [sophia](119-sophia.md)                 | 192.168.8.157 | priv | 2 | 1 GiB  | 10 GiB | `/mnt/library`        | —                                     | running  |
-| 120 | [mule-images](120-mule-images.md)       | 192.168.8.136 | priv | 4 | 8 GiB  | 60 GiB | `/mnt/library`        | `photos.hubris.network`               | running  |
+| 120 | [mule-images](120-mule-images.md)       | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU passthrough) | `photos.hubris.network` | running  |
 | 121 | [caddy](121-caddy.md)                   | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB  | —                     | (terminates all `*.hubris.network`)   | running  |
 | 122 | [arriman](122-arriman.md)               | 192.168.8.132 | priv | 4 | 8 GiB  | 24 GiB | `/mnt/library`        | `jellyseerr` / `qbit` / `sab`         | running  |
-| 123 | [claudio-bot](123-claudio-bot.md)       | 192.168.8.230 | unpriv | 1 | 512 MiB | 8 GiB | —                     | —                                     | running  |
 | 124 | [authentik](124-authentik.md)           | 192.168.8.180 | priv | 2 | 4 GiB  | 20 GiB | —                     | `auth.hubris.network`                 | running  |
 | 126 | [plato](126-plato.md)                   | 192.168.8.190 | priv | 2 | 2 GiB  | 8 GiB  | `/mnt/library/documents/plato` | `plato.hubris.network`       | running  |
-| 127 | [mule-photos-new](127-mule-photos-new.md) | 192.168.8.181 | unpriv | 6 | 8 GiB | 40 GiB | — (rsync scratch on rootfs) | `photos-new.hubris.network`         | running  |
 
 ## Recently destroyed (kept for archaeology)
 
 | ID  | Name             | Destroyed       | Reason                                        |
 | --- | ---------------- | --------------- | --------------------------------------------- |
+| 127 | mule-photos-new  | 2026-05-22      | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
 | 100 | arr (yunohost)   | ~2026-04-28     | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
 | 106 | flaresolverr     | ~2026-04-28     | Folded into the arriman docker compose       |
 | 116 | heaper           | 2026-05-14      | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
+| 123 | claudio-bot      | 2026-06-04      | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) |
 | 109 | syncthing        | 2026-05-14      | Decommissioned by user; `/mnt/library/syncthing` was already empty |
 | 125 | seafile          | 2026-05-13      | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
 | 107 | marimo           | between 2026-04-21 and 2026-04-28 | Decommissioned                |
@@ -41,7 +41,7 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
 - All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. IPs are stable via the LAN router's DHCP reservations.
 - `onboot=1` on every container — the host brings them up after `pve-guests.service`.
 - Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library`. Containers that don't mount `/mnt/library` don't need it.
-- Most containers are privileged. Unprivileged ones (`101`, `118`, `121`, `123`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
+- Most containers are privileged. Unprivileged ones (`101`, `118`, `121`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
 
 ## Related
 - [Hubris host](../hosts/hubris.md)

hosts/apps.yaml

@@ -8,14 +8,10 @@ role: docker-apps
 host: hubris
 pve_id: 105
 lan_ip: 192.168.8.205
-mesh:
-  netbird:
-    fqdn: apps.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_hosts:
@@ -35,16 +31,20 @@ services_hosted:
 - name: homelab_mcp
   backend: apps
   port: 9810
-  netbird_host: apps.netbird.selfhosted
-  endpoint: http://apps.netbird.selfhosted:9810/sse
-  note: MCP server. Read-only context + management. Mesh-only.
+  systemd_unit: homelab-mcp
+  public_host: mcp.hubris.network
+  endpoint: https://mcp.hubris.network/mcp
+  note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via
+    Netbird (192.168.8.0/24 is a network resource routed through hubris).
 - name: secrets_issuance
   backend: apps
   port: 9820
-  netbird_host: apps.netbird.selfhosted
-  endpoint: http://apps.netbird.selfhosted:9820/issue
-  note: Issues per-client age private keys. Mesh-only.
+  systemd_unit: secrets-issuance
+  public_host: secrets.hubris.network
+  endpoint: https://secrets.hubris.network/issue
+  note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS).
+age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
 see_also:
 - containers/105-apps.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/arriman.yaml

@@ -8,14 +8,10 @@ role: arr-stack
 host: hubris
 pve_id: 122
 lan_ip: 192.168.8.132
-mesh:
-  netbird:
-    fqdn: arriman.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_hosts:
@@ -30,5 +26,5 @@ services_hosted:
   note: jellyseerr / qbit / sab on docker compose
 see_also:
 - containers/122-arriman.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/authentik.yaml

@@ -1,35 +0,0 @@
-# Generated by mcp/build_host_files.py from inventory.yaml.
-# Do NOT edit by hand — your changes will be overwritten.
-# Source of truth: ../inventory.yaml
-name: authentik
-kind: lxc
-os: linux
-role: idp
-host: hubris
-pve_id: 124
-lan_ip: 192.168.8.180
-mesh:
-  netbird:
-    fqdn: authentik.netbird.selfhosted
-mesh_globals:
-  primary: netbird
-  accepted:
-  - netbird
-  - tailscale
-public_host: auth.hubris.network
-runs:
-- authentik
-- dnsmasq
-services_hosted:
-- name: authentik
-  url: https://auth.hubris.network
-  backend: authentik
-- name: dnsmasq
-  backend: authentik
-  note: split-horizon DNS, /etc/dnsmasq.d/hubris-split.conf
-notes:
-- 'Also hosts split-horizon dnsmasq: /etc/dnsmasq.d/hubris-split.conf'
-see_also:
-- containers/124-authentik.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue

hosts/caddy.yaml

@@ -8,14 +8,10 @@ role: reverse-proxy
 host: hubris
 pve_id: 121
 lan_ip: 192.168.8.175
-mesh:
-  netbird:
-    fqdn: caddy.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 peers:
 - authentik
 - gitea
@@ -31,5 +27,5 @@ notes:
 - /etc/caddy is a git checkout of dtoro/caddy-conf
 see_also:
 - containers/121-caddy.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/claudio-bot.yaml

@@ -1,24 +0,0 @@
-# Generated by mcp/build_host_files.py from inventory.yaml.
-# Do NOT edit by hand — your changes will be overwritten.
-# Source of truth: ../inventory.yaml
-name: claudio-bot
-kind: lxc
-os: linux
-role: matrix-agent
-host: hubris
-pve_id: 123
-lan_ip: 192.168.8.230
-mesh:
-  netbird:
-    fqdn: claudio-bot.netbird.selfhosted
-mesh_globals:
-  primary: netbird
-  accepted:
-  - netbird
-  - tailscale
-notes:
-- Reads /opt/homelab-context/ on startup
-see_also:
-- containers/123-claudio-bot.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue

hosts/elementsynapse.yaml

@@ -8,14 +8,10 @@ role: matrix-server
 host: hubris
 pve_id: 118
 lan_ip: 192.168.8.239
-mesh:
-  netbird:
-    fqdn: elementsynapse.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 public_host: matrix.hubris.network
 runs:
 - matrix
@@ -25,5 +21,5 @@ services_hosted:
   backend: elementsynapse
 see_also:
 - containers/118-elementsynapse.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/gitea.yaml

@@ -8,14 +8,10 @@ role: git-server
 host: hubris
 pve_id: 104
 lan_ip: 192.168.8.121
-mesh:
-  netbird:
-    fqdn: gitea.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_host: git.hubris.network
@@ -30,5 +26,5 @@ notes:
 - Bare repos live at /mnt/library/repos/dtoro/*.git
 see_also:
 - containers/104-gitea.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/haos.yaml

@@ -7,14 +7,11 @@ os: linux
 role: home-automation
 host: hubris
 pve_id: 108
-mesh:
-  netbird:
-    fqdn: haos.netbird.selfhosted
+lan_ip: 192.168.8.101
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 runs:
 - haos
 services_hosted:
@@ -22,5 +19,5 @@ services_hosted:
   backend: haos
 see_also:
 - vms/108-haos.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/hubris.md

@@ -6,8 +6,9 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
 - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
 - **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
 - **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
-- **LAN (primary):** `192.168.8.77/24` on bridge `vmbr0` (slave: `eno1`), gateway `192.168.8.1`. Default route metric 0.
-- **WiFi (failover):** `192.168.8.141/24` on `wlp3s0` (MediaTek MT7922, AX), DHCP from the same router. Default route metric 200. See [Phase 1 WiFi failover](#phase-1-wifi-failover) below.
+- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
+- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
+- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
 - **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
 - **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
 
@@ -36,17 +37,6 @@ See [containers/index](../containers/index.md). 13 active (109 syncthing destroy
 - **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
 - **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
 
-## Phase 1 WiFi failover
-
-Host is dual-homed on LAN (`eno1`/`vmbr0`) and WiFi (`wlp3s0`) so management/SSH stay reachable when LAN drops. **Guests are not yet failed over** — the LXC fleet remains on `vmbr0`/`eno1`. Phase 2 will migrate guest networking off the bridge so the homelab survives full LAN loss.
-
-- WiFi creds in `/etc/wpa_supplicant/wpa_supplicant-wlp3s0.conf` (hashed PSK, mode 600). SSID lives in `/etc/network/interfaces` as `wpa-conf`.
-- Both interfaces sit on the same `192.168.8.0/24`; cross-talk avoided with `arp_ignore=1` + `arp_announce=2` on `eno1`/`vmbr0`/`wlp3s0` (set via `post-up` in `/etc/network/interfaces`).
-- A second default route at metric 200 is added on `wlp3s0` (post-up). LAN wins while up.
-- **Carrier-based failover:** `vmbr0`'s carrier follows the LXC veth members, so it stays `1` even when `eno1` loses link. `ignore_routes_with_linkdown` is therefore not enough on its own. `wan-failover.service` (`/usr/local/sbin/wan-failover.sh`) watches `/sys/class/net/eno1/carrier` via `ip monitor link` and removes/restores the `vmbr0` default route on transitions. Logs to `journalctl -t wan-failover`.
-- Failover verified 2026-04-28: `ip link set eno1 down` → outbound HTTP keeps working via WiFi; `ip link set eno1 up` → vmbr0 default restored.
-- Reachable on `192.168.8.77` (LAN) and `192.168.8.141` (WiFi); SSH works on either.
-
 ## Network performance baseline (2026-05-14)
 
 | Path | Throughput | Notes |
@@ -99,6 +89,9 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
 
 ## Changelog
 
+### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
+Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
+
 ### 2026-05-14 — LXC 109 (syncthing) decommissioned
 User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
 

hosts/hubris.yaml

@@ -14,12 +14,12 @@ mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 ssh:
   port: 22
   netbird_port: 22022
+  user: root
 runs:
 - proxmox_ui
 services_hosted:
@@ -27,7 +27,8 @@ services_hosted:
   url: https://proxmox.hubris.network
   backend: hubris
   port: 8006
+age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
 see_also:
 - hosts/hubris.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/jellyfin.yaml

@@ -8,14 +8,10 @@ role: media-server
 host: hubris
 pve_id: 101
 lan_ip: 192.168.8.206
-mesh:
-  netbird:
-    fqdn: jellyfin.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_host: media.hubris.network
@@ -27,5 +23,5 @@ services_hosted:
   backend: jellyfin
 see_also:
 - containers/101-jellyfin.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/ludo-mini.yaml

@@ -5,6 +5,7 @@ name: ludo-mini
 kind: workstation
 os: linux
 role: dev
+lan_ip: 192.168.178.181
 mesh:
   netbird:
     fqdn: ludo-mini.netbird.selfhosted
@@ -12,6 +13,7 @@ mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+ssh:
+  user: dtoro
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/mac-mini.yaml

@@ -5,15 +5,18 @@ name: mac-mini
 kind: workstation
 os: macos
 role: dev
+lan_ip: 192.168.178.182
 mesh:
   netbird:
-    fqdn: mac-mini.netbird.selfhosted
+    fqdn: mac-mini-234-17.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
+ssh:
+  user: dtoro
 notes:
 - Only macOS in the fleet. Bootstrap uses launchd.
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+age_pubkey: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/mule-images.yaml

@@ -8,14 +8,10 @@ role: photo-management
 host: hubris
 pve_id: 120
 lan_ip: 192.168.8.136
-mesh:
-  netbird:
-    fqdn: mule-images.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_host: photos.hubris.network
@@ -27,5 +23,5 @@ services_hosted:
   backend: mule-images
 see_also:
 - containers/120-mule-images.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/mule-photos-new.yaml

@@ -1,29 +0,0 @@
-# Generated by mcp/build_host_files.py from inventory.yaml.
-# Do NOT edit by hand — your changes will be overwritten.
-# Source of truth: ../inventory.yaml
-name: mule-photos-new
-kind: lxc
-os: linux
-role: photo-management
-host: hubris
-pve_id: 127
-lan_ip: 192.168.8.181
-mesh:
-  netbird:
-    fqdn: mule-photos-new.netbird.selfhosted
-mesh_globals:
-  primary: netbird
-  accepted:
-  - netbird
-  - tailscale
-public_host: photos-new.hubris.network
-runs:
-- photos_new
-services_hosted:
-- name: photos_new
-  url: https://photos-new.hubris.network
-  backend: mule-photos-new
-see_also:
-- containers/127-mule-photos-new.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue

hosts/netbird-vps.yaml

@@ -0,0 +1,28 @@
+# Generated by mcp/build_host_files.py from inventory.yaml.
+# Do NOT edit by hand — your changes will be overwritten.
+# Source of truth: ../inventory.yaml
+name: netbird-vps
+kind: external
+os: linux
+role: netbird-mgmt
+mesh:
+  netbird:
+    ip: 100.122.165.149
+    fqdn: netbird-ionos.netbird.selfhosted
+mesh_globals:
+  primary: netbird
+  accepted:
+  - netbird
+ssh:
+  user: root
+notes:
+- "Public IONOS VPS \u2014 hosts the vanilla netbird mgmt+signal+relay+dashboard stack + host coturn (see\
+  \ infrastructure/vps-hardening.md + infrastructure/mesh.md changelog 2026-05-21)."
+- NOT a homelab client. No /etc/age/key.txt, no /opt/homelab-context clone. Managed via ssh from hubris;
+  sshd is locked to hubris's pubkey.
+- Public IPv4 82.165.190.79. Auto-patching via unattended-upgrades.
+- Configs rendered by `homelab render-vps-configs` from vps/turnserver.conf.tmpl + vps/management.json.tmpl,
+  with secrets decrypted from secrets/turn-shared-secret.yaml + secrets/netbird-authentik-oidc.yaml on
+  hubris.
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/nextcloud.yaml

@@ -8,14 +8,10 @@ role: file-sync
 host: hubris
 pve_id: 114
 lan_ip: 192.168.8.224
-mesh:
-  netbird:
-    fqdn: nextcloud.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_host: cloud.hubris.network
@@ -27,5 +23,5 @@ services_hosted:
   backend: nextcloud
 see_also:
 - containers/114-nextcloud.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/nfs-export.yaml

@@ -8,15 +8,11 @@ role: storage-export
 host: hubris
 pve_id: 102
 lan_ip: 192.168.8.200
-mesh:
-  netbird:
-    fqdn: nfs-export.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 see_also:
 - containers/102-nfs-export.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/paperless.yaml

@@ -8,14 +8,10 @@ role: document-archive
 host: hubris
 pve_id: 103
 lan_ip: 192.168.8.130
-mesh:
-  netbird:
-    fqdn: paperless.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 public_host: paperless.hubris.network
@@ -27,5 +23,5 @@ services_hosted:
   backend: paperless
 see_also:
 - containers/103-paperless.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/plato.yaml

@@ -8,14 +8,10 @@ role: app
 host: hubris
 pve_id: 126
 lan_ip: 192.168.8.190
-mesh:
-  netbird:
-    fqdn: plato.netbird.selfhosted
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library/documents/plato
 public_host: plato.hubris.network
@@ -27,5 +23,5 @@ services_hosted:
   backend: plato
 see_also:
 - containers/126-plato.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/republic-laptop.yaml

@@ -12,6 +12,7 @@ mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+ssh:
+  user: dtoro
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/sophia.yaml

@@ -7,18 +7,14 @@ os: linux
 role: workshop
 host: hubris
 pve_id: 119
-lan_ip: 192.168.8.157
-mesh:
-  netbird:
-    fqdn: sophia.netbird.selfhosted
+lan_ip: 192.168.8.109
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 mounts:
 - /mnt/library
 see_also:
 - containers/119-sophia.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

hosts/zimaos.yaml

@@ -7,14 +7,11 @@ os: linux
 role: nas-frontend-eval
 host: hubris
 pve_id: 100
-mesh:
-  netbird:
-    fqdn: zimaos.netbird.selfhosted
+lan_ip: 192.168.8.195
 mesh_globals:
   primary: netbird
   accepted:
   - netbird
-  - tailscale
 public_host: zimaos.hubris.network
 runs:
 - zimaos
@@ -24,5 +21,5 @@ services_hosted:
   backend: zimaos
 see_also:
 - vms/100-zimaos.md
-mcp_endpoint: http://apps.netbird.selfhosted:9810/sse
-secrets_issuance_endpoint: http://apps.netbird.selfhosted:9820/issue
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue

infrastructure/auto-deploy.md

@@ -20,7 +20,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
   - `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
   - `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
   - `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
-  - `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
+  - ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
   - `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
   - `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
   - `192.168.8.190` ([plato (126)](../containers/126-plato.md))
@@ -38,10 +38,16 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
 | `dtoro/mule-image`              | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
 | `dtoro/Artifacto`               | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
 | `dtoro/Plato`                   | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
-| `dtoro/claudio-bot`             | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
+| `dtoro/claudio-bot`             | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
 | `dtoro/backup-library`          | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy`  | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
+| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
+| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
 
-> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
+> Note: `dtoro/Homelab-Docs` has **two webhooks** firing on the same push.
+> Each owns its own clone on LXC 105. They don't conflict because each
+> deploy.sh only touches its own service unit + venv.
+
+> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
 
 ## When you change a tracked config
 
@@ -52,8 +58,9 @@ Always commit + push. Local-only edits drift. Common ones:
 - `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
 - `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
 - `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys)
-- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys)
+- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
 - `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
+- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
 
 ## Per-pipeline notes / gotchas
 
@@ -81,13 +88,44 @@ Always commit + push. Local-only edits drift. Common ones:
 - Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
 - `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
 
+### homelab-mcp / secrets-issuance
+- Both ride a single push to `dtoro/Homelab-Docs`. Two clones on LXC 105
+  (`/opt/homelab-mcp`, `/opt/secrets-issuance`) — each is an independent
+  Shape-B target with its own webhook receiver.
+- The deploy script restarts the service it just updated. Because the
+  webhook receiver itself is a separate systemd unit (`*-deploy.service`),
+  it does NOT restart itself — but `deploy.sh` running `systemctl
+  restart homelab-mcp-deploy.service` (or the secrets-issuance one)
+  would create a kill-self loop. The current `deploy.sh` is careful
+  to only restart the main service.
+- Both services consume `/opt/homelab-context` for their runtime data
+  (inventory, secret recipient lookup). That clone is **the same clone
+  every other client has** — kept fresh by `homelab-context-sync.timer`,
+  not by these webhooks.
+
+## Custom-built binaries that overlap apt-managed paths
+
+If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/<name>` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.
+
+Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:
+
+1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: <name>`, `Version: 1:<upstream>-hubris<n>`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)).
+
+2. **Hold the apt package.** `apt-mark hold <pkg>` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten.
+
+If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns.
+
 ## Related
 - [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
-- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
+- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
 - [Backups (disabled)](backups.md)
+- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
 
 ## Changelog
 
+### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
+Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
+
 ### 2026-05-13 — Plato pipeline added
 Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/126-plato.md)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
 
@@ -97,6 +135,9 @@ Initial documentation. Six active pipelines.
 ### 2026-04-22 — Artifacto pipeline added
 Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
 
+### 2026-06-04 — claudio-bot pipeline decommissioned
+LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
+
 ### 2026-04-21 — mule-image + claudio-bot pipelines added
 Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
 

infrastructure/backups.md

@@ -71,7 +71,7 @@ Baked into the systemd units:
 
 ## Notifications
 
-POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.
+~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
 
 `OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
 
@@ -95,7 +95,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
 
 ## Related
 - [Hubris host](../hosts/hubris.md)
-- [claudio-bot (123)](../containers/123-claudio-bot.md)
+- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
 - [Monitoring](monitoring.md)
 - [Auto-deploy](auto-deploy.md)
 - [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)

infrastructure/dns.md

@@ -7,14 +7,15 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
 ## Components
 
 - **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
-- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md), `192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`).
+- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53` — the **single DNS source** for LAN, household, and mesh. (Through 2026-06-21 it also synced A-records into a NetBird managed zone; that replica was retired in Phase 4, see changelog.) Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
 - **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
-- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
+- **All LXCs** now point at Technitium (`192.168.8.2`) directly (since 2026-06-21 — see changelog). The earlier mix of router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`) — which returned the public IONOS A record and forced `/etc/hosts` overrides — has been removed.
 
-## Live entries (as of 2026-04-28)
+## Live entries (as of 2026-06-04)
 
 ```
-address=/auth.hubris.network/192.168.8.175
+address=/auth.hubris.network/82.165.190.79   # → VPS, not Caddy (Authentik migrated 2026-05-31)
+address=/sso.hubris.network/192.168.8.175    # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
 address=/git.hubris.network/192.168.8.175
 address=/media.hubris.network/192.168.8.175
 address=/paperless.hubris.network/192.168.8.175
@@ -40,24 +41,20 @@ Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that point
 
 ## Why split-horizon
 
-The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
+The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
 
 ## The gotcha that cost a debug session (2026-04-22)
 
-Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
+Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
 
 ## Recipe — adding a new subdomain
 
 1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
-2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md):
-   ```
-   address=/<new>.hubris.network/192.168.8.175
-   ```
-3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path.
-4. Verify: `dig @192.168.8.180 +short <new>.hubris.network` → `192.168.8.175`.
-5. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
+2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)`. That's it — Technitium is the single DNS source; mesh peers forward to it live and LAN/household query it directly. (No managed-zone sync to wait for — retired 2026-06-21, Phase 4.)
+3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
+4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`. On a NetBird peer that caches a stale answer, `netbird service restart` clears its resolver cache.
 
-> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124).
+> Technitium on LXC 107 is the single source of truth. (Through 2026-06-21 a `dns-sync.py` cron mirrored it into a NetBird managed zone; that replica + cron were removed in Phase 4 once mesh peers were forwarding directly — see [changelog](#2026-06-21--dns-single-source-phase-4-complete--managed-zone-removed-technitium-is-the-single-source).)
 
 ## Public path — what does and doesn't follow the LAN map
 
@@ -75,10 +72,92 @@ Either:
 - [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
 - [Ingress (VPS traefik)](ingress.md) — public-side counterpart
 - [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
-- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq
+- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
 
 ## Changelog
 
+### 2026-06-21 — DNS single-source, Phase 4: COMPLETE — managed zone removed, Technitium is the single source
+After upgrading the Mac client `0.68.3 → 0.71.3` (matching mgmt) and clearing its NetBird resolver cache (`netbird service restart`), the managed-zone deletion was retried and **works**: with 0 managed-zone records, the Mac resolves `git`/`cloud`/`nfs-export`/`auth`.hubris.network entirely by forwarding to Technitium (`192.168.8.2`). The iPhone confirmed the same on **cellular** (no LAN/Fritz path — proof it's the mesh-forward path).
+
+So the first deletion attempt (the REVERTED entry below) was a **misdiagnosis**: forwarding wasn't broken — the old 0.68.3 client's resolver cache (`100.122.255.254`) held stale/empty answers and wouldn't clear on `down/up` (only a full daemon restart clears it), and the Mac's dual LAN+mesh resolver paths muddied the test. A direct query (`dig @100.122.255.254 <unsynced-name>`) had in fact shown forwarding working all along.
+
+**Done:**
+- Deleted all 23 NetBird managed-zone A-records via API.
+- Removed the `*/10` sync cron (`/etc/cron.d/dns-sync`). Kept `/opt/dns-sync/sync.py` + token + a pre-deletion backup as an emergency-restore tool only (run it once to rebuild the managed zone if ever reverting).
+
+**End state — one zone:** Technitium (`192.168.8.2`) is the single authoring + serving source. Mesh peers forward to it (route via `Core` → `192.168.8.0/24`); LAN/household query it directly (Fritz!Box DNSv4 → Technitium). No managed-zone replica, no sync.
+
+**Prereqs for it to keep working:** mesh clients on NetBird **0.71.x+**, and roaming peers in the `Core` group (route) — both satisfied. Other peers (laptops, proxmox-server) pick up forwarding as their resolver caches expire, or after a `netbird service restart`.
+
+**Optional, not done:** flip the Technitium wildcard `*.hubris.network` from `→ 82.165.190.79` (VPS, mirrors public) to `→ 192.168.8.175` (Caddy) so new Caddy services need zero DNS entries. Deferred — decide separately.
+
+### 2026-06-21 — DNS single-source, Phase 4: ATTEMPTED + REVERTED — managed zone is load-bearing
+Tried to collapse to a single zone by deleting the NetBird managed-zone replica (and pausing `dns-sync`). Result: **mesh-peer DNS broke.** With the managed zone gone, the Mac (NetBird 0.68.3) failed to resolve `git`/`cloud`/`nfs-export`.hubris.network via the NetBird resolver (`100.122.255.254`) — the `home-lab-dns` nameserver group (`→ 192.168.8.2`) reports `Available` but does **not** actually serve forwarded queries. Restored the managed zone via `dns-sync.py` (23 records) and re-enabled the cron; resolution recovered immediately.
+
+**Correction to the Phase 2 entry below:** the missing route was *a* real problem (it blocked roaming-peer *connectivity* to services — the actual user-facing win), but it was **not** the whole story. DNS forwarding to the routed-LAN IP `192.168.8.2` still does not work for mesh peers, so the original "NetBird won't forward to Technitium" finding stands and the managed zone stays.
+
+**To actually finish single-source later**, forwarding must first be made to work. Two candidates, untried:
+- Upgrade NetBird clients to 0.71.x (the Mac is 0.68.3 — version skew with mgmt 0.71.3 is a known source of resolver bugs), then retest.
+- Point `home-lab-dns` at a **mesh-native** DNS IP — join CT 107 to the mesh and use its `100.122.x` address instead of the routed-LAN `192.168.8.2` (the original Phase 2 hypothesis; needs a brief 107 restart for `/dev/net/tun`).
+
+Net state after today: Phase 1 (on-prem single resolver) and Phase 2 (roaming route → iPhone reaches the homelab) stand and deliver the practical goals. The managed-zone replica + sync are **retained** as load-bearing.
+
+### 2026-06-21 — DNS single-source, Phase 2: roaming-peer route fixed (see Phase 4 correction above — forwarding still doesn't serve queries; this fixed *connectivity*, not DNS)
+The long-standing belief that "NetBird won't forward to Technitium for mesh peers" (which is *why* the managed-zone sync was built) turned out to be **wrong**. The NetBird API showed the real cause:
+
+- The `home-lab-dns` nameserver group (→ `192.168.8.2`, match-domain `hubris.network`) was already applied to **all** peers (via the `All` group). So every peer *had* the forwarding rule.
+- But the `192.168.8.0/24` route (network resource `home-lab-network`) was distributed to the **`Services`** group only = `{netbird-ionos, proxmox-server}`. Roaming peers (`dtoro-iphone`, `mac-mini`, `republic-laptop`, `muli-laptop`, `ludo-mini`) are in **`Core`**, which had **no route to the homelab subnet** → they couldn't reach `192.168.8.2` → forwarding silently failed (`Networks: -`).
+
+**Fix:** added the `Core` group to the `home-lab-network` resource distribution (now `[Services, Core]`) via `PUT /api/networks/.../resources/...`. Roaming peers immediately picked up `Networks: 192.168.8.0/24` and `[192.168.8.2:53] for [hubris.network] is Available`. No CT 107 mesh-join / TUN / restart was needed (the original Phase 2 hypothesis is obsolete). This also gives roaming devices full homelab **service** access, not just DNS.
+
+The managed zone is still in place as a fallback pending the Phase 3 roaming test (iPhone on cellular); Phase 4 then deletes the managed zone + `dns-sync` cron. Rollback: PUT the resource back to `[Services]` only.
+
+### 2026-06-21 — DNS single-source, Phase 1: on-prem LXCs decoupled from NetBird
+Goal: collapse the three overlapping DNS sources (Technitium + NetBird managed zone + per-LXC band-aids) toward **one zone**, keeping NetBird. Phase 1 (the safe, mesh-independent half) is done:
+
+- **Every homelab LXC now resolves via Technitium (`192.168.8.2`).** Fixed 8 boxes that were on a dead resolver, the router, or Tailscale MagicDNS:
+  - `192.168.8.180` (dead ex-Authentik): 102, 106, 126
+  - `192.168.8.1` (router → public answer): 101
+  - `100.100.100.100` (Tailscale MagicDNS): 104, 105, 114, 119 — also ran `tailscale set --accept-dns=false` so `tailscaled` stops rewriting `/etc/resolv.conf`.
+  - Already correct (`.2`): 103, 118, 120, 121, 122. CT 107 stays on `1.1.1.1` by design (no self-dependency).
+- **Removed the redundant `/etc/hosts` band-aids** (`auth`/`mcp`/`secrets` → `192.168.8.175`) on 101, 103, 104, 105, 114, 118, 120, 121, 122, 126; disabled `hubris-hosts-override.service` where enabled. Backups at `/etc/hosts.bak-dnsplan`. PVE-managed lines and self-hostname maps preserved. Technitium already returns identical-or-better answers (verified: `auth → 82.165.190.79`, `mcp`/`secrets`/`cloud`/`git` → `192.168.8.175`).
+- **Nextcloud (114):** its documented Guzzle-workaround dnsmasq is not running; resolves correctly straight from Technitium, so no special-casing remains.
+- Net effect: **NetBird's DNS now only matters for off-LAN roaming peers** (Tier 2). On-prem (LXCs + household via Fritz!Box→Technitium) is fully NetBird-independent — so dropping the managed zone later can no longer break on-LAN resolution. Phases 2–4 (mesh-IP forwarding, roaming test, managed-zone + sync deletion) still pending.
+
+### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
+Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
+Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
+- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
+- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
+- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
+- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
+- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
+The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
+
+- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
+- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
+- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.
+
+**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
+
+> Reference: [scripts/dns-sync.py](../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
+
+### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
+The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
+
+**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
+
+Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
+
+### 2026-06-02 — 8 LXCs moved from DHCP to static IP
+All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
+
+### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
+Split-horizon DNS moved off [124](../containers/124-authentik.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/124-authentik.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
+
+### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
+Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
+
 ### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
 NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
 

infrastructure/homelab-context.md

@@ -0,0 +1,145 @@
+# Homelab context distribution
+
+The cross-client context-and-secrets system that makes every agent (Claude
+Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
+self-locating and able to read the same source of truth.
+
+Operational walkthrough for enrolling a new client lives in
+[operations/agent-enrollment.md](../operations/agent-enrollment.md); this
+page is the architecture reference.
+
+## What's where
+
+| Piece | Host | Path | Role |
+| --- | --- | --- | --- |
+| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
+| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
+| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab` → `/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
+| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
+| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
+| Secrets-issuance | [apps (105)](../containers/105-apps.md) | `secrets-issuance.service` on port 9820 (https://secrets.hubris.network/issue) | Generates per-client age keypair on first bootstrap; idempotent; admin-token-gated `/revoke` |
+| Sync timer | every enrolled client | `homelab-context-sync.timer` (Linux) / `network.hubris.homelab-context-sync.plist` (macOS) | `git pull --ff-only` every 5 min |
+| Encrypted secrets | `dtoro/Homelab-Docs` | `secrets/*.yaml` (SOPS+age) | Recipients declared in `.sops.yaml` |
+| Read-only context PAT | `dtoro` Gitea user, scope `read:repository` | given to operators out-of-band | Bootstrap-only — for the initial clone before SOPS works |
+| Write-scoped PAT | `secrets/gitea-pat.yaml` (SOPS) | `homelab refresh-creds` swaps it into `/etc/homelab-context/git-credentials` | All post-bootstrap pushes (client lifecycle, wiki edits) |
+
+## Data flow
+
+```
+                          dtoro/Homelab-Docs (gitea)
+                                │
+       ┌────────── push ────────┤ ◀── git push (write PAT or SSH)
+       │                        │
+       │    ┌────── push  ──────┘
+       │    │                   │
+       │    │                   ▼ webhook (push event)
+       │    │      ┌─── homelab-mcp-deploy   ──── (LXC 105:9811)
+       │    │      └─── secrets-issuance-deploy   (LXC 105:9821)
+       │    │                   │
+       │    │                   ▼
+       │    │           git pull → deploy.sh → restart service
+       │    │
+       │    └── on every client:
+       │           timer (5 min) → git pull --ff-only into /opt/homelab-context
+       │
+       ▼
+  homelab CLI / MCP server reads /opt/homelab-context for everything
+```
+
+## Why two clones on LXC 105
+
+The MCP server and secrets-issuance each have their own clone
+(`/opt/homelab-mcp`, `/opt/secrets-issuance`) **in addition to**
+`/opt/homelab-context`. Reasons:
+
+- The deploy webhook for each service updates its own clone, runs
+  `deploy.sh` from there, and re-installs the systemd unit. Mixing this
+  with the client-context clone would create a circular dependency
+  (deploy reinstalls the unit that pulled it).
+- The MCP server reads its data from `/opt/homelab-context` (the same path
+  every client uses) so changes to inventory propagate identically. Code
+  changes live in `/opt/homelab-mcp` and trigger a service restart.
+
+## Mesh / network gates
+
+- Both services bind `0.0.0.0:<port>`. The trust boundary is
+  `MESH_SUBNETS` in the service's environment + nftables (planned). Today
+  `MESH_SUBNETS=100.122.0.0/16,192.168.8.0/24` — Netbird + the homelab LAN.
+  (The legacy Tailscale CGNAT range `100.64.0.0/10` was dropped 2026-06-21
+  when Tailscale was decommissioned; secrets-issuance restarted. Only
+  secrets-issuance reads `MESH_SUBNETS` — homelab-mcp does not.) Adjust if
+  the LAN ever has untrusted devices.
+- Caddy fronts both with Let's Encrypt certs via the IONOS DNS challenge:
+  `mcp.hubris.network` → `192.168.8.205:9810`,
+  `secrets.hubris.network` → `192.168.8.205:9820`. Off-LAN clients on
+  Netbird reach them via the `192.168.8.0/24` network resource routed
+  through the PVE peer ([mesh.md](mesh.md)).
+- Clients with default-public DNS (workstations not on Netbird, LXCs
+  using router DNS) need a `/etc/hosts` override pointing
+  `mcp.hubris.network` and `secrets.hubris.network` at the caddy LXC
+  (`192.168.8.175`) — same caveat as every other `*.hubris.network`
+  service, see [dns.md](dns.md).
+
+## Secrets model
+
+- Each enrolled client gets one **age private key** issued by
+  secrets-issuance on first bootstrap. The key file stays root-only on
+  the client; the public key is committed to `inventory.yaml` (and
+  becomes a recipient on SOPS-encrypted files via `.sops.yaml`).
+- `secrets/*.yaml` are SOPS+age. Recipients per file are pinned in
+  `.sops.yaml` `creation_rules` by `path_regex`. Re-encrypting a file is
+  `sops updatekeys -y secrets/<name>.yaml`.
+- The MCP server's `list_my_secrets(caller_pubkey)` tool returns only
+  secret *names* a given pubkey can decrypt — the server never sees
+  plaintext. Decryption is local-on-client (`homelab secret <name>`
+  shells out to `sops -d` with the client's key).
+- The "all-clients" secrets (`hello.yaml` for the bootstrap decrypt
+  test, `gitea-pat.yaml` for the write-scoped PAT) are auto-granted to
+  every newly enrolled client by `homelab client add --finalize-pubkey`
+  (which appends the pubkey to the matching `.sops.yaml` rule and runs
+  `sops updatekeys`).
+- **Removal does not erase past disclosure.** Revoking a client via
+  `homelab client remove` shreds the issuance-side key, denylists the
+  hostname, removes them from the recipient list, and re-keys all
+  shared secrets — but anything they already decrypted to disk is out of
+  your control. Rotate the underlying credential if compromise is
+  suspected.
+
+## Why this design
+
+- **One source of truth** keeps inventory, code, secrets, and docs
+  versioned together. A `git log` of `inventory.yaml` is the history of
+  the homelab.
+- **Per-client age keys** scale better than a shared admin secret —
+  removing a client is a real revocation (for new ciphertext), not just
+  removing them from a wiki page.
+- **MCP layer over the same clone** gives MCP-capable agents structured
+  query (`find_service`, `search_docs`) without forcing non-MCP tools to
+  go without — anything can still `cat` the markdown.
+- **Sync timer rather than push fan-out** keeps the failure mode
+  contained: one client's webhook outage doesn't block a push from
+  landing on the others. Sub-5-min staleness is fine for docs and rare
+  enough for secrets that we don't need lower latency.
+
+## Related
+
+- [Operations: agent enrollment](../operations/agent-enrollment.md) — the
+  step-by-step for adding a new client
+- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
+  pipelines (and the rest of the lab's webhook pipelines)
+- [Mesh](mesh.md) — Netbird paths and the `192.168.8.0/24`
+  network resource
+- [Apps (105)](../containers/105-apps.md) — where both services run
+- [Gitea (104)](../containers/104-gitea.md) — the source of truth
+
+## Changelog
+
+### 2026-05-20 — system live across hubris, apps, republic-laptop
+Phase 1 of the [cross-client context plan](../README.md) merged. Three
+clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
+decrypt verified on each. Webhook auto-deploy for both LXC 105 services
+wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
+`client add --finalize-pubkey` grant flow live so new clients are one
+ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
+(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
+Hermes Agent integration so the agent uses inventory at chat-time.

infrastructure/ingress.md

@@ -15,7 +15,7 @@ Public client
 [Home backend on 192.168.8.x]
 ```
 
-Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
+LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
 
 ## Why this shape
 
@@ -45,6 +45,17 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
 
 `artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
 
+### `auth.hubris.network` — different pattern (local container, not cert-mirror)
+
+Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
+
+- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
+- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
+- Traefik reaches it over the `auth` Docker network (`172.30.1.0/24`); Postgres/Redis on that net are isolated from the netbird containers.
+- Admin UI is IP-gated: an `admin-allowlist` ipAllowList middleware on `PathPrefix(/if/admin/)` (currently `5.61.168.0/24`). Login/flow endpoints stay public.
+
+No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `auth`.
+
 ## Recipe — exposing another service
 
 1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
@@ -74,6 +85,9 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
 
 ## Changelog
 
+### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
+Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 

infrastructure/mesh.md

@@ -1,33 +1,52 @@
-# Mesh — Tailscale → Netbird migration
+# Mesh — Netbird
 
-The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target end-state. In-progress as of 2026-04-21.
+The hubris fleet runs on Netbird. Tailscale — the previous overlay — was **fully decommissioned on 2026-06-21**: removed from the 6 LXCs that still ran it (101, 103, 104, 105, 114, 119), apt package + state purged, `tailscaled` disabled. The fleet is now Netbird-only. (Historical migration notes below are kept for context.)
 
 ## Current state
 
 - **PVE host** uses Netbird (`wt0`, `100.122.38.109/16`). Its resolver is the local netbird daemon, which forwards to LAN/upstream — so the PVE host gets `*.hubris.network → 192.168.8.175` via the system resolver chain.
 - **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
-- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
+- **All LXCs** now resolve via Technitium (`192.168.8.2`) directly — as of the 2026-06-21 DNS single-source work (Phase 1). The previous mix of router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`) returned the *public* IONOS A record and is gone. See [dns.md changelog 2026-06-21](dns.md).
 
-## ICE / STUN — must use external STUN, not embedded
+## ICE / STUN / TURN
 
-The bundled netbird-server image runs an embedded STUN listener on UDP 3478. **IONOS drops inbound UDP 3478 to the VPS upstream of the host firewall** (verified 2026-05-10 via `tcpdump -ni any udp port 3478`: 0 packets captured during external probes from hubris). Without a reachable STUN server, the management API hands peers a STUN URI nothing can talk to → no srflx candidates → ICE always fails → every peer falls back to the websocket relay (`rels://netbird.hubris.network:443`). All cross-NAT traffic is then bottlenecked by the relay/VPS bandwidth (observed ~366 kB/s for Nextcloud uploads).
+**Today** (post-2026-05-21 migration):
 
-**Fix**: in `/opt/config.yaml` on the VPS, declare external STUN servers under `server:` — this disables the embedded STUN automatically:
+- External STUN servers (Google + Cloudflare) declared under `server.Stuns` in `/opt/management.json`. Embedded STUN is no longer in use.
+- coturn runs on the VPS host (apt-installed, systemd-managed). TURN URL `turn:netbird.hubris.network:3478?transport=tcp` is advertised to peers via mgmt's `TURNConfig.Turns` block. Long-term credentials at `user=netbird:<pwd from /root/turn-pass.txt>`.
+- For non-symmetric peers, ICE picks direct `srflx/srflx` (P2P). For symmetric-NAT peers, ICE falls through to TURN-relay before falling back to the WSS relay (`rels://netbird.hubris.network:443`).
 
-```yaml
-server:
-  stuns:
-    - uri: "stun:stun.l.google.com:19302"
-    - uri: "stun:stun1.l.google.com:19302"
-    - uri: "stun:stun.cloudflare.com:3478"
+**IONOS port-3478 caveat** (load-bearing — undocumented before 2026-05-21):
+
+IONOS upstream filters STUN-class traffic on port 3478 **for both UDP and TCP** by default. The UDP block was already known (verified 2026-05-10 with `tcpdump -ni any udp port 3478`). The TCP block was discovered 2026-05-21 when TURN-allocate retries timed out from a symmetric-NAT laptop: TCP handshake completed (carried by kernel-only SYN exchange) but data packets never reached the VPS's `ens6` interface.
+
+**Resolution**: operator must add an inbound firewall exception in the IONOS dashboard (Cloud Panel → Networks → Firewall policy on the VPS) for inbound TCP 3478 (and UDP 3478 if STUN-via-VPS-listener is ever needed; we use external STUN so this is optional). Without that exception, coturn is unreachable from outside even though host nftables + listener look fine.
+
+**Verifying TURN works** end-to-end from an outside peer:
+
+```python
+# python3
+import socket, struct, secrets
+s = socket.create_connection(("netbird.hubris.network", 3478), timeout=10)
+tid = secrets.token_bytes(12)
+attrs = struct.pack("!HHI", 0x0019, 4, 17 << 24)   # REQUESTED-TRANSPORT = UDP
+msg = struct.pack("!HHI", 0x0003, len(attrs), 0x2112A442) + tid + attrs  # Allocate
+s.sendall(msg)
+print(s.recv(4096).hex())  # expect 120 bytes starting 0x0113 (Allocate Error = auth challenge)
 ```
 
-`docker restart netbird-server`, then `netbird down && netbird up` on each peer to force a resync. Verify with `netbird status -d` — `Connection type:` should flip from `Relayed` to `P2P` for peers that aren't behind double-NAT/CGNAT.
+A 120-byte `0x0113`-typed response means coturn is reachable and responding. An indefinite TCP timeout (after `connected from ...` prints) means the IONOS rule was reverted or narrowed.
 
-If a peer is *still* relayed after this, it's a NAT-symmetry problem on its side, not a config bug — would need TURN to fix.
+**If a peer is still on the WSS relay after this** — verify in `netbird status -d`: `Connection type: P2P` with srflx/srflx is the cone-NAT happy path; `Connection type: P2P` with relay/srflx (or relay/prflx) means the peer's network is symmetric-NAT and it's using TURN. Falling back to `Relayed (rels://...)` only happens if TURN allocate also fails — investigate the IONOS exception first.
+
+**Old combined-server note (history, kept for context):**
+
+Pre-migration, the bundled `netbirdio/netbird-server` combined image silently ignored both `server.turns:` and top-level `TURNConfig:` YAML, so TURN was non-functional. That's why the architectural migration to the canonical multi-container stack (`mgmt + signal + relay + coturn`) happened. See the 2026-05-21 changelog entry below and the [Netbird-combined-no-TURN finding](https://github.com/netbirdio/netbird/issues) in homelab memory for the full discovery.
 
 ## Consequence — every LXC wired to Authentik needs an internal override
 
+> **RESOLVED 2026-06-21 (DNS single-source, Phase 1).** Every homelab LXC now points its resolver directly at **Technitium (`192.168.8.2`)**, which answers the full split-horizon zone (`auth → 82.165.190.79`, everything else → Caddy `192.168.8.175`). The per-LXC `/etc/hosts` overrides and Tailscale-MagicDNS/dead-`.180`/router resolvers below were removed; `hubris-hosts-override.service` disabled where present. The section is kept for history. See [dns.md changelog 2026-06-21](dns.md).
+
 Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`.
 
 Two techniques. Pick by HTTP-client behavior.
@@ -73,7 +92,7 @@ pct set <id> --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"
 ## Adding new LXCs
 
 - Don't add new LXCs to Tailscale; add them to Netbird. Tailscale is being decommissioned on hubris.
-- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. If nameserver is `192.168.8.1` or `100.100.100.100`, add the hosts override. If it's the netbird daemon IP, skip.
+- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. It should be `192.168.8.2` (Technitium), which returns correct split-horizon answers — no `/etc/hosts` override needed. (Historically, boxes on `192.168.8.1`/`100.100.100.100` needed an override; those resolvers were removed 2026-06-21.)
 
 ## Long-term fix
 
@@ -99,6 +118,68 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
 
 ## Changelog
 
+### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
+The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
+
+### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
+
+**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.
+
+**Root cause:** The 2026-05-21 migration configured `management.json` with `OIDCConfigEndpoint: https://auth.hubris.network/...`. The public DNS for `auth.hubris.network` already pointed to `82.165.190.79` (VPS), but the VPS Traefik had no route for that host → management got EOF on every boot, crash-looped.
+
+**Fix:**
+1. Added `auth-hubris` router to `/opt/traefik-dynamic.yaml` — `Host(auth.hubris.network)` with `certResolver: letsencrypt` → service `auth-authentik`.
+2. Service backend: `https://192.168.8.175` (Caddy on hubris LAN) via the Netbird mesh.
+3. ServersTransport `skip-verify` with `serverName: auth.hubris.network` + `insecureSkipVerify: true` so Traefik sends the correct SNI to Caddy.
+4. Upgraded VPS Netbird client `0.69.0 → 0.71.3` via apt (`apt install netbird=0.71.3`).
+5. Mesh fully recovered; management connected to peers within ~1 min.
+
+**Bootstrap note (if mesh is dark and management must restart):** If the WireGuard tunnels are fully dead AND management needs to restart (after e.g. a VPS reboot with no peer handshakes), the Traefik backend `192.168.8.175` will be unreachable and management will crash again. Recovery: temporarily open VPS port 22 via IONOS console (`nft insert rule inet hubris-fw input iifname "ens6" tcp dport 22 accept`), SSH in, run `socat TCP-LISTEN:8443,bind=172.30.0.1,reuseaddr,fork TCP:127.0.0.1:8443 &`, then from hubris `ssh -f -N -R 127.0.0.1:8443:192.168.8.175:443 root@82.165.190.79` — this bootstraps one management start, after which the mesh self-heals.
+
+**Architecture after this change:** `auth.hubris.network` is publicly accessible (HTTPS via VPS Traefik → Caddy on hubris → Authentik LXC 124). External devices authenticating to Netbird now hit this public path. Phase 6 (Authentik as Netbird IdP) is complete and live.
+
+### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
+The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
+
+Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
+
+Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
+
+The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/124-authentik.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
+
+**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):
+
+Existing peers — registered against the old combined image's embedded Dex IdP at `https://netbird.hubris.network/oauth2` — cache the OLD expected SSH-JWT issuer in the netbird daemon's in-memory state. After the migration, incoming `netbird ssh` connections were rejected with:
+
+```
+JWT authentication failed: validate token (
+  expected issuer=https://netbird.hubris.network/oauth2,
+            audiences=[netbird-dashboard netbird-cli],
+  actual    issuer=https://auth.hubris.network/application/o/netbird/,
+            audience=netbird-dashboard
+)
+```
+
+Neither `systemctl restart netbird` nor `netbird down && netbird up` clears the cache. Root cause: in `client/internal/engine_ssh.go`, `updateSSH()` bails out with `if e.sshServer != nil { return nil }` whenever the SSH server is already running, so mgmt-pushed JWT config updates are silently ignored. Only a full daemon-process tear-down lets the SSH server re-initialize with the new validator config:
+
+```
+sudo systemctl stop netbird
+sleep 3
+sudo systemctl start netbird
+```
+
+After that, `grep -iE "issuer|audience" /var/log/netbird/client.log | tail` shows the new Authentik issuer. Run this on every existing peer (PVE host + every LXC + every workstation) once after a future IdP swap.
+
+**Username gotcha (related):** `netbird ssh` defaults the remote username to the LOCAL one (e.g. `dtoro` from the operator's laptop). Hubris + the LXCs only have `root`, so the JWT is accepted but the session immediately fails with `user dtoro not found`. Always use the explicit `root@` prefix when invoking netbird-ssh manually:
+
+```
+netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted
+```
+
+The `homelab` CLI handles this automatically via the per-host `ssh.user` field in `inventory.yaml` (defaults to `root`; set explicitly only for workstations whose login user isn't `root`).
+
+Open follow-up: TURN-over-TLS on TCP 5349 (cert via certbot or extract Traefik's acme.json) for hostile-middlebox networks; plain TCP 3478 is sufficient for current usage.
+
 ### 2026-05-10 — ICE direct p2p restored (external STUN swap)
 All peers were `Connection type: Relayed` because the embedded STUN on the VPS was unreachable from outside (IONOS drops UDP 3478 inbound). Swapped to Google + Cloudflare STUN in `/opt/config.yaml`. After `netbird down/up`, peers now report `P2P` with srflx/host candidates. Backup of pre-change config at `/opt/config.yaml.bak-20260510-185050`. Fixes a real-world Nextcloud upload bottleneck (~366 kB/s through the relay → LAN-direct on same-network peers).
 

infrastructure/monitoring.md

@@ -1,75 +1,60 @@
-# Monitoring — `claudio-monitor`
+# Monitoring — Hermes health watchdog
 
-Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
+Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
+`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
 
-## Why
+## Current approach
 
-After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
+Two layers:
 
-## Components
+1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
+   the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
+   service reachability, and apt/docker drift across all hosts.
 
-- **Repo:** `dtoro/claudio-monitor`
-- **Checkout:** `/opt/claudio-monitor` on the PVE host
-- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
-- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
-- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
-- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
-- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
-- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
+2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
+   cron. Silent when healthy. When thresholds breach, sends an actionable alert
+   to Matrix (`@dtoro:avispero`) with options the user can reply to directly
+   (e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
+   the selected option via SSH.
 
-## Per-LXC data
+Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
+apt >10/>50 upgradable, services down.
 
-A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
+Home Assistant pulls PVE metrics independently via its Proxmox VE integration
+(unaffected by this change).
 
-## Matrix-side plugin
+## Legacy: claudio-monitor (deprecated 2026-06-04)
 
-`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
-- `/monitor status`
-- `/monitor history [N]`
-- `/monitor clear <key>`
+The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
+min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
+on LXC 123:9090. All components decommissioned:
 
-Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
+| Component | Fate |
+|-----------|------|
+| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
+| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
+| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
+| `claudio-monitor.timer` | Disabled on hubris |
+| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
+| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
 
-Events posted to claudio-bot `/notify`:
-```json
-{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
-```
-`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
-
-## Current thresholds & active alerts
-
-Active warnings at the time monitor was deployed (2026-04-21):
-- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
-- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
-
-## Testing
-
-- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
-- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
-- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
-
-## Not yet deployed (tracked in plan file)
-
-- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
-- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
-
-## Related repos
-
-- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
-- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
-- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
+For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
 
 ## Related pages
 - [Hubris host](../hosts/hubris.md)
-- [claudio-bot (123)](../containers/123-claudio-bot.md)
 - [HAOS VM (108)](../vms/108-haos.md)
 - [Backups (disabled)](backups.md)
-- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+- [Homelab context distribution](homelab-context.md)
 
 ## Changelog
 
+### 2026-06-04 — migrated to Hermes health watchdog
+claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
+On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
+Matrix alerts. LXC 123 destroyed, repos archived.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 
 ### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
-General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.
+General health monitor with per-LXC checks. MQTT/REST push paths ripped out.

infrastructure/network.md

@@ -0,0 +1,88 @@
+# Network
+
+Physical and logical network topology for the homelab.
+
+## Why
+
+The homelab runs on a dedicated internal subnet (`192.168.8.0/24`) isolated from the main household LAN (`192.168.178.0/24`). Isolation is enforced at Proxmox: LXC/VM traffic is bridged only on the internal `vmbr0` bridge; Proxmox routes packets out to Fritz!Box via `vmbr1`. The main LAN cannot reach homelab services directly without a Fritz!Box static route (which is configured to allow inbound).
+
+Fritz!OS 8.x does not support second IP networks on LAN ports, so Proxmox (`hubris`) acts as the subnet router rather than the Fritz!Box.
+
+## Hardware
+
+| Device | Role |
+|---|---|
+| Fritz!Box 7590 | Main router / ISP gateway (`192.168.178.1`) |
+| SODOLA 5-Port 2.5Gbit Managed | Homelab switch — flat L2, all ports native |
+| hubris (Proxmox) | Subnet router — routes between `192.168.8.0/24` and `192.168.178.0/24` |
+
+## Topology
+
+```
+ISP
+ └── Fritz!Box 7590 (192.168.178.1)
+      │   static route: 192.168.8.0/24 → 192.168.178.10
+      │
+      └── SODOLA 5-Port 2.5Gbit
+           ├── Port 1  uplink → Fritz!Box LAN
+           ├── Port 2  hubris eno1 → vmbr1 (192.168.178.10)
+           ├── Port 3  [device]
+           ├── Port 4  [device]
+           └── Port 5  spare
+
+hubris internal bridges:
+  vmbr1  192.168.178.10/24  eno1 (uplink, DHCP-reserved)  gateway 192.168.178.1
+  vmbr0  192.168.8.77/24    no physical port (internal)
+         192.168.8.1/24     alias — LXC default gateway
+            ├── all 16 LXCs
+            └── HAOS VM
+```
+
+## Subnets
+
+| Subnet | Gateway | Purpose |
+|---|---|---|
+| `192.168.178.0/24` | `192.168.178.1` | Household LAN — laptops, phones, Fritz!Box DHCP |
+| `192.168.8.0/24` | `192.168.8.1` (Proxmox `vmbr0` alias) | Homelab — all LXCs and VMs |
+
+## DHCP
+
+- **Household (`192.168.178.x`)**: Fritz!Box built-in DHCP. Proxmox `vmbr1` has a reservation: MAC `84:47:09:6b:e7:58` → `192.168.178.10`.
+- **Homelab (`192.168.8.x`)**: Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2`. Range `192.168.8.241–192.168.8.254`, gateway `192.168.8.1`, DNS `192.168.8.2`.
+
+Static IPs span `.101–.239` (all LXCs, VMs, and workstations). DHCP pool narrowed to `.241–.254` (2026-06-03) to avoid overlap and IP conflicts.
+
+## DNS
+
+Split-horizon DNS for `*.hubris.network` served by Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2:53`. See [dns.md](dns.md) for full detail.
+
+## Routing
+
+Proxmox has `net.ipv4.ip_forward=1` (already enabled by PVE). Packets from LXCs on `vmbr0` destined for the internet exit via `vmbr1` → Fritz!Box. Fritz!Box masquerades all outbound WAN traffic. Fritz!Box has a static route (`192.168.8.0/24 → 192.168.178.10`) so return traffic reaches the LXCs.
+
+No NAT on Proxmox — traffic flows without double-NAT.
+
+## Remote access
+
+- **NetBird mesh** — the remote-administration path. Authenticated via [Authentik on the VPS](../vps/).
+- Tailscale (the previous overlay) was **decommissioned 2026-06-21**. See [mesh.md](mesh.md).
+
+## Related
+
+- [DNS](dns.md) — split-horizon config and entry list
+- [Ingress](ingress.md) — public entry points via VPS traefik
+- [Mesh](mesh.md) — NetBird VPN overlay
+- [hosts/hubris.md](../hosts/hubris.md) — Proxmox host (vmbr0/vmbr1 config)
+- [CT 107 — dns](../containers/107-dns.md) — Technitium DNS + DHCP server
+
+## Changelog
+
+### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
+Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
+Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
+
+### 2026-06-02 — Executed migration; Proxmox as subnet router
+Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
+
+### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
+Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).

infrastructure/ssh-access.md

@@ -1,45 +1,199 @@
 # SSH access
 
-How to reach hubris and the VPS over SSH, and the dual-server gotcha.
+How to reach every host in the fleet from any workstation, with LAN as
+the primary path and Netbird as the automatic backup.
 
-## Hubris
+## Architecture
 
-Two SSH endpoints — easy to hit the wrong one.
+SSH access relies on three layers:
 
-| Server         | Listen                       | Auth                              | Notes |
-| -------------- | ---------------------------- | --------------------------------- | ----- |
-| OpenSSH        | `0.0.0.0:22`                 | `authorized_keys` at `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced; symlinked from `/root/.ssh/authorized_keys`) | Standard. |
-| Netbird SSH    | `100.122.38.109:22022`       | OIDC / browser auth — bypasses `authorized_keys` | If a client lands here it'll open a browser tab to authenticate, then sometimes hang. Force port 22 or use the LAN IP. |
+1. **Homelab inventory (`inventory.yaml`)** — the single source of truth
+   for every host's LAN IP, Netbird addresses, SSH user, and port.
+2. **Key distribution (`ssh/deploy-keys.sh`)** — deploys workstation SSH
+   public keys to hubris and every running LXC, so any key-authorized
+   workstation can log in anywhere.
+3. **Config generation (`homelab ssh-config --install`)** — generates
+   `~/.ssh/config.d/homelab` with short hostname aliases for every host,
+   using LAN IPs (routed via Netbird's `192.168.8.0/24` subnet route when
+   off-LAN) with Netbird FQDN fallbacks (`<name>-mesh`) for roaming
+   workstations.
 
-### Authorized root keys
-- `root@hubris` (self, RSA) — original.
-- `d.toro.v@pm.me` (ed25519) — user's iMac (`mac-mini.netbird.selfhosted`, LAN `192.168.8.174`), added 2026-04-22.
+### How it works
 
-### Notes
-- Password auth is enabled on hubris but the root password is **not** the one the user expects. Prefer key flows; don't try `ssh-copy-id` blind.
-- Off-LAN access from the iMac uses the LAN path. As of 2026-04-22 the iMac's Netbird tunnel to hubris was P2P healthy but no packets were captured on `wt0`; needs revisit if remote access becomes critical.
+- **From on-LAN:** `ssh gitea` resolves to `192.168.8.121` directly.
+- **From off-LAN (Netbird):** The same `192.168.8.121` works because
+  hubris routes the `192.168.8.0/24` subnet through Netbird.
+- **Roaming workstations:** `ssh mac-mini-mesh` or `ssh republic-laptop-mesh`
+  uses the Netbird FQDN as a fallback when the workstation is off its
+  home subnet.
 
-## VPS (`82.165.190.79` / `100.122.165.149`)
+The `homelab ssh <host>` CLI command also has built-in LAN probing:
+it tries a 1.5s TCP connect to the LAN IP, and if that fails, falls
+back to the Netbird FQDN.
 
-- **Mesh-only.** Public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`.
-- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
-- `ListenAddress` is still 0.0.0.0; gating is firewall-layer.
-- Authorized root keys: PVE (`root@hubris`), iMac (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer before disabling its access paths.
+## Key distribution
 
-See [VPS hardening](vps-hardening.md) for the firewall + fail2ban rules and recovery paths.
+Each workstation's SSH public key lives in the repo at:
+`ssh/authorized_keys/<hostname>.pub`
+
+To deploy or re-deploy all workstation keys to hubris + every running LXC:
+
+```bash
+# From hubris (or via homelab pct):
+sudo bash /opt/homelab-context/ssh/deploy-keys.sh
+
+# Or from any workstation:
+ssh root@192.168.8.77 "bash /opt/homelab-context/ssh/deploy-keys.sh"
+```
+
+This script:
+- Reads all `.pub` files from `ssh/authorized_keys/`
+- Adds any missing keys to `/etc/pve/priv/authorized_keys` on hubris
+- For each running LXC, appends keys to `/root/.ssh/authorized_keys`
+- Is idempotent — skips keys already present
+
+## Config generation
+
+To generate the SSH config on any workstation:
+
+```bash
+homelab ssh-config --install
+```
+
+This writes to `~/.ssh/config.d/homelab` and ensures
+`Include ~/.ssh/config.d/homelab` is present in `~/.ssh/config`.
+
+The config is regenerated automatically on every `homelab sync` (which
+kicks the 5-minute context sync timer).
+
+## Adding a new workstation
+
+When onboarding a new machine:
+
+1. Hostname must match an entry in `inventory.yaml`.
+2. If the workstation will be on the LAN, add its `lan_ip` to
+   `inventory.yaml` and push. This gives it a primary LAN entry in the
+   generated SSH config.
+3. Enable SSH Remote Login:
+   - **macOS:** `sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist`
+   - **Linux:** `sudo systemctl enable --now sshd`
+4. Generate an SSH keypair if one doesn't exist:
+   ```bash
+   ssh-keygen -t ed25519 -a 100
+   ```
+5. Publish the public key to the repo:
+   ```bash
+   cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/<hostname>.pub
+   cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add <hostname> pubkey' && git push
+   ```
+6. Deploy the key to all hosts:
+   ```bash
+   ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
+   ```
+7. Generate the local SSH config:
+   ```bash
+   homelab ssh-config --install
+   ```
+
+## Hosts
+
+### Hubris (PVE host)
+
+| Detail | Value |
+|--------|-------|
+| LAN IP | `192.168.8.77` |
+| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) |
+| Netbird SSH port | `22022` (mesh-only, OIDC auth) |
+| SSH user | `root` |
+| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) |
+
+Authorized root keys currently deployed:
+- `root@hubris` (self, RSA)
+- `d.toro.v@pm.me` (ed25519) — mac-mini
+
+### LXCs
+
+Every LXC at `192.168.8.x` accepts root SSH via authorized_keys. Keys
+are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
+
+| LXC | Name | LAN IP | Role |
+|-----|------|--------|------|
+| 101 | jellyfin | `192.168.8.206` | media-server |
+| 102 | nfs-export | `192.168.8.200` | storage-export |
+| 103 | paperless | `192.168.8.130` | document-archive |
+| 104 | gitea | `192.168.8.121` | git-server |
+| 105 | apps | `192.168.8.205` | docker-apps |
+| 106 | auth-outpost | `192.168.8.184` | authentik-outpost |
+| 107 | dns | `192.168.8.185` | dns-helper |
+| 114 | nextcloud | `192.168.8.224` | file-sync |
+| 118 | elementsynapse | `192.168.8.239` | matrix-server |
+| 119 | sophia | `192.168.8.157` | workshop |
+| 120 | mule-images | `192.168.8.136` | photo-management |
+| 121 | caddy | `192.168.8.175` | reverse-proxy |
+| 122 | arriman | `192.168.8.132` | arr-stack |
+| 126 | plato | `192.168.8.190` | app |
+
+### Workstations
+
+| Name | OS | LAN IP | Netbird FQDN | SSH user |
+|------|----|--------|--------------|----------|
+| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
+| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
+| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD |
+
+### VPS (external)
+
+| Detail | Value |
+|--------|-------|
+| Public IP | `82.165.190.79` |
+| Netbird | `100.122.165.149` (FQDN: `netbird-ionos.netbird.selfhosted`) |
+| SSH user | `root` |
+| Access | Mesh-only — public port 22 is blocked by nftables. Key-only auth. |
+
+## VPS
+
+Access is mesh-only. From a mesh-connected peer:
+
+```bash
+ssh root@100.122.165.149
+ssh root@netbird-ionos.netbird.selfhosted
+# or via homelab:
+homelab ssh netbird-vps
+```
+
+## Verification
+
+```bash
+# From any workstation after running homelab ssh-config --install:
+for name in hubris gitea apps sophia paperless caddy jellyfin nextcloud; do
+  ssh -o BatchMode=yes "$name" "hostname" && echo "$name OK"
+done
+```
 
 ## Related
-- [Hubris host](../hosts/hubris.md)
+
 - [Mesh migration](mesh.md)
 - [VPS hardening](vps-hardening.md)
+- [Agent enrollment](../operations/agent-enrollment.md)
+- [Homelab CLI](../bin/homelab)
 
 ## Changelog
 
+### 2026-06-02 — universal SSH reachability
+
+Replaced ad-hoc per-workstation SSH configs with inventory-generated
+configs (`ssh/gen-config.py`, `homelab ssh-config`). Added centralized
+key distribution (`ssh/deploy-keys.sh`, `ssh/authorized_keys/`). All
+LXCs now accept root SSH from any workstation whose pubkey is in the
+repo. mac-mini Remote Login enabled. Netbird subnet route
+(192.168.8.0/24 via hubris) provides off-LAN reachability for all LAN
+IPs.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 
 ### 2026-04-23 — VPS SSH hardened to mesh-only
-Public `:22` blocked at nftables. Key-only sshd. See [VPS hardening](vps-hardening.md).
+Public `:22` blocked at nftables. Key-only sshd.
 
 ### 2026-04-22 — iMac key authorized on hubris
 `d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.

infrastructure/vps-hardening.md

@@ -8,6 +8,9 @@ IONOS VPS that runs the Netbird control plane and the [public ingress traefik](i
 - **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)).
 - **Public:** `82.165.190.79` (`ens6`).
 - **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`.
+- **Docker stack** at `/opt/docker-compose.yml`: `traefik` (TLS/ACME) + `dashboard` + `mgmt` + `signal` + `relay` + `proxy` — netbird-mgmt 0.71.3 vanilla deploy since 2026-05-21 (see [mesh.md changelog](mesh.md#changelog)).
+- **Host services (outside docker):** `coturn` (TURN-TCP on :3478, long-term creds rendered into `/etc/turnserver.conf` by `homelab render-vps-configs` from sops-encrypted `secrets/turn-shared-secret.yaml`).
+- **Config rendering:** `/etc/turnserver.conf` + `/opt/management.json` are generated from templates in `vps/*.tmpl` on this repo by `homelab render-vps-configs`. Secret placeholders (`{{TURN_PASSWORD}}`, `{{AUTHENTIK_CLIENT_SECRET}}`) are substituted from sops-encrypted secrets decrypted on hubris and pushed over ssh. **Do not hand-edit those two files on the VPS** — the next render will overwrite them.
 
 ## SSH
 
@@ -20,8 +23,9 @@ IONOS VPS that runs the Netbird control plane and the [public ingress traefik](i
 Config at `/etc/nftables.conf`, service enabled.
 
 - Public iface `ens6`. Wireguard iface `wt0`.
-- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6. Everything else drops.
+- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6, **TCP 3478** (coturn TURN-TCP, added 2026-05-21). Everything else drops.
 - `wt0` fully accepted in INPUT. `lo` accepted.
+- **IONOS upstream firewall** also gates inbound traffic before it reaches `ens6`. Open ports today: TCP 80/443 (traefik), UDP 51820 (netbird-proxy), TCP 3478 (coturn, added 2026-05-21). UDP 3478 is dropped by IONOS upstream regardless of local nftables. See [mesh.md ICE/STUN](mesh.md) for the STUN/TURN port matrix.
 - **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below.
 - Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops.
 - Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too.
@@ -77,6 +81,9 @@ Ordered by preference:
 
 ## Changelog
 
+### 2026-05-21 — netbird stack migrated combined → vanilla; coturn added
+Replaced the `netbirdio/netbird-server` combined image with the canonical `mgmt + signal + relay + dashboard` containers (0.71.3). Added host-side `coturn` for external TURN, with nftables rule `iifname "ens6" tcp dport 3478 accept` and an IONOS upstream firewall exception. Authentik on LXC 124 now provides OIDC for the netbird dashboard. Full context in [mesh.md changelog](mesh.md#changelog).
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 

inventory.yaml

@@ -1,122 +1,105 @@
-# Homelab inventory — canonical structured topology
+# Homelab inventory — canonical structured topology.
 #
-# This file is the single source of truth for hosts, services, and mesh
-# addresses. Everything in hosts/*.yaml is generated from this file by
-# mcp/build_host_files.py — do not edit hosts/*.yaml by hand.
+# Single source of truth. hosts/*.yaml is generated from this file by
+# mcp/build_host_files.py; do NOT edit those by hand.
 #
 # Conventions:
-#   - hostname keys here MUST match the actual `hostname` of the machine
-#     (or, on macOS, `scutil --get LocalHostName` if set).
+#   - hostname keys MUST match the actual `hostname` of the machine (on
+#     macOS: `scutil --get LocalHostName` if set).
 #   - `os:` one of: linux, macos
-#   - `kind:` one of: proxmox-host, lxc, vm, workstation
-#   - `mesh:` lists addresses the host is reachable at. Both `netbird` and
-#     `tailscale` are accepted during the in-flight migration
-#     (see infrastructure/mesh.md). Prefer netbird FQDNs over raw IPs.
-#   - `age_pubkey:` is provisioned by the secrets-issuance service on first
-#     bootstrap and committed back here. New entries leave it empty.
+#   - `kind:` one of: proxmox-host, lxc, vm, workstation, external
+#     ("external" is reserved for hosts the homelab CLI manages via ssh but
+#      that aren't homelab clients themselves — e.g. the IONOS netbird VPS
+#      with no /etc/age/key.txt and no /opt/homelab-context clone.)
+#   - `mesh:` lists addresses the host is reachable at via `netbird`.
+#     (Tailscale was decommissioned 2026-06-21 — see infrastructure/mesh.md.)
+#     Prefer netbird FQDNs over raw IPs.
+#   - `age_pubkey:` provisioned by secrets-issuance on first bootstrap and
+#     committed back via `homelab client add --finalize-pubkey <key>`.
 #   - When a service moves hosts, update only the `services:` section here;
-#     never duplicate the address elsewhere.
-
-# ---------------------------------------------------------------------------
-# Static config (mesh + service endpoints)
-# ---------------------------------------------------------------------------
+#     never duplicate addresses elsewhere.
+#   - `ssh.user:` per-host login user. Default is `root` if omitted (matches
+#     every LXC + the PVE host). Set explicitly for workstations whose login
+#     user differs from `root`. Used by the `homelab` CLI to build
+#     `user@host` and to inform anyone running raw `netbird ssh` (which
+#     defaults to the LOCAL username — the gotcha that creates "user not
+#     found" errors when ssh'ing INTO machines that only have `root`).
+#
+# `homelab client add/remove` does surgical line-edits — comments survive.
+# Avoid round-tripping the file through yaml.safe_dump (it strips comments).
 
 mesh:
-  primary: netbird           # preferred for new clients
-  accepted: [netbird, tailscale]
+  primary: netbird
+  accepted:
+  - netbird
   netbird_subnet: 100.122.0.0/16
   netbird_domain: netbird.selfhosted
-
-# Top-level service registry. Each entry resolves to a host above plus a URL.
 services:
   proxmox_ui:
     url: https://proxmox.hubris.network
     backend: hubris
     port: 8006
-
   gitea:
     url: https://git.hubris.network
     backend: gitea
     backend_url: http://192.168.8.121:3000
-
   caddy:
     backend: caddy
     role: reverse-proxy
     note: terminates all *.hubris.network
-
   authentik:
     url: https://auth.hubris.network
     backend: authentik
-
-  dnsmasq:
-    backend: authentik
-    note: split-horizon DNS, /etc/dnsmasq.d/hubris-split.conf
-
+    dns:
+    backend: dns
+    note: Technitium DNS, split-horizon zone
   jellyfin:
     url: https://media.hubris.network
     backend: jellyfin
-
   nextcloud:
     url: https://cloud.hubris.network
     backend: nextcloud
-
   paperless:
     url: https://paperless.hubris.network
     backend: paperless
-
   matrix:
     url: https://matrix.hubris.network
     backend: elementsynapse
-
   photos:
     url: https://photos.hubris.network
     backend: mule-images
-
-  photos_new:
-    url: https://photos-new.hubris.network
-    backend: mule-photos-new
-
   plato:
     url: https://plato.hubris.network
     backend: plato
-
   arr_stack:
     backend: arriman
     note: jellyseerr / qbit / sab on docker compose
-
   artifacto:
     backend: apps
     url: https://artifacto.hubris.network
-
   zimaos:
     url: https://zimaos.hubris.network
     backend: zimaos
-
   haos:
     backend: haos
-
-  # New services introduced by this plan.
   homelab_mcp:
     backend: apps
     port: 9810
-    netbird_host: apps.netbird.selfhosted
-    endpoint: http://apps.netbird.selfhosted:9810/sse
-    note: MCP server. Read-only context + management. Mesh-only.
-
+    systemd_unit: homelab-mcp
+    public_host: mcp.hubris.network
+    endpoint: https://mcp.hubris.network/mcp
+    note: MCP server. Read-only context + management. Reachable on the LAN via Caddy
+      and from off-LAN via Netbird (192.168.8.0/24 is a network resource routed through
+      hubris).
   secrets_issuance:
     backend: apps
     port: 9820
-    netbird_host: apps.netbird.selfhosted
-    endpoint: http://apps.netbird.selfhosted:9820/issue
-    note: Issues per-client age private keys. Mesh-only.
-
-# ---------------------------------------------------------------------------
-# Hosts (Proxmox host + LXCs + VMs + workstations)
-# ---------------------------------------------------------------------------
-
+    systemd_unit: secrets-issuance
+    public_host: secrets.hubris.network
+    endpoint: https://secrets.hubris.network/issue
+    note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN
+      subnets in MESH_SUBNETS).
 hosts:
-
-  # ----- The hypervisor -----
   hubris:
     kind: proxmox-host
     os: linux
@@ -127,13 +110,12 @@ hosts:
         ip: 100.122.38.109
         fqdn: proxmox-server.netbird.selfhosted
     ssh:
-      port: 22                 # OpenSSH
-      netbird_port: 22022      # see infrastructure/ssh-access.md
+      port: 22
+      netbird_port: 22022
+      user: root
     mounts:
-      - /mnt/library           # shared lvmthin
-    age_pubkey: ""             # filled by issuance on first bootstrap
-
-  # ----- LXC containers (all on hubris) -----
+    - /mnt/library
+    age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
   jellyfin:
     kind: lxc
     pve_id: 101
@@ -142,10 +124,9 @@ hosts:
     role: media-server
     lan_ip: 192.168.8.206
     public_host: media.hubris.network
-    mesh: {netbird: {fqdn: jellyfin.netbird.selfhosted}}
-    mounts: [/mnt/library]
-    age_pubkey: ""
-
+    mounts:
+    - /mnt/library
+    age_pubkey: ''
   nfs-export:
     kind: lxc
     pve_id: 102
@@ -153,9 +134,6 @@ hosts:
     os: linux
     role: storage-export
     lan_ip: 192.168.8.200
-    mesh: {netbird: {fqdn: nfs-export.netbird.selfhosted}}
-    age_pubkey: ""
-
   paperless:
     kind: lxc
     pve_id: 103
@@ -164,10 +142,9 @@ hosts:
     role: document-archive
     lan_ip: 192.168.8.130
     public_host: paperless.hubris.network
-    mesh: {netbird: {fqdn: paperless.netbird.selfhosted}}
-    mounts: [/mnt/library]
-    age_pubkey: ""
-
+    mounts:
+    - /mnt/library
+    age_pubkey: ''
   gitea:
     kind: lxc
     pve_id: 104
@@ -177,12 +154,11 @@ hosts:
     lan_ip: 192.168.8.121
     public_host: git.hubris.network
     backend_port: 3000
-    mesh: {netbird: {fqdn: gitea.netbird.selfhosted}}
-    mounts: [/mnt/library]
+    mounts:
+    - /mnt/library
     notes:
-      - "Bare repos live at /mnt/library/repos/dtoro/*.git"
-    age_pubkey: ""
-
+    - Bare repos live at /mnt/library/repos/dtoro/*.git
+    age_pubkey: ''
   apps:
     kind: lxc
     pve_id: 105
@@ -190,16 +166,16 @@ hosts:
     os: linux
     role: docker-apps
     lan_ip: 192.168.8.205
-    public_hosts: [artifacto.hubris.network]
-    mesh: {netbird: {fqdn: apps.netbird.selfhosted}}
-    mounts: [/mnt/library]
+    public_hosts:
+    - artifacto.hubris.network
+    mounts:
+    - /mnt/library
     runs:
     - artifacto
     - plantuml
-      - homelab-mcp           # introduced by this plan
-      - secrets-issuance      # introduced by this plan
-    age_pubkey: ""
-
+    - homelab-mcp
+    - secrets-issuance
+    age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
   nextcloud:
     kind: lxc
     pve_id: 114
@@ -208,10 +184,9 @@ hosts:
     role: file-sync
     lan_ip: 192.168.8.224
     public_host: cloud.hubris.network
-    mesh: {netbird: {fqdn: nextcloud.netbird.selfhosted}}
-    mounts: [/mnt/library]
-    age_pubkey: ""
-
+    mounts:
+    - /mnt/library
+    age_pubkey: ''
   elementsynapse:
     kind: lxc
     pve_id: 118
@@ -220,20 +195,16 @@ hosts:
     role: matrix-server
     lan_ip: 192.168.8.239
     public_host: matrix.hubris.network
-    mesh: {netbird: {fqdn: elementsynapse.netbird.selfhosted}}
-    age_pubkey: ""
-
   sophia:
     kind: lxc
     pve_id: 119
     host: hubris
     os: linux
     role: workshop
-    lan_ip: 192.168.8.157
-    mesh: {netbird: {fqdn: sophia.netbird.selfhosted}}
-    mounts: [/mnt/library]
-    age_pubkey: ""
-
+    lan_ip: 192.168.8.109
+    mounts:
+    - /mnt/library
+    age_pubkey: ''
   mule-images:
     kind: lxc
     pve_id: 120
@@ -242,10 +213,9 @@ hosts:
     role: photo-management
     lan_ip: 192.168.8.136
     public_host: photos.hubris.network
-    mesh: {netbird: {fqdn: mule-images.netbird.selfhosted}}
-    mounts: [/mnt/library]
-    age_pubkey: ""
-
+    mounts:
+    - /mnt/library
+    age_pubkey: ''
   caddy:
     kind: lxc
     pve_id: 121
@@ -254,12 +224,11 @@ hosts:
     role: reverse-proxy
     lan_ip: 192.168.8.175
     notes:
-      - "Terminates all *.hubris.network"
-      - "/etc/caddy is a git checkout of dtoro/caddy-conf"
-    peers: [authentik, gitea]
-    mesh: {netbird: {fqdn: caddy.netbird.selfhosted}}
-    age_pubkey: ""
-
+    - Terminates all *.hubris.network
+    - /etc/caddy is a git checkout of dtoro/caddy-conf
+    peers:
+    - authentik
+    - gitea
   arriman:
     kind: lxc
     pve_id: 122
@@ -267,36 +236,14 @@ hosts:
     os: linux
     role: arr-stack
     lan_ip: 192.168.8.132
-    public_hosts: [jellyseerr.hubris.network, qbit.hubris.network, sab.hubris.network]
-    mesh: {netbird: {fqdn: arriman.netbird.selfhosted}}
-    mounts: [/mnt/library]
-    age_pubkey: ""
-
-  claudio-bot:
-    kind: lxc
-    pve_id: 123
-    host: hubris
-    os: linux
-    role: matrix-agent
-    lan_ip: 192.168.8.230
-    mesh: {netbird: {fqdn: claudio-bot.netbird.selfhosted}}
-    notes:
-      - "Reads /opt/homelab-context/ on startup"
-    age_pubkey: ""
-
-  authentik:
-    kind: lxc
-    pve_id: 124
-    host: hubris
-    os: linux
-    role: idp
-    lan_ip: 192.168.8.180
-    public_host: auth.hubris.network
-    notes:
-      - "Also hosts split-horizon dnsmasq: /etc/dnsmasq.d/hubris-split.conf"
-    mesh: {netbird: {fqdn: authentik.netbird.selfhosted}}
-    age_pubkey: ""
-
+    public_hosts:
+    - jellyseerr.hubris.network
+    - qbit.hubris.network
+    - sab.hubris.network
+    mounts:
+    - /mnt/library
+    age_pubkey: ''
+  # 123 (claudio-bot) — destroyed 2026-06-04, replaced by Hermes Agent
   plato:
     kind: lxc
     pve_id: 126
@@ -305,61 +252,74 @@ hosts:
     role: app
     lan_ip: 192.168.8.190
     public_host: plato.hubris.network
-    mesh: {netbird: {fqdn: plato.netbird.selfhosted}}
-    mounts: [/mnt/library/documents/plato]
-    age_pubkey: ""
-
-  mule-photos-new:
-    kind: lxc
-    pve_id: 127
-    host: hubris
-    os: linux
-    role: photo-management
-    lan_ip: 192.168.8.181
-    public_host: photos-new.hubris.network
-    mesh: {netbird: {fqdn: mule-photos-new.netbird.selfhosted}}
-    age_pubkey: ""
-
-  # ----- VMs (on hubris) -----
+    mounts:
+    - /mnt/library/documents/plato
+    age_pubkey: ''
   zimaos:
     kind: vm
     pve_id: 100
     host: hubris
     os: linux
     role: nas-frontend-eval
+    lan_ip: 192.168.8.195
     public_host: zimaos.hubris.network
-    mesh: {netbird: {fqdn: zimaos.netbird.selfhosted}}
-    age_pubkey: ""
-
   haos:
     kind: vm
     pve_id: 108
     host: hubris
     os: linux
     role: home-automation
-    mesh: {netbird: {fqdn: haos.netbird.selfhosted}}
-    age_pubkey: ""
-
-  # ----- Workstations (roaming, primary agent clients) -----
+    lan_ip: 192.168.8.101
   republic-laptop:
     kind: workstation
     os: linux
     role: primary-dev
-    mesh: {netbird: {fqdn: republic-laptop.netbird.selfhosted}}
-    age_pubkey: ""
-
+    mesh:
+      netbird:
+        fqdn: republic-laptop.netbird.selfhosted
+    ssh:
+      user: dtoro
   mac-mini:
     kind: workstation
     os: macos
     role: dev
-    mesh: {netbird: {fqdn: mac-mini.netbird.selfhosted}}
+    lan_ip: 192.168.178.182
+    mesh:
+      netbird:
+        fqdn: mac-mini-234-17.netbird.selfhosted
+    ssh:
+      user: dtoro
     notes:
-      - "Only macOS in the fleet. Bootstrap uses launchd."
-    age_pubkey: ""
-
+    - Only macOS in the fleet. Bootstrap uses launchd.
+    age_pubkey: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
   ludo-mini:
     kind: workstation
     os: linux
     role: dev
-    mesh: {netbird: {fqdn: ludo-mini.netbird.selfhosted}}
-    age_pubkey: ""
+    lan_ip: 192.168.178.181
+    mesh:
+      netbird:
+        fqdn: ludo-mini.netbird.selfhosted
+    ssh:
+      user: dtoro
+  netbird-vps:
+    kind: external
+    os: linux
+    role: netbird-mgmt
+    mesh:
+      netbird:
+        ip: 100.122.165.149
+        fqdn: netbird-ionos.netbird.selfhosted
+    ssh:
+      user: root
+    notes:
+    - Public IONOS VPS — hosts the vanilla netbird mgmt+signal+relay+dashboard
+      stack + host coturn (see infrastructure/vps-hardening.md +
+      infrastructure/mesh.md changelog 2026-05-21).
+    - NOT a homelab client. No /etc/age/key.txt, no /opt/homelab-context
+      clone. Managed via ssh from hubris; sshd is locked to hubris's pubkey.
+    - Public IPv4 82.165.190.79. Auto-patching via unattended-upgrades.
+    - Configs rendered by `homelab render-vps-configs` from
+      vps/turnserver.conf.tmpl + vps/management.json.tmpl, with secrets
+      decrypted from secrets/turn-shared-secret.yaml +
+      secrets/netbird-authentik-oidc.yaml on hubris.

investigations/2026-05-31-authentik-vps-migration.md

@@ -0,0 +1,109 @@
+# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
+
+## Summary
+
+The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
+
+Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
+
+The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
+
+## Timeline
+
+### Trigger
+`netbird status` on clients showed `Management: Disconnected`; `netbird.hubris.network` unreachable. VPS `docker logs netbird-mgmt`:
+```
+Error: failed reading provided config file: /etc/netbird/management.json:
+OIDC configuration request returned status 504 with response: Gateway Timeout
+```
+The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
+
+### Fix — Phase A (restore the mesh)
+1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
+2. Removed the `auth-authentik` → `192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
+3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
+4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
+
+### Fix — Phase B (full data migration)
+Once the mesh was back, the homelab DB was reachable:
+1. `ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik'` → gzip.
+2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
+3. **Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
+4. Restarted Authentik. All users / apps / passwords / groups present.
+
+Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
+
+### Fix — DNS cutover (the long tail)
+The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
+
+- Updated dnsmasq: `address=/auth.hubris.network/82.165.190.79`, `systemctl restart dnsmasq`.
+- `netbird down && netbird up` does **not** reliably flush the NetBird resolver cache. Immediate per-client fixes: `/etc/hosts` override (mac), `sudo resolvectl flush-caches` (Linux), `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` (mac).
+
+## Issues hit during the migration (and fixes)
+
+1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
+2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
+3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
+4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
+5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
+
+## Architecture delta
+
+| | Before | After |
+|---|---|---|
+| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
+| Version | `2026.2.2` | `2026.5.2` |
+| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
+| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
+| Reachability | mesh-only (split-horizon) | **public** (auth UI now exposed on the open internet) |
+| Redis | none (docs said 2026.x dropped it) | present (this deploy uses Redis) |
+| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
+
+> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
+
+## Lessons learned
+
+1. **Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
+2. **`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
+3. **Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
+4. **Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
+5. **Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
+6. **Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
+7. **Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
+8. **WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
+
+## Forward-auth outpost — RESOLVED 2026-06-01
+
+Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
+
+Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
+
+### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
+
+The default authentication flow was skipping the identification stage (showing only a password field with `pending_user: ""`), making login impossible. Root cause: a **"Trust me" reputation policy** (`threshold: -3`, `negate: true`) attached to the identification stage binding (#10). The `negate: true` + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's `ReevaluateMarker` skipped the stage entirely.
+
+**Fix:** Deleted from Postgres:
+```sql
+DELETE FROM authentik_policies_policybinding WHERE policy_id = '<uuid>';
+DELETE FROM authentik_policies_reputation_reputationpolicy WHERE policy_ptr_id = '<uuid>';
+DELETE FROM authentik_policies_policy WHERE policy_uuid = '<uuid>';
+```
+The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).
+
+**Note:** This is different from the 2026-05-31 issue where NetBird's `login_hint` OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.
+
+## Outstanding
+
+- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
+- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config` `AuthorizedKeysFile` suspect). DNS not yet flushed there either.
+- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
+- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
+- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
+- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
+- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
+
+## Related
+- [124 — authentik](../containers/124-authentik.md)
+- [DNS split-horizon](../infrastructure/dns.md)
+- [Public ingress (VPS traefik)](../infrastructure/ingress.md)
+- [Mesh migration](../infrastructure/mesh.md)

investigations/2026-06-01-mac-mini-onboarding.md

@@ -0,0 +1,191 @@
+# mac-mini onboarding — post-mortem & lessons learned
+
+Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris
+homelab context system with the `--with-hermes` profile. What follows is a
+chronological recap of every hitch, the fix, and the systemic improvements
+needed so the next workstation takes 5 min instead of an hour.
+
+## Session log
+
+### Step 1 — clone + symlink
+- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`.
+- Created `/opt/homelab-context` → `/Users/dtoro/Homelab-Docs` symlink.
+- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd
+  already cloned by hand. The bootstrap's `clone exists; pulling` path handled
+  it gracefully.
+
+### Step 2 — hostname mismatch
+- `scutil --get LocalHostName` → `Davids-Mac-mini`
+- `hostname -s` → `Mac`
+- Inventory file: `hosts/mac-mini.yaml`
+- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini`
+- **Lesson:** The bootstrap and `homelab whoami` use different hostname
+  resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS),
+  but the `homelab` CLI binary uses `hostname -s`. Both need to match the
+  inventory key. On a fresh macOS machine, neither does.
+
+### Step 3 — bootstrap dependencies
+- pyyaml was missing → `pip install pyyaml`
+- age and sops were missing → `brew install age sops`
+- Netbird was already installed and connected ✓
+- **Lesson:** The bootstrap preflight handles these, but only if running
+  `bootstrap.sh` from the start. Since we ran it after manual setup, some
+  steps (netbird install) were correctly skipped as already-present.
+
+### Step 4 — full bootstrap with `--with-mcp --with-hermes`
+- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes`
+- Age key issued ✓
+- Launchd sync timer installed ✓
+- Goose binary installed ✓
+- Hermes CLI linked ✓
+- MCP config merged ✓
+- `refresh-creds` skipped (not yet a recipient) ⚠️
+- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS
+  group-naming quirk)
+
+### Step 5 — finalize from hubris
+- Ran `homelab client add mac-mini --finalize-pubkey <age...> --with-hermes` on
+  hubris
+- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was
+  stale, bootstrap had already pushed from mac-mini
+- **Fix:** `git pull --rebase && git push` on hubris
+- **Lesson:** bootstrap pushes remote changes before hubris can finalize,
+  creating a race. The `homelab client add --finalize-pubkey` command should
+  pull before committing/pushing.
+
+### Step 6 — sops couldn't find the age key
+- `homelab secret hello` failed because sops looks in
+  `/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt`
+- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`,
+  but this requires passwordless sudo and the correct env var passthrough
+- **Fix:**
+  1. Added NOPASSWD sudo rules
+  2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly
+- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in
+  agent-enrollment troubleshooting. New clients can't assume `homelab secret`
+  works out of the gate — the sudo re-exec chain has permission pitfall.
+
+### Step 7 — OpenRouter key was a placeholder
+- `secrets/openrouter-api-key.yaml` contained
+  `api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY`
+- User ran `sops` on hubris, but got the same error (age key not found on
+  hubris either — `/root/.config/sops/age/keys.txt` didn't exist)
+- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris.
+  Later: the user pasted the real key, but the sops file showed
+  `sk-or-...5c55` — the literal content was truncated with ellipsis.
+
+### Step 8 — editor loaded the wrong data
+- Neovim on the system is configured with `clipboard+=unnamedplus`, which
+  points `*` and `+` registers to the macOS clipboard manager rather than
+  X11. When editing SOPS files, this caused the **system clipboard** to be
+  pasted instead of the actual ciphertext.
+- This wasn't diagnosed during the session — the sops file would load empty
+  or show the wrong content because the editor's idea of "paste" was
+  disconnected from what sops expected.
+- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't
+  hijack OS clipboards:
+  ```bash
+  EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
+  ```
+- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`:
+  macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing
+  because the paste register reads from the system clipboard instead of the
+  sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops
+  interactively. Alternatively, override the clipboard option with
+  `EDITOR='nvim -c "set clipboard="'`.
+- Also useful for the troubleshooting table in `agent-enrollment.md` under a
+  new row: "sops file loads empty / wrong content on macOS"
+
+### Step 9 — model doesn't support tool use
+- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter
+- Error: `No endpoints found that support tool use`
+- **Fix:** Switched to `deepseek/deepseek-v4-flash` in
+  `~/.config/goose/config.yaml`
+- Also updated `operations/hermes-agent.md` with the correct model
+- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was
+  never validated on OpenRouter for tool-use capability. Need to either:
+  (a) Pin a model known to work (`deepseek/deepseek-v4-flash`)
+  (b) Or have the bootstrap probe OpenRouter at install time
+
+### Step 10 — MCP homelab extension returns 404
+- Goose's homelab extension configured as `streamable_http` at
+  `https://mcp.hubris.network/mcp` returns HTTP 404
+- The actual MCP server runs on SSE (`/sse`), not streamable_http
+- **Known follow-up** from `hermes-agent.md` — server needs migration
+- **Workaround:** The agent works fine via the built-in `developer` extension
+  (shell + file) and the `homelab` CLI
+
+### Step 11 — git credential dance
+Several layers of issues with git authentication:
+
+1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro
+   couldn't access the credential file, and git's `store` helper couldn't
+   create its `.lock` file.
+   - **Fix 1a:** `sudo chmod 755 /etc/homelab-context`
+   - **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context`
+
+2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in
+   as a default credential helper. Even after configuring the `store` helper,
+   git calls osxkeychain after every successful `get`, which prompts for
+   macOS keychain access.
+   - `credential.helper = ""` does NOT disable the compiled-in default
+   - The `store` helper's `.lock` file in `/etc/homelab-context/` also failed
+     silently, causing fallback behaviour
+   - **Fix:** Embed the credential in the remote URL directly:
+     `https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git`
+   - This bypasses all credential helpers entirely
+
+3. **refresh-creds:** The credential is now managed by the remote URL.
+   If the PAT is rotated, `homelab refresh-creds` won't update the URL.
+   - **Workaround:** Run `git remote set-url origin` after `refresh-creds`
+
+### Step 12 — `homelab` CLI hostname detection
+- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
+  `hostname -s` (returning `Mac`) while the bootstrap uses
+  `scutil --get LocalHostName` (which we fixed to `mac-mini`)
+- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
+- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
+  LocalHostName` like the bootstrap does, or at least try both and pick the
+  one that matches an inventory key. This is a CLI bug.
+
+## Improvement backlog
+
+### High priority
+
+| # | Issue | Fix |
+|---|-------|-----|
+| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
+| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
+| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
+| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
+| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |
+
+### Medium priority
+
+| # | Issue | Fix |
+|---|-------|-----|
+| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
+| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
+| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
+| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |
+
+### Low priority
+
+| # | Issue | Fix |
+|---|-------|-----|
+| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
+| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
+| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |
+
+## Changelog
+
+### 2026-06-01 — initial post-mortem
+Captured the full onboarding of mac-mini into the hubris homelab context
+system, including the hostname mismatch, sudo/sops credential chain, model
+selection, git credential gymnastics, and editor clipboard interference.
+
+### 2026-06-02 — MCP server migrated to streamable-http
+Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
+not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
+and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
+re-deployed on apps LXC (105).

investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md

@@ -0,0 +1,84 @@
+# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
+
+## Summary
+
+[`ludo-mini`](../hosts/ludo-mini.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
+
+## Timeline
+
+### Pre-existing state
+- ludo-mini: wired Ethernet (eno1, 2.5 Gbps), 192.168.178.181/24
+- mac-mini: WiFi (en1, 802.11ac, 234 Mbps Tx rate, 1 stream, -60 dBm RSSI), 192.168.178.99/24
+- Both on the same subnet via a consumer router at 192.168.178.1
+- Sunshine configured: bitrate=80 Mbps, encoder=amf (AMD GPU), fec_percentage=5, hevc_mode=2
+- Moonlight configured: bitrate=57 Mbps, fps=60, localaddr=192.168.178.181 (correct)
+
+### 2026-06-03 — investigation
+**Network tests (iperf3 between mac-mini ↔ ludo-mini):**
+- TCP throughput: 49–86 Mbps, average ~65 Mbps — highly variable
+- UDP throughput: 1-second complete dropout during a 5-second test; 0% loss otherwise
+- Ping: 4.7 ms avg, but **28 ms stddev**, max **138 ms** — WiFi-level jitter
+- Netbird P2P connection also showed 7 ms latency over the tunnel (unnecessary given same-subnet direct connection)
+
+**Additional findings:**
+- mac-mini has a built-in Ethernet port (en0) but it is inactive — no cable connected
+- mac-mini also has Thunderbolt Ethernet adapters (en5, en6, en7) all inactive
+- ludo-mini's en0 shows `speed 2500` (2.5 GbE)
+- Netbird routes `192.168.8.0/24` via `utun100`, but 192.168.178.0/24 traffic stays on `en1`
+
+## Root cause
+
+mac-mini is on WiFi. WiFi introduces:
+1. **Throughput variance** — 49–86 Mbps TCP fluctuation
+2. **Dropouts** — 1-second UDP blackouts from radio retransmissions
+3. **Jitter** — 28 ms stddev with spikes to 138 ms
+
+Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
+
+## Mitigations applied
+
+### mac-mini — Moonlight (via `defaults write`)
+
+| Setting | Before | After | Rationale |
+||---------|--------|-------|-----------|
+|| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
+|| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
+|| fps | 30 | 60 | Restored to game-appropriate value |
+
+### mac-mini — Moonlight (current, wired — 2026-06-04)
+
+| Setting | WiFi value | Wired value | Rationale |
+||---------|-----------|-------------|-----------|
+|| bitrate | 60 Mbps | 80 Mbps | Wired 1 GbE can sustain comfortably |
+|| framepacing | on (1) | off (0) | Wired latency is consistent, no smoothing needed |
+|| fps | 60 | 60 | Unchanged |
+
+### ludo-mini — Sunshine (via SSH)
+
+| Setting | WiFi value | Wired value | Rationale |
+||---------|-----------|-------------|-----------|
+|| bitrate | 40 Mbps | 80 Mbps | Wired can handle full bandwidth |
+|| fec_percentage | 2 | 5 | Restored to default — wired has no dropout concern |
+|| packetsize | 1024 | 1316 | Restored to default (Ethernet MTU) |
+
+Sunshine restarted after changes; confirmed active.
+
+## Resolution
+
+### 2026-06-04 — mac-mini wired to LAN
+
+mac-mini plugged into Ethernet (en0, 192.168.178.182). This definitively resolves the WiFi jitter that caused streaming instability. All WiFi-era conservative settings reverted to appropriate wired values on both ends.
+
+## Open questions
+
+~~1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.~~ **RESOLVED** — mac-mini now wired via en0.
+2. **WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
+3. **Netbird route** — `192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
+
+## Changelog
+
+### 2026-06-04 — mac-mini wired to LAN, configs reverted from WiFi mitigations
+- mac-mini connected via en0 (wired Ethernet, 192.168.178.182), WiFi mitigations no longer needed
+- Moonlight: bitrate 60→80 Mbps, framepacing disabled
+- Sunshine: bitrate 40→80 Mbps, fec_percentage 2→5, packetsize restored to default (1316)
+- First root cause in investigation (Open questions #1) resolved

investigations/2026-06-06-authentik-session-lifetime.md

@@ -0,0 +1,101 @@
+# 2026-06-06 — Frequent Authentik login prompts fixed (session duration)
+
+## Summary
+
+User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
+
+## Timeline
+
+- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
+- **~23:10** — Investigation begins: check Authentik OAuth2 provider settings, proxy session table, Django session config
+- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
+- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)` → `_session_expire_at_browser_close=True`
+- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage` → `session_duration='days=30'`
+- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
+- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
+- **~23:45** — Verified both changes took effect
+
+## Root cause
+
+Two independent but compounding issues:
+
+1. **`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
+
+2. **`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
+
+3. **`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
+
+### Why forward-auth worked but OAuth2 didn't
+
+- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
+- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
+
+## Changes applied
+
+### 1. Database — `authentik_stages_user_login_userloginstage`
+
+```sql
+UPDATE authentik_stages_user_login_userloginstage
+SET session_duration = 'days=30'
+WHERE stage_ptr_id = (
+    SELECT stage_uuid FROM authentik_flows_stage 
+    WHERE name = 'default-authentication-login'
+);
+```
+
+This causes the login stage to call `session.set_expiry(timedelta(days=30))`, which:
+- Sets `_session_expire_at_browser_close=False` for that session
+- Sets the session cookie `Max-Age` to 30 days (persistent cookie)
+- Sets the server-side session expiry to 30 days
+
+### 2. Environment — `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE`
+
+Added to `/opt/authentik.env`:
+
+```
+AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30
+```
+
+This maps to config key `sessions.unauthenticated_age`, which Authentik's `settings.py` reads via:
+
+```python
+SESSION_COOKIE_AGE = timedelta_from_string(
+    CONFIG.get("sessions.unauthenticated_age", "days=1")
+).total_seconds()
+```
+
+Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
+
+## Verification
+
+```python
+# Inside authentik-server container:
+import os
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
+import django
+django.setup()
+from django.conf import settings
+print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)  # → 2592000.0
+print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)  # → True
+
+from authentik.stages.user_login.models import UserLoginStage
+stage = UserLoginStage.objects.filter(name="default-authentication-login").first()
+print("session_duration:", stage.session_duration)  # → "days=30"
+```
+
+## Open questions
+
+- `SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
+- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
+
+## Related
+
+- [Container 106 — auth-outpost](../containers/106-auth-outpost.md)
+- [Authentik VPS migration](2026-05-31-authentik-vps-migration.md)
+- [Ingress (VPS Traefik)](../infrastructure/ingress.md)
+- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
+
+## Changelog
+
+### 2026-06-06 — created
+Documented the session lifetime investigation, root cause, and applied fixes.

investigations/2026-06-06-caddyfile-truncation.md

@@ -0,0 +1,61 @@
+# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
+
+**Date:** 2026-06-06
+**Status:** resolved
+**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
+
+## Symptom
+
+All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
+
+## Root cause
+
+The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
+
+### Timeline
+
+| Time (UTC+2) | Event |
+|---|---|
+| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
+| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
+| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
+| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
+| Jun 06 22:34 | Investigation began |
+| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
+
+### Evidence
+
+- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
+- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
+- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
+- `git stash list` shows one auto-stash entry
+- `origin/master` at `1b977aa`: 260 lines, all site blocks present
+
+### Secondary root cause found during investigation
+
+**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
+
+This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
+
+## Fix applied
+
+1. **Caddyfile** → `git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
+2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
+
+## Permanent safeguards (all deployed)
+
+| Safeguard | Location | What it does |
+|---|---|---|
+| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
+| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
+| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
+| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
+| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
+
+## Related
+
+- [DHCP drift investigation (previous incident)](2026-06-05-homelab-dhcp-drift.md)
+- [Caddy (121)](../containers/121-caddy.md)
+- [elementsynapse (118)](../containers/118-elementsynapse.md)
+- [dns-sync script](../scripts/dns-sync.py)
+- [check-caddy-backends script](../scripts/check-caddy-backends.sh)

investigations/index.md

@@ -6,6 +6,9 @@ Time-stamped incident reports and experiments. One entry per incident; the entry
 
 | Date         | Title                                                              | Status        |
 | ------------ | ------------------------------------------------------------------ | ------------- |
+|| 2026-06-06   | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
+|| 2026-05-31   | [Authentik migrated from LXC 124 to the VPS](2026-05-31-authentik-vps-migration.md) | Resolved; LXC 124 Authentik in ~2-week dual-run before decommission |
+| 2026-06-03   | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
 | 2026-04-21   | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) |
 
 ## Conventions

mcp/deploy/homelab-mcp.service

@@ -8,7 +8,12 @@ Type=simple
 WorkingDirectory=/opt/homelab-context
 Environment=HOMELAB_CONTEXT_DIR=/opt/homelab-context
 Environment=HOMELAB_MCP_SSH_KEY=/etc/homelab-mcp/mcp-reader.key
-Environment=HOMELAB_MCP_SSH_USER=mcp-reader
+# SSH user is `root` on hubris — the restricted shell at command="..." in
+# authorized_keys (see mcp/mcp-reader-shell) provides the access boundary,
+# not a separate user account.
+Environment=HOMELAB_MCP_SSH_USER=root
+Environment=HOMELAB_MCP_HUBRIS_HOST=192.168.8.77
+Environment=HOMELAB_MCP_SSH_KNOWN_HOSTS=/etc/homelab-mcp/known_hosts
 ExecStart=/opt/homelab-mcp/.venv/bin/python /opt/homelab-mcp/mcp/server.py
 Restart=on-failure
 RestartSec=5

mcp/mcp-reader-shell

@@ -0,0 +1,62 @@
+#!/bin/bash
+# mcp-reader-shell — restricted SSH command for the homelab-mcp service.
+#
+# Authorized in /root/.ssh/authorized_keys on hubris via:
+#   command="/usr/local/bin/mcp-reader-shell",restrict ssh-ed25519 AAAA... mcp-reader@homelab-mcp
+#
+# `restrict` disables PTY/agent/forwarding/X11. This wrapper then validates
+# $SSH_ORIGINAL_COMMAND against a strict read-only allowlist before running
+# it. Anything outside the allowlist (interactive shell, file writes, pct
+# start/stop/destroy, etc.) is refused.
+#
+# Distributed via the homelab-context sync — symlink:
+#   /usr/local/bin/mcp-reader-shell -> /opt/homelab-context/mcp/mcp-reader-shell
+# so updates land on the next 5-min pull without a manual re-install.
+#
+# Argument shapes allowed (Bash glob, after rejecting shell metacharacters):
+#   systemctl is-active <unit>
+#   systemctl is-enabled <unit>
+#   journalctl -u <unit> [-n N] [--no-pager]
+#   pct list
+#   pct status <id>
+#   pct config <id>
+#   pct exec <id> -- systemctl is-active <unit>
+#   pct exec <id> -- systemctl is-enabled <unit>
+#   pct exec <id> -- journalctl -u <unit> [-n N] [--no-pager]
+#
+# Logs each call to syslog via `logger`. Deny entries are warnings.
+
+set -euo pipefail
+set -f   # disable glob expansion when we exec the command
+
+CMD="${SSH_ORIGINAL_COMMAND:-}"
+
+deny() {
+    logger -t mcp-reader -p auth.warning "DENY from=${SSH_CLIENT:-?}: ${CMD:-<empty>}"
+    echo "mcp-reader: command not allowed" >&2
+    exit 1
+}
+
+if [ -z "$CMD" ]; then
+    deny
+fi
+
+# Reject any shell metacharacter that would let an attacker chain or escape
+# from the patterns below.
+if [[ "$CMD" =~ [\;\&\|\>\<\`\$\\\(\)\{\}\*\?\~\!] ]]; then
+    deny
+fi
+
+case "$CMD" in
+    "systemctl is-active "*|"systemctl is-enabled "*) ;;
+    "journalctl -u "*)                                ;;
+    "pct list")                                       ;;
+    "pct status "*|"pct config "*)                    ;;
+    "pct exec "*" -- systemctl is-active "*)          ;;
+    "pct exec "*" -- systemctl is-enabled "*)         ;;
+    "pct exec "*" -- journalctl -u "*)                ;;
+    *) deny ;;
+esac
+
+logger -t mcp-reader -p auth.info "ALLOW from=${SSH_CLIENT:-?}: $CMD"
+exec $CMD

mcp/server.py

@@ -30,11 +30,23 @@ from mcp.server.fastmcp import FastMCP
 CONTEXT_DIR = Path(os.environ.get("HOMELAB_CONTEXT_DIR", "/opt/homelab-context"))
 INVENTORY = CONTEXT_DIR / "inventory.yaml"
 HOSTS_DIR = CONTEXT_DIR / "hosts"
+# All management tools proxy through hubris (the Proxmox host) via a single
+# restricted-shell SSH connection. The wrapper at mcp/mcp-reader-shell on
+# hubris validates each command against a strict read-only allowlist.
 SSH_IDENTITY = os.environ.get("HOMELAB_MCP_SSH_KEY", "/etc/homelab-mcp/mcp-reader.key")
-SSH_USER = os.environ.get("HOMELAB_MCP_SSH_USER", "mcp-reader")
+SSH_USER = os.environ.get("HOMELAB_MCP_SSH_USER", "root")
+SSH_KNOWN_HOSTS = os.environ.get("HOMELAB_MCP_SSH_KNOWN_HOSTS",
+                                 "/etc/homelab-mcp/known_hosts")
+HUBRIS_HOST = os.environ.get("HOMELAB_MCP_HUBRIS_HOST", "192.168.8.77")
 SSH_TIMEOUT = int(os.environ.get("HOMELAB_MCP_SSH_TIMEOUT", "10"))
 
 mcp = FastMCP("homelab")
+mcp.settings.host = os.environ.get("HOMELAB_MCP_HOST", "0.0.0.0")
+mcp.settings.port = int(os.environ.get("HOMELAB_MCP_PORT", "9810"))
+# FastMCP's DNS-rebinding protection only whitelists 127.0.0.1 / localhost / [::1]
+# by default, which breaks any LAN/mesh client. We're already mesh+LAN-gated at
+# nftables and the browser-attack threat doesn't apply to mesh-only services.
+mcp.settings.transport_security.enable_dns_rebinding_protection = False
 
 
 def _load_inventory() -> dict:
@@ -55,36 +67,49 @@ def inventory() -> dict:
     return _load_inventory()
 
 
-def _host_ssh_target(name: str) -> str:
-    """Return the ssh target (user@host) the MCP reader uses for `name`."""
-    inv = inventory()
-    h = inv.get("hosts", {}).get(name)
-    if h is None:
-        raise ValueError(f"unknown host: {name}")
-    mesh = h.get("mesh", {})
-    # Netbird FQDN if available, then lan_ip, then tailscale.
-    target = (
-        mesh.get("netbird", {}).get("fqdn")
-        or mesh.get("netbird", {}).get("ip")
-        or h.get("lan_ip")
-        or mesh.get("tailscale", {}).get("fqdn")
-    )
-    if not target:
-        raise ValueError(f"no reachable address for host {name}")
-    return f"{SSH_USER}@{target}"
-
-
-def _ssh(host: str, *cmd: str, timeout: int | None = None) -> subprocess.CompletedProcess:
-    """Run a command on a remote host as the restricted mcp-reader user."""
-    target = _host_ssh_target(host)
-    full = [
+def _run_via_hubris(remote_host: str, cmd: list[str],
+                    timeout: int | None = None) -> subprocess.CompletedProcess:
+    """SSH to hubris (single channel, restricted-shell key) and run `cmd`.
+    If remote_host is an LXC, the command is wrapped in `pct exec <id> --`
+    so hubris executes it inside the LXC. The wrapper at /usr/local/bin/
+    mcp-reader-shell on hubris validates the final command against an
+    allowlist before execution.
+    """
+    inv_hosts = inventory().get("hosts", {})
+    if remote_host == "hubris":
+        full = cmd
+    else:
+        host_entry = inv_hosts.get(remote_host)
+        if host_entry is None:
+            raise ValueError(f"unknown host: {remote_host}")
+        pve_id = host_entry.get("pve_id")
+        if pve_id is None:
+            raise ValueError(f"{remote_host} has no pve_id; can't pct-exec into it")
+        full = ["pct", "exec", str(pve_id), "--", *cmd]
+    # The restricted shell on hubris reads the joined command from
+    # $SSH_ORIGINAL_COMMAND and validates it as a single string.
+    joined = " ".join(full)
+    ssh_args = [
         "ssh", "-i", SSH_IDENTITY, "-o", "BatchMode=yes",
-        "-o", "StrictHostKeyChecking=accept-new",
+        # The systemd unit runs with ProtectHome=true so ~/.ssh is unreachable.
+        # Use a pre-populated known_hosts in /etc/homelab-mcp/.
+        "-o", f"UserKnownHostsFile={SSH_KNOWN_HOSTS}",
+        "-o", "StrictHostKeyChecking=yes",
         "-o", f"ConnectTimeout={SSH_TIMEOUT}",
-        target, *cmd,
+        f"{SSH_USER}@{HUBRIS_HOST}",
+        joined,
     ]
-    return subprocess.run(full, capture_output=True, text=True,
-                          timeout=timeout or SSH_TIMEOUT * 3)
+    proc = subprocess.run(
+        ssh_args, capture_output=True, text=True,
+        timeout=timeout or SSH_TIMEOUT * 3,
+    )
+    if proc.returncode != 0:
+        import logging as _l
+        _l.getLogger("homelab-mcp").warning(
+            "ssh failed rc=%s host=%s cmd=%r stderr=%r",
+            proc.returncode, remote_host, joined, proc.stderr.strip(),
+        )
+    return proc
 
 
 def _service_to_host(service: str) -> str:
@@ -244,8 +269,8 @@ def get_service_status(service: str) -> dict:
     host = _service_to_host(service)
     inv_svc = inventory()["services"][service]
     unit = inv_svc.get("systemd_unit", service)
-    active = _ssh(host, "systemctl", "is-active", unit)
-    enabled = _ssh(host, "systemctl", "is-enabled", unit)
+    active = _run_via_hubris(host, ["systemctl", "is-active", unit])
+    enabled = _run_via_hubris(host, ["systemctl", "is-enabled", unit])
     return {
         "service": service,
         "host": host,
@@ -261,14 +286,15 @@ def tail_log(service: str, lines: int = 200) -> str:
     host = _service_to_host(service)
     inv_svc = inventory()["services"][service]
     unit = inv_svc.get("systemd_unit", service)
-    proc = _ssh(host, "journalctl", "-u", unit, "-n", str(lines), "--no-pager")
+    proc = _run_via_hubris(host, ["journalctl", "-u", unit,
+                                  "-n", str(lines), "--no-pager"])
     return proc.stdout
 
 
 @mcp.tool()
 def list_lxcs() -> str:
     """Run `pct list` on hubris."""
-    return _ssh("hubris", "pct", "list").stdout
+    return _run_via_hubris("hubris", ["pct", "list"]).stdout
 
 
 @mcp.tool()
@@ -280,8 +306,8 @@ def get_lxc_state(lxc: str) -> dict:
     pve_id = inv[lxc].get("pve_id")
     if pve_id is None:
         raise ValueError(f"{lxc} has no pve_id (is it actually an LXC?)")
-    status = _ssh("hubris", "pct", "status", str(pve_id))
-    cfg = _ssh("hubris", "pct", "config", str(pve_id))
+    status = _run_via_hubris("hubris", ["pct", "status", str(pve_id)])
+    cfg = _run_via_hubris("hubris", ["pct", "config", str(pve_id)])
     return {"lxc": lxc, "pve_id": pve_id,
             "status": status.stdout.strip(),
             "config": cfg.stdout}
@@ -307,4 +333,4 @@ def ping_service(service: str) -> dict:
 
 
 if __name__ == "__main__":
-    mcp.run(transport="sse")
+    mcp.run(transport="streamable-http")

operations/agent-enrollment.md

@@ -0,0 +1,368 @@
+# Agent enrollment — bootstrap a client into the homelab context system
+
+This walks through enrolling a new machine (workstation, LXC, or VM) so it
+joins the cross-client context system: a `/opt/homelab-context/` clone of
+this repo that auto-syncs every 5 min, a per-client age key for SOPS
+decryption, the `homelab` CLI, and an MCP endpoint in Claude Code's config.
+
+> Onboarding a Nous-Hermes-powered Goose agent on top of standard enrollment?
+> See [hermes-agent.md](./hermes-agent.md). It uses the same `bootstrap.sh`
+> with an additional `--with-hermes` flag.
+
+Architecture in [project_homelab_context_plan](https://… memory link); the
+operational reference is here.
+
+## Prerequisites the client must satisfy
+
+| Requirement | Why | How to check |
+| --- | --- | --- |
+| Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
+| OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
+| On the Netbird mesh **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` |
+| `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
+| Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/mcp` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
+
+### Hostname mismatch is the most common bootstrap failure
+
+If the bootstrap exits with `no hosts/<name>.yaml in the repo`, the
+hostname doesn't match any inventory entry. Two fixes:
+
+- **Rename the host**: `sudo hostnamectl set-hostname <inventory-name>`
+  (Linux) or System Preferences → Sharing (macOS), then re-run.
+- **Rename the inventory entry**: edit `inventory.yaml` on hubris,
+  regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
+
+### Getting onto Netbird
+
+Bootstrap auto-installs netbird and drives `netbird up` if the mesh isn't already connected (since commit `<bootstrap-tier1>`). Both paths below produce the same end state: `netbird status` shows `Management: Connected`, peer IP `100.122.x.x/16`.
+
+**Path B — interactive OIDC (default; recommended):**
+
+The new client runs bootstrap straight from a fresh OS. Bootstrap installs netbird (apt/dnf/brew based on the OS), then runs `netbird up --management-url https://netbird.hubris.network --ssh-jwt-cache-ttl 86400`. A device-code URL prints inline. The operator opens it (in a browser logged into Authentik), goes through identification → password → consent, and the CLI returns `Connected`. Bootstrap then proceeds with the rest of preflight.
+
+Pre-condition: the operator must be a registered user in Authentik (typically the lab owner). The first user-login against a netbird account with existing peers is added as `pending_approval=1` and needs an sqlite promotion to `owner` — see [124-authentik.md First-time owner promotion gotcha](../containers/124-authentik.md). Only needed once per account.
+
+**Path A — setup-key (headless/scripted onboarding):**
+
+Useful for headless servers (no browser at all) or unattended cloud-init bootstraps.
+
+1. From an already-enrolled machine, log into the dashboard at `https://netbird.hubris.network/`.
+2. **Setup Keys** → Create → set reusable + expiry → copy.
+3. On the new client (after installing netbird, OR let bootstrap install it and skip its `netbird up` driver):
+   ```bash
+   sudo netbird up --setup-key <KEY> \
+                   --management-url https://netbird.hubris.network \
+                   --ssh-jwt-cache-ttl 86400
+   ```
+
+**Why we can't OIDC-login from the public internet (still open as a follow-up):**
+
+`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A brand-new client *off the mesh* hitting `auth.hubris.network` directly gets a Traefik default 404. In practice, Path B works fine because the operator's BROWSER (which clicks the device-code URL) is usually on a network that can reach Authentik through the public IONOS IP via some path. But "fresh laptop in a coffee shop with no prior session anywhere" still gets stuck. Future-session fix: add a Traefik route on the VPS forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124.
+
+### DNS prerequisite
+
+`*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124
+([dns.md](../infrastructure/dns.md)) for LAN clients, **but only if the
+client uses 192.168.8.180 as its resolver**. Most LXCs and roaming
+workstations don't by default. Options:
+
+- **LAN client**: set DNS to 192.168.8.180 (per-interface or
+  `/etc/resolv.conf`).
+- **Off-LAN workstation on Netbird**: configure Netbird DNS forwarder to
+  point `*.hubris.network` at LXC 124.
+- **Hack-fix anywhere**: append to `/etc/hosts`:
+  ```
+  192.168.8.175  mcp.hubris.network secrets.hubris.network
+  192.168.8.175  git.hubris.network
+  ```
+  (192.168.8.175 = caddy on LXC 121, terminates all `*.hubris.network`.)
+
+If DNS isn't an option at all, override the URLs at bootstrap time:
+
+```bash
+sudo HOMELAB_GITEA_TOKEN=... \
+     HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git \
+     HOMELAB_ISSUANCE_NETBIRD=http://192.168.8.205:9820/issue \
+     HOMELAB_MCP_URL=http://192.168.8.205:9810/mcp \
+     bash /tmp/bootstrap.sh --with-mcp
+```
+
+## Install dependencies
+
+Bootstrap auto-installs missing prerequisites (`git`, `python3` + PyYAML, `age`, `sops`, `netbird`) on Fedora/RHEL/Debian/Ubuntu/macOS — no manual `apt`/`dnf`/`brew` needed before running it. The only thing you must have on hand BEFORE the `curl ... | sudo bash` line is `curl` itself (used to pipe the script).
+
+Manual install is still possible (e.g. for air-gapped or unusual platforms); the per-OS recipes are below for reference but optional.
+
+<details>
+<summary>Manual recipes (Fedora / Debian / macOS)</summary>
+
+```bash
+# Fedora / RHEL / Nobara
+sudo dnf install -y git python3-pyyaml age curl
+SOPS_VERSION=v3.9.4
+sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
+  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
+
+# Debian / Ubuntu
+sudo apt update && sudo apt install -y git python3-yaml age curl
+SOPS_VERSION=v3.9.4
+sudo curl -fsSL https://github.com/getsops/sops/releases/download/$SOPS_VERSION/sops-$SOPS_VERSION.linux.amd64 \
+  -o /usr/local/bin/sops && sudo chmod +x /usr/local/bin/sops
+
+# macOS
+brew install git age sops
+pip3 install pyyaml   # if `python3 -c "import yaml"` fails
+```
+
+</details>
+
+## Run the bootstrap
+
+You need a Gitea read-only personal access token for the initial clone
+(the in-cluster shared PAT is encrypted at `secrets/gitea-readonly-pat.yaml`
+but a new client can't decrypt it before bootstrap — chicken-and-egg).
+Ask the operator (or generate in Gitea: Settings → Applications → Generate
+New Token → scope `read:repository`).
+
+```bash
+TOKEN=...   # your Gitea PAT, scope read:repository
+
+# Fetch bootstrap.sh from gitea (HTTPS uses split-DNS → caddy).
+curl -fsSL -u "dtoro:$TOKEN" \
+  https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
+  -o /tmp/bootstrap.sh
+
+# Run it.
+sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp
+```
+
+Flags:
+
+| Flag | Effect |
+| --- | --- |
+| `--with-mcp` | Merges the homelab MCP server into `~/.claude/.mcp.json` of the invoking user |
+| `--no-secrets` | Skips age-key issuance (use when bringing up the first hosts before secrets-issuance exists) |
+| `--dry-run` | Prints actions without executing |
+
+The bootstrap is idempotent: re-running on an enrolled client just
+verifies state, re-issues the age key only if it doesn't match the
+inventory pubkey, and refreshes the sync timer + symlinks.
+
+## Verify
+
+```bash
+homelab whoami                    # prints hosts/$(hostname).yaml
+homelab list                      # shows the full topology
+homelab status                    # ping + HTTP-check across hosts/services
+homelab secret hello              # decrypt the bootstrap-test secret
+systemctl list-timers homelab-context-sync.timer
+                                  # next run within ≤5 min
+```
+
+For Claude Code: start a new session — the `homelab` MCP server appears
+in `~/.claude/.mcp.json` and registers 14 tools (8 context, 5 management,
+1 secrets-metadata).
+
+## Post-bootstrap: SSH reachability
+
+A new workstation must be reachable from other workstations and must be
+able to reach every host by short hostname. Run these steps after the
+bootstrap verify passes:
+
+### 1. Enable SSH server
+
+```bash
+# macOS:
+sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist
+
+# Linux:
+sudo systemctl enable --now sshd
+```
+
+### 2. Generate SSH key (if missing)
+
+```bash
+ls ~/.ssh/id_ed25519.pub 2>/dev/null || ssh-keygen -t ed25519 -a 100
+```
+
+### 3. Publish pubkey to the repo
+
+```bash
+cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/$(hostname -s).pub
+cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add $(hostname -s) pubkey' && git push
+```
+
+### 4. Deploy keys to all hosts
+
+From any existing enrolled machine (hubris or another workstation):
+
+```bash
+ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
+```
+
+This adds the new workstation's pubkey to hubris and every running LXC.
+
+### 5. Generate SSH config
+
+```bash
+homelab ssh-config --install
+```
+
+Verify:
+
+```bash
+ssh hubris hostname     # should return "hubris" without password
+ssh gitea hostname      # should return "gitea" without password
+ssh mac-mini hostname   # should return "mac-mini" without password (workstation-to-workstation)
+```
+
+### 6. Add LAN IP to inventory (if on LAN)
+
+If the workstation has a static or reserved LAN IP, add it to
+`inventory.yaml`:
+
+```yaml
+hosts:
+  your-hostname:
+    lan_ip: 192.168.8.xxx
+```
+
+This gives it a primary LAN entry in the generated SSH config (faster
+than the Netbird fallback). Commit + push, then:
+
+```bash
+cd /opt/homelab-context && git pull --ff-only && homelab ssh-config --install
+```
+
+## Claude Code permissions for fleet ops
+
+By default Claude Code's auto-mode classifier asks for confirmation on every
+ssh into the mesh. The bootstrap already installs the ssh ControlMaster block
+so subsequent in-session sshes multiplex, but the *first* ssh of each session
+still gets classifier-evaluated. Pre-authorize the common fleet ssh patterns
+by adding to `~/.claude/settings.json`:
+
+```json
+{
+  "permissions": {
+    "defaultMode": "auto",
+    "allow": [
+      "Bash(ssh -p 22022 *)",
+      "Bash(homelab *)"
+    ]
+  }
+}
+```
+
+The first rule covers any ssh to a mesh peer on the homelab netbird port; the
+second covers all `homelab` CLI invocations. Both are scoped tight enough that
+the classifier doesn't gate them but loose enough to handle the variety of
+arguments.
+
+If you also want the netbird `--ssh-jwt-cache-ttl` flag rationale to be
+visible to the classifier (it's not actually durable in 0.71.2, but the
+ControlMaster block is — see [runbook-dpkg-interrupted](runbook-dpkg-interrupted.md)
+for context), drop a free-text rule into `autoMode.allow` describing the
+authorization. Optional.
+
+## Adding a new client to inventory
+
+If the hostname you want isn't yet in inventory, enrollment is a two-step
+ceremony driven from an existing enrolled client (e.g. hubris). The
+`homelab` CLI handles steps 1 + 4; you provide steps 2 + 3.
+
+```bash
+# 1. On hubris (or any existing client): add the inventory entry.
+homelab client add my-new-machine
+# Prompts for kind, os, netbird FQDN, role. Commits + pushes.
+
+# 2. Join the new machine to Netbird (out-of-band, Netbird console / setup key).
+
+# 3. On the new machine: install deps + run bootstrap (above).
+#    Bootstrap calls /issue, receives a fresh age keypair, and prints the
+#    public key for the operator to commit back to inventory.
+
+# 4. On hubris: finalize the age public key.
+homelab client add my-new-machine --finalize-pubkey age1...
+# Updates inventory.yaml hosts.my-new-machine.age_pubkey, regenerates
+# hosts/*.yaml, commits + pushes. The 5-min sync propagates.
+```
+
+## Granting a secret to a new client
+
+Adding a client doesn't grant them every secret. Recipients are explicit
+per file via `.sops.yaml` glob rules. To grant a client access to (say)
+`secrets/hello.yaml`:
+
+1. Edit `.sops.yaml` at the repo root, add the client's `age_pubkey` to
+   the matching `creation_rules` block.
+2. Re-key the existing ciphertext for the new recipient list:
+   ```bash
+   sops updatekeys -y secrets/hello.yaml
+   ```
+3. Commit + push. On the next sync (≤5 min), the client can decrypt.
+
+## Removing a client
+
+```bash
+# From any existing client:
+homelab client remove my-old-machine
+```
+
+This:
+1. Removes the inventory entry and `hosts/my-old-machine.yaml`.
+2. Runs `sops updatekeys -y` against every file in `secrets/` (operator
+   must first remove the pubkey from `.sops.yaml` rules).
+3. Calls `secrets-issuance` `/revoke` (admin-token-gated, on LXC 105) to
+   shred the key file and add the hostname to the denylist.
+4. Commits + pushes.
+
+The CLI prints a follow-up checklist that the operator must do manually:
+
+- Revoke the peer in the Netbird console (denies future mesh access).
+- **Rotate any credentials whose ciphertext the removed client already
+  has on disk.** The age key revocation only protects *future*
+  ciphertext; what's already been pulled is still decryptable until the
+  underlying credential changes.
+- Optional: `homelab nuke my-old-machine` SSHes in, shreds
+  `/etc/age/key.txt`, removes `/opt/homelab-context`, disables sync.
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| `no hosts/<hostname>.yaml in the repo` | Hostname doesn't match inventory entry | Rename either side (see above) |
+| `fatal: could not read Username for 'http://192.168.8.121:3000'` | bootstrap.sh's credentials file has wrong scheme | Fixed in commit `de6f8be`; pull latest `bootstrap.sh` |
+| `gnutls_handshake() failed: TLS connection was non-properly terminated` cloning `git.hubris.network` | Client DNS resolves `*.hubris.network` to the public VPS IP | Configure split-DNS (LXC 180 / Netbird forwarder) or `/etc/hosts` override; or use `HOMELAB_REPO_URL=http://192.168.8.121:3000/dtoro/Homelab-Docs.git` |
+| `TLS/SSL connection has been closed (EOF)` connecting MCP | Same — `mcp.hubris.network` resolves to public VPS without this vhost | Same DNS fix |
+| `Invalid Host header` from MCP server | FastMCP's DNS-rebinding protection (default whitelist is 127.0.0.1 only) | Fixed in commit `6848640`; pull latest `mcp/server.py` and redeploy |
+| `python3-yaml` install fails on Fedora | Wrong package name | Use `python3-pyyaml` (Fedora) instead of `python3-yaml` (Debian) |
+| `address already in use` for FastMCP | FastMCP defaults to 127.0.0.1:8000 | Fixed: server now sets `mcp.settings.host/port` from env (default `0.0.0.0:9810`) |
+| `homelab: no age key at /etc/age/key.txt` even after bootstrap | `/etc/age` is 0700 root, so non-root users couldn't even stat the key file; existence check returned False under regular users | Fixed in commit `df6aca8`: the CLI re-execs `sops -d` via sudo when invoked as a non-root user. On older deployments, re-link the CLI with `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab` after the 5-min sync. |
+| `homelab` CLI doesn't pick up repo updates | Pre-`02db…` bootstrap copied the binary instead of symlinking | One-time migration: `sudo ln -sfn /opt/homelab-context/bin/homelab /usr/local/bin/homelab`. New bootstraps use the symlink, which auto-tracks the synced repo. |
+| `homelab-context-sync.service` journal shows `fatal: could not read Username for 'https://git.hubris.network'` | Pre-fix bootstrap set the gitea credential helper via `git config --global`, which writes to `/root/.gitconfig` — invisible to the systemd timer's git process (no HOME set). | One-time migration: `sudo git config --system credential.helper "store --file=/etc/homelab-context/git-credentials"`. New bootstraps store the helper in `/etc/gitconfig` instead. |
+| Chat-mode `!` shell can't `sudo` (`a terminal is required to read the password`) | Claude Code's `!` invocation doesn't allocate a tty, and standard `sudo` won't read its password from stdin or a non-tty pipe. | Run the sudo'd command in a real terminal outside chat. For commands the agent issues repeatedly, configure passwordless sudo for the narrow set (e.g. `/etc/sudoers.d/homelab-self` with `<user> ALL=(ALL) NOPASSWD: /usr/bin/dnf upgrade -y, /usr/bin/apt-get *`). |
+| `netbird status -d` reports `192.168.8.180:53 ... is Unavailable` but DNS actually works | netbird's UDP-53 probe times out over the relay latency (~90ms), but actual queries still flow through systemd-resolved. Cosmetic. | Ignore unless `dig @192.168.8.180 git.hubris.network` also fails — then check dnsmasq on [LXC 124](../containers/124-authentik.md). |
+| `netbird ssh` rejected with `JWT authentication failed: validate token (expected issuer=https://netbird.hubris.network/oauth2 ...)` | Peer's SSH JWT validator cached the OLD embedded-Dex issuer from before the 2026-05-21 Authentik migration. `systemctl restart netbird` and `netbird down/up` don't clear it — `client/internal/engine_ssh.go` bails out of `updateSSH()` if the SSH server is already running. | Full daemon bounce: `sudo systemctl stop netbird; sleep 3; sudo systemctl start netbird`. Verify with `grep -iE "issuer\|audience" /var/log/netbird/client.log \| tail`. Apply once per peer post-migration. |
+| `netbird ssh` JWT passes but session closes with `user privilege check failed: user dtoro not found: unknown user dtoro` | netbird-ssh defaults the remote username to the LOCAL one (operator's laptop user). Hubris and LXCs only have `root`. | Always use explicit `root@` prefix manually: `netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted`. `homelab ssh <host>` does this automatically via `inventory.yaml`'s per-host `ssh.user` field (defaults to `root`). |
+| `homelab ssh hubris` (or any host on the LAN) fails with `Connection refused` or hangs, despite mesh routing being up | Off-LAN networks (operator on a VPN / coffee shop / symmetric NAT) sometimes can't reach the LAN IP even with the netbird subnet route. | Newer homelab CLIs probe the LAN with a 1.5s TCP connect and transparently fall back to the netbird FQDN. If your `/usr/local/bin/homelab` is a symlink to `/opt/homelab-context/bin/homelab` it'll pick up the fix on the next 5-min context sync. Otherwise pull the latest from gitea. |
+
+## Changelog
+
+### 2026-06-02 — SSH reachability post-bootstrap steps
+Added a new "Post-bootstrap: SSH reachability" section covering SSH key
+generation, pubkey publication, deployment to hosts, SSH config generation,
+and LAN IP registration. New workstations enrolled via this doc will
+automatically join the universal SSH mesh.
+
+### 2026-05-31 — cross-link to hermes-agent.md
+Added a sibling page covering Nous-Hermes-on-Goose enrollment ([hermes-agent.md](./hermes-agent.md)) and noted it at the top of this page. The Hermes flow extends `bootstrap.sh` with `--with-hermes` and `homelab client add` with the same flag; it does not change the underlying enrollment steps documented here.
+
+### 2026-05-21 — netbird-ssh JWT issuer + username + LAN-fallback troubleshooting rows
+Added three rows to the troubleshooting table covering issues surfaced during the netbird vanilla migration: (1) post-migration SSH JWT validator cache stuck on old Dex issuer (full `systemctl stop/start` required, not `restart`), (2) `user not found` from netbird-ssh's local-username default (use explicit `root@`), and (3) homelab CLI's LAN→netbird-FQDN fallback for off-LAN operators. Companion code change: per-host `ssh.user` field in `inventory.yaml` + `homelab` CLI's `ssh_target()` helper.
+
+### 2026-05-20 — initial page
+Captures the enrollment flow validated during Phase 2 of the homelab
+context distribution rollout. hubris + LXC 105 (apps) enrolled; first
+workstation (republic-laptop) blocked on hostname mismatch, documented
+the resolution.

operations/commands.md

@@ -26,9 +26,11 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root
 
 ## Reverse proxy
 
-- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). Tracked in `dtoro/caddy-conf`. Edits should be committed + pushed — see [auto-deploy](../infrastructure/auto-deploy.md).
+- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md).
+- **CRITICAL:** This file is tracked in `dtoro/caddy-conf` (https://git.hubris.network/dtoro/caddy-conf). Never edit it directly on the LXC — commit + push to the repo instead. Caddy auto-deploys on push (see [auto-deploy](../infrastructure/auto-deploy.md)). If you edit directly, the change will be lost on the next pull and agents won't know about it.
 - Hot reload: `pct exec 121 -- systemctl reload caddy`.
 - Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
+- Git workflow shortcut: `pct exec 121 -- "cd /etc/caddy && git add Caddyfile && git commit -m '...' && git push"`.
 
 ## DNS
 
@@ -48,9 +50,25 @@ Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root
 - `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
 - `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md))
 
+## Fleet apt operations
+
+Two `homelab` subcommands wrap the common patterns; both fan out to hubris + every LXC.
+
+| Command | What it does |
+| --- | --- |
+| `homelab apt-audit [--target HOST]` | Per-host table: dpkg-interrupted state, holds, upgradable count, non-apt binaries in system paths, DNS health. Exits nonzero if any host has dpkg-interrupted state. |
+| `homelab apt-upgrade --target HOST` | Launch `apt update && apt upgrade` inside a transient `systemd-run --collect` unit on the target. Survives ssh teardown. Apt configured with `Acquire::Retries=3` + `ForceIPv4=true`. |
+| `homelab apt-upgrade --all` | Same, fanned out across the standard targets. |
+| `homelab apt-upgrade ... --status` | Show running unit + tail `/var/log/homelab-apt-upgrade.log` on each target. |
+| `homelab apt-upgrade ... --safe` | Take a pre-upgrade snapshot per LXC first (`pct snapshot` → `vzdump` fallback for bind-mounted LXCs). Refuses if any snapshot fails unless `--force`. |
+| `homelab apt-upgrade ... --force` | Skip both the dpkg-audit gate and snapshot-failure refusal. |
+
+PVE/kernel deferral on hubris: `homelab apt-upgrade --target hubris` will try every upgrade, including kernel + `pve-*`. To skip those, `apt-mark hold` the relevant packages on hubris first; `homelab apt-audit` shows held packages so you can confirm.
+
 ## Related
 - [Hubris host](../hosts/hubris.md)
 - [Containers index](../containers/index.md)
 - [DNS](../infrastructure/dns.md)
 - [Monitoring](../infrastructure/monitoring.md)
 - [Auto-deploy](../infrastructure/auto-deploy.md)
+- [Runbook: dpkg-interrupted recovery](runbook-dpkg-interrupted.md) — what to do when apt got killed mid-transaction

operations/hermes-agent.md

@@ -0,0 +1,210 @@
+# Hermes agent — Nous-Hermes-powered Goose sessions on a homelab client
+
+Onboards [Nous Research's Hermes](https://nousresearch.com/) (a fine-tuned
+Llama variant) as a working terminal agent on a homelab client. Builds on top
+of standard client enrollment (see [agent-enrollment.md](./agent-enrollment.md))
+— this page covers only the Hermes-specific additions.
+
+The agent runs as a [Goose](https://goose-docs.ai/) session. Goose provides:
+
+- The chat loop, multi-turn history, and streaming
+- The OpenRouter provider that routes to Nous Hermes
+- The built-in `developer` extension (shell + file editor — same surface Claude
+  Code has)
+- A remote MCP extension pointed at `mcp.hubris.network` for read-only
+  homelab context (`list_lxcs`, `tail_log`, `search_docs`, etc.)
+
+The persona is `/opt/homelab-context/HERMES.md`, symlinked as Goose's global
+`.goosehints` so it's injected into the system prompt on every session.
+
+## Prerequisites
+
+| Requirement | How |
+| --- | --- |
+| Standard enrollment complete (`homelab whoami` works) | [agent-enrollment.md](./agent-enrollment.md) |
+| `secrets/openrouter-api-key.yaml` exists with a real `sk-or-...` value | See "Seeding the OpenRouter key" below |
+| The host's `age_pubkey` is on the openrouter-api-key.yaml sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` |
+
+## Onboarding flow
+
+```bash
+# 1. On hubris (or any enrolled client): reserve the inventory entry.
+homelab client add new-machine
+
+# 2. Join new-machine to Netbird (setup-key or OIDC).
+
+# 3. On new-machine: bootstrap with --with-hermes.
+TOKEN=...   # gitea PAT, read:repository
+curl -fsSL -u "dtoro:$TOKEN" \
+  https://git.hubris.network/dtoro/Homelab-Docs/raw/branch/main/bootstrap.sh \
+  -o /tmp/bootstrap.sh
+sudo HOMELAB_GITEA_TOKEN=$TOKEN bash /tmp/bootstrap.sh --with-mcp --with-hermes
+
+# 4. Back on hubris: finalize the age pubkey AND grant the Hermes secret.
+homelab client add new-machine \
+  --finalize-pubkey age1... \
+  --with-hermes
+
+# 5. Wait ≤5 min for sync, then on new-machine:
+hermes "what LXCs are running?"
+```
+
+The bootstrap `--with-hermes` flag does five things, all idempotent:
+
+1. Downloads the latest Goose binary into the operator's `~/.local/bin/goose`
+   (upstream installer) and symlinks `/usr/local/bin/goose` to it.
+2. Symlinks `/opt/homelab-context/bin/hermes` → `/usr/local/bin/hermes`.
+3. Symlinks `/opt/homelab-context/HERMES.md` → `/root/HERMES.md` (Linux) or
+   `/etc/HERMES.md` (macOS) for `cat`-as-operator convenience.
+4. Drops `~/.config/goose/config.yaml` pinning the provider, model, and
+   extensions (preserves any keys the operator added by hand).
+5. Symlinks `~/.config/goose/.goosehints` → HERMES.md, so the persona is
+   injected as the system prompt on every session.
+
+## Seeding the OpenRouter key
+
+The first time anyone enrolls with `--with-hermes`, the encrypted file
+`secrets/openrouter-api-key.yaml` contains a placeholder. On hubris (or any
+existing recipient):
+
+```bash
+sops secrets/openrouter-api-key.yaml
+# editor opens; replace api_key value with the real sk-or-... key, save, close.
+git -C /opt/homelab-context add secrets/openrouter-api-key.yaml
+git -C /opt/homelab-context commit -m 'openrouter-api-key: seed real key'
+git -C /opt/homelab-context push
+```
+
+Until this step happens, `hermes …` exits with `openrouter-api-key.yaml still
+contains the placeholder`. Subsequent enrollees get the real key automatically
+via `--with-hermes` (which adds them as a sops recipient on
+`secrets/openrouter-api-key.yaml`).
+
+## Granting the OpenRouter key to an already-enrolled host
+
+If a host was enrolled without `--with-hermes` and you want to add it later:
+
+```bash
+# On hubris:
+PUBKEY=$(homelab whoami --hostname <host> | grep age_pubkey | awk '{print $2}')
+homelab client add <host> --finalize-pubkey "$PUBKEY" --with-hermes
+```
+
+`--finalize-pubkey` is required by the existing flow even when the pubkey is
+unchanged — it's also the trigger that runs the sops grant.
+
+After ≤5 min sync the host can decrypt the key. Bootstrap doesn't need to
+re-run; only the secret recipient list changed.
+
+## Verifying
+
+```bash
+homelab whoami                                # standard enrollment OK
+homelab secret openrouter-api-key | head -c 8 # decrypts (prints `api_key:`)
+which goose && which hermes                   # binaries present
+goose info -v                                  # provider/model wiring sane
+hermes "what LXCs are running?"                # interactive Goose session
+
+# Non-interactive smoke test:
+echo "List the homelab MCP tools you have available" | hermes
+```
+
+## Configuration
+
+The bootstrap-managed keys in `~/.config/goose/config.yaml`:
+
+```yaml
+GOOSE_PROVIDER: openrouter
+GOOSE_MODEL: deepseek/deepseek-v4-flash
+GOOSE_MODE: smart_approve         # asks before destructive tool calls
+extensions:
+  developer:
+    type: builtin
+    bundled: true
+    enabled: true
+    name: developer
+    timeout: 300
+  homelab:
+    type: streamable_http
+    enabled: true
+    name: homelab
+    uri: https://mcp.hubris.network/mcp
+    timeout: 60
+```
+
+Override via env on a single bootstrap run:
+
+```bash
+HOMELAB_HERMES_MODEL=nousresearch/hermes-3-llama-3.1-405b \
+HOMELAB_HERMES_MCP_URI=https://mcp.hubris.network/mcp \
+  sudo bash /tmp/bootstrap.sh --with-hermes
+```
+
+Any keys you add by hand (e.g. `GOOSE_TEMPERATURE`, extra `extensions.*`) are
+preserved across re-bootstraps — the merge only overwrites the keys it manages.
+
+## Tool permissions
+
+`GOOSE_MODE: smart_approve` is the bootstrap default: Goose runs read-only
+shell commands without prompting and asks for confirmation before destructive
+ones. To make the agent fully unattended (e.g. for scheduled jobs), set
+`GOOSE_MODE: auto` in `~/.config/goose/config.yaml`. To require confirmation on
+every tool call, use `approve`. See
+[goose-permissions](https://goose-docs.ai/docs/guides/managing-tools/goose-permissions/).
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+| --- | --- | --- |
+| `hermes: could not decrypt secrets/openrouter-api-key.yaml` | Host isn't a recipient on the sops rule | `homelab client add <host> --finalize-pubkey <age1...> --with-hermes` from hubris |
+| `hermes: openrouter-api-key.yaml still contains the placeholder` | No real key has been seeded yet | See "Seeding the OpenRouter key" above |
+| Goose hangs on first `hermes` invocation with no output | Goose's interactive `configure` ran on first launch and is awaiting input | Re-run; the installer is supposed to skip it (CONFIGURE=false). If it persists, run `goose configure` once manually in a real terminal to commit the config. |
+| `homelab` extension fails to connect / no MCP tools listed | MCP server still runs SSE-only; Goose requires `streamable_http`. See follow-up #1 below. | Either: (a) migrate the FastMCP server to streamable_http (one-line change in `mcp/server.py` — `mcp.run(transport="streamable_http")` — then redeploy), or (b) accept that the agent works via the developer extension alone (shell + `homelab` CLI cover everything MCP would). |
+| `goose: command not found` after bootstrap | Upstream installer dropped binary in `~/.local/bin/` but `/usr/local/bin/goose` symlink didn't land | Re-run bootstrap with `--with-hermes`; the symlink step is at the end of the install block. If still missing, `ln -sfn ~/.local/bin/goose /usr/local/bin/goose` manually. |
+| Tool calls hit OpenRouter rate limits | One shared key across many hosts | Future: per-host keys; for now, see the rate-limits guide referenced in `goose info -v`. |
+
+## Cross-references
+
+- [agent-enrollment.md](./agent-enrollment.md) — base client onboarding the
+  Hermes flow assumes is done.
+- [`HERMES.md`](../HERMES.md) — the persona the Hermes agent reads on every
+  session start (via `~/.config/goose/.goosehints`).
+- [`bin/hermes`](../bin/hermes) — the wrapper that decrypts the OpenRouter key
+  and execs `goose session`.
+- [`bootstrap.sh`](../bootstrap.sh) — the `--with-hermes` flag's install block.
+
+## Follow-ups
+
+1. **Migrate the MCP server to streamable_http.** Goose 1.x deprecated SSE
+   (`"SSE transport is no longer supported - kept only for config file
+   compatibility"` in `crates/goose/src/agents/extension.rs`). Our FastMCP
+   server at `mcp/server.py:336` still calls `mcp.run(transport="sse")`. Until
+   that's changed, the `homelab` MCP extension in Goose will fail to connect.
+   The developer extension (shell + edit) covers most ops without it; this is
+   a polish item, not a blocker.
+2. **Per-host OpenRouter keys** for billing attribution. Today all Hermes
+   hosts share one key.
+3. **Pin the model version** rather than tracking `nousresearch/hermes-4-405b`
+   directly — OpenRouter periodically rotates the underlying weights.
+4. **Local-inference fallback** (ollama / vllm) once the homelab has a GPU
+   node. The wrapper, persona, and MCP wiring stay unchanged; only
+   `GOOSE_PROVIDER`/`GOOSE_MODEL` change.
+
+7. **Caveman auto-setup via post-pull hook.** The sync timer now calls
+   `tools/post-pull.sh`, which runs any `tools/*.setup.sh` after git pull.
+   Currently this auto-installs the Caveman npm package, wrapper scripts, and
+   compact output templates on all agent hosts (*token efficiency*).
+
+## Changelog
+
+### 2026-06-01 — caveman + post-pull auto-setup
+Added `tools/post-pull.sh` sync hook that auto-runs `tools/*.setup.sh`
+after every git pull. First user: `tools/setup-caveman.sh` installed Caveman
+templating + `~/bin/caveman_wrapper.sh` + `~/templates/*.txt` for token-
+efficient CLI output. Replaces raw `git pull` in launchd/systemd timers.
+Also created `tools/caveman/` with the wrapper script, JS renderer, and
+templates — the canonical source for all agent hosts.
+Captures the Hermes-on-Goose onboarding flow added in the same commit as
+`bootstrap.sh --with-hermes`, `bin/hermes`, the sops rule for
+`secrets/openrouter-api-key.yaml`, and the `homelab client add --with-hermes`
+extension. MCP streamable_http migration is queued as follow-up #1.

operations/runbook-dpkg-interrupted.md

@@ -0,0 +1,108 @@
+# Runbook — recover from dpkg-interrupted state
+
+You're here because an apt run got killed mid-transaction and the target now
+has packages that are **unpacked but not configured**. Symptoms:
+
+- `apt` refuses to do anything new: `Error: dpkg was interrupted, you must
+  manually run 'dpkg --configure -a' to correct the problem.`
+- `dpkg --audit` lists packages with header
+  `The following packages have been unpacked but not yet configured.`
+- `homelab apt-audit` shows `DPKG: DIRTY(N)` for the host.
+
+The system is still running the **old** binaries (still in memory), but the
+**new** binaries are unpacked and waiting for their postinst to run. Two
+worst-case manifestations from the 2026-05-21 sweep:
+
+- LXC 121 caddy: leftover state from a prior aborted apt run; caddy itself was
+  still serving but the new caddy binary on disk hadn't been wired up.
+- hubris: ssh master died mid-Wave-6 → 135 packages unpacked-not-configured,
+  including `systemd`, `openssh-server`, `sudo`, `netbird`. The half-
+  configured netbird daemon dropped the mesh peer, and we got locked out
+  until we recovered from the PVE web UI Shell.
+
+**Do not reboot until dpkg is clean.** A reboot tries to start the new
+binaries' services, which may fail because postinst never ran (missing users,
+config dirs, capabilities, etc.). The system might not come back up cleanly.
+
+## Path A — target is still reachable over ssh (preferred)
+
+```
+homelab ssh <host> -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
+```
+
+Or for an LXC by name:
+
+```
+homelab pct <lxc> exec -- bash -c 'DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y -o Dpkg::Options::=--force-confold upgrade'
+```
+
+When that returns, confirm:
+
+```
+homelab apt-audit --target <host>
+```
+
+Expect `DPKG: ok` and the remaining `UPGR` count to match what's intentionally
+deferred (kernel/PVE on hubris, 0 elsewhere).
+
+## Path B — target locked out (mesh broken / ssh dead)
+
+Most common for hubris when netbird itself went half-configured: the daemon
+crashed on the new binary, the mesh peer dropped, port 22022 stopped listening,
+and you can't ssh in.
+
+1. Open `https://proxmox.hubris.network` in a browser.
+2. Datacenter → node `hubris` → `>_ Shell` (or `_ Console`). That's a root
+   shell on hubris served by the PVE web UI, independent of the netbird mesh.
+3. Run the recovery one-liner:
+
+```
+DEBIAN_FRONTEND=noninteractive dpkg --configure -a \
+  && DEBIAN_FRONTEND=noninteractive apt -y -o Dpkg::Options::=--force-confold upgrade \
+  && systemctl restart netbird \
+  && dpkg --audit \
+  && echo RECOVERY_OK
+```
+
+   Wait for `RECOVERY_OK`. The `systemctl restart netbird` is the bit that
+   heals the mesh — once netbird's daemon comes back up clean, your client's
+   peer state moves from `Connecting` to `Connected` within ~30 seconds and
+   the rest of your tooling works again.
+
+4. For an **LXC** that's locked out (less common — LXCs reach the world via
+   netbird routed through hubris, so unless hubris itself is broken, you can
+   still `pct enter` from the hubris shell):
+
+   From the PVE web UI shell on hubris:
+
+   ```
+   pct enter <id>
+   DEBIAN_FRONTEND=noninteractive dpkg --configure -a && apt -y upgrade
+   exit
+   ```
+
+## Prevention
+
+The `homelab apt-upgrade` wrapper launches apt inside a `systemd-run --collect`
+unit on the target, so it survives ssh teardown — the failure mode that put
+hubris into this state in the first place is no longer reachable through the
+standard tool. If you absolutely need to run apt manually over ssh, wrap it:
+
+```
+ssh <host> systemd-run --unit=apt-recovery --collect bash -c 'apt -y upgrade'
+```
+
+Then `systemctl status apt-recovery` from a fresh ssh to check progress.
+
+## Related
+
+- [Operations cheatsheet](commands.md)
+- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
+- [Hubris host page](../hosts/hubris.md)
+
+## Changelog
+
+### 2026-05-21 — initial page
+Documents the dpkg-interrupted recovery path that came out of the
+fleet apt sweep (Wave 6 killed mid-transaction; hubris recovered via PVE
+web Shell).

plans/2026-06-01-slate-ax-to-sodola-migration.md

@@ -0,0 +1,131 @@
+# 2026-06-01 — Slate AX → SODOLA Migration
+
+**Status:** Done — 2026-06-02  
+**Hardware:** SODOLA 5-Port 2.5Gbit Managed Switch replacing GL.iNet Slate AX  
+**Router:** Fritz!Box 7590
+
+## Goal
+
+Remove the Slate AX sub-router. It adds double-NAT, double port-forward complexity, and a second DHCP/gateway hop. Replace with the SODOLA switch so Proxmox and homelab devices attach directly to the Fritz!Box—without changing any of their IPs.
+
+## Current topology
+
+```
+ISP
+ └── Fritz!Box 7590 (192.168.178.1)
+      └── GL.iNet Slate AX  WAN: 192.168.178.x  LAN: 192.168.8.1  [NAT]
+           ├── hubris (Proxmox)   192.168.8.77
+           ├── + all 16 LXCs on vmbr0 bridged to the same subnet
+           └── other homelab devices
+```
+
+## Target topology
+
+```
+ISP
+ └── Fritz!Box 7590
+      ├── VLAN 1  192.168.178.0/24  (main LAN — laptops, phones)
+      └── VLAN 10 192.168.8.0/24   gateway 192.168.8.1  (homelab, no NAT)
+           │
+           └── SODOLA 5-Port 2.5Gbit
+                ├── Port 1  uplink → Fritz!Box LAN port
+                ├── Port 2  hubris (Proxmox)     192.168.8.77
+                ├── Port 3  [homelab device]
+                ├── Port 4  [homelab device]
+                └── Port 5  spare
+```
+
+Fritz!Box takes over `192.168.8.1` — the same gateway IP the Slate AX used. No static IPs or gateway entries change on any LXC or VM.
+
+See [network architecture](../infrastructure/network.md) for the permanent topology reference.
+
+## Pre-flight checklist
+
+- [ ] Note Slate AX gateway IP on the homelab subnet (expected `192.168.8.1`)
+- [ ] Note Slate AX upstream DNS setting (what it forwards to)
+- [ ] List all port forwards currently on the Slate AX → move these to Fritz!Box
+- [ ] List all port forwards on Fritz!Box that point at the Slate AX WAN IP → collapse these into direct homelab-IP rules
+- [ ] Confirm Fritz!OS version supports "second IP network" (Fritz!OS 7.x required)
+- [ ] SSH into hubris: verify `grep gateway /etc/network/interfaces` → `192.168.8.1`
+- [ ] SODOLA web UI accessible and ports configurable before physical move
+
+## Fritz!Box configuration
+
+**Path:** Home Network → Network → Network Settings → IPv4 Addresses → Add IP network
+
+| Field | Value |
+|---|---|
+| IP address | `192.168.8.1` |
+| Subnet mask | `255.255.255.0` |
+| DHCP range | `192.168.8.100 – 192.168.8.240` |
+| Assign to | LAN port that connects to SODOLA |
+| Network isolation | Enabled (blocks main LAN from initiating into homelab) |
+| DNS for DHCP clients | `192.168.8.2` (Technitium on [CT 107](../containers/107-dns.md)) |
+
+After creating the network, move any port forwards from the Slate AX into Fritz!Box → Internet → Permits (target IPs are now directly reachable on `192.168.8.x`).
+
+## SODOLA configuration
+
+All ports in VLAN 10, untagged (simple access-port setup):
+
+| Port | PVID | Role |
+|---|---|---|
+| 1 | 10 | Uplink → Fritz!Box |
+| 2 | 10 | hubris (Proxmox) |
+| 3 | 10 | homelab device |
+| 4 | 10 | homelab device |
+| 5 | 10 | spare |
+
+## Proxmox / LXC changes
+
+If the Slate AX gateway was `192.168.8.1`: **no changes needed**. Fritz!Box takes the same IP.
+
+If Slate AX used a different gateway (e.g. `.254`), update:
+```bash
+# on hubris
+nano /etc/network/interfaces   # change gateway line
+ifreload -a
+# for each LXC with a static gateway
+pct config <id>                # check net0: line for gw=
+pct set <id> --net0 name=eth0,bridge=vmbr0,ip=<ip>/24,gw=192.168.8.1
+```
+
+## DNS after migration
+
+Technitium ([CT 107](../containers/107-dns.md)) at `192.168.8.2` continues to serve split-horizon DNS for `hubris.network`. The Fritz!Box DHCP server for VLAN 10 hands out `192.168.8.2` as the DNS server. This fixes the "update router DHCP DNS from dead .180 → .2" outstanding item in [dns.md](../infrastructure/dns.md).
+
+## Cutover procedure
+
+1. Configure Fritz!Box second IP network (VLAN 10 / `192.168.8.1`) — verify DHCP works via a test laptop on the target port.
+2. Configure SODOLA port VLAN assignments.
+3. Connect SODOLA port 1 → Fritz!Box target LAN port.
+4. Move hubris ethernet → SODOLA port 2. Proxmox should stay reachable immediately (same gateway IP).
+5. Move remaining homelab devices to SODOLA ports 3–5.
+6. Decommission Slate AX.
+
+## Verification
+
+```bash
+# internet from Proxmox
+ssh proxmox "ping -c3 1.1.1.1"
+
+# Fritz!Box main LAN reachable (routing between VLANs works)
+ssh proxmox "ping -c3 192.168.178.1"
+
+# split-horizon DNS via Technitium still resolves
+ssh proxmox "dig @192.168.8.2 +short git.hubris.network"
+# expected: 192.168.8.175
+
+# Caddy reverse proxy responds
+curl -sk https://git.hubris.network | head -5
+
+# Authentik on VPS still reachable
+curl -sk https://auth.hubris.network/if/flow/default-authentication-flow/ | head -5
+```
+
+## Post-migration
+
+- Update [network.md](../infrastructure/network.md) topology to reflect new state.
+- Add changelog entries to [hosts/hubris.md](../hosts/hubris.md) and any affected container pages.
+- Update status in [plans/index.md](index.md) to `Done`.
+- If anything went sideways, open an investigation in `investigations/`.

plans/2026-06-04_130000-deprecate-claudio-bot.md

@@ -0,0 +1,318 @@
+# Deprecate claudio-bot (LXC 123) — Hermes Agent now serves as control plane
+
+## Goal
+
+Phase out the **claudio-bot ecosystem** (LXC 123, claudio-monitor, IPC server)
+now that Hermes Agent is configured and running on mac-mini. Hermes replaces
+claudio-bot's role as the homelab control plane — accepting commands, relaying
+alerts, and answering queries about infrastructure state.
+
+## What is claudio-bot
+
+A three-component ecosystem deployed 2026-04-20:
+
+| Component | Location | Purpose |
+|-----------|----------|---------|
+| **claudio-bot** | LXC 123 (192.168.8.230) | Matrix bot `@claudio:avispero` — slash commands, NL queries, infra notifications |
+| **claudio-monitor** | hubris (PVE host) | 5-min watchdog → sends alerts to claudio-bot's IPC server |
+| **IPC server** | LXC 123:9090 | `/notify`, `/propose`, `/status` endpoints — used by monitor + backup wrapper |
+
+Dependencies:
+- Matrix synapse on LXC 118 (elementsynapse) — claudio-bot connects to `192.168.8.239:8008`
+- Gitea auto-deploy webhook → LXC 123:9797 (pipeline `dtoro/claudio-bot`)
+- LLM backend: LM Studio on mac-mini (192.168.8.174:1234) or Anthropic API
+- Secrets at `/etc/claudio-bot/` (config.yaml, matrix.token, anthropic.key, lmstudio.key, ipc.token)
+
+## What Hermes replaces
+
+| claudio-bot function | Hermes replacement |
+|---------------------|-------------------|
+| Matrix slash commands | Hermes CLI + `send_message` for outbound notifications |
+| NL infrastructure queries | Hermes with `homelab` CLI + MCP tools (already working) |
+| Monitor alert relay | Hermes-native: (a) on-demand `homelab health` via extended skill, (b) cronjob every 15 min with actionable alerts (not just passive notifications) |
+| IPC server (port 9090) | **Retired** — Hermes doesn't need a separate IPC layer; monitoring runs directly via SSH + `send_message` |
+| LLM backend | Already handled by Hermes (OpenRouter → deepseek/deepseek-v4-pro) |
+| Auto-deploy webhook (9797) | **Retired** — repo archived |
+
+## Full blast radius (files referencing claudio)
+
+### Homelab-context repo (`/opt/homelab-context/`)
+
+#### Core topology
+- `inventory.yaml:275` — `claudio-bot:` host entry (LXC 123, matrix-agent)
+- `hosts/claudio-bot.yaml` — auto-generated host file (delete)
+- `.sops.yaml:47` — placeholder comment listing claudio-bot/claudio-monitor
+
+#### Documentation (containers)
+- `containers/123-claudio-bot.md` — full container doc
+- `containers/index.md:17` — LXC table row
+- `containers/118-elementsynapse.md:3,36,39` — `@claudio:avispero` reference, disk reading, direct connection note
+- `containers/103-paperless.md:27` — disk reading from claudio-monitor
+
+#### Documentation (infrastructure)
+- `infrastructure/monitoring.md` — **ENTIRE FILE** is about claudio-monitor → claudio-bot pipeline
+- `infrastructure/backups.md:74,94,98` — IPC notify, thermal monitoring moved out, related link
+- `infrastructure/auto-deploy.md:23,41,50,61,120,138-139` — Gitea webhook target, pipeline table, port 9797
+- `infrastructure/ssh-access.md:134` — LXC table row
+- `infrastructure/homelab-context.md:4,16,143` — client list, integration note
+
+#### Other
+- `operations/commands.md:17` — `pvesh` bulk query note (passive reference, can stay)
+- `README.md:32,40,64` — LXC table, monitoring link, tracked configs mention
+- `mcp/deploy/webhook/webhook.py:5-6` — port comment (9797, 9799)
+- `hosts/hubris.md:57,96` — `/opt/claudio-monitor` checkout, LXC 109 cleanup note (passive)
+- `vms/108-haos.md:30` — claudio-monitor MQTT removal note (historical, can stay)
+- `investigations/2026-04-21-hubris-crash-loop.md:27` — historical reference (keep)
+
+### External repos (Gitea)
+- `dtoro/claudio-bot` — the bot itself
+- `dtoro/claudio-monitor` — the watchdog scripts
+
+### External state (hubris)
+- `/opt/claudio-bot/` — bot checkout (LXC 123)
+- `/opt/claudio-monitor/` — monitor checkout (hubris host)
+- `/etc/claudio-bot/` — secrets directory
+- `/etc/claudio-deploy/` — deploy git credentials
+- `/etc/claudio-monitor/` — thresholds + bot.token
+- `/var/lib/claudio-monitor/` — state files (.alert, .counter)
+- `claudio-bot.service` — systemd unit (LXC 123)
+- `claudio-monitor.timer` + `.service` — systemd timer (hubris)
+- Gitea webhook id 6 (claudio-bot-deploy) + webhook for claudio-monitor (9799) on apps LXC 105
+
+### External state (workstations)
+- Nothing on mac-mini, republic-laptop, or ludo-mini references claudio directly
+
+## Phased deprecation plan
+
+### Phase 1 — Stop & verify (low-risk, reversible)
+
+**Goal**: Stop claudio services, verify nothing breaks, keep LXC 123 paused (not destroyed).
+
+1. **Stop claudio-monitor on hubris**
+   ```bash
+   ssh hubris systemctl disable --now claudio-monitor.timer
+   ```
+   The 5-minute watchdog stops. Home Assistant still gets PVE metrics independently via its own integration — no alerting gap there.
+
+2. **Stop claudio-bot on LXC 123**
+   ```bash
+   ssh claudio-bot systemctl stop claudio-bot
+   ```
+   The Matrix bot goes offline. Nobody can send slash commands to `@claudio:avispero`.
+
+3. **Pause LXC 123** (not destroy yet)
+   ```bash
+   ssh hubris pct shutdown 123
+   ```
+   IPC server (port 9090) and deploy webhook (port 9797) become unreachable.
+
+4. **Verify no breakage** (run for 24-48h)
+   - Check Gitea webhook logs — the `dtoro/claudio-bot` webhook will 404 on pushes (non-fatal)
+   - Check Caddy logs — no upstream errors for claudio-related paths (there shouldn't be any — no public hostname)
+   - Check DNS — no queries for claudio-bot.hubris.network (doesn't have a public hostname)
+   - Confirm Hermes works normally for all homelab operations
+
+### Phase 2 — Hermes-native health monitoring (replaces claudio-monitor)
+
+**Goal**: Two capabilities that together exceed what claudio-monitor did:
+
+1. **On-demand health check** — user asks "how's the homelab?" → Hermes runs a
+   comprehensive check across all hosts/LXCs and reports results conversationally.
+2. **Cron-based watchdog** — runs every 15 minutes, checks thresholds, raises
+   alarms with actionable options (not just passive alerts). User can respond
+   and Hermes takes action directly.
+
+**Key difference from claudio-monitor**: The old system was a bash script that
+POSTed JSON to an IPC server that a Matrix bot reformatted into a flat message.
+The Hermes-native approach gives the LLM the raw metrics and lets it reason about
+what's important before alerting — and crucially, the user can reply to an alert
+and Hermes can SSH into hubris to resize a rootfs, restart a service, or
+investigate deeper.
+
+#### 2a. Extend `homelab-hardware-health` skill
+
+The existing skill covers CPU/NVMe temps. Expand it to be the **single
+homelab health check skill** covering:
+
+- **Hardware**: CPU temps (sysfs), NVMe/SATA SSD temps (smartctl), block device inventory
+- **LXC resources**: `pvesh get /cluster/resources --type vm --output-format json` → disk %, CPU %, RAM usage per container
+- **Service health**: Check key services respond (gitea:3000, caddy:80/443, synapse:8008, paperless, nextcloud, jellyfin)
+- **System drift**: `apt list --upgradable` counts per host, docker image pull digests on apps LXC 105
+
+Thresholds (from old `/etc/claudio-monitor/thresholds.env`):
+
+| Metric | Warning | Critical |
+|--------|---------|----------|
+| LXC disk usage | > 80% | > 90% |
+| NVMe temp | > 60°C | > 70°C |
+| CPU temp | > 70°C | > 80°C |
+| Apt upgradable | > 10 | > 50 |
+| Service down | any | 2+ services |
+
+Update the skill file at `devops/homelab-hardware-health/SKILL.md` with:
+- Expanded trigger phrases ("check homelab health", "how's the lab", "health report")
+- LXC resource check commands
+- Service health check commands
+- Thresholds table
+- "If you find issues, present options to the user" instruction
+
+#### 2b. Create Hermes cronjob `homelab-health-watchdog`
+
+```bash
+hermes cron create \
+  --schedule 'every 15m' \
+  --name 'homelab-health-watchdog' \
+  --prompt 'Load skill homelab-hardware-health. Run a full health check across all hosts (hubris + all LXCs). Check: (1) hardware temps, (2) LXC disk/CPU/RAM via pvesh, (3) service reachability for gitea, caddy, synapse, paperless, nextcloud, jellyfin, (4) apt upgradable counts. Compare against thresholds in the skill. If nothing is critical, say nothing (no message = healthy). If anything breaches thresholds, send a concise alert with: what broke, current value vs threshold, and 2-3 actionable options the user can reply to. DO NOT send routine "all clear" messages.'
+```
+
+This runs every 15 minutes (Hermes cron is LLM-driven, heavier than a bash
+script — 15 min is appropriate vs claudio-monitor's 5 min). When thresholds
+are clean, the agent stays silent. When something breaches, the user gets a
+message like:
+
+> ⚠️ LXC 103 (paperless) disk at 91% (> 90% critical)
+>
+> Options: [resize rootfs to 16 GiB] [investigate what's using space] [snooze 24h]
+
+User picks an option → Hermes takes action (SSH into hubris, run `pct resize`,
+grow the filesystem inside the container).
+
+#### 2c. Add `homelab-health` alias to `homelab` CLI
+
+For quick on-demand checks from the terminal:
+
+```bash
+homelab health          # runs the skill inline via Hermes
+# or equivalently:
+hermes --prompt "Load homelab-hardware-health, run full health check"
+```
+
+### Phase 3 — Documentation & inventory cleanup
+
+**Goal**: Update homelab-context to reflect claudio deprecation, without destroying history.
+
+#### Files to edit
+
+| File | Action |
+|------|--------|
+| `inventory.yaml` | Mark `claudio-bot` entry as deprecated (comment + `status: deprecated`). Do NOT remove — keeps historical ID mapping. |
+| `containers/123-claudio-bot.md` | Add deprecation banner at top, changelog entry |
+| `containers/index.md` | Change status from "running" to "stopped (deprecated)" |
+| `infrastructure/monitoring.md` | Rewrite to document new Hermes-based monitoring. Keep claudio-monitor section as "legacy (deprecated 2026-06-04)". |
+| `infrastructure/auto-deploy.md` | Mark claudio-bot and claudio-monitor webhook entries as deprecated |
+| `infrastructure/backups.md` | Update notification section (IPC path no longer available). Note: backups themselves are still disabled — this is forward-looking. |
+| `infrastructure/ssh-access.md` | Mark LXC 123 as "offline (deprecated)" |
+| `infrastructure/homelab-context.md` | Remove claudio-bot from client list |
+| `containers/118-elementsynapse.md` | Note `@claudio:avispero` is decommissioned |
+| `containers/103-paperless.md` | Remove stale disk reading reference (already resolved — rootfs grew to 16 GiB 2026-05-15 per 118 doc) |
+| `README.md` | Update LXC table, remove monitoring link |
+| `hosts/claudio-bot.yaml` | Add deprecation banner (or delete if auto-generated — it is, so just update inventory) |
+| `.sops.yaml` | Remove claudio-bot from placeholder comment |
+
+#### Files to leave untouched (historical)
+
+| File | Reason |
+|------|--------|
+| `investigations/2026-04-21-hubris-crash-loop.md` | Historical investigation — claudio-monitor is factual context |
+| `vms/108-haos.md:30` | Historical note about MQTT removal — factual |
+| `mcp/deploy/webhook/webhook.py:5-6` | Comment only — harmless, keep for port mapping context |
+| `operations/commands.md:17` | Passive reference — harmless |
+
+### Phase 4 — Purge (after confirmation window)
+
+**Goal**: Full removal after confirming Phase 1-3 is stable for ≥2 weeks.
+
+1. **Destroy LXC 123**
+   ```bash
+   ssh hubris pct destroy 123 --purge
+   ```
+   Frees 8 GiB rootfs, 512 MiB RAM, 1 core.
+
+2. **Remove claudio-monitor from hubris**
+   ```bash
+   ssh hubris rm -rf /opt/claudio-monitor /etc/claudio-monitor /var/lib/claudio-monitor
+   ssh hubris systemctl disable claudio-monitor.timer  # already done in phase 1
+   ```
+
+3. **Archive Gitea repos**
+   - Archive `dtoro/claudio-bot` (mark read-only in Gitea)
+   - Archive `dtoro/claudio-monitor` (mark read-only in Gitea)
+
+4. **Remove Gitea webhooks**
+   - Remove claudio-bot-deploy webhook (id 6) from Gitea
+   - Remove claudio-monitor webhook (9799) from apps LXC 105
+   - Remove `192.168.8.230` from Gitea `app.ini` `ALLOWED_HOST_LIST`
+
+5. **Final documentation cleanup**
+   - Remove `containers/123-claudio-bot.md` (or move to `containers/_destroyed/`)
+   - Remove `hosts/claudio-bot.yaml`
+   - Remove claudio-bot row from `containers/index.md`
+   - Update all cross-references (see blast radius above)
+
+## Files changed by phase
+
+### Phase 3 (documentation)
+- `/opt/homelab-context/inventory.yaml`
+- `/opt/homelab-context/containers/123-claudio-bot.md`
+- `/opt/homelab-context/containers/index.md`
+- `/opt/homelab-context/infrastructure/monitoring.md`
+- `/opt/homelab-context/infrastructure/auto-deploy.md`
+- `/opt/homelab-context/infrastructure/backups.md`
+- `/opt/homelab-context/infrastructure/ssh-access.md`
+- `/opt/homelab-context/infrastructure/homelab-context.md`
+- `/opt/homelab-context/containers/118-elementsynapse.md`
+- `/opt/homelab-context/containers/103-paperless.md`
+- `/opt/homelab-context/README.md`
+- `/opt/homelab-context/hosts/claudio-bot.yaml`
+- `/opt/homelab-context/.sops.yaml`
+
+### Phase 4 (destruction)
+- Proxmox: LXC 123 destroyed
+- Gitea: `dtoro/claudio-bot` archived, `dtoro/claudio-monitor` archived
+- Gitea webhooks removed (apps LXC 105)
+- hubris: `/opt/claudio-monitor/`, `/etc/claudio-monitor/`, `/var/lib/claudio-monitor/` removed
+- `containers/123-claudio-bot.md` → moved to `_destroyed/` or deleted
+
+## Risks & tradeoffs
+
+| Risk | Mitigation |
+|------|-----------|
+| **Monitoring gap** — no alerts during Phase 1-2 transition | Phase 1 stops the old timer but Phase 2 builds the Hermes-native replacement immediately. If there's a gap, keep claudio-monitor.timer running until the Hermes cronjob is verified with a test alert. |
+| **Backup notifications** — the backup wrapper's IPC notify path is dead | Backups are already DISABLED. When re-enabled, wire them to the new Hermes monitoring pipeline instead. |
+| **Matrix `@claudio:avispero`** — users may still try to interact | The bot just won't respond. Could leave a notice in the Matrix room. |
+| **Gitea webhook failures** — pushes to `dtoro/claudio-bot` will 404 | Non-fatal. Archive the repo in Phase 4 to stop pushes entirely. |
+| **Losing monitoring history** — state files on hubris | Keep `/var/lib/claudio-monitor/` until Phase 4. No rush. |
+| **LM Studio dependency** — claudio-bot was using mac-mini's LM Studio | Hermes already uses OpenRouter. LM Studio on mac-mini can stay for other uses or be reclaimed. |
+
+## Open questions
+
+1. **Should the Matrix `@claudio:avispero` account be deactivated or left as-is?** It has no public hostname and was invite-only to a private room.
+
+2. **Are you OK archiving `dtoro/claudio-bot` and `dtoro/claudio-monitor` on Gitea (read-only, preserved), or do you want them fully deleted?**
+
+3. **Should LXC 123 be paused (reversible) or destroyed immediately?** Recommend pause for at least 2 weeks as a safety net.
+
+4. **Which messaging platform should the health watchdog deliver alerts to?** Currently Hermes is connected via CLI (mac-mini). To get push alerts, we need a `send_message` target — Telegram, Discord, or Matrix are available. Which do you prefer?
+
+## Verification
+
+After Phase 1:
+- `ssh hubris systemctl is-active claudio-monitor.timer` → `inactive`
+- `ssh claudio-bot systemctl is-active claudio-bot` → `inactive` (or unreachable if LXC stopped)
+- `ssh hubris pct status 123` → `stopped`
+- `curl -s http://192.168.8.230:9090/status` → connection refused (IPC dead)
+
+After Phase 2:
+- `hermes cron list` shows `homelab-health-watchdog` running every 15 min
+- "check homelab health" → Hermes runs comprehensive check, reports temps + LXC resources + service status + apt drift
+- Force an alert: `NVME_WARN_TEMP_C=1` on hubris → watchdog catches it on next 15-min tick → alert arrives via Hermes messaging with actionable options
+- Alert format includes 2-3 options user can reply to, not just passive notification
+
+After Phase 3:
+- All cross-references in homelab-context point to deprecated/removed status
+- `grep -r 'claudio' /opt/homelab-context/ --include='*.md' --include='*.yaml' | grep -v '_destroyed\|investigations\|changelog'` → only historical references remain
+
+After Phase 4:
+- `ssh hubris pct list | grep 123` → no output
+- `ssh hubris ls /opt/claudio-monitor` → no such file
+- Gitea `dtoro/claudio-bot` → archived badge

plans/index.md

@@ -0,0 +1,16 @@
+# Plans
+
+Pre-flight runbooks for planned changes that haven't happened yet. Once executed, move the outcome to `investigations/` (if anything interesting happened) or just a changelog entry on the affected node pages.
+
+## Index
+
+| Date | Title | Status |
+| ---- | ----- | ------ |
+| 2026-06-01 | [Slate AX → SODOLA managed switch migration](2026-06-01-slate-ax-to-sodola-migration.md) | Done |
+
+## Conventions
+
+- File name: `YYYY-MM-DD-<slug>.md`. Use the *target* date if known, otherwise the planning date.
+- Status: `Planned` → `In Progress` → `Done` (update index entry as it moves).
+- When done: add a changelog entry on every affected node page, then update status to `Done` here. If things went sideways, open an investigation.
+- Plans are append-only once execution starts — don't rewrite pre-flight intent after the fact.

scripts/check-caddy-backends.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+# check-caddy-backends.sh — validate all Caddy reverse_proxy targets are reachable
+# Run this on hubris (192.168.8.77) or any host on the homelab LAN.
+# Returns non-zero if any backend is unreachable.
+# Uses curl with a short timeout for reliable TCP checks.
+set -o pipefail
+
+CADDY_HOST="192.168.8.175"
+
+echo "=== Caddy backend connectivity check ==="
+echo "Date: $(date)"
+echo ""
+
+TMPFILE=$(mktemp /tmp/caddy-checks.XXXXXX)
+trap "rm -f $TMPFILE" EXIT
+
+# Extract unique IP:port targets from Caddyfile
+ssh root@"$CADDY_HOST" "grep reverse_proxy /etc/caddy/Caddyfile \
+  | grep -oP '\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d+' \
+  | sort -u" > "$TMPFILE"
+
+TOTAL=0
+FAILED=0
+
+while read -r target; do
+  [ -z "$target" ] && continue
+  TOTAL=$((TOTAL + 1))
+  # Use curl with 3s connect timeout for reliable TCP check
+  if timeout 3 curl -s -o /dev/null --connect-timeout 2 "$target" 2>/dev/null; then
+    echo "  ✅ $target"
+  elif timeout 2 bash -c "echo >/dev/tcp/${target/:/\/}" 2>/dev/null; then
+    echo "  ⚠️ $target — port open, no HTTP response"
+  else
+    echo "  ❌ $target — unreachable"
+    FAILED=$((FAILED + 1))
+  fi
+done < "$TMPFILE"
+
+echo ""
+echo "Checked $TOTAL targets, $FAILED failures"
+exit $FAILED

scripts/dns-sync.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""Sync Technitium hubris.network A-records -> NetBird managed DNS zone.
+Technitium is the single authoring source; this reconciles NetBird to match."""
+import json, urllib.request, urllib.parse, ssl, sys
+
+NB_API   = "https://netbird.hubris.network/api"
+ZONE_ID  = "d7gaad00qfrc73er3n1g"
+NB_TOKEN = open("/opt/dns-sync/netbird-token").read().strip()
+TECH     = "http://127.0.0.1:5380/api"
+TECH_PW  = open("/opt/technitium/admin_password.txt").read().strip()
+
+ctx = ssl.create_default_context(); ctx.check_hostname=False; ctx.verify_mode=ssl.CERT_NONE
+
+def jget(url):
+    with urllib.request.urlopen(url, context=ctx, timeout=15) as r: return json.load(r)
+
+def nb(method, path, data=None):
+    req = urllib.request.Request(NB_API+path, method=method)
+    req.add_header("Authorization", "Token "+NB_TOKEN)
+    body=None
+    if data is not None:
+        req.add_header("Content-Type","application/json"); body=json.dumps(data).encode()
+    with urllib.request.urlopen(req, data=body, context=ctx, timeout=15) as r:
+        t=r.read(); return json.loads(t) if t else {}
+
+# 1. source of truth: Technitium named A-records
+tok  = jget(f"{TECH}/user/login?user=admin&pass={urllib.parse.quote(TECH_PW)}&includeInfo=false")["token"]
+recs = jget(f"{TECH}/zones/records/get?token={tok}&zone=hubris.network&domain=hubris.network&listZone=true")["response"]["records"]
+source = { r["name"]: r["rData"]["ipAddress"]
+           for r in recs
+           if r.get("type")=="A" and not r["name"].startswith("*") and r["name"]!="hubris.network" }
+
+# 2. current NetBird records
+cur = { r["name"]: (r["id"], r["content"]) for r in nb("GET", f"/dns/zones/{ZONE_ID}/records") if r.get("type")=="A" }
+
+# 3. reconcile
+c=u=d=0; changes=[]
+for name, ip in source.items():
+    if name not in cur:
+        nb("POST", f"/dns/zones/{ZONE_ID}/records", {"name":name,"type":"A","content":ip,"ttl":300}); c+=1; changes.append(f"+ {name} {ip}")
+    elif cur[name][1] != ip:
+        nb("DELETE", f"/dns/zones/{ZONE_ID}/records/{cur[name][0]}")
+        nb("POST", f"/dns/zones/{ZONE_ID}/records", {"name":name,"type":"A","content":ip,"ttl":300}); u+=1; changes.append(f"~ {name} {cur[name][1]}->{ip}")
+for name,(rid,ip) in cur.items():
+    if name not in source:
+        nb("DELETE", f"/dns/zones/{ZONE_ID}/records/{rid}"); d+=1; changes.append(f"- {name} {ip}")
+
+print(f"dns-sync: {len(source)} source records | +{c} ~{u} -{d}")
+for ch in changes: print("  "+ch)

scripts/sync/linux/homelab-context-sync.service

@@ -1,13 +1,12 @@
 [Unit]
-Description=Pull /opt/homelab-context from Gitea
+Description=Pull /opt/homelab-context from Gitea + auto-setup tooling
 After=network-online.target
 Wants=network-online.target
 
 [Service]
 Type=oneshot
-ExecStart=/usr/bin/git -C /opt/homelab-context pull --ff-only --quiet
+ExecStart=/usr/bin/env bash /opt/homelab-context/tools/post-pull.sh
 TimeoutStartSec=60
-# Don't fail aggressively — a missed pull just retries next tick.
 SuccessExitStatus=0 1
 Nice=10
 

scripts/sync/macos/network.hubris.homelab-context-sync.plist

@@ -7,12 +7,9 @@
 
     <key>ProgramArguments</key>
     <array>
-        <string>/usr/bin/git</string>
-        <string>-C</string>
-        <string>/opt/homelab-context</string>
-        <string>pull</string>
-        <string>--ff-only</string>
-        <string>--quiet</string>
+        <string>/usr/bin/env</string>
+        <string>bash</string>
+        <string>/opt/homelab-context/tools/post-pull.sh</string>
     </array>
 
     <key>StartInterval</key>

secrets-issuance/backup.service

@@ -0,0 +1,9 @@
+[Unit]
+Description=Snapshot secrets-issuance state to /mnt/library
+After=network-online.target
+
+[Service]
+Type=oneshot
+ExecStart=/opt/secrets-issuance/secrets-issuance/backup.sh
+Nice=10
+TimeoutStartSec=300

secrets-issuance/backup.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+# secrets-issuance backup — snapshot the per-client age keys + denylist +
+# admin token to /mnt/library so they survive an LXC-105-only failure.
+#
+# Destination is under /mnt/library/.secrets-issuance-backup/ (dot-prefix to
+# stay out of any indexer's path). Mode 0700 root-owned. Privileged LXCs
+# that mount /mnt/library would still be able to read it as their root maps
+# to host root — accept that trade today; an encrypted-tarball variant
+# (separate offline age key) is a follow-up if the LAN trust model changes.
+#
+# Retention: 14 daily snapshots. Each snapshot is the WHOLE /var/lib/
+# secrets-issuance directory (small — ~few KiB per client), so restore is
+# just `tar xf <snap> -C /`.
+
+set -euo pipefail
+
+SRC=/var/lib/secrets-issuance
+DEST=/mnt/library/.secrets-issuance-backup
+RETAIN=14
+
+if [ ! -d "$SRC" ]; then
+    echo "[backup] source $SRC missing — nothing to do" >&2
+    exit 0
+fi
+if [ ! -d /mnt/library ]; then
+    echo "[backup] /mnt/library not mounted — can't write backup" >&2
+    exit 1
+fi
+
+install -d -m 0700 "$DEST"
+chown root:root "$DEST"
+
+stamp=$(date +%Y%m%d-%H%M%S)
+out="$DEST/secrets-issuance-$stamp.tar.gz"
+
+# Include the admin token too (recoverable separately, but trivial size and
+# saves a step when restoring after total LXC 105 loss).
+tar czf "$out" \
+    -C / \
+    var/lib/secrets-issuance \
+    $([ -f /etc/secrets-issuance/admin-token ] && echo etc/secrets-issuance/admin-token || true)
+chmod 0600 "$out"
+
+# Drop oldest beyond retention.
+ls -1t "$DEST"/secrets-issuance-*.tar.gz 2>/dev/null \
+    | tail -n +$((RETAIN + 1)) \
+    | xargs -r rm -f
+
+count=$(ls -1 "$DEST"/secrets-issuance-*.tar.gz 2>/dev/null | wc -l)
+size=$(du -sh "$out" | cut -f1)
+echo "[backup] wrote $out ($size); kept $count snapshots"

secrets-issuance/backup.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Daily secrets-issuance backup
+
+[Timer]
+OnCalendar=*-*-* 03:30:00
+RandomizedDelaySec=10min
+Persistent=true
+Unit=secrets-issuance-backup.service
+
+[Install]
+WantedBy=timers.target

secrets-issuance/deploy/deploy.sh

@@ -36,6 +36,11 @@ install -m 644 secrets-issuance/server.service \
                /etc/systemd/system/secrets-issuance.service
 install -m 644 secrets-issuance/deploy/webhook/secrets-issuance-deploy.service \
                /etc/systemd/system/secrets-issuance-deploy.service
+install -m 755 secrets-issuance/backup.sh /usr/local/bin/secrets-issuance-backup
+install -m 644 secrets-issuance/backup.service \
+               /etc/systemd/system/secrets-issuance-backup.service
+install -m 644 secrets-issuance/backup.timer \
+               /etc/systemd/system/secrets-issuance-backup.timer
 
 if [ ! -d /opt/homelab-context/.git ]; then
     echo "  /opt/homelab-context is not a git clone — run bootstrap.sh first." >&2
@@ -50,6 +55,10 @@ if systemctl is-active --quiet secrets-issuance-deploy.service; then
     systemctl restart secrets-issuance-deploy.service
 fi
 
+echo "[deploy] daemon-reload + enable backup timer"
+systemctl daemon-reload
+systemctl enable --now secrets-issuance-backup.timer
+
 echo "[deploy] done"
 echo "First-time enable:"
 echo "  systemctl enable --now secrets-issuance.service secrets-issuance-deploy.service"

secrets-issuance/server.service

@@ -8,7 +8,7 @@ Type=simple
 WorkingDirectory=/opt/secrets-issuance
 Environment=HOMELAB_CONTEXT_DIR=/opt/homelab-context
 Environment=SECRETS_ISSUANCE_STATE=/var/lib/secrets-issuance
-Environment=MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10
+Environment=MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10,192.168.8.0/24
 ExecStart=/opt/secrets-issuance/.venv/bin/python /opt/secrets-issuance/secrets-issuance/server.py
 Restart=on-failure
 RestartSec=5

secrets/gitea-pat.yaml

@@ -0,0 +1,53 @@
+user: ENC[AES256_GCM,data:e6Dj7CE=,iv:Bf4tvdWQr0vdF3PShG2cRTryxqRC5rN6UdGd1qwFg7g=,tag:22rrPMAF97C2GUhRGgutOw==,type:str]
+token: ENC[AES256_GCM,data:98bMZO2t0qgOO/rTYmyDBZdRTHu8iPgUsctS4dc0B2BCML6tBxBNhA==,iv:oXehq2FJy1pQxzK8M3LqDtjpoFbdfAPvYcKhygwf51A=,tag:RMYdimBwxw247OARznBu3g==,type:str]
+scopes:
+    - ENC[AES256_GCM,data:S3nz8buLTp1PCHpf8F3Y,iv:IkHU+QwIlsxrNrGEu3tRS0zlnuCyp9Fs8ILxUaZrabw=,tag:Ufd+MVYXAAtbSbfvRi3jCg==,type:str]
+    - ENC[AES256_GCM,data:/UG/8LQD21l/Ca0wfEodKw==,iv:f3tW1ZNLLkzE5Oy8acQ6GfyDvzEH3++kp8duBOI8+Is=,tag:IgfhvEaX9EGyPHhr6FSAjQ==,type:str]
+note: ENC[AES256_GCM,data:vRDjHXh5fYlhwJ4LyS/hcqMGdS+Z/D1W2oH6NaJak/XRUr+nFVQqcXSH/zUgqEBWNrMv6j8n2Ouxex/q2mr+p3Ds1jfThJTobX/MFqMkxvxk0TmB7ucui7CDa4mcVgSo3wyhcaba2o3erObByZ3o2uay3mmqzAUdo/wy2bCgQTkTG9/xqVSf3EYuR1j/A+SAzSvpsOORMNUjC5NRP1DUL+3hZkPXsfkXuy7sV6T2vF+oOrbEI0WxR8o4/JcF1B/4ezFAJ+f+nukCMz6eSCwur/wHffbcYa8mtD6BIRuV1pTd/Zob8arhe2ir19MB0fR27E3UkcCwyCkQ+QWuGzisvciCPtwR4JidH7xOpPW+4AK3vqEjmpEeMSWlyxbZadQFuSi8f3bPH+8eB2h/UFvT0BU6xw==,iv:r2dUnKyUWhyafIeDIjHOdqh+PNFzGtVZfcHc2TAhvvI=,tag:Cmg6esU4mQ8/9UbIQQL/gw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXaWJhdFAxdThGSVZRN2I1
+            VjE1djFtdjdnSFVJU3lMZC9YSzNSeGVLaVJBCjlId3pvcFRadUdwamxaa00zNUVl
+            VUJFaDRRLzdSNDNVekR3eWdmREh3eUEKLS0tIEVMMmh2eUlSL0FhRzNnZXJyWk9I
+            MEY3dlVDQktRR1VUajM4WmVwYS80TVUKA0bkns4IE093PvF5Ka3HNWmi+Htqs66H
+            BBEAETnQZOdM/Ca+oySDnhLYYT4rD46m4d3H0YQfXQhlVk/h9CTshQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5bVZma09OVWkrVmtVNEda
+            MnRxdDZuRTV3RmhieTB2ZXVTcjY2VWtTOW5rCkw2STNWTHluRGZhT3B4TmtFWHUv
+            L2hyd3QwMW1iaHRCRWMycmxHVW5qMlEKLS0tIHNrTjdYb3NCdU05WjR3RFdvdFBR
+            Z3NRRVRRcnBhdXVmRmprbk1iL0lWS28KTvN7Z3VyXKCnbJBD9N+FtC5UDCxofPIU
+            WERW/eV7+2F6SeClSu7iE+pDMbjiKrFBVKBAU8OA8yd8VmcnD8IWdw==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBublRwajRDeUtmdmtpN3pj
+            VVE0VWJhVWRXeE1UM0xHdEJKUVhHVWpNTG00CnZjNUg3ZkUvcFZTRHc4UDMrOVhV
+            QTNPMmphTkhxZjgzNXN6REtTTTZUU00KLS0tIEk0MXFBNS8wVmpMZFhoYUpsOGlG
+            ZE1OaXdhZVVGQXUrdUhJT09UU21DNkUK4DcLYPqf3ojRotaa95KIUhKYQwWorsum
+            IletEBZtyJdOJgpwN/eZLe12a4E5thI245jSjQlw2B8RKFqOCDyASw==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPVEdGR0VwUjk3MUROOHRJ
+            aXdFMTFDRFZBek5FSWJSbWJDU2F2ODFyTVFzCndTbStrQmFSSWJwbFc0M1Nia2FS
+            eGZEMGsrYkNnT3M2aEhOS0IzOVNOVncKLS0tIENtOEx4SVJQTFcxR3NIM0NHVDI5
+            Z1RIMWRBNGhCOFF4MkdHRlpya25ybGMKsyLYsmqxuXvJ4ZF97Jh5D8BoepehSdKi
+            yGzLaaQo0gW/wu7n0fq7S7HhbiTcPZ8lQboVvhYWU2lTx1p8npUsAQ==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-05-20T16:24:18Z"
+    mac: ENC[AES256_GCM,data:d3qcjEqpPIFeut634ImvISJLdit0MQWcdFYomrsqriqJ0NUKdyq3XCk85+vvPvDWikB4WEApk6HMXssSY8mhDci95q5Ssmr+JRAhLebMZjs9yXUf9A7vXpNFswmdldu3CKBiDrhm2GE08qUfsCkcf3jjihEqsfHhxMuWSd5fZpU=,iv:VZgvZ3bBArRChgRX38U6/43XoX5aTnts7Kd4S9spBPk=,tag:/5oOw5yvulvFT6KXv2zj0g==,type:str]
+    pgp: []
+    unencrypted_suffix: _unencrypted
+    version: 3.9.4

secrets/hello.yaml

@@ -0,0 +1,49 @@
+greeting: ENC[AES256_GCM,data:eNMtZcpv04PHl/lGBZGBp7xAcO/5Sw==,iv:A4jtwePX0p/eDTP5+ztfV6gege9RwdPfLBY2N+DPfx0=,tag:MU8HpvLt/PUrNMw7BNUVIg==,type:str]
+phase: ENC[AES256_GCM,data:oIeo+kmkwaMDQrf10iY2PqOXJnq+hCIwOROQ86xgpkU7EJEeJKFssrNgbvQPHhIRZUZblQg9ZPamiZixQbYSTOXQSTWWu2FOnu6XFb24wCay,iv:TYKqoHcN+V324/roNopYhkwRpmv3cPrTFcmni5TajYQ=,tag:g8DXqEyORbQZ9B7PaFWgOQ==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUanM5MzFZYnkxVHVHSlYy
+            U1Q1ejNJb3ZORklKM2xjK0p2cytRU0NNRUZJClBTUDJ5dFM2eUFjV0t1YzBCcXcv
+            cmE3czJST1RuMHJVN0tmVTZNdGNRS1kKLS0tIFUydktvK0tBU2U3czRhWHVySGZR
+            ck9KTVR6RS9pOHQ4WkZneG1Tbk5OSWMKwOQ+CWkuLSqfhle9fgDw4XXIp0ojZssw
+            9YWK/suEAb6u9nzbw7zuEmZxhhDV0Y61UAeSSbSmygy7MD7dvxrvgQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCbm5MOG1Bd0FZbVZTbUU1
+            VGpGVWNnUk03RURPNURNNjdQejY2WkxkNzBRClpuMGFaQzZhaENJckZvbUI3VmUy
+            ekV0eFMza1p4S2w5L0hvNndjN3RpdTQKLS0tIDRTdExxTlZQK0FKV294dnh4dmd4
+            a21yK0VabzhOcDhRajlxbVE1dTdtcWcKQorqkwUs7lppaDJuCDn+KtH+76xPIvsr
+            9axkrDLumeY9LtDhunlaEpax7zdvvAiC4DzJdbJX0LgFCaurtgubOA==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaS1JrL3lpa1c4QUwxWlZT
+            VWQ1ZDVPVjN6QTdFNGQzNkV5WllOK3dYWHhzCllTbVdzRVFTbDZ4TU9PZWpKNnJu
+            cWJ6YWs5amVsR2xZZFpaQlErOXBKZVEKLS0tIFFzVG9QWmdNWGl5UEkxaXZvMUxx
+            SExXL1gxNzVud2JrZ04zTnI4cUlNSVEKPF2zoSnYEt/zeG8QW1454Mcr8u8JTCl7
+            jDyghcRw94enbPbA43zsKGn6QALQ40PfXmZ/MlEbdnf6U3zI7zvygQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwMlVSajFiT2loclFzeEQ5
+            ekxqUjBkSEh5M2FJMXA2VGkycllYa1R1Qm40Clh0L0RwSzZqeXlQVXMvRVlsZ05L
+            RHkyMlJlc1BmV0tjQk1uaWJLemR2cmMKLS0tIDlXT1dhRkpUdDRpRkJhL2NHSjN5
+            QU5Pc25lSUV4OWROZENXQzB4Y0o2Y1kKzhOY5jN4gi+u5tl9rAHGRb8Bh9DESl1K
+            qNiD+wauApal8iRrIkGdGsrWrWThSwy7vjnUsoB8JDrBaqLj2usuFA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-05-20T14:39:14Z"
+    mac: ENC[AES256_GCM,data:CAoIW1sJkIbNoXMaVlXwfHdREzATNqDKpSN5ecaSiRARThB0/tcLNbkazXKyV1Rx5b+b3s+yCpBzwmNnH6ytaCzG5pKPcnfd6YJq7C5rm/EQHyaVGrPdBsSfqiz4yBXKClej//v3+qkD5Ls2PMppv5KmqigOKNQC/ot3ht2c7lM=,iv:eOmDDclksr9f9CDu6dJprSFwlWuoggzVFjyfqWf5Uhg=,tag:IU3sMlIi92ohM86zOIJdAg==,type:str]
+    pgp: []
+    unencrypted_suffix: _unencrypted
+    version: 3.9.4

secrets/netbird-authentik-oidc.yaml

@@ -0,0 +1,48 @@
+client_secret: ENC[AES256_GCM,data:p8csBIMAuDb1BYLLL9aYIifShpw9uIWm1Z0c59SlzBDP3SJD4I8Z6ipYBETisgxcECr+78zs2wtuebtcHZ3DBqF5PZaG87TJEGotTkjOVSjDLKgelMq0TDfnpWyR1yedsi6/SiRhm4ggTz0Vwwo+TLDNkw3Gv7EP7fZQphYdMyA=,iv:MZcvhiBwDc8FWcNqQvCiTL+9OVEfHpZrzt1kmcKcxdA=,tag:FmEIrw7u2wYiF3ksymC5lA==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvdUxUdzd2VmN1MWU2dXdM
+            N0FWcXkveVdHYk5aYzFrOVZaRzVYZWs1RFZBCkFwRy9xVFBrVjU0NmhkMlpyN0pp
+            ZlZKenhoNFVRdGg4Q0ZTMS9YUDFKMHcKLS0tIG1GYkZaYkhJNk5zdWZBMEtNL041
+            MWhEQ1RmQ1k0d2Y2dzh2YnRvRm1IOEUKJ64/tNyLe2qBIL4CetRlpLaxhfOL93D5
+            BjeHLwtfXB2fNCJSdCY8H4KsQP+epwriUQPtZVId32n9xDEt71obSw==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRNi9kMkt5Vk4rS0pqY2ZP
+            ODErMlRxNmlWaHBvYjNaQ3lRT1RSQitMVlFBCk5HQ2o0QUg4dWFodTNEVGJFTlF0
+            Y2JIVEFPZGVUdXpNS2lGOVQzeFRvMFEKLS0tIGtQY0dicE91R0J3eEM3bEdMZHJ4
+            TndiYVFQWGZyRkF1bWYrRlZ6N0JBRm8KEUKY3chev13KjnGKdTR8tvyYV3s0W1rI
+            8LeJSIocSX58PexqgcKsT8pGpuIiOetEzjzv2WPa0MzEwtFCVgwKYQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFcktzeGx5VWZaeFVNc2Mv
+            aWt1dzNUVko5YytOTTdxMWVEVm85N29TMDBrClc3ODdVVDRVRFI3K3BiUGd1TFp1
+            SE5nalY0RkwzNmZQOWU1THhGQ3Q0UkUKLS0tIHNZdHhxdDY0a1AyY0g3WUx3bUJh
+            U0RqRWRFRXNLTWFwemNOYk8vTmxZeTAKaZI6WSNM022xtZQx5yOYXj5mesAjWsu/
+            ZIDG6MayoyrhwAkuZsWOKtKp6zbRCcnm7s9OHdlNsx7PWG1ODEFpOQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVbFFxSkhscUV2bm93Z01O
+            WDl3MU11K2lrNzAwUVN5MDhodDlveFZ6QlYwCjNaNjd3bG9QMTQxcVA5TTNQU3Za
+            bWdZTDdGZzJObnJ6NlBxKzVYcmpVNk0KLS0tIEJTcFVTRmkxQlJWSzVIM0ErOVNN
+            c290WkI1ZzBlYkg3QzJJUlQwOFZPYmMKQFp4vqFwh0GpYOur4kGMCeENnuNZnN0x
+            sbNmM9rRm/10T4tHF9aR2/WIjEqm6M6+yXd3phhB8rIggDPlVk2bCg==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-05-21T20:13:20Z"
+    mac: ENC[AES256_GCM,data:bi4o17skIOtZoxJGzLFJ7IAv+X265qIhgnQz6wr4bch2xofwPZZfzECck0I0xDhcwxcDtx3VZ5E2VBdvHWk+iSan+clRq7735k9+DIpjdWaGxIHuylmelZl6aE5kE36UO7fQdnwmiqCPHxiVq/sKuqknFXkl1+FvO1ryrtSHQsU=,iv:xxm+5rK7o2c43iGS1j2Q274QCRsbDn+k+wskpp5tzeU=,tag:sZHk9eth/TwqM8tIWSrv3w==,type:str]
+    pgp: []
+    unencrypted_suffix: _unencrypted
+    version: 3.9.4

secrets/netbird-pat.yaml

@@ -0,0 +1,36 @@
+#ENC[AES256_GCM,data:yB1EyrABkGl4G+Mq67djNXCuySBS1TDZjKwtko8hfZi15br6+L9dlwhVTHrzvHAdGYOvkZu/dCssii0kKXC+3QeJws+SfwUV,iv:r7bb8QjkyfeaATPKN3VzO4H+ykQTyUKRMxoE9bmGo+E=,tag:TA0actRKQmYsgCMAcvQkjA==,type:comment]
+#ENC[AES256_GCM,data:oVH2uanu0pyv7B0SKiaSKfAXWmKAimXKR2mHKi/+hvFIQGzf3mYTZ5zGMQdBCLoy,iv:pgQrEqMZmynEHfIIHHi52xSNwn0/uQ2ccnK0T9T/Z7U=,tag:8hdflKb6l7HlqHTto8eEyQ==,type:comment]
+netbird_pat: ENC[AES256_GCM,data:cz9pdu7aLR+vKV9QbQ2PtKWf5j5PWxcHEWZ8IEseHS/qPdR4M8282g==,iv:3MCOU0q/7nqQCD6+7qWq++XR9uHi5QHZoenCIT2aYIk=,tag:m4ZE4pDycjwGxoLi1Li1bg==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpYVlocHFacWxtTnhPeTV1
+            SnZqVUUwbXVId0dKNTdmaVQzSUcrT3gxYTA0CloyYm1oQlU3QXVUNVIxTXpGQkN6
+            cTBxTHViZ0E5bEF0dmx4S3l5WVJZOW8KLS0tIEQ5UmJGZG5aVDhGblFDREc5M1dR
+            ZlJ1UEVLeWFsQm0zQ0Q1Z2pTbEpFenMKsw+gfYZ+sby1crIyfoy+yGIaQ2WZ2KYU
+            YgsXY4Vt5b5DC0u/sHKaqXH3bw8ooBQ7tR4WX3JvU6emIAJU0rAuwg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaM3NXTExiRzZ0VWdRR1RH
+            RUlBVVBTWFpUQWtCTnc2T0VqMDJITVFiOFNnClZRbElxRFlGb1orbThmdGI3dFRM
+            V1Jqck9pRzVjWGhVQlIvVUFLMXkyT1EKLS0tIGY5K3g3VDJjUHpjSTNrOTRBUEpz
+            dHBmQ0FpeHRSc1ZyMjJHdU1pM1FzbFEKqIJ0qC0SY0aSceh0ZhGVDMb/WCkh1JZZ
+            E3cP7r4Sa6rMZXRR4u23wTHnrckdHgwxSSXYYymy4ilXhgiFQ3ZTBA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhSElYV3dkMU5hR1Vnb0tl
+            L1R0SG0zeFRuZUd2aTVVaU1leU14bHJjVWl3CjhibVp4NWFsazRYWkl4R1NWTUdk
+            TFdrOXc2OWY0aWxUb3NqM3R1T2xVVEkKLS0tIHlUZGUrRmM5VVVTOEZNYTBSZ2s0
+            VWVTdXlZcmJqeU4xc2NLSDY3MWNUUVkKix4IZEbRI7icZciI7bYzIvDiW4M3Nb7O
+            uopkLc9bC/E+qK9H7jfySkk/zIeW1s/s7fBEKB5utUVpvmKzUB24pA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
+    lastmodified: "2026-06-03T19:57:46Z"
+    mac: ENC[AES256_GCM,data:zez2CaP1NPJJnUbEWT58If6jKxFiAudRRPKVJCzbjtp0loFohvOHt1nFY3g0XldqkNngsSZ83GOeJ9g+CvdD7/plkmjz0C04uHWahnNhjAwNRiN32xt31XFqEejScOHW4dbOunX8wXuQ0Q4SrSLTl5G9O/qbLwlwEiqp8KxwRWA=,iv:ctqQ9TDUM65POxn4v1iILubygU2lloCLz9u0bAP6gsY=,tag:05FpkqIhPRMX5hYopFNK2Q==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.13.1

secrets/openrouter-api-key.yaml

@@ -0,0 +1,34 @@
+api_key: ENC[AES256_GCM,data:TbPkuLCidS7cg52DYb0MOCui3TsaiwkOWyYLhp6gcvXI8TqvXej1tNsCF0TNAIr9vvnFaYno/UTEPiOKe/EQqR8uQMyWuvsmMg==,iv:bahy9ae4Qxvkv1OAjx6LwzK5ggqDh2GVHDj95PVP1Mc=,tag:J7PajLaeqFOdz3FESYZmSQ==,type:str]
+sops:
+    age:
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3djYvT3JJUWh3cE1ZbFc0
+            SiswOGlKNEdFNlJhTVlVSkhqZGlJdGl4U2pvCnV0aEhud0VaOHVGbkVVVkJ3b2Js
+            K1liaEFscGJxMXVVNjNJNW9NeGR6Um8KLS0tIEt2TlFCV09CWDJKbmFySm1EUXc3
+            Zm9IT21QN0lkSlh0UVdhYkVrTDdiUWcK8IO+gylIYKDlSADOtj2gSpM7Af/JG/7Q
+            6y1dT2d82cEu7KuXXuog6gP3sADe/6SDHGgC6Ot6EWM+5dwMN6WwTA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKY2xVSUVBUTd0dVhxekhY
+            d2t1U00xNnRjTC9PT3puVGo1U2ViekhmQng4CnJlZWYxK0tyQ2U0Y3lXRUZSTEti
+            cU9QdWhjalJhb3dJSk4vK1hWZGFGNEkKLS0tIG51aFRGcW9xdmdmSkt0TTZXSll4
+            dkJMYjdPbUgyL1Rmc0tQbHgyR3RLZTAKllFoX3m/zntAtfGkSGFmRzXuk4pHALkR
+            XeQaAl33n5dMiZHtynoDNN3eBXtDDWiKZAhZeWI5SMoLpWBEXapNag==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsdUpxd0RYRld3L1BYdllp
+            R3RwVStLUysyR1lXQ0RlcW4vWGRINzB3WUd3CktTSXJzUWsxcEdVajVjS3N1d0NE
+            WHU4QkVvU3dyYWZ0VVpmYnB2blVWaGcKLS0tIG5sa1I2YW1tS0hLeGc1SWtjV0NM
+            TkFoL0ZZbmhXdElFNytLclByK0s1LzgKVrBvlaryZXYn43v1ukjGZnhSPuwfslf7
+            Ruy7z7EzVaCXRn+F9gABD4mhIQaUIRswiut7aF9lrPOR72/y8s/1JA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+    lastmodified: "2026-06-01T21:23:34Z"
+    mac: ENC[AES256_GCM,data:DXvUZUOMKQA0aDpN1foKy+aHhBh6daGSRJM1FagTh76qutk4XF9Een9iyy9sQ7olabCjf3oUezvU7XWB83bSDD3e23CFuweOl0RreRFqIsbXBi55Dbcxw8+9fvMyOK4PXu6mkT/PRao/XR5sSMqiHt3FoJbjazSaJlI5teeVXbM=,iv:IICO5ay+KqkTikfxW4kqB2qcSRpbsOP1NjGWtCB9uqI=,tag:MX2TqUxre+ff3RszCfXeFg==,type:str]
+    unencrypted_suffix: _unencrypted
+    version: 3.9.4

secrets/turn-shared-secret.yaml

@@ -0,0 +1,48 @@
+password: ENC[AES256_GCM,data:C9uA7LHAc/SmWJsICGMNLu2OOYyowIhLq/sTfuNo/w==,iv:211oFm7MdphnXwYszRxWrHLeKw7qeh0BNzP0yomTUTk=,tag:zfhfkzLm0AivKwD7TZTEqw==,type:str]
+sops:
+    kms: []
+    gcp_kms: []
+    azure_kv: []
+    hc_vault: []
+    age:
+        - recipient: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGbTFyeCtJVWNOUXlSNXQ0
+            MEF6aG84NEVNUzlrMnoxcEF0ZFpEVC9LK1ZvCk04TkNkVWtVMWhhL2QwVEVEMi9X
+            M3Jkd1NoVFBFaDczMGhtN0svR284UzgKLS0tIE5OdTlXWXptTS9TZWdUNzFKU1VW
+            QXl2N2I3dFFORlFEbUVSK2ppSmNHSk0KzeS7uJCvQLd97XI/MfKuhJi4pQ/jKx2d
+            5veiGfnmI/j/WCBdH35PMTEotbBBQ26uFt+uvcShK2gO62MVbht8PQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjWG9PME1XQ1dSNkVEME9N
+            ZlBNL0FnclpoeEVLTVk5emlnTmpFVWtLcVF3CmQrY1ExOG5wMmRoMkV4Y3hXSjU5
+            OTZidFVWcklmZ0dHU0pJeTdiUTM5RzQKLS0tIHBGbncxRkpnUTVlNitRZkhyRjBE
+            bjdVSkZRZ241OE9Xd240OExWVkFrczAK+loPUAs5aBMC6XRY+yu0r3bqnWozldP6
+            m6mnRhUl2+JGm5RHglDWibUwPZ9I1EVlufDwDCABBnoXnY/HWuOlAw==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1vf8h7s8mqsn2q5eadgpdupsj4mwn8zguc77d85ws3xj40sl9rgksx2rxw6
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxOG96YzNTQk1sQitQNnV3
+            V3FsUEN5b00wcys5TmJSY0lTTm1qaEV1MGtZCnlYa3ZMNTFTdkliM0NjVW9BLzhj
+            ZytCMDQrcjVmdEx0cjhNRllZY2xuWDAKLS0tIHhoMVR0KzhZb1dVMldIUjUyc2JS
+            OHBSbVliajNSTDR5VW9odkdpN01iT0UKzifxPVQc98lEB4sXoDcDw7t3R396iwa7
+            RENR54u5GYUHHySq14v4k+7zhV42+xaXW45ZPhcA4BKuaAw8ygSDGQ==
+            -----END AGE ENCRYPTED FILE-----
+        - recipient: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
+          enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5R1EvVlZOOVMxSSsya2Fn
+            Zm5iWWN6d2RFNzU5aXBNbHNkcnYwM1VLY1Z3CjlEazNpcnhQdTRFMTVxdWliV0Ju
+            eEZMMnliS3BnU0RMSy9lNDZhVUw4K28KLS0tIG4vN01xZjRNSmVMVWx0VEJhd29Q
+            QTR3cEhvTVowU2ZXNDY3ZjhSVktSdEEKTxL2M/OgOMcptliInoED6aRDEQaqGnXF
+            N7dg2UnQsuFplJIfsW8KRoDQkcuXbdjn/W5aQh76+OC8wuRPVNWcEA==
+            -----END AGE ENCRYPTED FILE-----
+    lastmodified: "2026-05-21T20:13:20Z"
+    mac: ENC[AES256_GCM,data:PXXEpDbJIoIwWuNwMetpALtxrcB7yJhDYqp6LSKV8WFQI2TJSDgDHiNQ2cngfB9PXKmUNNhlvhmB52Hi6WQSWQdXjiy79T1fOjMhbUAKmykTaZDrwvlrDGkPgiZ1cFSb8+hx/5aZw7YhdQO9+7LAmymeIldZIFEpEO2pvuPZ6MM=,iv:mJb3K8TnsnY183OX7o9pF7oVMDcwVU/rOgX74KTRpr4=,tag:f+M+FT0Vjozm7a8LFDI8lA==,type:str]
+    pgp: []
+    unencrypted_suffix: _unencrypted
+    version: 3.9.4

ssh/authorized_keys/mac-mini.pub

@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMtgjgpljZcu98NOW1ibry8ZBQSoUg0IaxK8iW10w2+ d.toro.v@pm.me

ssh/deploy-keys.sh

@@ -0,0 +1,91 @@
+#!/bin/bash
+# deploy-keys.sh — deploy workstation SSH pubkeys to hubris + all LXCs
+#
+# Run from hubris (PVE host) as root.
+# Reads keys from /opt/homelab-context/ssh/authorized_keys/*.pub
+# and appends them to /root/.ssh/authorized_keys in each running LXC,
+# and to /etc/pve/priv/authorized_keys on hubris.
+#
+# Idempotent — skips keys already present.
+
+set -euo pipefail
+
+KEYS_DIR="/opt/homelab-context/ssh/authorized_keys"
+
+if ! [ -d "$KEYS_DIR" ]; then
+  echo "ERROR: $KEYS_DIR not found. Is homelab context synced?"
+  exit 1
+fi
+
+# Collect all pubkeys into a single variable, one per line
+ALL_KEYS=""
+for f in "$KEYS_DIR"/*.pub; do
+  [ -f "$f" ] || continue
+  key=$(cat "$f" | head -1)
+  ALL_KEYS="${ALL_KEYS}${key}
+"
+done
+
+if [ -z "$ALL_KEYS" ]; then
+  echo "ERROR: no .pub files found in $KEYS_DIR"
+  exit 1
+fi
+
+echo "= Deploying SSH keys to hubris ="
+AUTH_FILE="/etc/pve/priv/authorized_keys"
+touch "$AUTH_FILE"
+added=0
+while IFS= read -r key; do
+  [ -z "$key" ] && continue
+  if ! grep -qF "$key" "$AUTH_FILE" 2>/dev/null; then
+    echo "  + Adding key to hubris: ${key:0:40}..."
+    echo "$key" >> "$AUTH_FILE"
+    added=$((added + 1))
+  fi
+done <<< "$ALL_KEYS"
+echo "  hubris: $added key(s) added"
+
+# Deploy to LXCs
+echo ""
+echo "= Deploying SSH keys to LXCs ="
+
+# pct list output: VMID Status Lock Name (Lock may be empty)
+# Use awk to grab VMID, Status, and Name (last field)
+pct list | tail -n +2 | while read -r line; do
+  vmid=$(echo "$line" | awk '{print $1}')
+  status=$(echo "$line" | awk '{print $2}')
+  name=$(echo "$line" | awk '{print $NF}')
+  # Skip header if any
+  [ "$vmid" = "VMID" ] && continue
+  [ -z "$vmid" ] && continue
+  if [ "$status" != "running" ]; then
+    echo "  SKIP $name ($vmid): status=$status"
+    continue
+  fi
+
+  echo "  -> $name ($vmid)"
+
+  # Ensure .ssh directory exists
+  pct exec "$vmid" -- mkdir -p /root/.ssh 2>/dev/null
+
+  # For each key, check if already present, append if not
+  while IFS= read -r key; do
+    [ -z "$key" ] && continue
+    if ! pct exec "$vmid" -- grep -qF "$key" /root/.ssh/authorized_keys 2>/dev/null; then
+      echo "     + ${key:0:40}..."
+      # Use tee to append (pct exec preserves stdin)
+      echo "$key" | pct exec "$vmid" -- tee -a /root/.ssh/authorized_keys >/dev/null 2>&1
+    fi
+  done <<< "$ALL_KEYS"
+
+  # Fix permissions
+  pct exec "$vmid" -- chmod 600 /root/.ssh/authorized_keys 2>/dev/null || true
+  pct exec "$vmid" -- chmod 700 /root/.ssh 2>/dev/null || true
+
+  echo "  <- $name done"
+done
+
+echo ""
+echo "=== Deploy complete ==="
+echo "Keys deployed. Test from any workstation with:"
+echo "  ssh root@<lxc-ip>"

ssh/gen-config.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""Generate ~/.ssh/config.d/homelab from inventory.yaml.
+
+Usage:
+  python3 ssh/gen-config.py                        # print to stdout
+  python3 ssh/gen-config.py --install              # write to ~/.ssh/config.d/homelab
+
+The generated config provides short hostname aliases for every host in
+the homelab inventory. LAN IPs are preferred (they work directly on-LAN
+and are routed via Netbird 192.168.8.0/24 off-LAN); mesh FQDNs are
+available as <name>-mesh fallbacks for roaming workstations.
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import sys
+from pathlib import Path
+
+try:
+    import yaml
+except ImportError:
+    print("PyYAML required (pip install pyyaml)", file=sys.stderr)
+    sys.exit(2)
+
+INVENTORY_PATH = Path(os.environ.get(
+    "HOMELAB_CONTEXT_DIR", "/opt/homelab-context"
+)) / "inventory.yaml"
+
+SSH_CONFIG_DIR = Path.home() / ".ssh" / "config.d"
+SSH_CONFIG_FILE = SSH_CONFIG_DIR / "homelab"
+SSH_MAIN_CONFIG = Path.home() / ".ssh" / "config"
+INCLUDE_LINE = f"Include ~/.ssh/config.d/homelab"
+
+# Hosts to skip in the generated config
+SKIP_HOSTS = {"ludo-mini", "authentik"}  # offline / not enrolled
+
+
+def inventory() -> dict:
+    if not INVENTORY_PATH.exists():
+        print(f"ERROR: no inventory at {INVENTORY_PATH}", file=sys.stderr)
+        sys.exit(1)
+    return yaml.safe_load(INVENTORY_PATH.read_text())
+
+
+def gen_config() -> str:
+    inv = inventory()
+    hosts = inv.get("hosts", {})
+    mesh_globals = inv.get("mesh", {})
+
+    lines: list[str] = []
+    lines.append("# Homelab SSH config — generated from inventory.yaml")
+    lines.append(f"# Source: {INVENTORY_PATH}")
+    lines.append("# Do not edit by hand. Run: homelab ssh-config")
+    lines.append("")
+
+    # --- Common defaults (before any Host block) ---
+    lines.append("# --- Defaults ---")
+    lines.append("Host *")
+    lines.append("  IdentityFile ~/.ssh/id_ed25519")
+    lines.append("  IdentitiesOnly yes")
+    lines.append("  ServerAliveInterval 30")
+    lines.append("  StrictHostKeyChecking accept-new")
+    lines.append("")
+
+    def add_entry(
+        tag: str,
+        hostname: str,
+        user: str = "root",
+        port: int = 22,
+        extra_lines: list[str] | None = None,
+    ) -> None:
+        lines.append(f"Host {tag}")
+        lines.append(f"  HostName {hostname}")
+        lines.append(f"  User {user}")
+        if port != 22:
+            lines.append(f"  Port {port}")
+        if extra_lines:
+            lines.extend(f"  {el}" for el in extra_lines)
+        lines.append("")
+
+    # Sort hosts by kind for a logical output order
+    def sort_key(item):
+        name, h = item
+        kind = h.get("kind", "")
+        order = {
+            "workstation": 0,
+            "proxmox-host": 1,
+            "lxc": 2,
+            "vm": 3,
+            "external": 4,
+        }
+        return (order.get(kind, 9), name)
+
+    sorted_hosts = sorted(
+        [(n, h) for n, h in hosts.items() if n not in SKIP_HOSTS],
+        key=sort_key,
+    )
+
+    for name, h in sorted_hosts:
+        kind = h.get("kind", "")
+        ssh_config = h.get("ssh", {}) or {}
+        user = ssh_config.get("user", "root")
+        port = ssh_config.get("port", 22)
+        lan_ip = h.get("lan_ip")
+        mesh = h.get("mesh", {}) or {}
+        nb = mesh.get("netbird", {}) or {}
+        ts = mesh.get("tailscale", {}) or {}
+        nb_fqdn = nb.get("fqdn") or nb.get("ip") or ""
+        nb_port = ssh_config.get("netbird_port")
+
+        # Determine primary address
+        if nb_port and not lan_ip:
+            # Hosts with netbird SSH but no LAN IP: mesh-only
+            primary = nb_fqdn
+        elif lan_ip:
+            primary = lan_ip
+        elif nb_fqdn:
+            primary = nb_fqdn
+        elif ts.get("fqdn"):
+            primary = ts["fqdn"]
+        else:
+            continue  # no address found
+
+        extra = []
+        if kind == "proxmox-host":
+            extra = [
+                "ControlMaster auto",
+                "ControlPath ~/.ssh/cm/%C",
+                "ControlPersist 2h",
+            ]
+
+        add_entry(name, primary, user, port, extra_lines=extra)
+
+        # Mesh fallback for workstations and hubris
+        if nb_fqdn and (kind == "workstation" or nb_port):
+            mesh_tag = f"{name}-mesh"
+            mesh_port = nb_port or port
+            mesh_extra = []
+            if kind == "proxmox-host":
+                mesh_extra = [
+                    "ControlMaster auto",
+                    "ControlPath ~/.ssh/cm/%C",
+                    "ControlPersist 2h",
+                ]
+            add_entry(mesh_tag, nb_fqdn, user, mesh_port, extra_lines=mesh_extra)
+
+    # --- *.hubris.network ControlMaster ---
+    lines.append("# --- Mesh ControlMaster (speeds up repeated mesh ops) ---")
+    lines.append("Host *.netbird.selfhosted")
+    lines.append("  ControlMaster auto")
+    lines.append("  ControlPath ~/.ssh/cm/%C")
+    lines.append("  ControlPersist 2h")
+    lines.append("")
+
+    return "\n".join(lines)
+
+
+def ensure_include() -> bool:
+    """Add 'Include ~/.ssh/config.d/homelab' to the main SSH config if missing."""
+    if not SSH_MAIN_CONFIG.exists():
+        SSH_MAIN_CONFIG.parent.mkdir(parents=True, exist_ok=True)
+        with open(SSH_MAIN_CONFIG, "w") as f:
+            f.write(f"{INCLUDE_LINE}\n")
+        return True
+
+    content = SSH_MAIN_CONFIG.read_text()
+    for line in content.splitlines():
+        stripped = line.strip()
+        if stripped.startswith("Include") and "homelab" in stripped:
+            return False  # already present
+
+    # Prepend to existing config
+    updated = f"{INCLUDE_LINE}\n\n{content}"
+    SSH_MAIN_CONFIG.write_text(updated)
+    return True
+
+
+def install() -> None:
+    """Write config to ~/.ssh/config.d/homelab and ensure Include is set."""
+    SSH_CONFIG_DIR.mkdir(parents=True, exist_ok=True)
+    config = gen_config()
+    SSH_CONFIG_FILE.write_text(config)
+    SSH_CONFIG_FILE.chmod(0o644)
+    changed = ensure_include()
+    print(f"Wrote {SSH_CONFIG_FILE} ({len(config.splitlines())} lines)")
+    if changed:
+        print(f"Added '{INCLUDE_LINE}' to {SSH_MAIN_CONFIG}")
+    print("Done. Run: ssh <hostname>  (e.g. ssh gitea)")
+
+
+def main() -> int:
+    p = argparse.ArgumentParser(description=__doc__)
+    p.add_argument("--install", "-i", action="store_true",
+                   help=f"write to {SSH_CONFIG_FILE} and wire Include into main config")
+    args = p.parse_args()
+
+    if args.install:
+        install()
+    else:
+        print(gen_config())
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tools/caveman/caveman.js

@@ -0,0 +1,23 @@
+#!/usr/bin/env node
+// Caveman template renderer — reads template + data JSON files and renders output
+// Installed automatically via homelab-context post-pull hook
+const caveman = require("caveman");
+const fs = require("fs");
+const path = require("path");
+
+const args = process.argv.slice(2);
+if (args.length === 0) {
+  console.error("Usage: caveman <template> [data.json]");
+  process.exit(1);
+}
+
+const templatePath = args[0];
+let data = {};
+if (args.length > 1) {
+  data = JSON.parse(fs.readFileSync(args[1], "utf8"));
+}
+
+const template = fs.readFileSync(templatePath, "utf8");
+const templateName = path.basename(templatePath, path.extname(templatePath));
+caveman.register(templateName, template);
+console.log(caveman.render(templateName, data).trim());

tools/caveman/caveman_wrapper.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+# Caveman + RTK Wrapper - Automated token-efficient output formatting
+# Installed automatically via homelab-context post-pull hook
+# Source: https://github.com/adityahimaone/hermes-agent-rtk-caveman
+# Usage: caveman_wrapper.sh <workflow> [options]
+
+set -e
+
+WORKFLOW="$1"
+shift
+
+CAVEMAN=~/bin/caveman
+TEMPLATES_DIR=~/templates
+DATA_DIR=/tmp/caveman_data
+
+mkdir -p "$DATA_DIR"
+
+case "$WORKFLOW" in
+  git-status)
+    git status --porcelain | awk '
+    BEGIN { staged=0; modified=0; untracked=0; deleted=0 }
+    /^[MARC]./ { staged_arr[staged++] = substr($0, 4) }
+    /^.[MARC]/ { modified_arr[modified++] = substr($0, 4) }
+    /^\?\?/ { untracked_arr[untracked++] = substr($0, 4) }
+    /^D/ || /^.D/ { deleted_arr[deleted++] = substr($0, 4) }
+    END {
+      printf "{"
+      printf "\"staged\":["
+      for(i=0;i<staged;i++) printf "%s\"%s\"", (i>0?",":""), staged_arr[i]
+      printf "],\"modified\":["
+      for(i=0;i<modified;i++) printf "%s\"%s\"", (i>0?",":""), modified_arr[i]
+      printf "],\"untracked\":["
+      for(i=0;i<untracked;i++) printf "%s\"%s\"", (i>0?",":""), untracked_arr[i]
+      printf "],\"deleted\":["
+      for(i=0;i<deleted;i++) printf "%s\"%s\"", (i>0?",":""), deleted_arr[i]
+      printf "]}"
+    }' > "$DATA_DIR/git_status.json"
+
+    if command -v rtk &>/dev/null; then
+      rtk "$CAVEMAN" "$TEMPLATES_DIR/git_status.txt" "$DATA_DIR/git_status.json"
+    else
+      node "$CAVEMAN" "$TEMPLATES_DIR/git_status.txt" "$DATA_DIR/git_status.json"
+    fi
+    ;;
+
+  git-log)
+    LIMIT="${1:-10}"
+    git log --oneline -"$LIMIT" --format='{"hash":"%h","author":"%an","date":"%ad","message":"%s"}' --date=short | \
+      jq -s '.' > "$DATA_DIR/git_log.json"
+    jq '{commits: .}' "$DATA_DIR/git_log.json" > "$DATA_DIR/git_log_final.json"
+    if command -v rtk &>/dev/null; then
+      rtk "$CAVEMAN" "$TEMPLATES_DIR/git_log.txt" "$DATA_DIR/git_log_final.json"
+    else
+      node "$CAVEMAN" "$TEMPLATES_DIR/git_log.txt" "$DATA_DIR/git_log_final.json"
+    fi
+    ;;
+
+  test-results)
+    TEST_CMD="${1:-npx vitest run}"
+    $TEST_CMD --reporter json 2>/dev/null | \
+      jq '{total: .numTotalTests, passed: .numPassedTests, failed: .numFailedTests, suites: [.testResults[] | {name: .name, status: .status, duration: .duration}]}' > "$DATA_DIR/test_results.json" || true
+    if command -v rtk &>/dev/null; then
+      rtk "$CAVEMAN" "$TEMPLATES_DIR/test_results.txt" "$DATA_DIR/test_results.json"
+    else
+      node "$CAVEMAN" "$TEMPLATES_DIR/test_results.txt" "$DATA_DIR/test_results.json"
+    fi
+    ;;
+
+  *)
+    echo "Usage: caveman_wrapper.sh <workflow> [options]"
+    echo "  git-status            - Compact git status"
+    echo "  git-log [limit]       - Recent git commits"
+    echo "  test-results [cmd]    - Compact test results"
+    exit 1
+    ;;
+esac

tools/caveman/templates/git_log.txt

@@ -0,0 +1,3 @@
+Recent Commits:
+{{- for d.commits as commit }}  {{commit.hash}} {{commit.date}} {{commit.message}}
+{{- end }}

tools/caveman/templates/git_status.txt

@@ -0,0 +1,12 @@
+{{- if d.staged }}Staged:
+{{- for d.staged as file }}  + {{file}}
+{{- end }}{{- end }}
+{{- if d.modified }}Modified:
+{{- for d.modified as file }}  ~ {{file}}
+{{- end }}{{- end }}
+{{- if d.untracked }}Untracked:
+{{- for d.untracked as file }}  ? {{file}}
+{{- end }}{{- end }}
+{{- if d.deleted }}Deleted:
+{{- for d.deleted as file }}  - {{file}}
+{{- end }}{{- end }}