db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion
- Migrations 010 (content_hash) + 011 (search tsvector column) - new: internal/knowledge/seed.go — knowledge seed ingest engine - new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge - wire knowledge ingest into oikos seed pipeline - convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml - archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/ - delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py - remove empty knowledge/ directory tree
This commit is contained in:
@@ -0,0 +1,277 @@
|
||||
# Plan: Migrate library SSD to ludo-mini + Proxmox gaming/media server
|
||||
|
||||
## Goal
|
||||
|
||||
Split the homelab into two Proxmox hosts:
|
||||
|
||||
| Host | Role | Storage |
|
||||
|------|------|---------|
|
||||
| **hubris** | Core services (reverse-proxy, SSO, Matrix, git, documents, HA) | SSD 1 — boot + LXC rootfs (unchanged) |
|
||||
| **ludo-mini** | Gaming server + media/library services | SSD 2 — Samsung 990 EVO Plus 4 TB (moved from hubris) |
|
||||
|
||||
The library SSD physically moves from hubris to ludo-mini. hubris LXCs that still need `/mnt/library` access it over NFS from ludo-mini.
|
||||
|
||||
## Current state
|
||||
|
||||
### hubris hardware
|
||||
- GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS, 12 vCPU, ~28 GiB RAM
|
||||
- 2× Samsung 990 EVO Plus NVMe:
|
||||
- nvme0: `local` (95G) + `local-lvm` (856G) — boot, ISOs, LXC rootfs
|
||||
- nvme1: `library` LVM (3.7T) — `/mnt/library` ext4 via `/dev/mapper/library-library`
|
||||
|
||||
### LXCs binding `/mnt/library` (host-level bind-mount)
|
||||
| ID | Name | Role | I/O profile |
|
||||
|----|------|------|-------------|
|
||||
| 101 | jellyfin | Media streaming | Read-heavy, sequential |
|
||||
| 103 | paperless | Document archive | Mixed, OCR writes |
|
||||
| 104 | gitea | Git server | Mixed, lots of small files |
|
||||
| 105 | apps | Docker (booklore, audiobookshelf, artifacto, MCP) | Mixed, depends on container |
|
||||
| 114 | nextcloud | File sync | Mixed, WebDAV |
|
||||
| 119 | sophia | Workshop | Low I/O |
|
||||
| 120 | mule-images | Photo management | Write-heavy (processing), iGPU |
|
||||
| 122 | arriman | *arr stack + downloads | Write-heavy (downloads) |
|
||||
| 126 | plato | App (sub-mount: `/mnt/library/documents/plato`) | Light |
|
||||
|
||||
### NFS export chain (for VM 100 zimaos)
|
||||
```
|
||||
/mnt/library (ext4, host) → bind-mount → LXC 102 (nfs-export) → NFSv4 → VM 100 (zimaos)
|
||||
```
|
||||
|
||||
### ludo-mini current
|
||||
- Linux workstation, wired Ethernet 2.5 Gbps, `192.168.178.181` (household LAN)
|
||||
- Runs Sunshine for game streaming
|
||||
- No Proxmox, no LVM config
|
||||
- Connected to SODOLA switch (same switch as hubris eno1)
|
||||
|
||||
### Network topology
|
||||
```
|
||||
Fritz!Box 7590 (192.168.178.1)
|
||||
└── SODOLA 2.5G switch
|
||||
├── hubris eno1 → vmbr1 (192.168.178.10)
|
||||
│ └── routes to vmbr0 (192.168.8.0/24) — all LXCs
|
||||
└── ludo-mini (192.168.178.181)
|
||||
```
|
||||
|
||||
hubris routes between `192.168.8.0/24` (vmbr0) and `192.168.178.0/24` (vmbr1). So LXCs can reach ludo-mini via hubris as a router.
|
||||
|
||||
## Key decisions
|
||||
|
||||
### 1. Service split — what moves, what stays
|
||||
|
||||
**Move to ludo-mini** (high I/O, benefits from data locality + GPU):
|
||||
- 101 jellyfin — media streaming, GPU transcoding
|
||||
- 120 mule-images — photo processing, iGPU passthrough
|
||||
- 122 arriman — *arr stack, downloads write to library
|
||||
|
||||
**Stay on hubris, NFS-mount library from ludo-mini:**
|
||||
- 103 paperless — documents, moderate I/O
|
||||
- 104 gitea — git repos (small files, some I/O sensitivity but acceptable over NFS)
|
||||
- 105 apps — Docker apps, mixed workloads
|
||||
- 114 nextcloud — file sync
|
||||
- 119 sophia — workshop, light use
|
||||
- 126 plato — app, light use
|
||||
- 100 zimaos — NAS frontend, already NFS-mounted
|
||||
|
||||
### 2. NFS architecture
|
||||
|
||||
Instead of changing every LXC's mount config, keep the bind-mount pattern on hubris:
|
||||
|
||||
```
|
||||
ludo-mini: /mnt/library (ext4, local NVMe)
|
||||
│
|
||||
└── NFSv4 export to 192.168.8.0/24
|
||||
│
|
||||
└── hubris host: NFS-mount at /mnt/library
|
||||
│
|
||||
└── LXCs: bind-mount /mnt/library (unchanged!)
|
||||
```
|
||||
|
||||
This is transparent to all hubris LXCs — no container config changes needed. Only the hubris host changes from ext4 local mount to NFS mount. The LXC bind-mounts "just work" because `/mnt/library` is still at the same path on the host.
|
||||
|
||||
### 3. Network — ludo-mini reachability from hubris LXCs
|
||||
|
||||
LXCs on `192.168.8.0/24` reach ludo-mini (`192.168.178.181`) through hubris routing:
|
||||
- `vmbr0` → hubris kernel routing → `vmbr1` → SODOLA → ludo-mini
|
||||
- Already works (IP forwarding enabled on hubris)
|
||||
|
||||
**Alternative (cleaner):** Add a secondary IP `192.168.8.x` on ludo-mini's physical interface so it's directly on the homelab subnet. This avoids the router hop and keeps NFS traffic off kernel forwarding path. Worth considering but not required.
|
||||
|
||||
### 4. Gaming on ludo-mini with Proxmox
|
||||
|
||||
ludo-mini runs Sunshine (game streaming). Under Proxmox:
|
||||
- **Option A:** Gaming VM with GPU passthrough — Sunshine + games in a VM, full GPU access
|
||||
- **Option B:** LXC with GPU device passthrough (`/dev/dri`) — lighter, shares kernel
|
||||
- **Option C:** Keep Sunshine on the Proxmox host itself (not recommended, but simplest)
|
||||
|
||||
Option A is the cleanest for isolation. Games need a full desktop environment and GPU drivers; a VM with GPU passthrough gives them that.
|
||||
|
||||
### 5. What about nfs-export (LXC 102)?
|
||||
|
||||
Currently exports `/mnt/library` to zimaos. After migration:
|
||||
- If zimaos stays on hubris and accesses library via host NFS → bind-mount → LXC 102, that's triple-hop (ludo-mini → NFS → hubris → bind-mount → LXC 102 → NFS → zimaos). Terrible.
|
||||
- Better: zimaos NFS-mounts directly from ludo-mini.
|
||||
- So LXC 102 gets decommissioned (or repurposed).
|
||||
- zimaos gets a new NFS mount pointing directly at ludo-mini.
|
||||
|
||||
## Migration phases
|
||||
|
||||
### Phase 1 — Preparation (no downtime)
|
||||
|
||||
1. **Document current state** on hubris:
|
||||
- `pct list` — full container inventory
|
||||
- `pct config <id>` for every library-mounting LXC
|
||||
- `cat /etc/fstab` — capture the library mount line
|
||||
- `df -h /mnt/library` — confirm space usage
|
||||
- `lsblk -f` — UUID, filesystem
|
||||
- Identify the exact NVMe device (`nvme1n1`)
|
||||
|
||||
2. **Pre-flight on ludo-mini:**
|
||||
- Confirm hardware: CPU, RAM, available M.2 slots, GPU model
|
||||
- Confirm it can take the Samsung 990 EVO Plus (M.2 NVMe, PCIe 4.0 x4)
|
||||
- Verify BIOS supports virtualization (VT-d/AMD-Vi for PCIe passthrough)
|
||||
- Check: does ludo-mini have a second drive for Proxmox OS? If not, we need to partition the library SSD for Proxmox boot + library LVM, which complicates things significantly
|
||||
|
||||
3. **Install Proxmox on ludo-mini:**
|
||||
- Download Proxmox VE 9.x ISO
|
||||
- Install to ludo-mini's system drive (NOT the library SSD)
|
||||
- Configure networking: bridge for Proxmox, IP on 192.168.178.x
|
||||
- Test: web UI accessible
|
||||
|
||||
4. **Prepare NFS server on ludo-mini Proxmox:**
|
||||
- Create NFS-export LXC (or serve from host — simpler for now)
|
||||
- Prepare `/etc/exports`: `192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)`
|
||||
- Same squash params as current nfs-export LXC 102
|
||||
|
||||
### Phase 2 — Physical SSD move (planned downtime)
|
||||
|
||||
1. **Graceful shutdown on hubris:**
|
||||
- Stop all library-mounting LXCs (101, 103, 104, 105, 114, 119, 120, 122, 126)
|
||||
- Unmount `/mnt/library` on hubris host
|
||||
- Edit `/etc/fstab` to comment out the library mount line
|
||||
- Power off hubris
|
||||
|
||||
2. **Physical drive swap:**
|
||||
- Remove Samsung 990 EVO Plus (library SSD) from hubris
|
||||
- Install into ludo-mini M.2 slot
|
||||
- Power on ludo-mini
|
||||
|
||||
3. **Bring library online on ludo-mini:**
|
||||
- Detect the new NVMe device
|
||||
- If it's the whole device with LVM, activate the VG:
|
||||
```
|
||||
vgscan && vgchange -ay library
|
||||
mount /dev/mapper/library-library /mnt/library
|
||||
```
|
||||
- Add to `/etc/fstab` for auto-mount
|
||||
- Verify content: `ls /mnt/library` — same tree as before
|
||||
|
||||
4. **Start NFS export on ludo-mini:**
|
||||
- `exportfs -ra`
|
||||
- Verify: `showmount -e <ludo-mini-ip>`
|
||||
|
||||
### Phase 3 — Reconnect hubris LXCs
|
||||
|
||||
1. **Power on hubris** (without library SSD — it'll boot fine, just won't mount library)
|
||||
|
||||
2. **Mount NFS on hubris host:**
|
||||
- Install `nfs-common` if not present
|
||||
- Add to `/etc/fstab`:
|
||||
```
|
||||
192.168.178.181:/mnt/library /mnt/library nfs rw,vers=4,soft,timeo=30,retrans=3 0 0
|
||||
```
|
||||
Use `soft` to prevent hangs if ludo-mini is down; `hard` with `intr` is safer for data integrity but can block processes.
|
||||
- `mount /mnt/library`
|
||||
- **Verify permissions:** `ls -la /mnt/library` — should show `www-data:media` ownership for shared subtrees (same uid 33, gid 10000). The NFS all_squash guarantees this.
|
||||
|
||||
3. **Start LXCs:**
|
||||
- Start the LXCs that stayed on hubris (103, 104, 105, 114, 119, 126)
|
||||
- Their bind-mounts should work — `/mnt/library` is populated via NFS
|
||||
- Verify each service: web UIs, git clone, document access
|
||||
|
||||
4. **Update DNS/Caddy** for services that moved:
|
||||
- If jellyfin, arr services moved to ludo-mini, update Caddyfile to point to ludo-mini IPs
|
||||
- Update DNS entries if needed
|
||||
|
||||
### Phase 4 — Migrate services to ludo-mini
|
||||
|
||||
1. **Create LXCs/VM on ludo-mini Proxmox:**
|
||||
- 101 jellyfin — privileged LXC, mount `/mnt/library`, add `media` group
|
||||
- 120 mule-images — privileged LXC, mount `/mnt/library` + `/dev/dri` passthrough
|
||||
- 122 arriman — privileged LXC, mount `/mnt/library`
|
||||
|
||||
2. **Migrate configs:**
|
||||
- Copy LXC configs from hubris (`/etc/pve/lxc/<id>.conf`) as templates
|
||||
- Adjust network (IPs on household subnet or Proxmox bridge)
|
||||
- Restore app data from backups or copy over NFS
|
||||
|
||||
3. **Gaming VM:**
|
||||
- Create VM with GPU passthrough
|
||||
- Pass through the dGPU for gaming performance
|
||||
- Install Sunshine + game libraries
|
||||
- Storage: VM disk on Proxmox storage, games on library SSD
|
||||
|
||||
4. **Update reverse proxy:**
|
||||
- Caddy on hubris (121): update backend IPs for jellyfin, jellyseerr, qbit, sab, mule-images → point to ludo-mini
|
||||
- Test: `media.hubris.network` serves from ludo-mini jellyfin
|
||||
|
||||
### Phase 5 — Cleanup
|
||||
|
||||
1. **Decommission nfs-export LXC 102** on hubris (no longer needed)
|
||||
2. **Update zimaos (100)** — change NFS mount from `192.168.8.200` → `192.168.178.181`
|
||||
3. **Remove old LXCs** from hubris (101, 120, 122) after confirming migration works
|
||||
4. **Update inventory.yaml:**
|
||||
- ludo-mini: `kind: proxmox-host`, add mounts/storage, add LXCs
|
||||
- Move services from hubris to ludo-mini
|
||||
- Remove nfs-export
|
||||
5. **Update DNS:** `nfs-export.hubris.network` → ludo-mini IP (or remove)
|
||||
6. **Run `homelab sync`** to propagate changes
|
||||
|
||||
## Open questions / unknowns
|
||||
|
||||
1. **Does ludo-mini have a second drive for Proxmox OS?** If not, we'd need to repartition the library SSD — carve out ~100 GB for Proxmox, then the rest for library LVM. This is risky (data loss if partitioning goes wrong) and requires a full backup first. **Alternative:** Buy a small SSD for ludo-mini's OS.
|
||||
|
||||
2. **What GPU does ludo-mini have?** Proxmox GPU passthrough requires IOMMU support and a GPU that doesn't have the reset bug. Need to check the exact GPU model.
|
||||
|
||||
3. **NFS performance for git (gitea)?** Git operations over NFS can be problematic (locking, stat() storms). Gitea bare repos at `/mnt/library/repos/*.git` might need testing. Worst case: move gitea's repo storage to local disk and keep `/mnt/library` for large file/LFS storage only.
|
||||
|
||||
4. **Media permission drift.** NFS `all_squash,anonuid=33,anongid=10000` ensures all writes from hubris LXCs (over NFS) and ludo-mini LXCs (local) land as `www-data:media`. This is the same squash currently used by nfs-export (102). Should be fine.
|
||||
|
||||
5. **ludo-mini network — add 192.168.8.x address?** Adding a secondary IP on ludo-mini's interface directly on the homelab subnet avoids routing through hubris for NFS traffic. Cleaner, but requires Proxmox bridge setup. Worth doing during Proxmox install.
|
||||
|
||||
6. **Sunshine migration.** Currently runs on ludo-mini bare metal. After Proxmox install, it needs to run in a VM. What happens to existing Sunshine configs, game libraries, save files? Need to preserve these during the Proxmox install.
|
||||
|
||||
7. **Backup before moving.** The library SSD holds 3.7 TB of irreplaceable data (documents, photos, repos). Restic backups are currently disabled. **Before physically moving the drive, verify the data is readable and consider doing one backup** — or at minimum, `rsync` critical directories.
|
||||
|
||||
## Files affected
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/opt/homelab-context/inventory.yaml` | ludo-mini: workstation → proxmox-host; add LXCs, mounts; remove nfs-export; move service backends |
|
||||
| `/opt/homelab-context/hosts/hubris.md` | Remove library storage, add NFS mount note |
|
||||
| `/opt/homelab-context/hosts/ludo-mini.yaml` | Complete rewrite — Proxmox host, storage, tenants |
|
||||
| `/opt/homelab-context/containers/102-nfs-export.md` | Mark decommissioned |
|
||||
| `/opt/homelab-context/containers/index.md` | Move 101, 120, 122 to ludo-mini; remove 102 |
|
||||
| `/opt/homelab-context/infrastructure/dns.md` | Update nfs-export entry |
|
||||
| `/opt/homelab-context/infrastructure/media-permissions.md` | Note NFS squash from ludo-mini, not hubris |
|
||||
| hubris `/etc/fstab` | Replace ext4 mount with NFS mount |
|
||||
| ludo-mini `/etc/fstab` | Add library ext4 mount |
|
||||
| ludo-mini `/etc/exports` | Add NFS export config |
|
||||
| caddy (LXC 121) Caddyfile | Backend IPs for moved services |
|
||||
| DNS (LXC 107 Technitium) | Update entries for moved services |
|
||||
|
||||
## Validation checklist
|
||||
|
||||
- [ ] ludo-mini Proxmox web UI accessible
|
||||
- [ ] Library SSD detected and mountable on ludo-mini
|
||||
- [ ] NFS export from ludo-mini: `showmount -e <ip>` shows `/mnt/library`
|
||||
- [ ] hubris host NFS mount: `df -h /mnt/library` shows NFS, not ext4
|
||||
- [ ] hubris LXCs start and bind-mount /mnt/library (content visible)
|
||||
- [ ] gitea: `git clone` over SSH works, repos readable
|
||||
- [ ] paperless: document ingestion works, OCR processing
|
||||
- [ ] nextcloud: file sync, WebDAV
|
||||
- [ ] jellyfin: media plays from ludo-mini, transcoding works
|
||||
- [ ] arriman: downloads write to library, jellyfin picks up new media
|
||||
- [ ] mule-images: photo import and processing
|
||||
- [ ] zimaos: NFS mount from ludo-mini works, Files UI shows library
|
||||
- [ ] Sunshine: game streaming from ludo-mini VM works
|
||||
- [ ] All `*.hubris.network` services resolve and load through Caddy
|
||||
@@ -0,0 +1,271 @@
|
||||
# Homelab structure revision & improvement plan
|
||||
|
||||
## Goal
|
||||
|
||||
Identify structural issues in the current hubris homelab topology and propose an
|
||||
actionable improvement roadmap — DNS consolidation, monitoring gaps, backup
|
||||
recovery, mesh completion, resource rightsizing, and operational hygiene.
|
||||
|
||||
---
|
||||
|
||||
## Current state summary
|
||||
|
||||
| Dimension | Status |
|
||||
|-----------|--------|
|
||||
| Hypervisor | hubris (single-node Proxmox) — also does subnet routing (192.168.8.0/24) |
|
||||
| LXCs | 14 active, 1 retired (124), 1 new (107 dns) |
|
||||
| VMs | HAOS (108), ZimaOS (100) |
|
||||
| Workstations | mac-mini (macOS), republic-laptop, ludo-mini |
|
||||
| VPS | 1 IONOS box — netbird mgmt+signal+relay, traefik, authentik, coturn |
|
||||
| Switch | SODOLA 5-Port 2.5Gbit (L2, flat) |
|
||||
| Networking | 192.168.8.0/24 internal, fr!tz box main LAN via vmbr0→vmbr1 routing |
|
||||
| Mesh | Netbird + Tailscale (migrating), ~mixed state |
|
||||
| DNS | 3 sources: Technitium (CT 107), NetBird managed zone, public IONOS |
|
||||
| Backups | DISABLED since 2026-04-22 |
|
||||
| Monitoring | claudio-monitor (host health only, no apt/docker checks yet) |
|
||||
| Agent enrollment | 3/19 hosts enrolled (hubris, apps, republic-laptop) |
|
||||
|
||||
---
|
||||
|
||||
## Issues identified
|
||||
|
||||
### 1. Three overlapping DNS sources (highest risk)
|
||||
|
||||
**Problem:** Technitium on CT 107, NetBird managed DNS zone, and the public
|
||||
IONOS wildcard all answer `*.hubris.network` queries. The dns.md doc still
|
||||
references the old dnsmasq on LXC 124 (though the change log says it moved).
|
||||
NetBird's managed DNS bypasses Technitium entirely for some app names — there
|
||||
is no single source of truth for DNS.
|
||||
|
||||
**Risk:** Mismatched answers → services unreachable → "works on some clients
|
||||
but not others" debugging sessions. Already cost time when `auth.hubris.network`
|
||||
re-pointed to the VPS.
|
||||
|
||||
**Proposal:**
|
||||
- Phase out NetBird managed DNS zone for `hubris.network` — Technitium is the
|
||||
authoritative answerer for mesh & LAN clients
|
||||
- Set Technitium as the sole DNS for all LXCs (remove router DNS / Tailscale
|
||||
MagicDNS fallbacks)
|
||||
- Document the full authoritative chain: Technitium → upstream forwarders → public
|
||||
- Track Technitium config in git (dtoro/technitium-config or equivalent)
|
||||
|
||||
### 2. Backups disabled with no alternative (data loss risk)
|
||||
|
||||
**Problem:** The only backup was restic to an external USB that caused host
|
||||
crashes. It was disabled 2026-04-22 as an A/B test — host stability was
|
||||
confirmed by the SODOLA migration (no more Slate AX double-NAT hangs). The
|
||||
drive is still removed.
|
||||
|
||||
**Risk:** `/mnt/library` (~429 GB of irreplaceable data: photos, documents,
|
||||
git repos, gitea data) has no off-host copy. Single-disk failure = total loss.
|
||||
|
||||
**Proposal:**
|
||||
- Re-evaluate the USB drive stability with the new SODOLA switch topology
|
||||
(direct rear USB 3.0 port, no hub chain)
|
||||
- OR adopt a cloud-backed strategy: `rclone`-to-Hetzner Storage Box or
|
||||
Backblaze B2 for the irreplaceable subset (docs, photos, gitea data)
|
||||
- OR use hubris's own `zfs send` to a second host/disk if ZFS is feasible
|
||||
- Minimum viable: at minimum restore gitea backups + sops-encrypted secrets
|
||||
via an off-site cron (cheap B2 bucket)
|
||||
|
||||
### 3. Mesh migration still incomplete
|
||||
|
||||
**Problem:** Most LXCs still use Tailscale. This forces per-LXC DNS workarounds
|
||||
(/etc/hosts overrides, local dnsmasq) and runs two VPN stacks in parallel.
|
||||
Mesh migration doc (mesh.md) is comprehensive but execution stalled.
|
||||
|
||||
**Proposal:**
|
||||
- Batch-migrate all LXCs from Tailscale to Netbird in one maintenance window
|
||||
- Remove Tailscale from the PVE host
|
||||
- Ensure all LXCs resolve `*.hubris.network` via Technitium (no more /etc/hosts
|
||||
overrides)
|
||||
- Document Netbird client on each LXC (netbird version, setup key rotation)
|
||||
|
||||
### 4. LXC resource imbalance & disk pressure
|
||||
|
||||
**Problem:**
|
||||
| LXC | Cores | RAM | Rootfs | Disk usage |
|
||||
|-----|-------|-----|--------|------------|
|
||||
| mule-images (120) | 6 | 12 GiB | 60 GiB | photo AI, reasonable |
|
||||
| arriman (122) | 4 | 8 GiB | 24 GiB | media stack, OK |
|
||||
| nextcloud (114) | 4 | 6 GiB | 25 GiB | file sync, OK |
|
||||
| apps (105) | 2 | 4 GiB | 30 GiB | 6+ services, tight |
|
||||
| paperless (103) | 2 | 3 GiB | 8 GiB | 86.9% disk |
|
||||
| elementsynapse (118) | 1 | 2 GiB | 8 GiB | 86.8% disk |
|
||||
| gitea (104) | 1 | 1 GiB | 8 GiB | git server, adequate |
|
||||
| caddy (121) | 1 | 512 MiB | 6 GiB | fine for reverse proxy |
|
||||
|
||||
Paperless (103) and elementsynapse (118) are at critical disk levels. apps (105)
|
||||
is undersized for 6+ services.
|
||||
|
||||
**Proposal:**
|
||||
- Resize rootfs on paperless (8→16 GiB) and elementsynapse (8→16 GiB)
|
||||
- Bump apps (105) to 4 cores / 6 GiB RAM / 40 GiB rootfs
|
||||
- Enable claudio-monitor's disk check to alert before next crisis
|
||||
|
||||
### 5. VPS is a single point of failure
|
||||
|
||||
**Problem:** One IONOS VM runs netbird management (control plane), traefik
|
||||
(public ingress), authentik (identity), and coturn (TURN relay). If it goes
|
||||
down: no remote mesh, no public services, no auth.
|
||||
|
||||
**Proposal:**
|
||||
- Document a VPS recovery runbook (how to restore from a known-working backup)
|
||||
- Consider splitting authentik into a separate host or at minimum having a
|
||||
standby configuration
|
||||
- Not a high priority (the VPS has been stable) but worth documenting the
|
||||
blast radius and recovery path
|
||||
|
||||
### 6. No centralized logging
|
||||
|
||||
**Problem:** Each LXC has independent journald. Cross-service debugging
|
||||
involves hopping between `pct exec <id> -- journalctl -u <service>`. There is
|
||||
no aggregation or retention.
|
||||
|
||||
**Proposal:**
|
||||
- Deploy a lightweight log shipper (Loki + promtail, or vector.dev) on each LXC
|
||||
- Ship logs to a central Loki instance on apps (105) or a new small LXC
|
||||
- Grafana dashboard optional — even a simple `logcli` query saves time
|
||||
|
||||
### 7. Agent enrollment incomplete
|
||||
|
||||
**Problem:** Only hubris, apps, and republic-laptop are enrolled in the
|
||||
homelab-context system (age keys, sync timers, MCP access). mac-mini,
|
||||
ludo-mini, claudio-bot, and all other LXCs are not.
|
||||
|
||||
**Proposal:**
|
||||
- Batch-enroll remaining LXCs (jellyfin, paperless, gitea, nextcloud, etc.)
|
||||
- Enroll mac-mini (macOS — exercises the launchd timer path)
|
||||
- Enroll ludo-mini (needs SSH user config in inventory first)
|
||||
- Wire claudio-bot into inventory-aware queries
|
||||
|
||||
### 8. Configuration drift on untracked configs
|
||||
|
||||
**Problem:** Technitium config, dnsmasq (legacy), and several service-specific
|
||||
configs are not git-tracked.
|
||||
|
||||
**Proposal:**
|
||||
- Track Technitium zone backup + compose config in a git repo
|
||||
- Apply the same pattern as caddy-conf: dtoro/technitium-conf with auto-deploy
|
||||
|
||||
### 9. No capacity planning / resource monitoring
|
||||
|
||||
**Problem:** No trend data on CPU, RAM, or disk growth. The 86% disk alerts
|
||||
were discovered reactively. Rootfs resize is painful (requires Proxmox stop +
|
||||
resize + growfs inside).
|
||||
|
||||
**Proposal:**
|
||||
- Enable the missing claudio-monitor checks (disk growth trend, apt upgradable
|
||||
counts, docker image drift)
|
||||
- Set up a simple Prometheus + node_exporter on hubris or use the PVE API
|
||||
directly
|
||||
- At minimum, surface disk usage in the existing homelab-mcp management tools
|
||||
|
||||
### 10. No standard deploy / orchestration for bare-metal LXCs
|
||||
|
||||
**Problem:** Some services are bare-metal CLI apps (sophia, claudio-bot),
|
||||
some are Docker on apps (105), some are Portainer-managed. No consistent
|
||||
deploy pattern means every new service reinvents the deployment.
|
||||
|
||||
**Proposal:**
|
||||
- Don't over-engineer this — the current pragmatism works
|
||||
- Just document the decision tree:
|
||||
- Needs `/mnt/library` mount + heavy I/O → dedicated LXC
|
||||
- Small stateless web service → Docker on apps (105)
|
||||
- Media stack → dedicated LXC (arriman, jellyfin)
|
||||
- Everything else → judge by complexity
|
||||
|
||||
---
|
||||
|
||||
## Phased implementation plan
|
||||
|
||||
### Phase 1 — Critical fixes (this week)
|
||||
1. Resize rootfs on paperless (103) 8→16 GiB, elementsynapse (118) 8→16 GiB
|
||||
2. Enable claudio-monitor disk check + disk-growth alerting
|
||||
3. Pick one backup strategy and implement minimum viable (e.g. nightly
|
||||
gitea dump + sops-encrypted secrets to B2 via rclone)
|
||||
4. Verify Technitium is the sole DNS for all LXCs (remove NetBird managed zone
|
||||
for hubris.network)
|
||||
|
||||
### Phase 2 — Mesh consolidation (next week)
|
||||
5. Batch-migrate remaining LXCs from Tailscale to Netbird
|
||||
6. Remove Tailscale from PVE host
|
||||
7. Remove all per-LXC /etc/hosts DNS overrides
|
||||
8. Update DNS documentation to reflect Technitium as single source
|
||||
|
||||
### Phase 3 — Agent enrollment & logging (next 2 weeks)
|
||||
9. Enroll all LXCs in homelab-context (age keys, sync timers)
|
||||
10. Enroll mac-mini (macOS launchd path — exercises untested code path)
|
||||
11. Enroll ludo-mini
|
||||
12. Deploy log shipper (Loki + promtail) on apps (105) + all LXCs
|
||||
|
||||
### Phase 4 — Resource & monitoring hardening (next month)
|
||||
13. Resize apps (105) rootfs, bump RAM
|
||||
14. Deploy Prometheus + node_exporter or equivalent for trend data
|
||||
15. Track Technitium config in git with auto-deploy
|
||||
16. Write VPS recovery runbook
|
||||
|
||||
### Phase 5 — Drive re-evaluation (optional, behind host-stability gate)
|
||||
17. Re-attach USB backup drive with the new SODOLA topology (direct port)
|
||||
18. If stable for 7 days, re-enable restic backup schedule (chunked)
|
||||
19. If not stable, finalize cloud backup as permanent strategy
|
||||
|
||||
---
|
||||
|
||||
## Files likely to change
|
||||
|
||||
| Path | Change |
|
||||
|------|--------|
|
||||
| `/opt/homelab-context/inventory.yaml` | LXCs enrolled, resource updates |
|
||||
| `/opt/homelab-context/secrets/*.yaml` | New recipients for enrolled LXCs |
|
||||
| `/opt/homelab-context/infrastructure/dns.md` | Reflect Technitium as sole source |
|
||||
| `/opt/homelab-context/infrastructure/mesh.md` | Remove Tailscale references post-migration |
|
||||
| `/opt/homelab-context/infrastructure/backups.md` | New strategy |
|
||||
| `/opt/homelab-context/infrastructure/monitoring.md` | Enable missing checks |
|
||||
| `/opt/homelab-context/containers/103-paperless.md` | Rootfs resize |
|
||||
| `/opt/homelab-context/containers/118-elementsynapse.md` | Rootfs resize |
|
||||
| `/opt/homelab-context/containers/105-apps.md` | Resource bump, logging addition |
|
||||
| `/opt/homelab-context/containers/index.md` | Updated resource table |
|
||||
| `.sops.yaml` | New age pubkeys for enrolled LXCs |
|
||||
|
||||
## Verification
|
||||
|
||||
Each phase ends with a verification milestone:
|
||||
- Phase 1: `claudio-monitor` triggers on paperless disk → confirmed alert. Backup
|
||||
of gitea data lands in B2 (or equivalent). DNS query from any LXC returns
|
||||
Technitium answer.
|
||||
- Phase 2: `netbird status` shows all LXCs connected. Tailscale not running on
|
||||
PVE host. `curl auth.hubris.network` from any LXC resolves correctly without
|
||||
/etc/hosts.
|
||||
- Phase 3: Every LXC has `/opt/homelab-context/` + `/etc/age/key.txt`. MCP
|
||||
tools return valid host info for all enrolled LXC names. `journalctl` shows
|
||||
promtail shipping to Loki.
|
||||
- Phase 4: `claudio-monitor` shows disk growth trend. apps (105) can run all
|
||||
6+ services without OOM.
|
||||
|
||||
## Risks & tradeoffs
|
||||
|
||||
- **Netbird migration window:** All LXCs will briefly lose mesh connectivity
|
||||
during the Tailscale→Netbird cutover. Schedule in off-hours.
|
||||
- **Backup cost:** B2/e2 costs ~$5/month for ~500 GB. The USB drive was free
|
||||
but unstable — trade money for reliability.
|
||||
- **DNS consolidation:** Removing the NetBird managed DNS zone means any
|
||||
NetBird-specific names stop resolving for hubris.network — verify nothing
|
||||
depends on that path.
|
||||
- **Loki on apps (105):** Adds another container to an already-loaded host.
|
||||
May need to bump resources before deploying.
|
||||
- **Agent enrollment on every LXC:** Each enrollment creates an age keypair
|
||||
and commits a pubkey to inventory. Process is scriptable via `homelab client
|
||||
add` but still takes ~2 min per host for verification.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Is the USB backup drive still physically attached to hubris? If not, the
|
||||
simplest "re-enable" path requires physically re-attaching it.
|
||||
2. Authentik is now on the VPS — is LXC 124 (old Authentik) still running or
|
||||
was it fully decommissioned? The dns.md changelog says "shut down" but
|
||||
index.md lists it as "running".
|
||||
3. What's the actual disk layout on hubris? nvme0n1, ZFS pool, mount structure
|
||||
— needed to plan rootfs resizes safely.
|
||||
4. Does the user want to keep Tailscale on any host for a specific reason, or
|
||||
is full Netbird migration the clear goal?
|
||||
@@ -0,0 +1,246 @@
|
||||
# Plan: Narrow Technitium DHCP Pool to Avoid Static-IP Conflicts
|
||||
|
||||
> **For Hermes:** Use subagent-driven-development skill to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Eliminate the IP conflict risk created by the Technitium DHCP pool (`.100–.240`) overlapping with all static LXC/VM IPs (`.101–.239`).
|
||||
|
||||
**Architecture:** Shrink the DHCP pool range on Technitium so it only covers IPs that no static host uses. No LXC/VM IPs change. Single server-side change (Technitium API), plus documentation updates.
|
||||
|
||||
**Tech Stack:** Technitium DNS API (`/api/dhcp/scopes/set`), bash/curl, homelab-context repo for docs.
|
||||
|
||||
---
|
||||
|
||||
## Problem statement
|
||||
|
||||
The Technitium DHCP server on [CT 107](../../knowledge/wiki/containers/107-dns.md) serves `192.168.8.100–192.168.8.240`. **Every static homelab IP except hubris (`.77`) sits inside that range:**
|
||||
|
||||
| Host | IP | Inside pool? |
|
||||
|---|---|---|
|
||||
| hubris (Proxmox) | .77 | No — below `.100` |
|
||||
| haos (VM 108) | .101 | YES |
|
||||
| gitea (104) | .121 | YES |
|
||||
| paperless (103) | .130 | YES |
|
||||
| arriman (122) | .132 | YES |
|
||||
| mule-images (120) | .136 | YES |
|
||||
| sophia (119) | .157 | YES |
|
||||
| mac-mini | .174 | YES |
|
||||
| caddy (121) | .175 | YES |
|
||||
| authentik (124) | .180 | YES |
|
||||
| plato (126) | .190 | YES |
|
||||
| zimaos (VM 100) | .195 | YES |
|
||||
| nfs-export (102) | .200 | YES |
|
||||
| apps (105) | .205 | YES |
|
||||
| jellyfin (101) | .206 | YES |
|
||||
| nextcloud (114) | .224 | YES |
|
||||
| claudio-bot (123) | .230 | YES |
|
||||
| elementsynapse (118) | .239 | YES |
|
||||
|
||||
The docs claim "Static-IP LXCs (below `.100`) are unaffected" — this is **false**. Static IPs span `.101–.239`, the DHCP pool spans `.100–.240`. They overlap almost entirely.
|
||||
|
||||
If the DHCP server hands out `.121/.136/.224` (or any of the above) to a new dynamic client before the static LXC claims it on boot, the static service will fail to bind and the service goes dark.
|
||||
|
||||
---
|
||||
|
||||
## Proposed approach: Shrink the pool
|
||||
|
||||
**Move the DHCP pool start from `.100` to `.241`**, resulting in:
|
||||
- **New pool:** `192.168.8.241 – 192.168.8.254` (14 dynamic IPs)
|
||||
- **Reserved:** `.100–.240` stays for static hosts, `.2` for Technitium, `.1` for gateway
|
||||
- **Zero changes to any LXC, VM, Caddy, or Proxmox config.**
|
||||
|
||||
Why `.241–.254`:
|
||||
- Highest static IP is `.239` (elementsynapse) — `.241` gives a 1-IP gap
|
||||
- `.255` is the broadcast address (unusable)
|
||||
- 14 IPs is plenty for truly dynamic clients (new transient containers, test VMs)
|
||||
- If more are ever needed, the pool can easily be widened back down
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1: Verify current Technitium DHCP scope from the API
|
||||
|
||||
**Objective:** Confirm the active pool range matches what's documented.
|
||||
|
||||
**Step 1: Log in to Technitium API and get a token**
|
||||
|
||||
```bash
|
||||
TOKEN=$(curl -sk -X POST http://192.168.8.2:5380/api/user/login \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"user":"admin","pass":"'$(cat /opt/technitium/admin_password.txt)'","includeInfo":false}' \
|
||||
| jq -r '.token')
|
||||
echo "Token: ${TOKEN:0:10}..."
|
||||
```
|
||||
|
||||
**Step 2: Fetch current DHCP scopes**
|
||||
|
||||
```bash
|
||||
curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq .
|
||||
```
|
||||
|
||||
**Expected:** One scope named `homelab` with `startingAddress: "192.168.8.100"` and `endingAddress: "192.168.8.240"`.
|
||||
|
||||
**Verification:** If the scope is NOT `.100–.240`, note the actual range and adjust the plan.
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Update the DHCP scope to `.241–.254`
|
||||
|
||||
**Objective:** Shrink the pool so it no longer overlaps static IPs.
|
||||
|
||||
**Step 1: Update the scope via API**
|
||||
|
||||
```bash
|
||||
curl -sk -X POST "http://192.168.8.2:5380/api/dhcp/scopes/set?token=$TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"name": "homelab",
|
||||
"startingAddress": "192.168.8.241",
|
||||
"endingAddress": "192.168.8.254",
|
||||
"subnetMask": "255.255.255.0",
|
||||
"gatewayAddress": "192.168.8.1",
|
||||
"dnsServerAddresses": ["192.168.8.2"],
|
||||
"leaseTime": 86400
|
||||
}'
|
||||
```
|
||||
|
||||
**Step 2: Verify the change took effect**
|
||||
|
||||
```bash
|
||||
curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq '.response.scopes[0] | {startingAddress, endingAddress}'
|
||||
```
|
||||
|
||||
**Expected:**
|
||||
```json
|
||||
{
|
||||
"startingAddress": "192.168.8.241",
|
||||
"endingAddress": "192.168.8.254"
|
||||
}
|
||||
```
|
||||
|
||||
**Pitfall:** If the API returns `{"status":"error"}`, the scope name or parameter format may differ. Inspect the response body. Technitium's API might use `rangeStart`/`rangeEnd` instead of `startingAddress`/`endingAddress`. Adjust if needed (check the full scope object from Task 1 step 2 for exact key names).
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Check for active DHCP leases in the old pool that would be stranded
|
||||
|
||||
**Objective:** Ensure no DHCP client is currently holding an IP in `.100–.240` that it will lose when its lease expires.
|
||||
|
||||
**Step 1: List active DHCP leases**
|
||||
|
||||
```bash
|
||||
curl -sk "http://192.168.8.2:5380/api/dhcp/leases/list?token=$TOKEN" | jq '.response.leases[] | {ip: .ipAddress, client: .clientHostname, mac: .hardwareAddress, expires: .leaseExpires}'
|
||||
```
|
||||
|
||||
**Step 2: Interpret results**
|
||||
|
||||
- If the only leases are from static LXCs that configured themselves before the DHCP move (e.g., old leases from before the 2026-06-02 static-IP migration), these leases are stale and harmless.
|
||||
- If a *dynamic* client (e.g., a test laptop, transient VM) holds `.195` or similar, note it — it will lose its IP on next renew and should be moved to a static assignment or into the `.241+` pool.
|
||||
- **ZimaOS (VM 100) at `.195` is a DHCP lease, not static** — this is the one host that needs attention. Either:
|
||||
- Set a static IP inside ZimaOS (preferred), or
|
||||
- Add a DHCP reservation for MAC in Technitium to pin `.195`
|
||||
|
||||
**Verification:** No "surprise" dynamic clients that would break on lease expiry.
|
||||
|
||||
---
|
||||
|
||||
### Task 4: Fix ZimaOS IP stability (if needed)
|
||||
|
||||
**Objective:** Ensure ZimaOS at `.195` won't float or break when the pool shrinks.
|
||||
|
||||
**If ZimaOS already has a static IP configured inside the VM:** Nothing to do.
|
||||
|
||||
**If ZimaOS is DHCP-only (likely — doc says "DHCP lease, not a reservation"):**
|
||||
|
||||
Option A (preferred): Set a static IP inside ZimaOS via its web UI at `http://192.168.8.195` → Settings → Network → Static IP → `192.168.8.195/24`, gateway `192.168.8.1`, DNS `192.168.8.2`.
|
||||
|
||||
Option B: Add a DHCP reservation in Technitium for ZimaOS's MAC address:
|
||||
```bash
|
||||
ZIMAMAC=$(ssh root@hubris "qm config 100 | grep net0 | grep -oE '([0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}'")
|
||||
curl -sk -X POST "http://192.168.8.2:5380/api/dhcp/reservations/add?token=$TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"hardwareAddress\":\"$ZIMAMAC\",\"ipAddress\":\"192.168.8.195\"}"
|
||||
```
|
||||
|
||||
**Pitfall:** The `/api/dhcp/reservations/add` endpoint signature is unverified — confirm the exact endpoint name from Technitium's API docs or the web UI before running it. The web console at `http://192.168.8.2:5380` → DHCP → Reservations can be used as a manual fallback.
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Update documentation in homelab-context
|
||||
|
||||
**Objective:** Fix the now-wrong claims about static IPs being "below .100".
|
||||
|
||||
**Files to edit:**
|
||||
|
||||
1. **`infrastructure/network.md`** — Line 53
|
||||
- Old: `Most homelab LXCs use static IPs below \`.100\`. DHCP only covers new/transient containers.`
|
||||
- New: `Static IPs span \`.101–.239\` (all LXCs + VMs + workstations). DHCP pool narrowed to \`.241–.254\` to avoid overlap.`
|
||||
|
||||
2. **`containers/107-dns.md`** — Lines 37, 42, 55
|
||||
- Line 37: Update pool range: `192.168.8.241 – 192.168.8.254`
|
||||
- Line 42: `Static-IP LXCs (below \`.100\`)` → `Static-IP LXCs (\`.101–.239\`) are excluded from the pool.`
|
||||
- Line 55: Add changelog entry for the pool shrink
|
||||
|
||||
3. **`containers/107-dns.md`** — Add changelog entry:
|
||||
```markdown
|
||||
### 2026-06-03 — DHCP pool narrowed to `.241–.254` to exclude static IPs
|
||||
Previous pool `.100–.240` overlapped with all static LXCs/VMs (\`.101–.239\`), creating IP conflict risk. Shrunk pool to `.241–.254`. No services re-IP'd. See [plan](../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
|
||||
```
|
||||
|
||||
4. **`infrastructure/network.md`** — Line 51: Update pool range in the DHCP table row.
|
||||
|
||||
5. **`plans/2026-06-01-slate-ax-to-sodola-migration.md`** — Line 60: Optionally update the pool range in the config table (or add a post-migration note). This is the historical migration plan, so a footnote rather than an edit may be better.
|
||||
|
||||
**Commit:**
|
||||
```bash
|
||||
cd /opt/homelab-context
|
||||
git add infrastructure/network.md containers/107-dns.md plans/
|
||||
git commit -m "docs: DHCP pool narrowed to .241-.254 to exclude static IPs"
|
||||
git push
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 6: Verify no regressions
|
||||
|
||||
**Objective:** Smoke-test that DNS and key services still work after the scope change.
|
||||
|
||||
```bash
|
||||
# 1. DNS resolution via Technitium
|
||||
dig @192.168.8.2 +short git.hubris.network
|
||||
# Expected: 192.168.8.175
|
||||
|
||||
# 2. Caddy reverse-proxy chain
|
||||
curl -sI https://git.hubris.network | head -1
|
||||
# Expected: HTTP/2 200
|
||||
|
||||
# 3. All app names resolve
|
||||
for name in git cloud media paperless photos matrix auth plato artifacto; do
|
||||
result=$(dig @192.168.8.2 +short ${name}.hubris.network)
|
||||
printf "%-20s → %s\n" "${name}.hubris.network" "$result"
|
||||
done
|
||||
|
||||
# 4. Technitium DHCP scope is correct
|
||||
curl -sk "http://192.168.8.2:5380/api/dhcp/scopes/list?token=$TOKEN" | jq '.response.scopes[0] | {startingAddress, endingAddress}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risk assessment
|
||||
|
||||
| Risk | Likelihood | Impact | Mitigation |
|
||||
|---|---|---|---|
|
||||
| API call fails (wrong field names) | Medium | Low | Inspect live scope object first (Task 1); adjust payload |
|
||||
| ZimaOS loses IP on next boot | Low | Medium | Task 4 makes ZimaOS static or reserved |
|
||||
| Active DHCP client in `.100–.240` gets stranded | Low | Low | Task 3 surfaces this; client just requests a new IP from `.241+` |
|
||||
| Technitium admin password file missing | Low | Medium | `/opt/technitium/admin_password.txt` was created during setup; verify existence |
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Is zimaos (VM 100) currently DHCP or static? The doc says DHCP lease, but it's listed as `lan_ip: 192.168.8.195` in inventory. If it's actually DHCP, it's the one host that needs a static assignment before the pool shrinks.
|
||||
2. Are there any transient DHCP clients (test laptops, phones) on the homelab subnet that hold `.100–.240` addresses? Check leases before cutting over.
|
||||
3. Should we widen the pool slightly (e.g., `.230–.254`) for more headroom? Currently 14 IPs. If 3+ transient devices are expected, `.230–.254` = 25 IPs — still safe since the highest static is `.239` and `.230–.239` could be excluded.
|
||||
|
||||
## Execution preference
|
||||
|
||||
All changes are on the Technitium API + homelab-context repo. No LXC/VM restarts needed. The pool shrink takes effect immediately for NEW DHCP requests; existing leases in the old range continue until expiry (24h max).
|
||||
129
archive/hermes-plans/2026-06-05_170000-prevent-dhcp-ip-drift.md
Normal file
129
archive/hermes-plans/2026-06-05_170000-prevent-dhcp-ip-drift.md
Normal file
@@ -0,0 +1,129 @@
|
||||
# Plan: Prevent DHCP IP drift from breaking Caddy backends
|
||||
|
||||
**Date:** 2026-06-05
|
||||
**Slug:** prevent-dhcp-ip-drift
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Eliminate the root cause of services becoming unreachable when DHCP lease renewals change backend IPs that Caddy's `reverse_proxy` directives hardcode.
|
||||
|
||||
**Triggering incident:** Paperless (LXC 103) and HAOS (VM 108) had DHCP-assigned IPs change from `.130→.243` and `.101→.241` respectively. Caddyfile still pointed at the old IPs → services unreachable from iPhone on Netbird.
|
||||
|
||||
## Current context
|
||||
|
||||
### DHCP vs static IP inventory
|
||||
|
||||
| Machine | Type | PVE ID | Current IP | Allocation method | Status |
|
||||
|---------|------|--------|-----------|-------------------|--------|
|
||||
| **jellyfin** | LXC | 101 | 192.168.8.206 | Static (`ip=.../24`) | ✅ |
|
||||
| **paperless** | LXC | 103 | 192.168.8.243 | **DHCP** (`ip=dhcp`) | ❌ broken, hotfixed to .243 |
|
||||
| **gitea** | LXC | 104 | 192.168.8.121 | Static | ✅ |
|
||||
| **apps** | LXC | 105 | 192.168.8.205 | Static | ✅ |
|
||||
| **nextcloud** | LXC | 114 | 192.168.8.224 | Static | ✅ |
|
||||
| **elementsynapse** | LXC | 118 | 192.168.8.239 | Static | ✅ |
|
||||
| **mule-images** | LXC | 120 | 192.168.8.136 | Static | ✅ |
|
||||
| **caddy** | LXC | 121 | 192.168.8.175 | Static | ✅ |
|
||||
| **arriman** | LXC | 122 | 192.168.8.132 | Static | ✅ |
|
||||
| **sophia** | LXC | 119 | 192.168.8.157 | Static? | ? (not in 2026-06-02 list) |
|
||||
| **nfs-export** | LXC | 102 | 192.168.8.200 | Static? | ? |
|
||||
| **plato** | LXC | 126 | 192.168.8.190 | Static? | ? |
|
||||
| **HAOS** | VM | 108 | 192.168.8.241 | **DHCP** (VM — OS-managed) | ❌ broken, hotfixed to .241 |
|
||||
| **zimaos** | VM | 100 | 192.168.8.195 | DHCP (known stale lease, see note) | ⚠️ open issue |
|
||||
| **authentik** | — | — | — | migrated to VPS (external) | N/A |
|
||||
|
||||
### Infrastructure facts
|
||||
|
||||
- **DHCP server:** Technitium on CT 107 (192.168.8.2), pool `.241–.254`
|
||||
- **Static IP range:** `.101–.239`
|
||||
- **DNS:** Split-horizon on Technitium — `*.hubris.network → 192.168.8.175` (Caddy itself)
|
||||
- **Caddyfile:** Has 29 `reverse_proxy` directives, all using **hardcoded IP:port**
|
||||
- **Caddy reload:** Auto-deployed via webhook on git push to `dtoro/caddy-conf`
|
||||
- **Documentation:** `inventory.yaml` updated, `hosts/*.yaml` stale-before-regenerate
|
||||
|
||||
### Why it happened
|
||||
|
||||
1. Paperless LXC 103 was **missed** during the 2026-06-02 static-IP migration (Proxmox config still shows `ip=dhcp`)
|
||||
2. HAOS VM 108 uses DHCP internally (HAOS manages its own network — can't just `pct set`)
|
||||
3. Both got new IPs from the Technitium `.241–.254` pool after a lease renewal
|
||||
4. Caddyfile still pointed at the old static-range IPs → connection refused
|
||||
|
||||
---
|
||||
|
||||
## Proposed approach
|
||||
|
||||
Three-layer solution:
|
||||
|
||||
### Layer 1: Fix the immediate offenders (static assignment)
|
||||
|
||||
**Paperless LXC 103:**
|
||||
- `pct set 103 --net0 name=eth0,bridge=vmbr0,gw=192.168.8.1,hwaddr=BC:24:11:0A:8D:C2,ip=192.168.8.130/24,ip6=auto,type=veth`
|
||||
- Inside the LXC, update `/etc/network/interfaces` to match
|
||||
- Reboot/restart networking
|
||||
|
||||
**HAOS VM 108:**
|
||||
- Set a **DHCP reservation** in Technitium for the VM's MAC address, pinning it to `192.168.8.101`
|
||||
- This avoids needing to reconfigure HAOS internally (which is tedious)
|
||||
- Alternatively: use `ha network update` from the HAOS console to set a static IP
|
||||
|
||||
### Layer 2: Audit and fix all remaining DHCP hosts
|
||||
|
||||
Check every LXC/VM hosted on hubris:
|
||||
```bash
|
||||
for ct in $(pct list | awk 'NR>1{print $1}'); do
|
||||
echo "=== CT $ct ==="
|
||||
pct config $ct | grep "^net0"
|
||||
done
|
||||
```
|
||||
Any with `ip=dhcp` that Caddy reverse-proxies to → convert to static.
|
||||
|
||||
**Known candidates to check:**
|
||||
- CT 102 (nfs-export) — `.200` but not in Caddy. May not need static.
|
||||
- CT 119 (sophia) — `.157` — is this static or DHCP? Not sure.
|
||||
- CT 126 (plato) — `.190` — same question.
|
||||
- VM 100 (zimaos) — `.195` but known to have a stale lease (see 2026-06-03 changelog)
|
||||
|
||||
### Layer 3: Add validation
|
||||
|
||||
Create a script that runs periodically (cron or homelab cronjob):
|
||||
|
||||
**`/opt/homelab-context/scripts/check-caddy-backends.sh`:**
|
||||
1. Parse `/etc/caddy/Caddyfile` on CT 121 to extract all `reverse_proxy IP:port` targets
|
||||
2. For each `IP:port`, attempt a TCP connect (timeout 3s)
|
||||
3. Report any that fail
|
||||
|
||||
Could also run as a homelab cron job that notifies dtoro on Matrix if a backend is unreachable.
|
||||
|
||||
This catches any future drift proactively (before a user reports it).
|
||||
|
||||
### Files likely to change
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `inventory.yaml` | May update paperless/HAOS IPs if we choose different static IPs |
|
||||
| LXC 103 Proxmux config (via `pct set`) | Set static IP |
|
||||
| Technitium DHCP reservations | Add HAOS reservation |
|
||||
| `/etc/caddy/Caddyfile` on CT 121 | Already fixed — only changes again if we re-assign paperless IP to `.130` |
|
||||
| `scripts/check-caddy-backends.sh` | New validation script (new file in homelab-context) |
|
||||
|
||||
### Risks / Tradeoffs
|
||||
|
||||
- **Pinning paperless to `.130`** — if the LXC was reinstalled since then, `.130` may already be in use. Verify first with `arp-scan` or `nmap`.
|
||||
- **HAOS static IP via Technitium reservation** vs **inside HAOS**: Technitium reservation is simpler (no HA config changes), but if HAOS's DHCP lease expires and the Technitium server is down, the reservation won't help. A static IP inside HAOS is more robust but requires poking the HA console.
|
||||
- **Validation script false positives** — a service might be legitimately down for maintenance. The script should be a warning, not an alert.
|
||||
- **Caddy reload** — each Caddyfile edit triggers an auto-reload via webhook. If the backend is down during reload, Caddy itself stays up (it's just a reverse_proxy target).
|
||||
|
||||
### Verification
|
||||
|
||||
1. After setting paperless static: `ssh root@192.168.8.175 "curl -s -o /dev/null -w '%{http_code}' http://192.168.8.130:8000"` → 302
|
||||
2. After Technitium HAOS reservation: `curl -s -o /dev/null -w '%{http_code}' http://192.168.8.101:8123` → 200
|
||||
3. Run validation script → all targets reachable
|
||||
4. Confirm from iPhone: both `paperless.hubris.network` and `home.hubris.network` load
|
||||
|
||||
### Open questions
|
||||
|
||||
1. Should paperless go back to `.130` (its original), or stay at `.243` (current)? Going back to `.130` means updating the Caddyfile again, but keeps the static range allocation consistent.
|
||||
2. HAOS: Technitium reservation or HAOS-internal static config? Reservation is easier; HAOS-internal is more robust.
|
||||
3. Should the Caddyfile validation script run as a homelab cron job, or as a cron on the caddy LXC itself?
|
||||
4. ZimaOS (VM 100) — should we also pin its IP while we're at it?
|
||||
@@ -0,0 +1,130 @@
|
||||
# Plan: Fix Frequent Authentik Login Prompts
|
||||
|
||||
## Goal
|
||||
|
||||
Stop requiring repeated login to Authentik (several times per day) by fixing session and cookie expiry settings so the user stays logged in for longer periods (e.g., 7–30 days, or until explicit logout).
|
||||
|
||||
## Current Context
|
||||
|
||||
Authentik runs on the VPS (`82.165.190.79`) in Docker Compose. Traffic flows:
|
||||
|
||||
```
|
||||
Browser → Caddy (LXC 121) → VPS Traefik → Authentik
|
||||
```
|
||||
|
||||
Caddy's `forward_auth` uses the `(authentik)` snippet which proxies to `auth.hubris.network/outpost.goauthentik.io/auth/caddy`. The Authentik server version is **2026.5.2**.
|
||||
|
||||
## Root Cause Found
|
||||
|
||||
### Primary: `SESSION_EXPIRE_AT_BROWSER_CLOSE = True`
|
||||
|
||||
The Authentik Django session (`authentik_session` cookie) is configured to **expire on browser close**. Every time the user closes and reopens their browser, the session cookie is cleared. The next visit to a service that requires OAuth2 authorization (Gitea, Jellyfin, etc.) will redirect to the Authentik login page.
|
||||
|
||||
### Secondary: `SESSION_COOKIE_AGE = 86400` (24 hours)
|
||||
|
||||
Even with the browser left open continuously, the session expires after 24 hours. Combined with `SESSION_SAVE_EVERY_REQUEST = False`, activity does NOT extend the session.
|
||||
|
||||
### Session configuration (from Docker Python environment):
|
||||
|
||||
| Setting | Current Value | Default in Django |
|
||||
|---------|---------------|-------------------|
|
||||
| `SESSION_EXPIRE_AT_BROWSER_CLOSE` | `True` | `False` |
|
||||
| `SESSION_COOKIE_AGE` | `86400` (24h) | `1209600` (14d) |
|
||||
| `SESSION_SAVE_EVERY_REQUEST` | `False` | `False` |
|
||||
| `SESSION_COOKIE_SAMESITE` | `Lax` | `Lax` |
|
||||
|
||||
### What ISN'T the problem:
|
||||
|
||||
- **Proxy cookie validity** — `hubris-forward-auth` has `access_token_validity = hours=24`, which is reasonable for the forward-auth token.
|
||||
- **Server-side session duration** — The `user_login` stage has `session_duration = seconds=0` (indefinite).
|
||||
- **Refresh tokens** — All OAuth2 providers have `refresh_token_validity = days=30`, which is fine.
|
||||
- **Caddy configuration** — The forward-auth chain is correctly set up.
|
||||
- **Outpost health** — All containers healthy, up for 6 days.
|
||||
|
||||
## Proposed Approach
|
||||
|
||||
Change two Django session settings via Authentik environment variables:
|
||||
|
||||
1. **`AUTHENTIK_SESSION_COOKIE_AGE` = 604800** (7 days) — extends session cookie lifetime from 24h to 7 days
|
||||
2. **`AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` = false** — prevents session cookie from being cleared on browser close
|
||||
|
||||
This keeps users logged in for up to 7 days with normal browser use (close/reopen, daily usage). The session still expires after 7 days of inactivity (`SESSION_SAVE_EVERY_REQUEST` stays False).
|
||||
|
||||
## Step-by-step Plan
|
||||
|
||||
### Step 1: Add environment variables to Docker compose
|
||||
|
||||
Edit `/opt/docker-compose.yml` on the VPS to add these env vars to the `authentik-server` service:
|
||||
|
||||
```yaml
|
||||
authentik-server:
|
||||
environment:
|
||||
# ... existing vars ...
|
||||
AUTHENTIK_SESSION_COOKIE_AGE: "604800" # 7 days (was 86400 / 24h)
|
||||
AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE: "false" # was true
|
||||
```
|
||||
|
||||
Note: the Authentik config system uses `__` (double underscore) for nesting. The env vars map to the Django settings via the config YAML path. The correct Authentik env var for `SESSION_COOKIE_AGE` would be `AUTHENTIK_SESSION__COOKIE_AGE` if it goes through the config system, or just `SESSION_COOKIE_AGE` if it's passed directly. Need to verify the exact variable name Authentik expects.
|
||||
|
||||
### Step 2: Verify variable naming
|
||||
|
||||
Check the Authentik config YAML (`/authentik/lib/default.yml` inside the container) to confirm the exact env var name mapping. Authentik uses a custom config layer that maps env vars to settings.
|
||||
|
||||
**Alternative if env vars don't work:** Some Authentik settings need to be set via the admin UI (under System Settings or Tenant settings). The Django session settings might need to be configured differently in this version.
|
||||
|
||||
### Step 3: Restart Authentik server
|
||||
|
||||
```bash
|
||||
ssh root@82.165.190.79
|
||||
docker compose -f /opt/docker-compose.yml restart authentik-server
|
||||
```
|
||||
|
||||
### Step 4: Verify the fix
|
||||
|
||||
```bash
|
||||
# Check session settings took effect
|
||||
ssh root@82.165.190.79 'docker exec -i authentik-server python3 << "PYEOF"
|
||||
import os
|
||||
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
|
||||
import django
|
||||
django.setup()
|
||||
from django.conf import settings
|
||||
print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE)
|
||||
print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE)
|
||||
PYEOF'
|
||||
```
|
||||
|
||||
### Step 5: Functional test
|
||||
|
||||
1. Login to Authentik at `auth.hubris.network`
|
||||
2. Close the browser completely
|
||||
3. Re-open browser, navigate to a forward-auth-gated service (e.g., paperless.hubris.network)
|
||||
4. Verify you're NOT redirected to login
|
||||
5. Verify OAuth2 services (Gitea) also maintain the session
|
||||
|
||||
## Files Likely to Change
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/opt/docker-compose.yml` | Add `AUTHENTIK_SESSION_COOKIE_AGE` and `AUTHENTIK_SESSION_EXPIRE_AT_BROWSER_CLOSE` env vars |
|
||||
|
||||
## Tests / Validation
|
||||
|
||||
1. **Config verification** — Run Python snippet inside container to confirm Django settings changed
|
||||
2. **Browser test** — Close/reopen browser, verify session persists (Step 5 above)
|
||||
3. **24-hour test** — Check session is still alive after 24h of normal use
|
||||
|
||||
## Risks, Tradeoffs, and Open Questions
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Env var names don't match Authentik's config schema | First verify in the container's `default.yml` config file |
|
||||
| 7-day persistent cookie is a security concern (stolen cookie = 7 days of access) | This is the same risk as any "Remember Me" feature on any web app. The tradeoff is convenience vs. security. |
|
||||
| The proxy cookie (`authentik_proxy_*`) may still have its own 24h limit | That's managed separately via the OAuth2 provider's `access_token_validity` setting. If we also want to extend that, we can update `hubris-forward-auth` provider's `access_token_validity` from `hours=24` to `days=7`. |
|
||||
| `SESSION_COOKIE_SECURE = False` | Should be `True` since Authentik is served behind HTTPS. However, the forward-auth subrequest from Caddy to the outpost is HTTP internally (`http://127.0.0.1:8099`), so `False` may be intentional for the outpost check. |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **What environment variable name does Authentik use for Django session settings?** Need to check `default.yml`. The config layer may use `AUTHENTIK_SESSION__COOKIE_AGE` (double underscore) or the raw Django setting name.
|
||||
2. **Should we also extend the proxy token validity?** The `hubris-forward-auth` provider has `access_token_validity = hours=24`. If we want users to not need re-login for more than 24h, we should also bump this to match the session cookie age.
|
||||
3. **Which specific service triggers the most login prompts?** The forward-auth (Caddy-gated) services use proxy cookies. OAuth2 services (Gitea, Jellyfin) use the Django session. Understanding which one the user is hitting most could narrow the fix scope.
|
||||
@@ -0,0 +1,149 @@
|
||||
# Plan: Fix Caddyfile truncation + prevent recurring outages
|
||||
|
||||
**Date:** 2026-06-06
|
||||
**Slug:** caddyfile-truncation-permanent-fix
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Restore all `*.hubris.network` services that went offline when the Caddyfile on LXC 121 was truncated to only 3 photo-related site blocks, and implement automated safeguards to prevent this class of outage from recurring.
|
||||
|
||||
## Root cause
|
||||
|
||||
The Caddyfile at `/etc/caddy/Caddyfile` on LXC 121 was manually edited locally (not via the `dtoro/caddy-conf` git repo), overwriting ~260 lines (30+ site blocks + forward-auth infrastructure) with only 43 lines covering `photos.hubris.network`, `prism.hubris.network`, and a manually-added `photos2.hubris.network`.
|
||||
|
||||
**Evidence:**
|
||||
- `git diff HEAD -- Caddyfile` shows `+3 / -159` lines diff — all other blocks deleted
|
||||
- Git reflog shows HEAD at `32575ce` (`fix: sab... port 8081→8082`), but working tree diverges
|
||||
- Deploy webhook log: Jun 06 12:39 — `deploy failed: git pull` (dirty tree blocks merge)
|
||||
- Backup file `Caddyfile.bak.1780263919` (225 lines) confirms the full original was intact before truncation
|
||||
- `origin/master` at `1b977aa` is the authoritative source — 260 lines, all blocks present
|
||||
|
||||
**Why "third time this week":**
|
||||
| Incident | Date | Cause |
|
||||
|---|---|---|
|
||||
| 1 | Jun 02 | DHCP IP drift — paperless (130→243), HAOS (101→241) |
|
||||
| 2 | Jun 05 | More DHCP drift — apps (205), mule-images (136 overridden by dhclient) |
|
||||
| 3 | Jun 06 | **Caddyfile truncated** — unrelated to IPs, much worse |
|
||||
|
||||
The Caddyfile truncation is the most severe: it took down **all LAN services** except `photos.hubris.network` and `auth.hubris.network` (VPS-hosted).
|
||||
|
||||
## Immediate fix
|
||||
|
||||
### Step 1: Restore Caddyfile from origin/master and reload
|
||||
|
||||
On LXC 121:
|
||||
|
||||
```bash
|
||||
cd /etc/caddy
|
||||
# Stash any local changes
|
||||
git stash
|
||||
# Reset to origin/master
|
||||
git checkout --force origin/master -- Caddyfile
|
||||
# Caddyfile now has all 30+ sites
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
systemctl reload caddy
|
||||
```
|
||||
|
||||
This restores all service blocks including: media, git, paperless, books, home, cloud, matrix, proxmox, docker, jellyseerr, qbit, sab, blog, auth, artifacto, plato, zimaos, mcp, secrets, sso + authentik forward-auth infrastructure.
|
||||
|
||||
### Step 2: Add `photos2.hubris.network` via git (if still needed)
|
||||
|
||||
The `photos2.hubris.network` block was manually added locally and is NOT in origin/master. If the user wants to keep it, submit a PR/commit to the `dtoro/caddy-conf` repo.
|
||||
|
||||
### Step 3: Verify
|
||||
|
||||
- From any LAN/mesh client: `curl -sk https://media.hubris.network/` → 200
|
||||
- Run `bash /opt/homelab-context/scripts/check-caddy-backends.sh` from hubris → all targets reachable
|
||||
- Flush mac-mini DNS: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`
|
||||
|
||||
## Permanent safeguards
|
||||
|
||||
### Layer 1: Caddyfile integrity check (deploy hook)
|
||||
|
||||
Add a site-count validation to the deploy script (`/etc/caddy/scripts/deploy.sh`):
|
||||
|
||||
```bash
|
||||
# Count site blocks (lines matching *.hubris.network {)
|
||||
SITE_COUNT=$(grep -c '^[a-z].*hubris.network {' Caddyfile)
|
||||
if [ "$SITE_COUNT" -lt 20 ]; then
|
||||
echo "[deploy] ERROR: Only $SITE_COUNT sites found (expected 20+). Refusing to reload."
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
This catches any future truncation before `caddy reload` runs.
|
||||
|
||||
### Layer 2: Caddyfile backup on deploy
|
||||
|
||||
Add to deploy script before git pull:
|
||||
|
||||
```bash
|
||||
cp Caddyfile "Caddyfile.bak.$(date +%s)"
|
||||
```
|
||||
|
||||
Keep last 3 backups, auto-rotate.
|
||||
|
||||
### Layer 3: Dirty-tree handling in deploy webhook
|
||||
|
||||
The deploy webhook currently hard-fails when the working tree is dirty. Change the receiver script to handle this gracefully:
|
||||
|
||||
```bash
|
||||
cd /etc/caddy
|
||||
# If dirty, stash local changes
|
||||
if ! git diff --quiet; then
|
||||
echo "[deploy] Working tree dirty — stashing"
|
||||
git stash push -m "auto-stash by deploy webhook $(date)"
|
||||
fi
|
||||
git pull --ff-only
|
||||
```
|
||||
|
||||
This prevents the webhook from blocking on future local edits.
|
||||
|
||||
### Layer 4: Scheduled Caddyfile health check
|
||||
|
||||
Add a homelab cron job that runs `check-caddy-backends.sh` every 10 minutes and notifies if any Caddy backend is unreachable.
|
||||
|
||||
```yaml
|
||||
# In homelab context: cronjob
|
||||
schedule: "*/10 * * * *"
|
||||
script: /opt/homelab-context/scripts/check-caddy-backends.sh
|
||||
```
|
||||
|
||||
### Layer 5: DNS sync cron (fix already-deployed sync)
|
||||
|
||||
The `dns-sync.py` on LXC 107 at `/opt/dns-sync/sync.py` is installed but has **no crontab** — the sync never runs automatically. The NetBird managed DNS zone has drifted from Technitium. Add a systemd timer or crontab:
|
||||
|
||||
```bash
|
||||
echo "*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1" > /etc/cron.d/dns-sync
|
||||
```
|
||||
|
||||
## Files likely to change
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/etc/caddy/Caddyfile` on LXC 121 | Restore from origin/master |
|
||||
| `/etc/caddy/scripts/deploy.sh` on LXC 121 | Add site-count validation + backup + dirty-tree handling |
|
||||
| `caddy-conf` git repo | PR with deploy.sh improvements + photos2 (if wanted) |
|
||||
| `cronjob` in Hermes | Schedule `check-caddy-backends.sh` |
|
||||
| `/etc/cron.d/dns-sync` on LXC 107 | New — add dns-sync cron |
|
||||
|
||||
## Verification
|
||||
|
||||
1. All `*.hubris.network` URLs load from mac-mini: `media`, `git`, `paperless`, `cloud`, `home`, `proxmox`, etc.
|
||||
2. `check-caddy-backends.sh` exits 0 on hubris
|
||||
3. `systemctl status caddy` shows active on LXC 121
|
||||
4. `dns-sync` runs and writes to `/var/log/dns-sync.log`
|
||||
|
||||
## Risks / Tradeoffs
|
||||
|
||||
- **Restoring from origin/master overwrites photos2.hubris.network** — recreate it via proper git commit
|
||||
- **Caddy staging ACME certs for prism/photos2**: The `tls dns ionos` directive uses staging env (`acme-staging-v02.api.letsencrypt.org`), which fails DNS propagation check (VPS port 53 unreachable from LXC). Once restored, these two subdomains will have the same issue. Move them to production IONOS DNS-01 by removing the staging CA directive or setting the correct `acme_issuer` in Caddyfile.
|
||||
- **Dirty-tree stash could lose edits** — mitigated by `git stash push --message` + backup file creation before stash
|
||||
|
||||
## Open questions
|
||||
|
||||
1. Keep `photos2.hubris.network`? If yes, add via proper git push.
|
||||
2. `prism.hubris.network` and `photos2` certs fail on staging ACME — set production `acme_issuer` in Caddyfile?
|
||||
3. Should `check-caddy-backends.sh` run as a homelab cron job or as a regular cron on LXC 121?
|
||||
358
archive/hermes-plans/2026-07-05_strong-migration-assessment.md
Normal file
358
archive/hermes-plans/2026-07-05_strong-migration-assessment.md
Normal file
@@ -0,0 +1,358 @@
|
||||
# Assessment: Which nodes can move to `strong`
|
||||
|
||||
## Executive summary
|
||||
|
||||
hubris is **memory-starved**: 28 GiB RAM, 71.9 GiB allocated across 18 LXC + 2 VM
|
||||
(2.5× overcommit), 10 GiB swap in active use. strong sits **completely empty** —
|
||||
28 GiB RAM, 25 GiB free, 0 guests, 2.7 TiB unused storage. The single most
|
||||
effective decongestion move is to shift guests off hubris onto strong.
|
||||
|
||||
This document assesses every guest for move-readiness, grouped by constraints
|
||||
(library dependency, GPU, core-infra status), and proposes a phased migration
|
||||
that does **not** require the physical library-SSD move (the blocker of the
|
||||
original plan) — library access from strong is provided via NFS from hubris.
|
||||
|
||||
---
|
||||
|
||||
## Current resource state (live, 2026-07-05)
|
||||
|
||||
### hubris — overloaded
|
||||
|
||||
| Resource | Capacity | Allocated (all guests) | Actual use | Status |
|
||||
|----------|----------|----------------------|------------|--------|
|
||||
| RAM | 28 GiB | 70.2 GiB (2.5× overcommit) | 18 GiB used + 10 GiB swap | ⚠️ heavy swap pressure |
|
||||
| CPU | 12 vCPU (6c/12t) | 43 vCPU (3.6× overcommit) | ~43% scaling MHz | OK (shares) |
|
||||
| local-lvm | 856 GiB | 582 GiB allocated (29% thin) | — | OK |
|
||||
| library (lvmthin) | 3.7 TiB | — | 1.2 TiB used (34%) | OK, 2.3 TiB free |
|
||||
|
||||
### strong — empty, ready
|
||||
|
||||
| Resource | Capacity | Used | Status |
|
||||
|----------|----------|------|--------|
|
||||
| RAM | 28 GiB | 2.3 GiB (host only) | 25 GiB free |
|
||||
| CPU | 16 vCPU (8c/16t) | idle (0.08 load) | 100% free |
|
||||
| local-lvm | 856 GiB | 0 | empty |
|
||||
| ludo-lvm | 1.8 TiB | 0 | empty |
|
||||
| Guests | — | 0 LXC, 0 VM | nothing running |
|
||||
|
||||
### Network topology constraint
|
||||
|
||||
```
|
||||
Fritz!Box (192.168.178.1)
|
||||
└── SODOLA 2.5G switch
|
||||
├── hubris eno1 → vmbr1 (192.168.178.10) → vmbr0 (192.168.8.0/24)
|
||||
│ └── all 20 guests on 192.168.8.x
|
||||
└── strong vmbr0 (192.168.178.181)
|
||||
└── no internal bridge yet, guests would be on 192.168.178.x
|
||||
```
|
||||
|
||||
strong reaches `192.168.8.0/24` via the Fritz static route through hubris.
|
||||
Guests on strong get `192.168.178.x` IPs unless we add an internal bridge
|
||||
on strong (Phase 0 prerequisite — see below).
|
||||
|
||||
---
|
||||
|
||||
## Per-guest assessment
|
||||
|
||||
### Tier 1 — Move immediately (no library dependency, no core-infra)
|
||||
|
||||
These guests mount **no** `/mnt/library` and are not part of the core
|
||||
infrastructure spine (caddy/dns/auth/mcp). They are the easiest wins.
|
||||
|
||||
| ID | Name | Cores | RAM | Library? | GPU? | Notes |
|
||||
|----|------|-------|-----|----------|------|-------|
|
||||
| 118 | elementsynapse | 2 | 4 GiB | ❌ | ❌ | Matrix homeserver. Public via Caddy (`matrix.hubris.network`). Only change: Caddy backend IP. **Easiest move in the fleet.** |
|
||||
| 129 | house | 2 | 3 GiB | ❌ | ❌ | Yuvomi family planner (Docker). No public Caddy route yet (uses VPS traefik directly). Self-contained. |
|
||||
|
||||
**Combined RAM freed from hubris: 7 GiB.** No NFS, no library, no GPU.
|
||||
|
||||
### Tier 2 — Move with library NFS (high resource consumers)
|
||||
|
||||
These are the heaviest guests and the original migration plan's primary
|
||||
targets. They mount `/mnt/library` and two use the iGPU. Moving them
|
||||
requires an NFS export from hubris → strong (reverse of the original
|
||||
plan's direction, since the physical SSD hasn't moved).
|
||||
|
||||
| ID | Name | Cores | RAM | Library? | GPU? | I/O profile | Notes |
|
||||
|----|------|-------|-----|----------|------|-------------|-------|
|
||||
| 120 | mule-images | 6 | 12 GiB | ✅ mp0 | ✅ iGPU | Write-heavy (photo processing) | **#1 RAM consumer.** Strong has Radeon 680M iGPU (VAAPI works). |
|
||||
| 122 | arriman | 4 | 8 GiB | ✅ mp0 | ❌ | Write-heavy (downloads) | *arr stack + qbit + sab. Mounts library for download writes. |
|
||||
| 101 | jellyfin | 4 | 8 GiB | ✅ mp0 | ✅ iGPU | Read-heavy sequential | Media streaming + transcode. Strong 680M handles VAAPI. |
|
||||
|
||||
**Combined RAM freed: 28 GiB.** This alone would eliminate hubris's swap
|
||||
pressure entirely.
|
||||
|
||||
### Tier 3 — Could move, low urgency
|
||||
|
||||
| ID | Name | Cores | RAM | Library? | Notes |
|
||||
|----|------|-------|-----|----------|-------|
|
||||
| 130 | grimmory | 1 | 2 GiB | ✅ mp0 | Book library (Docker). Migrated from apps LXC recently. |
|
||||
| 131 | teddycloud | 1 | 1 GiB | ✅ mp0 | New (not in inventory.yaml yet). |
|
||||
| 132 | rclone | 1 | 2 GiB | ✅ mp0 (ro) | Backup container. Read-only library mount. |
|
||||
| 128 | trmnl | 1 | 768 MiB | ❌ | TRMNL middleware. No library. Could move but tiny. |
|
||||
| 119 | sophia | 2 | 1 GiB | ✅ mp0 | Workshop. Light use. |
|
||||
|
||||
### Stay on hubris (core infrastructure)
|
||||
|
||||
| ID | Name | Cores | RAM | Why it stays |
|
||||
|----|------|-------|-----|--------------|
|
||||
| 121 | caddy | 1 | 512 MiB | Reverse proxy — terminates all `*.hubris.network`. Must stay on hubris for LAN-side reachability. **Needs backend IP updates** when guests move. |
|
||||
| 107 | dns | 1 | 1 GiB | Technitium DNS, split-horizon. Core. |
|
||||
| 106 | auth-outpost | 1 | 512 MiB | Authentik SSO enforcement. Core. |
|
||||
| 105 | apps | 2 | 4 GiB | homelab MCP + secrets-issuance + artifacto. Core infra. Mounts library. |
|
||||
| 104 | gitea | 1 | 1 GiB | Git server. NFS would hurt git lock/stat perf. Mounts library (bare repos). |
|
||||
| 103 | paperless | 2 | 3 GiB | Document archive. Moderate I/O, OCR writes. Mounts library. |
|
||||
| 102 | nfs-export | 1 | 512 MiB | Exports library to zimaos via NFS. Must stay with the physical library. |
|
||||
| 100 | zimaos | 4 | 8 GiB (VM) | NAS frontend eval. Already NFS-mounts library from 102. |
|
||||
| 108 | haos | 2 | 4 GiB (VM) | Home Assistant OS. Hardware access, low latency. |
|
||||
|
||||
---
|
||||
|
||||
## Constraints & prerequisites
|
||||
|
||||
### 1. Network — strong needs an internal bridge (Phase 0)
|
||||
|
||||
strong currently has only `vmbr0` on `192.168.178.0/24`. Guests created there
|
||||
get household-LAN IPs, not homelab-subnet IPs. Two options:
|
||||
|
||||
- **Option A (recommended):** Add `vmbr1` on strong as a portless internal
|
||||
bridge with a `192.168.8.x/24` address (e.g. `192.168.8.3`). Route between
|
||||
strong's `vmbr0` and `vmbr1` the same way hubris does. Guests go on `vmbr1`
|
||||
and get `192.168.8.x` IPs — transparent to Caddy, DNS, and inter-LXC refs.
|
||||
Requires adding a static route on Fritz (or relying on hubris's existing
|
||||
route — strong would need IP forwarding + a route to 192.168.8.0/24 via vmbr1).
|
||||
|
||||
- **Option B (simpler, messier):** Put guests on `192.168.178.x` directly.
|
||||
Caddy can still reach them (hubris routes to 192.168.178.0/24). But DNS
|
||||
records, inter-LXC references, and firewall rules all assume `192.168.8.x`.
|
||||
More config churn per guest.
|
||||
|
||||
### 2. Storage — rootfs migration (no shared storage)
|
||||
|
||||
`local-lvm` is per-node (not shared). Moving an LXC requires either:
|
||||
- `vzdump` → restore on strong (clean, but needs temp disk space + downtime)
|
||||
- `rsync` the rootfs to a new LXC on strong (faster for large rootfs like 120's 100G)
|
||||
- `pct migrate` only works with shared storage — **not applicable here**
|
||||
|
||||
For VMs (100, 108): `qm migrate` also needs shared storage. Not moving VMs.
|
||||
|
||||
### 3. Library access — NFS from hubris to strong
|
||||
|
||||
Since the physical library SSD is still on hubris, strong's guests that need
|
||||
`/mnt/library` must NFS-mount it from hubris. Options:
|
||||
|
||||
- **Export from hubris host directly** (simplest): add `/mnt/library` to
|
||||
`/etc/exports` on hubris with the same squash params as LXC 102
|
||||
(`rw,all_squash,anonuid=33,anongid=10000,no_subtree_check`). Mount on strong
|
||||
at `/mnt/library`. Strong's guests bind-mount it just like hubris's guests do.
|
||||
|
||||
- **Use existing nfs-export LXC 102**: strong NFS-mounts from `192.168.8.200`
|
||||
(LXC 102). This already has the right squash config. Less host-level change.
|
||||
**This is the path of least resistance.**
|
||||
|
||||
### 4. GPU — iGPU passthrough on strong
|
||||
|
||||
strong has a Ryzen 7 PRO 6850U with Radeon 680M iGPU. For jellyfin (VAAPI
|
||||
transcoding) and mule-images (photo processing), we need:
|
||||
- `/dev/dri/renderD128` passed to the LXC (`lxc.cgroup2.devices.allow` +
|
||||
`lxc.mount.entry` or Proxmox's `dev0:` passthrough)
|
||||
- `video` / `render` group membership inside the container
|
||||
- Confirm `amdgpu` driver loads on strong's host kernel (it should — same APU family)
|
||||
|
||||
### 5. Quorum — 2-node cluster, no QDevice
|
||||
|
||||
Moving guests to strong does NOT fix the quorum issue but **reduces blast
|
||||
radius**: if hubris reboots (its known thermal instability), the guests on
|
||||
strong keep running independently. Consider adding a QDevice as a separate
|
||||
follow-up — it's orthogonal to this migration.
|
||||
|
||||
---
|
||||
|
||||
## Revised migration phases
|
||||
|
||||
The original plan's NFS-over-LAN approach has been superseded. Instead,
|
||||
**media library data moves to ludo-lvm** on strong so migrated guests access
|
||||
it as a local ext4 mount. Data is split by origin:
|
||||
|
||||
```
|
||||
hubris (stays): library SSD (3.7T, 1.2T used)
|
||||
└── /mnt/library/{documents,images,cloud,homecloud,notes,repos,sophia}
|
||||
↑ user-generated content (docs, photos, cloud sync, notes, repos, workshop)
|
||||
|
||||
strong (moves): ludo-lvm (1.8T, 0 used at start)
|
||||
└── /mnt/media_local ← 1.5T thin volume
|
||||
└── {downloads,movies,music,tv,anime,books}
|
||||
↑ non-user-generated content (media arr stack, book library)
|
||||
```
|
||||
|
||||
| Category | Stays on hubris | Moves to strong |
|
||||
|----------|----------------|-----------------|
|
||||
| Media | — | downloads (25G), movies (51G), music (29G), tv (30G), anime (206G) |
|
||||
| Books | — | books (2.6G) |
|
||||
| Docs/Photos | documents (249M), images (4K) | — |
|
||||
| Cloud sync | cloud (287G), homecloud (367G) | — |
|
||||
| Personal | notes (6.7M), repos (84M), sophia (151G) | — |
|
||||
| **Total** | **~805G** | **~344G** |
|
||||
|
||||
ludo-lvm (1.8T) fits all media + books with ~1.15T headroom for growth.
|
||||
hubris library SSD (3.7T, 1.2T used) retains the user-generated content.
|
||||
Both sides keep their data local — no cross-node NFS needed for daily I/O.
|
||||
|
||||
---
|
||||
|
||||
### Phase 2a — Prepare ludo-lvm on strong
|
||||
|
||||
1. Create a ext4 filesystem on ludo-lvm for media:
|
||||
```bash
|
||||
lvcreate -n media -L 1.5T ludo-lvm
|
||||
mkfs.ext4 /dev/ludo-lvm/media
|
||||
```
|
||||
2. Mount at `/mnt/media_local` on strong, add to `/etc/fstab`
|
||||
3. rsync media directories from hubris → strong:
|
||||
```bash
|
||||
rsync -av --progress /mnt/library/{movies,tv,anime,downloads,music,books} strong:/mnt/media_local/
|
||||
```
|
||||
|
||||
### Phase 2b — Migrate arriman (122) to strong
|
||||
|
||||
1. Stop arriman on hubris, dump rootfs (24G)
|
||||
2. Restore on strong with IP `192.168.8.245/28` on vmbr1
|
||||
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (downloads land locally)
|
||||
4. Update Caddy: jellyseerr/qbit/sab backends → new IP
|
||||
5. Update inventory.yaml
|
||||
|
||||
### Phase 2c — Migrate jellyfin (101) to strong
|
||||
|
||||
1. Stop jellyfin on hubris, dump rootfs (16G)
|
||||
2. Restore on strong with IP `192.168.8.246/28` on vmbr1
|
||||
3. Pass `/dev/dri/renderD128` + `/dev/dri/card0` (Radeon 680M + RX 7600)
|
||||
4. Mount `/mnt/media_local` → `/mnt/library` via mp0 (media reads locally)
|
||||
5. Update Caddy: `media.hubris.network` → new IP
|
||||
6. Reinstall `sso-inject.js` in web dir (lost on every apt upgrade)
|
||||
7. Test VAAPI transcoding, SSO login, media playback
|
||||
|
||||
### Phase 2d — Migrate grimmory (130) to strong
|
||||
|
||||
1. Stop grimmory on hubris, dump rootfs (16G)
|
||||
2. Restore on strong with IP `192.168.8.247/28` on vmbr1
|
||||
3. Mount `/mnt/media_local` → `/mnt/library` via mp0 (books read locally)
|
||||
4. Update Caddy: `books.hubris.network` → new IP
|
||||
5. Update inventory.yaml
|
||||
6. Test: book browsing, calibre-web access
|
||||
|
||||
### No NFS export needed
|
||||
|
||||
With the data split by origin, hubris guests that only need user-generated
|
||||
content (documents, images, cloud, repos, sophia) still access them from the
|
||||
original library SSD — no cross-node NFS required. The two sides are
|
||||
independent.
|
||||
|
||||
**Result after Phase 2: hubris frees 26 GiB RAM (4 migrated guests) + 344G of
|
||||
library I/O burden. Strong becomes the media/books powerhouse.**
|
||||
|
||||
---
|
||||
|
||||
### Phase 3 — Migrate mule-images (120) to strong
|
||||
|
||||
Move photo management (12 GiB RAM, 6 cores, iGPU) last because it needs:
|
||||
- `/mnt/library` access (now NFS from strong — already set up in Phase 2d)
|
||||
- `/dev/dri/renderD128` (Radeon 680M — confirm VAAPI compatibility first)
|
||||
|
||||
Steps:
|
||||
1. Stop mule-images on hubris, rsync the 100G rootfs to strong (faster than vzdump)
|
||||
2. Restore on strong with IP on vmbr1
|
||||
3. Pass Radeon 680M iGPU
|
||||
4. Reconfigure library paths → `/mnt/media_local` (or keep NFS mount)
|
||||
5. Update Caddy: `photos.hubris.network` → new IP
|
||||
6. Test photo import + processing pipeline
|
||||
|
||||
---
|
||||
|
||||
### Phase 4 — Tier 3 moves (optional)
|
||||
|
||||
Migrate grimmory (130), teddycloud (131), rclone (132), trmnl (128), sophia (119)
|
||||
as needed — each frees 1–2 GiB. Not urgent; do when convenient.
|
||||
|
||||
---
|
||||
|
||||
### Phase 5 — Follow-up
|
||||
|
||||
- **QDevice**: add a tiebreaker for 2-node quorum
|
||||
- **Gaming VM**: strong's 6850U has enough cores alongside migrated LXCs
|
||||
- **Hubris library cleanup**: after all guests are confirmed working, decide
|
||||
whether to keep the original library SSD as backup or repurpose it
|
||||
|
||||
---
|
||||
|
||||
## Resource math after Phase 3 (all Tier 1 + 2 moved)
|
||||
|
||||
| | hubris | strong |
|
||||
|---|--------|--------|
|
||||
| Guests | 11 LXC + 2 VM | 5 LXC |
|
||||
| RAM allocated | ~25 GiB | ~45 GiB |
|
||||
| RAM capacity | 28 GiB | 28 GiB |
|
||||
| Overcommit | 0.9× (under-committed) | 1.6× (manageable) |
|
||||
| Library disk | Local ext4 (3.7T) → NFS client | Local ext4 on ludo-lvm (1.8T) |
|
||||
| GPU | Radeon 760M (idle) | Radeon 680M (jellyfin + mule-images) |
|
||||
|
||||
strong becomes the media/library powerhouse. hubris becomes a lean core-infra
|
||||
node (DNS, auth, git, docs, caddy, HA).
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## Risk register
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|------|--------|------------|
|
||||
| NFS latency for library reads (jellyfin, arriman) | Media playback stutter, slow downloads | Test iperf between strong↔hubris first. If 2.5G link, NFS throughput is fine (~1 Gbit/s). |
|
||||
| GPU passthrough on strong (680M vs 760M) | Transcode quality/compat differences | Both are AMD VAAPI — same driver stack. Test `vainfo` inside LXC before going live. |
|
||||
| Caddy backend IP churn | Service outage if IP wrong | Update Caddyfile in git repo (caddy-conf), test each route before destroying old LXC. |
|
||||
| vzdump/restore downtime | Service unavailable during migration | Schedule off-hours. Use rsync for large rootfs (120's 100G) to minimize freeze window. |
|
||||
| 2-node quorum still fragile | If hubris goes down, strong /etc/pve goes read-only | Guests keep running. Add QDevice as follow-up. |
|
||||
| Library data integrity during NFS transition | Permission drift | NFS `all_squash,anonuid=33,anongid=10000` matches existing LXC 102 config. Verify with `ls -la /mnt/library` after mount. |
|
||||
|
||||
---
|
||||
|
||||
## Open questions for operator
|
||||
|
||||
1. **Internal bridge on strong**: proceed with `vmbr1` on `192.168.8.3/24`
|
||||
(Option A), or use `192.168.178.x` guest IPs (Option B)?
|
||||
2. **Migration method**: `vzdump`/restore (clean, downtime) vs `rsync` rootfs
|
||||
(faster for large disks, needs manual config copy)?
|
||||
3. **Phase 1 priority**: move elementsynapse + house first (quick wins), or
|
||||
go straight to Phase 2 (mule-images/jellyfin/arriman) for maximum relief?
|
||||
4. **Should we add a QDevice now** before moving anything, to protect
|
||||
management plane during the migration?
|
||||
|
||||
---
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-07-05 — Phase 2d complete (grimmory migrated; media NFS to zimaos)
|
||||
grimmory (130) → 192.168.8.247 on strong. Rsync'd /books (2.6G) to ludo-lvm.
|
||||
LXC 102 (nfs-export) now mounts strong's NFS at /mnt/media and exports it as
|
||||
a second share alongside /mnt/library. Zimaos mounts both: /media/library
|
||||
(hubris user-generated) and /media/media (strong media+books).
|
||||
See hosts/strong.md changelog.
|
||||
|
||||
### 2026-07-05 — Phase 2 complete (arriman + jellyfin migrated; library on ludo-lvm)
|
||||
arriman (122) → 192.168.8.245, jellyfin (101) → 192.168.8.246. Created 1.5T
|
||||
thin volume on ludo-lvm, rsync'd 363G of media data. Both containers use local
|
||||
ext4 mount — no NFS. Jellyfin has 680M + RX 7600 GPU passthrough.
|
||||
Caddy backends updated. See hosts/strong.md changelog.
|
||||
|
||||
### 2026-07-05 — Phase 1 complete (elementsynapse + house migrated to strong)
|
||||
Both Tier 1 guests moved: elementsynapse (118) → 192.168.8.242, house (129) → 192.168.8.244.
|
||||
Strong now has vmbr1 at 192.168.8.241/28. Hubris has proxy ARP + /32 routes for strong
|
||||
guest range. DHCP scope narrowed to 192.168.8.100-239 to avoid conflicts.
|
||||
Teddycloud (LXC 131) given static IP 192.168.8.150 due to IP conflict with
|
||||
previous DHCP allocation at 192.168.8.243.
|
||||
See hosts/strong.md changelog for full steps.
|
||||
|
||||
### 2026-07-05 — assessment created
|
||||
Built from live `pct config` + `pvesm status` + `free -h` data pulled from
|
||||
both nodes. Supersedes the storage-migration framing of the original
|
||||
library-SSD plan — this assessment treats the SSD move as optional and
|
||||
focuses on guest relocation via NFS.
|
||||
Reference in New Issue
Block a user