oikos/.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md

# Plan: Migrate library SSD to ludo-mini + Proxmox gaming/media server

## Goal

Split the homelab into two Proxmox hosts:

| Host | Role | Storage |
|------|------|---------|
| **hubris** | Core services (reverse-proxy, SSO, Matrix, git, documents, HA) | SSD 1 — boot + LXC rootfs (unchanged) |
| **ludo-mini** | Gaming server + media/library services | SSD 2 — Samsung 990 EVO Plus 4 TB (moved from hubris) |

The library SSD physically moves from hubris to ludo-mini. hubris LXCs that still need `/mnt/library` access it over NFS from ludo-mini.

## Current state

### hubris hardware
- GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS, 12 vCPU, ~28 GiB RAM
- 2× Samsung 990 EVO Plus NVMe:
  - nvme0: `local` (95G) + `local-lvm` (856G) — boot, ISOs, LXC rootfs
  - nvme1: `library` LVM (3.7T) — `/mnt/library` ext4 via `/dev/mapper/library-library`

### LXCs binding `/mnt/library` (host-level bind-mount)
| ID | Name | Role | I/O profile |
|----|------|------|-------------|
| 101 | jellyfin | Media streaming | Read-heavy, sequential |
| 103 | paperless | Document archive | Mixed, OCR writes |
| 104 | gitea | Git server | Mixed, lots of small files |
| 105 | apps | Docker (booklore, audiobookshelf, artifacto, MCP) | Mixed, depends on container |
| 114 | nextcloud | File sync | Mixed, WebDAV |
| 119 | sophia | Workshop | Low I/O |
| 120 | mule-images | Photo management | Write-heavy (processing), iGPU |
| 122 | arriman | *arr stack + downloads | Write-heavy (downloads) |
| 126 | plato | App (sub-mount: `/mnt/library/documents/plato`) | Light |

### NFS export chain (for VM 100 zimaos)
```
/mnt/library (ext4, host) → bind-mount → LXC 102 (nfs-export) → NFSv4 → VM 100 (zimaos)
```

### ludo-mini current
- Linux workstation, wired Ethernet 2.5 Gbps, `192.168.178.181` (household LAN)
- Runs Sunshine for game streaming
- No Proxmox, no LVM config
- Connected to SODOLA switch (same switch as hubris eno1)

### Network topology
```
Fritz!Box 7590 (192.168.178.1)
  └── SODOLA 2.5G switch
       ├── hubris eno1 → vmbr1 (192.168.178.10)
       │     └── routes to vmbr0 (192.168.8.0/24) — all LXCs
       └── ludo-mini (192.168.178.181)
```

hubris routes between `192.168.8.0/24` (vmbr0) and `192.168.178.0/24` (vmbr1). So LXCs can reach ludo-mini via hubris as a router.

## Key decisions

### 1. Service split — what moves, what stays

**Move to ludo-mini** (high I/O, benefits from data locality + GPU):
- 101 jellyfin — media streaming, GPU transcoding
- 120 mule-images — photo processing, iGPU passthrough
- 122 arriman — *arr stack, downloads write to library

**Stay on hubris, NFS-mount library from ludo-mini:**
- 103 paperless — documents, moderate I/O
- 104 gitea — git repos (small files, some I/O sensitivity but acceptable over NFS)
- 105 apps — Docker apps, mixed workloads
- 114 nextcloud — file sync
- 119 sophia — workshop, light use
- 126 plato — app, light use
- 100 zimaos — NAS frontend, already NFS-mounted

### 2. NFS architecture

Instead of changing every LXC's mount config, keep the bind-mount pattern on hubris:

```
ludo-mini: /mnt/library (ext4, local NVMe)
    │
    └── NFSv4 export to 192.168.8.0/24
              │
              └── hubris host: NFS-mount at /mnt/library
                        │
                        └── LXCs: bind-mount /mnt/library (unchanged!)
```

This is transparent to all hubris LXCs — no container config changes needed. Only the hubris host changes from ext4 local mount to NFS mount. The LXC bind-mounts "just work" because `/mnt/library` is still at the same path on the host.

### 3. Network — ludo-mini reachability from hubris LXCs

LXCs on `192.168.8.0/24` reach ludo-mini (`192.168.178.181`) through hubris routing:
- `vmbr0` → hubris kernel routing → `vmbr1` → SODOLA → ludo-mini
- Already works (IP forwarding enabled on hubris)

**Alternative (cleaner):** Add a secondary IP `192.168.8.x` on ludo-mini's physical interface so it's directly on the homelab subnet. This avoids the router hop and keeps NFS traffic off kernel forwarding path. Worth considering but not required.

### 4. Gaming on ludo-mini with Proxmox

ludo-mini runs Sunshine (game streaming). Under Proxmox:
- **Option A:** Gaming VM with GPU passthrough — Sunshine + games in a VM, full GPU access
- **Option B:** LXC with GPU device passthrough (`/dev/dri`) — lighter, shares kernel
- **Option C:** Keep Sunshine on the Proxmox host itself (not recommended, but simplest)

Option A is the cleanest for isolation. Games need a full desktop environment and GPU drivers; a VM with GPU passthrough gives them that.

### 5. What about nfs-export (LXC 102)?

Currently exports `/mnt/library` to zimaos. After migration:
- If zimaos stays on hubris and accesses library via host NFS → bind-mount → LXC 102, that's triple-hop (ludo-mini → NFS → hubris → bind-mount → LXC 102 → NFS → zimaos). Terrible.
- Better: zimaos NFS-mounts directly from ludo-mini.
- So LXC 102 gets decommissioned (or repurposed).
- zimaos gets a new NFS mount pointing directly at ludo-mini.

## Migration phases

### Phase 1 — Preparation (no downtime)

1. **Document current state** on hubris:
   - `pct list` — full container inventory
   - `pct config <id>` for every library-mounting LXC
   - `cat /etc/fstab` — capture the library mount line
   - `df -h /mnt/library` — confirm space usage
   - `lsblk -f` — UUID, filesystem
   - Identify the exact NVMe device (`nvme1n1`)

2. **Pre-flight on ludo-mini:**
   - Confirm hardware: CPU, RAM, available M.2 slots, GPU model
   - Confirm it can take the Samsung 990 EVO Plus (M.2 NVMe, PCIe 4.0 x4)
   - Verify BIOS supports virtualization (VT-d/AMD-Vi for PCIe passthrough)
   - Check: does ludo-mini have a second drive for Proxmox OS? If not, we need to partition the library SSD for Proxmox boot + library LVM, which complicates things significantly

3. **Install Proxmox on ludo-mini:**
   - Download Proxmox VE 9.x ISO
   - Install to ludo-mini's system drive (NOT the library SSD)
   - Configure networking: bridge for Proxmox, IP on 192.168.178.x
   - Test: web UI accessible

4. **Prepare NFS server on ludo-mini Proxmox:**
   - Create NFS-export LXC (or serve from host — simpler for now)
   - Prepare `/etc/exports`: `192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)`
   - Same squash params as current nfs-export LXC 102

### Phase 2 — Physical SSD move (planned downtime)

1. **Graceful shutdown on hubris:**
   - Stop all library-mounting LXCs (101, 103, 104, 105, 114, 119, 120, 122, 126)
   - Unmount `/mnt/library` on hubris host
   - Edit `/etc/fstab` to comment out the library mount line
   - Power off hubris

2. **Physical drive swap:**
   - Remove Samsung 990 EVO Plus (library SSD) from hubris
   - Install into ludo-mini M.2 slot
   - Power on ludo-mini

3. **Bring library online on ludo-mini:**
   - Detect the new NVMe device
   - If it's the whole device with LVM, activate the VG:
     ```
     vgscan && vgchange -ay library
     mount /dev/mapper/library-library /mnt/library
     ```
   - Add to `/etc/fstab` for auto-mount
   - Verify content: `ls /mnt/library` — same tree as before

4. **Start NFS export on ludo-mini:**
   - `exportfs -ra`
   - Verify: `showmount -e <ludo-mini-ip>`

### Phase 3 — Reconnect hubris LXCs

1. **Power on hubris** (without library SSD — it'll boot fine, just won't mount library)

2. **Mount NFS on hubris host:**
   - Install `nfs-common` if not present
   - Add to `/etc/fstab`:
     ```
     192.168.178.181:/mnt/library /mnt/library nfs rw,vers=4,soft,timeo=30,retrans=3 0 0
     ```
     Use `soft` to prevent hangs if ludo-mini is down; `hard` with `intr` is safer for data integrity but can block processes.
   - `mount /mnt/library`
   - **Verify permissions:** `ls -la /mnt/library` — should show `www-data:media` ownership for shared subtrees (same uid 33, gid 10000). The NFS all_squash guarantees this.

3. **Start LXCs:**
   - Start the LXCs that stayed on hubris (103, 104, 105, 114, 119, 126)
   - Their bind-mounts should work — `/mnt/library` is populated via NFS
   - Verify each service: web UIs, git clone, document access

4. **Update DNS/Caddy** for services that moved:
   - If jellyfin, arr services moved to ludo-mini, update Caddyfile to point to ludo-mini IPs
   - Update DNS entries if needed

### Phase 4 — Migrate services to ludo-mini

1. **Create LXCs/VM on ludo-mini Proxmox:**
   - 101 jellyfin — privileged LXC, mount `/mnt/library`, add `media` group
   - 120 mule-images — privileged LXC, mount `/mnt/library` + `/dev/dri` passthrough
   - 122 arriman — privileged LXC, mount `/mnt/library`

2. **Migrate configs:**
   - Copy LXC configs from hubris (`/etc/pve/lxc/<id>.conf`) as templates
   - Adjust network (IPs on household subnet or Proxmox bridge)
   - Restore app data from backups or copy over NFS

3. **Gaming VM:**
   - Create VM with GPU passthrough
   - Pass through the dGPU for gaming performance
   - Install Sunshine + game libraries
   - Storage: VM disk on Proxmox storage, games on library SSD

4. **Update reverse proxy:**
   - Caddy on hubris (121): update backend IPs for jellyfin, jellyseerr, qbit, sab, mule-images → point to ludo-mini
   - Test: `media.hubris.network` serves from ludo-mini jellyfin

### Phase 5 — Cleanup

1. **Decommission nfs-export LXC 102** on hubris (no longer needed)
2. **Update zimaos (100)** — change NFS mount from `192.168.8.200` → `192.168.178.181`
3. **Remove old LXCs** from hubris (101, 120, 122) after confirming migration works
4. **Update inventory.yaml:**
   - ludo-mini: `kind: proxmox-host`, add mounts/storage, add LXCs
   - Move services from hubris to ludo-mini
   - Remove nfs-export
5. **Update DNS:** `nfs-export.hubris.network` → ludo-mini IP (or remove)
6. **Run `homelab sync`** to propagate changes

## Open questions / unknowns

1. **Does ludo-mini have a second drive for Proxmox OS?** If not, we'd need to repartition the library SSD — carve out ~100 GB for Proxmox, then the rest for library LVM. This is risky (data loss if partitioning goes wrong) and requires a full backup first. **Alternative:** Buy a small SSD for ludo-mini's OS.

2. **What GPU does ludo-mini have?** Proxmox GPU passthrough requires IOMMU support and a GPU that doesn't have the reset bug. Need to check the exact GPU model.

3. **NFS performance for git (gitea)?** Git operations over NFS can be problematic (locking, stat() storms). Gitea bare repos at `/mnt/library/repos/*.git` might need testing. Worst case: move gitea's repo storage to local disk and keep `/mnt/library` for large file/LFS storage only.

4. **Media permission drift.** NFS `all_squash,anonuid=33,anongid=10000` ensures all writes from hubris LXCs (over NFS) and ludo-mini LXCs (local) land as `www-data:media`. This is the same squash currently used by nfs-export (102). Should be fine.

5. **ludo-mini network — add 192.168.8.x address?** Adding a secondary IP on ludo-mini's interface directly on the homelab subnet avoids routing through hubris for NFS traffic. Cleaner, but requires Proxmox bridge setup. Worth doing during Proxmox install.

6. **Sunshine migration.** Currently runs on ludo-mini bare metal. After Proxmox install, it needs to run in a VM. What happens to existing Sunshine configs, game libraries, save files? Need to preserve these during the Proxmox install.

7. **Backup before moving.** The library SSD holds 3.7 TB of irreplaceable data (documents, photos, repos). Restic backups are currently disabled. **Before physically moving the drive, verify the data is readable and consider doing one backup** — or at minimum, `rsync` critical directories.

## Files affected

| File | Change |
|------|--------|
| `/opt/homelab-context/inventory.yaml` | ludo-mini: workstation → proxmox-host; add LXCs, mounts; remove nfs-export; move service backends |
| `/opt/homelab-context/hosts/hubris.md` | Remove library storage, add NFS mount note |
| `/opt/homelab-context/hosts/ludo-mini.yaml` | Complete rewrite — Proxmox host, storage, tenants |
| `/opt/homelab-context/containers/102-nfs-export.md` | Mark decommissioned |
| `/opt/homelab-context/containers/index.md` | Move 101, 120, 122 to ludo-mini; remove 102 |
| `/opt/homelab-context/infrastructure/dns.md` | Update nfs-export entry |
| `/opt/homelab-context/infrastructure/media-permissions.md` | Note NFS squash from ludo-mini, not hubris |
| hubris `/etc/fstab` | Replace ext4 mount with NFS mount |
| ludo-mini `/etc/fstab` | Add library ext4 mount |
| ludo-mini `/etc/exports` | Add NFS export config |
| caddy (LXC 121) Caddyfile | Backend IPs for moved services |
| DNS (LXC 107 Technitium) | Update entries for moved services |

## Validation checklist

- [ ] ludo-mini Proxmox web UI accessible
- [ ] Library SSD detected and mountable on ludo-mini
- [ ] NFS export from ludo-mini: `showmount -e <ip>` shows `/mnt/library`
- [ ] hubris host NFS mount: `df -h /mnt/library` shows NFS, not ext4
- [ ] hubris LXCs start and bind-mount /mnt/library (content visible)
- [ ] gitea: `git clone` over SSH works, repos readable
- [ ] paperless: document ingestion works, OCR processing
- [ ] nextcloud: file sync, WebDAV
- [ ] jellyfin: media plays from ludo-mini, transcoding works
- [ ] arriman: downloads write to library, jellyfin picks up new media
- [ ] mule-images: photo import and processing
- [ ] zimaos: NFS mount from ludo-mini works, Files UI shows library
- [ ] Sunshine: game streaming from ludo-mini VM works
- [ ] All `*.hubris.network` services resolve and load through Caddy