Promote NFS export to rw + capture ZimaOS Drives-API finding + mesh reachability

After Files-UI evaluation passed (library renders as folder, thumbnails work), flipped /etc/exports from ro to rw on LXC 102. Tested: write from ZimaOS appears on /mnt/library as www-data:media, confirming the all_squash,anonuid=33,anongid=10000 design works. Documented two architectural findings discovered this session: - ZimaOS Drives panel sources from GET /v2/local_storage/storages (read-only API). Network shares cannot become Drives — Files-as-folder is supported. - Mesh peers reach ZimaOS via hubris's existing 192.168.8.0/24 netbird subnet advertisement; no new infra needed, just DNS (Management nameserver group for hubris.network or per-device /etc/hosts override).
2026-05-14 13:13:46 +02:00
parent 060eff826c
commit 82d4e35fcb
4 changed files with 175 additions and 0 deletions
--- a/README.md
+++ b/README.md
@@ -10,6 +10,7 @@ Living documentation for the **hubris** Proxmox homelab. Every node, every cross
 - [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77`

 ### VMs
+- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation)
 - [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS

 ### LXC containers
@@ -18,6 +19,7 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links:
 | ID  | Name             | IP              | Role                                          |
 | --- | ---------------- | --------------- | --------------------------------------------- |
 | 101 | [jellyfin](containers/101-jellyfin.md)             | 192.168.8.206 | Media server                                  |
+| 102 | [nfs-export](containers/102-nfs-export.md)         | 192.168.8.200 | NFSv4 re-export of /mnt/library for ZimaOS    |
 | 103 | [paperless](containers/103-paperless.md)           | 192.168.8.130 | Document mgmt                                 |
 | 104 | [gitea](containers/104-gitea.md)                   | 192.168.8.121 | Git server                                    |
 | 105 | [apps](containers/105-apps.md)                     | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) |
--- a/containers/102-nfs-export.md
+++ b/containers/102-nfs-export.md
@@ -0,0 +1,89 @@
+# 102 — `nfs-export`
+
+Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clients that can't use the host's PVE virtiofs path — currently only [100-zimaos](../vms/100-zimaos.md), which ships a kernel without virtiofs support.
+
+## At a glance
+- **Hostname:** `nfs-export`
+- **IP:** `192.168.8.200` (static; LAN-only, no Caddy in front because NFS is L4)
+- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy)
+- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
+- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
+- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, plato, syncthing, apps)
+
+## What it does
+
+```
+/mnt/library (host ext4 on nvme1n1)
+        │
+        ├── bind-mounted into 8 other LXCs (jellyfin, paperless, …)
+        └── bind-mounted into LXC 102
+                    │
+                    └── nfs-kernel-server exports /mnt/library
+                          │
+                          └── consumed by VM 100 (ZimaOS)
+```
+
+Same inodes, same page cache. The NFS server is just one more access path on top of a tree that 8 other consumers already share — see [media permissions](../infrastructure/media-permissions.md).
+
+## Export config
+
+`/etc/exports`:
+
+```
+/mnt/library  192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)
+```
+
+Initially started as `ro`; promoted to `rw` on 2026-05-14 after the Files-UI evaluation confirmed (a) the library renders correctly as a folder under `/DATA`, (b) thumbnails are generated, (c) the squash works — a write from ZimaOS appears on `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention used by Nextcloud and mule-images.
+
+### Guardrails (in order of importance)
+
+1. **`all_squash,anonuid=33,anongid=10000`.** Every write from ZimaOS records on disk as `www-data:media` (uid 33, gid 10000), the same identity Nextcloud and mule-images use. Keeps the existing tree convention from drifting. See [`project_media_perms`](../infrastructure/media-permissions.md).
+3. **Subnet restriction `192.168.8.0/24`.** No public/mesh access; LAN only.
+4. **`no_subtree_check`** + **`sec=sys`** — standard performance/auth pair for a homelab.
+5. **No `crossmnt`** because `/mnt/library` is a single ext4 filesystem on the host (no nested mounts to traverse).
+
+### What we're *not* doing yet
+
+- No per-subdir export. ZimaOS sees the whole tree; access is controlled by filesystem permissions (`drwxr-x---` private dirs like `documents/`, `notes/`, `heaper/` aren't readable by `media` group, so ZimaOS-as-squashed-uid won't see them either).
+- No write-back. Until we promote to `rw`, ZimaOS can't write — eliminates the lock-domain split concern between NFS clients (NLM/v4) and local LXCs (POSIX flock).
+- No Authentik / forward-auth. NFS doesn't sit behind HTTP, so the standard caddy+authentik path doesn't apply. Subnet ACL is the only auth.
+
+## Why this LXC exists (vs serving from host)
+
+We considered three options before building this:
+
+| Option | Outcome |
+|---|---|
+| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. |
+| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
+| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
+
+Rationale lives in [the install plan](https://git.hubris.network/dtoro/Homelab-Docs) `/root/.claude/plans/i-wannt-you-to-nifty-muffin.md` on hubris.
+
+## Operations
+
+- **Reload exports** after editing `/etc/exports`: `pct exec 102 -- exportfs -ra`
+- **List active exports:** `pct exec 102 -- exportfs -v`
+- **Watch from outside:** `showmount -e 192.168.8.200`
+- **Service health:** `pct exec 102 -- systemctl is-active nfs-server rpcbind`
+- **Restart cleanly:** `pct restart 102` (ZimaOS will retry the mount via `nofail`)
+- **Destroy + rebuild:** `pct stop 102 && pct destroy 102 --purge` — reversible in seconds; only ZimaOS notices
+
+## Open items
+- Consider tightening the export to subdirs (e.g. `movies`, `tv`, `music`, `audiobooks`, `books`, `images`, `podcasts`, `roms`) if you don't want ZimaOS reachable into `documents/`, `notes/`, `heaper/`, etc. — though those private subdirs are already invisible to `media`-group perms.
+- ZimaOS architecture finding: the Drives panel only enumerates physical/block devices via `GET /v2/local_storage/storages` (read-only API, no POST). Network shares **cannot** appear as Drives — they show up as folders in Files. This is intentional in CasaOS's design; don't try to work around it. Library-as-folder is the supported model.
+- Consider adding Samba to this LXC if a future Mac/iOS client needs SMB on the same tree — same LXC, no host changes.
+- No PBS backup (no PBS configured on hubris); the container is fully described in this page + `pct config 102`, rebuild from scratch in <2 min if lost.
+
+## Related
+- [100-zimaos](../vms/100-zimaos.md) — the only consumer today
+- [media permissions](../infrastructure/media-permissions.md) — uid 33 / gid 10000 standard
+- [DNS](../infrastructure/dns.md) — `nfs-export.hubris.network` entry (direct, no Caddy)
+
+## Changelog
+
+### 2026-05-14 — Promoted to rw; squash behaviour verified
+After ZimaOS Files UI evaluation passed (library renders as folder under `/DATA`, thumbnails work, ZimaOS Drives panel ignores NFS by design), flipped export to `rw`. Tested: writing `/DATA/library/.zimaos-rw-test` from ZimaOS appears on hubris's `/mnt/library` owned `www-data:media` (uid 33, gid 10000), confirming `all_squash,anonuid=33,anongid=10000` works as designed. Also discovered the dead end: ZimaOS's `GET /v2/local_storage/storages` is the source of the Drives panel; it returns only physical storage and rejects POST/PUT — network shares cannot be promoted to Drives.
+
+### 2026-05-14 — LXC built; NFS export live (read-only)
+Privileged Debian 13 container created with bind-mount `/mnt/library`. `nfs-kernel-server` installed and enabled; export `/mnt/library` to `192.168.8.0/24` with `ro,all_squash,anonuid=33,anongid=10000`. Smoke-tested from hubris host (mounted, listed library tree, confirmed RO). `nfs-export.hubris.network` added to LXC 124 dnsmasq.
--- a/infrastructure/dns.md
+++ b/infrastructure/dns.md
@@ -31,8 +31,12 @@ address=/blog.hubris.network/192.168.8.175
 address=/photos.hubris.network/192.168.8.175
 address=/artifacto.hubris.network/192.168.8.175
 address=/plato.hubris.network/192.168.8.175
+address=/zimaos.hubris.network/192.168.8.175
+address=/nfs-export.hubris.network/192.168.8.200
 ```

+Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
+
 ## Why split-horizon

 The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
@@ -74,6 +78,12 @@ Either:

 ## Changelog

+### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
+NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
+
+### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern)
+New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
+
 ### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
 New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).

--- a/vms/100-zimaos.md
+++ b/vms/100-zimaos.md
@@ -0,0 +1,74 @@
+# 100 — `zimaos`
+
+ZimaOS (IceWhale / CasaOS-family NAS distro), installed as a Proxmox VM to evaluate it as a potential primary NAS frontend in front of `/mnt/library` — alongside the existing fleet ([nextcloud (114)](../containers/114-nextcloud.md), [jellyfin (101)](../containers/101-jellyfin.md), [syncthing (109)](../containers/109-syncthing.md), [mule-images (120)](../containers/120-mule-images.md)).
+
+## At a glance
+- **Type:** QEMU VM
+- **ZimaOS version:** `v1.6.1` (build 2026-04-21, kernel 6.12.25)
+- **IP:** `192.168.8.195` (DHCP lease from the LAN router — see "Open items" below)
+- **Hostname (LAN DNS):** [`zimaos.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) `192.168.8.175` → VM `192.168.8.195:80`. TLS terminates at Caddy (Let's Encrypt via IONOS DNS-01).
+- **Resources:** 4 vCPU, 8 GiB RAM (balloon min 2 GiB), 64 GiB OS disk on `local-lvm`
+- **BIOS / machine:** OVMF (UEFI), `q35`, `virtio-scsi-pci`
+- **EFI disk:** none — ZimaOS installer writes a removable-media fallback (`/EFI/BOOT/BOOTX64.EFI`), so a persistent EFI vars disk isn't required. PVE warns "no efidisk configured! Using temporary efivars disk" on start; harmless.
+- **ISO:** `/var/lib/vz/template/iso/zimaos-x86_64-1.6.1_installer.iso` (1.48 GB, from `IceWhaleTech/ZimaOS` GitHub releases; upstream publishes SHA256 only for the `.img`/`.raucb` variants, not the `.iso`)
+
+## Web UI
+
+- **LAN:** <https://zimaos.hubris.network> (via [caddy (121)](../containers/121-caddy.md))
+- **Netbird mesh (off-LAN):** same URL — works because hubris advertises `192.168.8.0/24` into the mesh, and once a peer resolves `zimaos.hubris.network → 192.168.8.175`, the request tunnels home. DNS resolution from a mesh peer needs **either** a netbird Management nameserver group (`hubris.network` match domain → `192.168.8.180`), or an `/etc/hosts` override on the peer (`192.168.8.175 zimaos.hubris.network`). See [mesh](../infrastructure/mesh.md).
+- **Direct fallback:** <http://192.168.8.195> — backend ZimaOS bundles its own Caddy (`Server: Caddy`, `Via: ZimaOS-Gateway`), which the front-end Caddy on LXC 121 reverse-proxies to. Two Caddys in series.
+
+## SSH
+
+Disabled by default. To enable:
+1. Toggle SSH on in the ZimaOS web UI ("Settings → SSH"), OR
+2. From the VM console, switch to TTY2 with **Alt+F2** and run `passwd` to set a root password first.
+
+## Storage strategy
+
+**Initial assumption (didn't work).** PVE 9's native virtiofs share with the host's existing `library` dir mapping (used by [108-haos](108-haos.md)) was the obvious path. Attaching it took one line — but ZimaOS's kernel ships *without* the virtiofs module (`modinfo virtiofs` → `Module virtiofs not found`). The kvm-side device was attached fine; the guest just can't mount it. Detached on 2026-05-14.
+
+**What we ended up with.** A dedicated NFS re-export LXC [102-nfs-export](../containers/102-nfs-export.md) bind-mounts `/mnt/library` from the host and serves it over NFSv4 to ZimaOS. Performance is within ~2% of host-served NFS (LXC adds namespace isolation only, not an IO hop), with the upside that no NFS/RPC daemons run on the bare-metal host. Read-only during evaluation; promote to rw once the UI test confirms behavior.
+
+**Filesystem layout inside ZimaOS.** `/etc/fstab` lives on a writable overlay so persistent mounts via fstab DO survive reboots, but the *standard* path to add a network share is the ZimaOS Files UI (Connect Network Storage → NFS). UID/GID mapping is squashed at the server to `www-data:media` (33:10000) — the same identity Nextcloud and mule-images use — so any rw activity from ZimaOS records as a normal `media`-group write.
+
+**Caveats discovered during install.**
+- Root filesystem is **squashfs / read-only**; only `/DATA` is writable.
+- `dtoro` identity is `uid 999, gid 1000(samba), groups samba+wheel`. Primary group is **`samba`** (not `dtoro` — that group doesn't exist). SSH key needs `chown dtoro:samba` to satisfy sshd StrictModes.
+- `dtoro`'s home directory is literally `/DATA` (not `/home/dtoro`). Authorized keys go at `/DATA/.ssh/authorized_keys`.
+- `/etc` is a writable overlay rooted at `/mnt/overlay/upper_etc`. Mounts added to `/etc/fstab` persist across reboots.
+- `mount.nfs4` is a stripped busybox variant — rejects `_netdev`, `nofail`, `actimeo=N`. The working minimal fstab line: `192.168.8.200:/mnt/library /DATA/library nfs ro,vers=4 0 0`.
+- No `qemu-guest-agent` is bundled — `qm guest cmd` calls fail. IP discovery via `qm monitor` + `screendump` instead.
+- No package manager at runtime — no `apt`, no `opkg`. Apps come via ZimaOS's CasaOS-style appstore only.
+
+The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. `qm set 100 --scsi1 library:1024`) was rejected because it would start empty and force content migration; the NFS path keeps `/mnt/library` as the single source of truth shared with the existing 8 LXCs.
+
+## Open items
+
+- **DHCP lease, not a reservation.** `192.168.8.195` came from the LAN router; the [LAN-side dnsmasq](../infrastructure/dns.md) only does DNS, not DHCP. If the lease rotates, the Caddy upstream needs updating too (`/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md)). Better fix: pin a static lease on the LAN router or set a static config inside ZimaOS.
+- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
+- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
+- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`.
+
+## Related
+- [108 — `haos-16.3`](108-haos.md) — the existing VM, same q35/OVMF pattern; also a virtiofs `library` consumer
+- [DNS](../infrastructure/dns.md) — split-horizon entry lives here
+- [Caddy (121)](../containers/121-caddy.md) — would front this if/when promoted
+- [Media permissions](../infrastructure/media-permissions.md) — relevant when wiring `/mnt/library`
+
+## Changelog
+
+### 2026-05-14 — Mesh-reachable from netbird peers
+Verified `https://zimaos.hubris.network` reachable from `republic-laptop` (100.122.78.140) through the existing hubris→netbird subnet advertisement (`192.168.8.0/24`). No new infrastructure required — the path was already wired by [mesh](../infrastructure/mesh.md). Open: configure a netbird Management nameserver group for `hubris.network` so peers don't need per-device `/etc/hosts` overrides.
+
+### 2026-05-14 — Phase 2 promoted to rw
+NFS export on [nfs-export (102)](../containers/102-nfs-export.md) flipped from `ro` to `rw` after Files UI evaluation passed. Verified the squash: a write from ZimaOS appears on hubris's `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention.
+
+### 2026-05-14 — Phase 2: NFS mount via new LXC 102 (read-only)
+Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs module (verified by `modinfo virtiofs` returning "Module not found"). Pivoted to NFSv4: built [nfs-export (102)](../containers/102-nfs-export.md) as a dedicated 512-MiB privileged Debian LXC bind-mounting `/mnt/library` and serving it on `192.168.8.200:2049` with `ro,all_squash,anonuid=33,anongid=10000` to the LAN subnet. Detached the now-useless `virtiofs0` from VM 100. ZimaOS-side mount via Files UI (NFS protocol, server `192.168.8.200`, path `/mnt/library`). Promote export to `rw` after Files UI evaluation passes.
+
+### 2026-05-14 — Wired through Caddy (HTTPS)
+Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
+
+### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
+`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.