diff --git a/README.md b/README.md index f03cf11..56caa2e 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,7 @@ Living documentation for the **hubris** Proxmox homelab. Every node, every cross - [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77` ### VMs +- [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation) - [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS ### LXC containers @@ -18,6 +19,7 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links: | ID | Name | IP | Role | | --- | ---------------- | --------------- | --------------------------------------------- | | 101 | [jellyfin](containers/101-jellyfin.md) | 192.168.8.206 | Media server | +| 102 | [nfs-export](containers/102-nfs-export.md) | 192.168.8.200 | NFSv4 re-export of /mnt/library for ZimaOS | | 103 | [paperless](containers/103-paperless.md) | 192.168.8.130 | Document mgmt | | 104 | [gitea](containers/104-gitea.md) | 192.168.8.121 | Git server | | 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) | diff --git a/containers/102-nfs-export.md b/containers/102-nfs-export.md new file mode 100644 index 0000000..b57be10 --- /dev/null +++ b/containers/102-nfs-export.md @@ -0,0 +1,89 @@ +# 102 — `nfs-export` + +Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clients that can't use the host's PVE virtiofs path — currently only [100-zimaos](../vms/100-zimaos.md), which ships a kernel without virtiofs support. + +## At a glance +- **Hostname:** `nfs-export` +- **IP:** `192.168.8.200` (static; LAN-only, no Caddy in front because NFS is L4) +- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy) +- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server` +- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap +- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, plato, syncthing, apps) + +## What it does + +``` +/mnt/library (host ext4 on nvme1n1) + │ + ├── bind-mounted into 8 other LXCs (jellyfin, paperless, …) + └── bind-mounted into LXC 102 + │ + └── nfs-kernel-server exports /mnt/library + │ + └── consumed by VM 100 (ZimaOS) +``` + +Same inodes, same page cache. The NFS server is just one more access path on top of a tree that 8 other consumers already share — see [media permissions](../infrastructure/media-permissions.md). + +## Export config + +`/etc/exports`: + +``` +/mnt/library 192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys) +``` + +Initially started as `ro`; promoted to `rw` on 2026-05-14 after the Files-UI evaluation confirmed (a) the library renders correctly as a folder under `/DATA`, (b) thumbnails are generated, (c) the squash works — a write from ZimaOS appears on `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention used by Nextcloud and mule-images. + +### Guardrails (in order of importance) + +1. **`all_squash,anonuid=33,anongid=10000`.** Every write from ZimaOS records on disk as `www-data:media` (uid 33, gid 10000), the same identity Nextcloud and mule-images use. Keeps the existing tree convention from drifting. See [`project_media_perms`](../infrastructure/media-permissions.md). +3. **Subnet restriction `192.168.8.0/24`.** No public/mesh access; LAN only. +4. **`no_subtree_check`** + **`sec=sys`** — standard performance/auth pair for a homelab. +5. **No `crossmnt`** because `/mnt/library` is a single ext4 filesystem on the host (no nested mounts to traverse). + +### What we're *not* doing yet + +- No per-subdir export. ZimaOS sees the whole tree; access is controlled by filesystem permissions (`drwxr-x---` private dirs like `documents/`, `notes/`, `heaper/` aren't readable by `media` group, so ZimaOS-as-squashed-uid won't see them either). +- No write-back. Until we promote to `rw`, ZimaOS can't write — eliminates the lock-domain split concern between NFS clients (NLM/v4) and local LXCs (POSIX flock). +- No Authentik / forward-auth. NFS doesn't sit behind HTTP, so the standard caddy+authentik path doesn't apply. Subnet ACL is the only auth. + +## Why this LXC exists (vs serving from host) + +We considered three options before building this: + +| Option | Outcome | +|---|---| +| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../investigations/index.md)). Rejected. | +| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. | +| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. | + +Rationale lives in [the install plan](https://git.hubris.network/dtoro/Homelab-Docs) `/root/.claude/plans/i-wannt-you-to-nifty-muffin.md` on hubris. + +## Operations + +- **Reload exports** after editing `/etc/exports`: `pct exec 102 -- exportfs -ra` +- **List active exports:** `pct exec 102 -- exportfs -v` +- **Watch from outside:** `showmount -e 192.168.8.200` +- **Service health:** `pct exec 102 -- systemctl is-active nfs-server rpcbind` +- **Restart cleanly:** `pct restart 102` (ZimaOS will retry the mount via `nofail`) +- **Destroy + rebuild:** `pct stop 102 && pct destroy 102 --purge` — reversible in seconds; only ZimaOS notices + +## Open items +- Consider tightening the export to subdirs (e.g. `movies`, `tv`, `music`, `audiobooks`, `books`, `images`, `podcasts`, `roms`) if you don't want ZimaOS reachable into `documents/`, `notes/`, `heaper/`, etc. — though those private subdirs are already invisible to `media`-group perms. +- ZimaOS architecture finding: the Drives panel only enumerates physical/block devices via `GET /v2/local_storage/storages` (read-only API, no POST). Network shares **cannot** appear as Drives — they show up as folders in Files. This is intentional in CasaOS's design; don't try to work around it. Library-as-folder is the supported model. +- Consider adding Samba to this LXC if a future Mac/iOS client needs SMB on the same tree — same LXC, no host changes. +- No PBS backup (no PBS configured on hubris); the container is fully described in this page + `pct config 102`, rebuild from scratch in <2 min if lost. + +## Related +- [100-zimaos](../vms/100-zimaos.md) — the only consumer today +- [media permissions](../infrastructure/media-permissions.md) — uid 33 / gid 10000 standard +- [DNS](../infrastructure/dns.md) — `nfs-export.hubris.network` entry (direct, no Caddy) + +## Changelog + +### 2026-05-14 — Promoted to rw; squash behaviour verified +After ZimaOS Files UI evaluation passed (library renders as folder under `/DATA`, thumbnails work, ZimaOS Drives panel ignores NFS by design), flipped export to `rw`. Tested: writing `/DATA/library/.zimaos-rw-test` from ZimaOS appears on hubris's `/mnt/library` owned `www-data:media` (uid 33, gid 10000), confirming `all_squash,anonuid=33,anongid=10000` works as designed. Also discovered the dead end: ZimaOS's `GET /v2/local_storage/storages` is the source of the Drives panel; it returns only physical storage and rejects POST/PUT — network shares cannot be promoted to Drives. + +### 2026-05-14 — LXC built; NFS export live (read-only) +Privileged Debian 13 container created with bind-mount `/mnt/library`. `nfs-kernel-server` installed and enabled; export `/mnt/library` to `192.168.8.0/24` with `ro,all_squash,anonuid=33,anongid=10000`. Smoke-tested from hubris host (mounted, listed library tree, confirmed RO). `nfs-export.hubris.network` added to LXC 124 dnsmasq. diff --git a/infrastructure/dns.md b/infrastructure/dns.md index 68d436d..e9b6c2f 100644 --- a/infrastructure/dns.md +++ b/infrastructure/dns.md @@ -31,8 +31,12 @@ address=/blog.hubris.network/192.168.8.175 address=/photos.hubris.network/192.168.8.175 address=/artifacto.hubris.network/192.168.8.175 address=/plato.hubris.network/192.168.8.175 +address=/zimaos.hubris.network/192.168.8.175 +address=/nfs-export.hubris.network/192.168.8.200 ``` +Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do. + ## Why split-horizon The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy. @@ -74,6 +78,12 @@ Either: ## Changelog +### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP) +NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful. + +### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern) +New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS). + ### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed New LAN-only entry for [plato (126)](../containers/126-plato.md). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend). diff --git a/vms/100-zimaos.md b/vms/100-zimaos.md new file mode 100644 index 0000000..d8a7981 --- /dev/null +++ b/vms/100-zimaos.md @@ -0,0 +1,74 @@ +# 100 — `zimaos` + +ZimaOS (IceWhale / CasaOS-family NAS distro), installed as a Proxmox VM to evaluate it as a potential primary NAS frontend in front of `/mnt/library` — alongside the existing fleet ([nextcloud (114)](../containers/114-nextcloud.md), [jellyfin (101)](../containers/101-jellyfin.md), [syncthing (109)](../containers/109-syncthing.md), [mule-images (120)](../containers/120-mule-images.md)). + +## At a glance +- **Type:** QEMU VM +- **ZimaOS version:** `v1.6.1` (build 2026-04-21, kernel 6.12.25) +- **IP:** `192.168.8.195` (DHCP lease from the LAN router — see "Open items" below) +- **Hostname (LAN DNS):** [`zimaos.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) `192.168.8.175` → VM `192.168.8.195:80`. TLS terminates at Caddy (Let's Encrypt via IONOS DNS-01). +- **Resources:** 4 vCPU, 8 GiB RAM (balloon min 2 GiB), 64 GiB OS disk on `local-lvm` +- **BIOS / machine:** OVMF (UEFI), `q35`, `virtio-scsi-pci` +- **EFI disk:** none — ZimaOS installer writes a removable-media fallback (`/EFI/BOOT/BOOTX64.EFI`), so a persistent EFI vars disk isn't required. PVE warns "no efidisk configured! Using temporary efivars disk" on start; harmless. +- **ISO:** `/var/lib/vz/template/iso/zimaos-x86_64-1.6.1_installer.iso` (1.48 GB, from `IceWhaleTech/ZimaOS` GitHub releases; upstream publishes SHA256 only for the `.img`/`.raucb` variants, not the `.iso`) + +## Web UI + +- **LAN:** (via [caddy (121)](../containers/121-caddy.md)) +- **Netbird mesh (off-LAN):** same URL — works because hubris advertises `192.168.8.0/24` into the mesh, and once a peer resolves `zimaos.hubris.network → 192.168.8.175`, the request tunnels home. DNS resolution from a mesh peer needs **either** a netbird Management nameserver group (`hubris.network` match domain → `192.168.8.180`), or an `/etc/hosts` override on the peer (`192.168.8.175 zimaos.hubris.network`). See [mesh](../infrastructure/mesh.md). +- **Direct fallback:** — backend ZimaOS bundles its own Caddy (`Server: Caddy`, `Via: ZimaOS-Gateway`), which the front-end Caddy on LXC 121 reverse-proxies to. Two Caddys in series. + +## SSH + +Disabled by default. To enable: +1. Toggle SSH on in the ZimaOS web UI ("Settings → SSH"), OR +2. From the VM console, switch to TTY2 with **Alt+F2** and run `passwd` to set a root password first. + +## Storage strategy + +**Initial assumption (didn't work).** PVE 9's native virtiofs share with the host's existing `library` dir mapping (used by [108-haos](108-haos.md)) was the obvious path. Attaching it took one line — but ZimaOS's kernel ships *without* the virtiofs module (`modinfo virtiofs` → `Module virtiofs not found`). The kvm-side device was attached fine; the guest just can't mount it. Detached on 2026-05-14. + +**What we ended up with.** A dedicated NFS re-export LXC [102-nfs-export](../containers/102-nfs-export.md) bind-mounts `/mnt/library` from the host and serves it over NFSv4 to ZimaOS. Performance is within ~2% of host-served NFS (LXC adds namespace isolation only, not an IO hop), with the upside that no NFS/RPC daemons run on the bare-metal host. Read-only during evaluation; promote to rw once the UI test confirms behavior. + +**Filesystem layout inside ZimaOS.** `/etc/fstab` lives on a writable overlay so persistent mounts via fstab DO survive reboots, but the *standard* path to add a network share is the ZimaOS Files UI (Connect Network Storage → NFS). UID/GID mapping is squashed at the server to `www-data:media` (33:10000) — the same identity Nextcloud and mule-images use — so any rw activity from ZimaOS records as a normal `media`-group write. + +**Caveats discovered during install.** +- Root filesystem is **squashfs / read-only**; only `/DATA` is writable. +- `dtoro` identity is `uid 999, gid 1000(samba), groups samba+wheel`. Primary group is **`samba`** (not `dtoro` — that group doesn't exist). SSH key needs `chown dtoro:samba` to satisfy sshd StrictModes. +- `dtoro`'s home directory is literally `/DATA` (not `/home/dtoro`). Authorized keys go at `/DATA/.ssh/authorized_keys`. +- `/etc` is a writable overlay rooted at `/mnt/overlay/upper_etc`. Mounts added to `/etc/fstab` persist across reboots. +- `mount.nfs4` is a stripped busybox variant — rejects `_netdev`, `nofail`, `actimeo=N`. The working minimal fstab line: `192.168.8.200:/mnt/library /DATA/library nfs ro,vers=4 0 0`. +- No `qemu-guest-agent` is bundled — `qm guest cmd` calls fail. IP discovery via `qm monitor` + `screendump` instead. +- No package manager at runtime — no `apt`, no `opkg`. Apps come via ZimaOS's CasaOS-style appstore only. + +The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. `qm set 100 --scsi1 library:1024`) was rejected because it would start empty and force content migration; the NFS path keeps `/mnt/library` as the single source of truth shared with the existing 8 LXCs. + +## Open items + +- **DHCP lease, not a reservation.** `192.168.8.195` came from the LAN router; the [LAN-side dnsmasq](../infrastructure/dns.md) only does DNS, not DHCP. If the lease rotates, the Caddy upstream needs updating too (`/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md)). Better fix: pin a static lease on the LAN router or set a static config inside ZimaOS. +- **No Authentik wiring.** [authentik (124)](../containers/124-authentik.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists. +- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up. +- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`. + +## Related +- [108 — `haos-16.3`](108-haos.md) — the existing VM, same q35/OVMF pattern; also a virtiofs `library` consumer +- [DNS](../infrastructure/dns.md) — split-horizon entry lives here +- [Caddy (121)](../containers/121-caddy.md) — would front this if/when promoted +- [Media permissions](../infrastructure/media-permissions.md) — relevant when wiring `/mnt/library` + +## Changelog + +### 2026-05-14 — Mesh-reachable from netbird peers +Verified `https://zimaos.hubris.network` reachable from `republic-laptop` (100.122.78.140) through the existing hubris→netbird subnet advertisement (`192.168.8.0/24`). No new infrastructure required — the path was already wired by [mesh](../infrastructure/mesh.md). Open: configure a netbird Management nameserver group for `hubris.network` so peers don't need per-device `/etc/hosts` overrides. + +### 2026-05-14 — Phase 2 promoted to rw +NFS export on [nfs-export (102)](../containers/102-nfs-export.md) flipped from `ro` to `rw` after Files UI evaluation passed. Verified the squash: a write from ZimaOS appears on hubris's `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention. + +### 2026-05-14 — Phase 2: NFS mount via new LXC 102 (read-only) +Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs module (verified by `modinfo virtiofs` returning "Module not found"). Pivoted to NFSv4: built [nfs-export (102)](../containers/102-nfs-export.md) as a dedicated 512-MiB privileged Debian LXC bind-mounting `/mnt/library` and serving it on `192.168.8.200:2049` with `ro,all_squash,anonuid=33,anongid=10000` to the LAN subnet. Detached the now-useless `virtiofs0` from VM 100. ZimaOS-side mount via Files UI (NFS protocol, server `192.168.8.200`, path `/mnt/library`). Promote export to `rw` after Files UI evaluation passes. + +### 2026-05-14 — Wired through Caddy (HTTPS) +Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`. + +### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1) +`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/124-authentik.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.