infra: join ludo-mini to hubris as Homelab cluster node "strong"

Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed the wiki/inventory identity from ludo-mini to strong to match the OS/ cluster hostname, since bootstrap's client-enrollment lookup depends on that match. Also regenerated hosts/grimmory.yaml, which was missing from git despite being referenced by inventory.yaml. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-01 12:28:24 +02:00
parent 41df77fb20
commit c012a9124f
9 changed files with 242 additions and 41 deletions
--- a/.sops.yaml
+++ b/.sops.yaml
@@ -41,7 +41,7 @@ creation_rules:
  - path_regex: ^secrets/gitea-tokens\.yaml$
    # Workstations only.
    age: >-
-      # placeholder — fill with age_pubkey of: republic-laptop, mac-mini, ludo-mini, hubris
+      # placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris

  - path_regex: ^secrets/webhook-hmacs\.yaml$
    # LXCs that run a webhook receiver.
--- a/README.md
+++ b/README.md
@@ -7,7 +7,8 @@ Living documentation for the **hubris** Proxmox homelab. Every node, every cross
 ## Map

 ### Hosts
- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77`
+- [`hubris`](hosts/hubris.md) — Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77` — runs everything today
+- [`strong`](hosts/strong.md) — Proxmox VE node (cluster hostname `strong`), `192.168.178.181` — 2nd member of the `Homelab` cluster as of 2026-07-01, hosts no guests yet

 ### VMs
 - [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation)
--- a/hosts/grimmory.yaml
+++ b/hosts/grimmory.yaml
@@ -0,0 +1,26 @@
+# Generated by mcp/build_host_files.py from inventory.yaml.
+# Do NOT edit by hand — your changes will be overwritten.
+# Source of truth: ../inventory.yaml
+name: grimmory
+kind: lxc
+os: linux
+role: book-library
+host: hubris
+pve_id: 130
+lan_ip: 192.168.8.213
+mesh_globals:
+  primary: netbird
+  accepted:
+  - netbird
+  - tailscale
+mounts:
+- /mnt/library
+public_host: books.hubris.network
+notes:
+- Docker host for Grimmory (community fork of Booklore). Created 2026-06-29.
+- Migrated from apps LXC 105; MariaDB data carried over (schema-compatible fork).
+age_pubkey: age1uellsemnjrzgfg9fxw4jefpy05laxzggwnwhh6ny3wl7alyp6v8q0muxet
+see_also:
+- containers/130-grimmory.md
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue
--- a/hosts/hubris.md
+++ b/hosts/hubris.md
@@ -1,6 +1,9 @@
 # `hubris` — Proxmox host

-Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.
+Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
+workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
+`Homelab` cluster (see [Cluster](#cluster)); the second node is
+[strong](strong.md), which hosts nothing yet.

 ## At a glance
 - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
@@ -22,6 +25,30 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab.

 `/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).

+## Cluster
+
+Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
+(cluster/OS hostname `strong`), formed 2026-07-01.
+
+- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
+  strong reaches it via the existing Fritz!Box static route
+  (`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
+  household LAN. Fine for a home cluster; not latency-isolated.
+- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
+  votes — if either node is down (reboot, maintenance, network hiccup), the
+  survivor's running guests keep working but `/etc/pve` goes read-only:
+  no start/stop/create/edit until quorum returns. Decided to skip a QDevice
+  for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
+  Quirks below) make this painful in practice.
+- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
+  (every node has its own, not actually shared). The `library` lvmthin pool
+  is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
+  it's a physical thinpool that only exists on this host's hardware.
+- strong currently hosts no LXCs/VMs — it exists solely as a cluster
+  member so far. See [strong.md](strong.md) and the [library-SSD
+  migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
+  for what comes next (physical drive move, service migration — not started).
+
 ## Tenants

 ### VMs
@@ -62,7 +89,7 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b

 ## Quirks

- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install.
+- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
 - ZFS is **not** in use; storage is LVM-thin + ext4.
 - Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
 - Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
@@ -73,6 +100,8 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b

 - `root@hubris` (self, RSA) — local
 - `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
+- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
+  `pvecm add` could authenticate without a password prompt

 OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.

@@ -86,9 +115,13 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220
 - [Backups (disabled)](../infrastructure/backups.md)
 - [Operations cheatsheet](../operations/commands.md)
 - [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+- [strong — Proxmox host](strong.md)

 ## Changelog

+### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
+User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
+
 ### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
 Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).

--- a/hosts/ludo-mini.yaml
+++ b/hosts/ludo-mini.yaml
@@ -1,20 +0,0 @@
-# Generated by mcp/build_host_files.py from inventory.yaml.
-# Do NOT edit by hand — your changes will be overwritten.
-# Source of truth: ../inventory.yaml
-name: ludo-mini
-kind: workstation
-os: linux
-role: dev
-lan_ip: 192.168.178.181
-mesh:
-  netbird:
-    fqdn: ludo-mini.netbird.selfhosted
-mesh_globals:
-  primary: netbird
-  accepted:
-  - netbird
-  - tailscale
-ssh:
-  user: dtoro
-mcp_endpoint: https://mcp.hubris.network/mcp
-secrets_issuance_endpoint: https://secrets.hubris.network/issue
--- a/hosts/strong.md
+++ b/hosts/strong.md
@@ -0,0 +1,107 @@
+# `strong` — Proxmox host
+
+Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
+a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
+Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
+
+## At a glance
+
+- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`)
+- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
+  kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
+  2026-07-01 — earlier the same day the inventory entry was briefly named
+  `ludo-mini`, but that was renamed so the hostname bootstrap needs
+  (`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
+  the machine's everyday nickname; some older docs (investigations, the
+  library-SSD migration plan) still refer to it that way — that's fine,
+  those are historical.
+- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
+  - `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
+    (thinpool) 815G, 8G swap.
+  - `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
+    existing APFS partition (previously used in a Mac) — wiped via
+    `pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
+    into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
+    (handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
+    step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
+    `ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
+    was actually created live; not worth renaming), restricted `nodes strong`
+    in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
+    Empty so far; this is separate from the [library-SSD migration
+    plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
+    planned drive move from hubris (that hasn't happened) — this is
+    general-purpose VM/CT capacity.
+- **Network:** `vmbr0` is bridged straight onto the household LAN —
+  `192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
+  internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
+  `vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
+  via the existing Fritz!Box static route through hubris; no new cabling
+  or routing was needed to join the cluster.
+- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
+  with hubris by convention, actually separate per-node volumes). Hubris's
+  `library` lvmthin pool is restricted to `nodes hubris` in
+  `/etc/pve/storage.cfg` and does not exist here.
+- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
+  workstation had. Reachable today only via LAN routing (confirmed DNS for
+  `*.hubris.network` already resolves correctly here via `192.168.8.2`).
+  Add to Netbird if off-LAN access to this host itself (distinct from any
+  future guests) is needed.
+- **Homelab-context client enrollment:** not done yet — no
+  `/opt/homelab-context`, no age key, no `homelab` CLI. Separate from the
+  Proxmox cluster join. See [agent-enrollment.md](../operations/agent-enrollment.md).
+
+## Cluster membership
+
+Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
+2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
+cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
+either node going down freezes management on the survivor).
+
+## SSH
+
+Root login via the same key set trusted on hubris (`root@hubris`,
+`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
+`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
+post cluster-join, so it's cluster-synced same as hubris). No password auth
+needed going forward.
+
+## Related
+
+- [hubris — Proxmox host](hubris.md)
+- [Library SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
+- [Network](../infrastructure/network.md)
+- [SSH access](../infrastructure/ssh-access.md)
+- [Agent enrollment](../operations/agent-enrollment.md)
+
+## Changelog
+
+### 2026-07-01 — inventory identity renamed ludo-mini → strong
+Discovered while starting client enrollment: `bootstrap.sh` looks up
+`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
+`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
+member — Proxmox doesn't support in-place node rename, only leave+rejoin).
+Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
+(`hosts/ludo-mini.md` → `hosts/strong.md`), README, ssh-access.md all now
+say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
+
+### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
+User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
+(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
+Used Proxmox's own disk-management API rather than raw LVM commands:
+`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
+partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
+--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
+--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
+storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
+rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
+pattern — this pool only physically exists here). Empty — no VM/CT disks
+placed on it yet.
+
+### 2026-07-01 — Proxmox install; joined Homelab cluster
+Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
+hubris's trusted set (root password used once, then discarded). Joined the
+existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
+this node's side, using key-based SSH pre-authorized in both directions —
+no interactive password prompt needed for the join itself. Cluster now 2
+nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
+as `strong` and skip a QDevice for now — both revisitable later.
--- a/hosts/strong.yaml
+++ b/hosts/strong.yaml
@@ -0,0 +1,32 @@
+# Generated by mcp/build_host_files.py from inventory.yaml.
+# Do NOT edit by hand — your changes will be overwritten.
+# Source of truth: ../inventory.yaml
+name: strong
+kind: proxmox-host
+os: linux
+role: hypervisor
+lan_ip: 192.168.178.181
+mesh_globals:
+  primary: netbird
+  accepted:
+  - netbird
+  - tailscale
+ssh:
+  user: root
+notes:
+- Reformatted from Linux workstation ("ludo-mini" in this wiki, still the machine's nickname) to Proxmox
+  VE 9.2.3 on 2026-07-01. Renamed the inventory/wiki identity from ludo-mini to strong on the same day
+  so it matches the OS/cluster hostname everywhere (bootstrap looks up hosts/$(hostname).yaml, so a mismatch
+  would break enrollment).
+- "Joined hubris's \"Homelab\" cluster same day. 2-node, no QDevice tiebreaker yet \u2014 see hosts/hubris.md\
+  \ quorum note."
+- Netbird not yet installed (fresh OS wiped prior enrollment); reachable today only via the household
+  LAN / existing Fritz static route to 192.168.8.0/24. Re-enroll in mesh as a follow-up if off-LAN access
+  to this host itself (not just its future guests) is needed.
+- "First step of the planned library-SSD migration \u2014 see .hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md\
+  \ (filename kept as-is, it's a historical planning doc). Only Phase 1 (Proxmox install + cluster join)\
+  \ is done; no physical drive move, service migration, or GPU passthrough has happened yet."
+see_also:
+- hosts/strong.md
+mcp_endpoint: https://mcp.hubris.network/mcp
+secrets_issuance_endpoint: https://secrets.hubris.network/issue
--- a/infrastructure/ssh-access.md
+++ b/infrastructure/ssh-access.md
@@ -97,19 +97,23 @@ When onboarding a new machine:

 ## Hosts

-### Hubris (PVE host)
+### Hubris + strong (PVE cluster: `Homelab`)

-| Detail | Value |
-|--------|-------|
-| LAN IP | `192.168.8.77` |
-| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) |
-| Netbird SSH port | `22022` (mesh-only, OIDC auth) |
-| SSH user | `root` |
-| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) |
+Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox
+cluster-synced, so a key added on either node is authorized on both.

-Authorized root keys currently deployed:
+| Detail | hubris | strong |
+|--------|--------|-----------|
+| LAN IP | `192.168.8.77` | `192.168.178.181` |
+| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) |
+| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet |
+| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a |
+| SSH user | `root` | `root` |
+
+Authorized root keys currently deployed (cluster-wide):
 - `root@hubris` (self, RSA)
 - `d.toro.v@pm.me` (ed25519) — mac-mini
+- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join

 ### LXCs

@@ -138,7 +142,8 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
 |------|----|--------|--------------|----------|
 | mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
 | republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
-| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD |
+
+strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above.

 ### VPS (external)

@@ -178,6 +183,9 @@ done

 ## Changelog

+### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected
+strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md).
+
 ### 2026-06-02 — universal SSH reachability

 Replaced ad-hoc per-workstation SSH configs with inventory-generated
--- a/inventory.yaml
+++ b/inventory.yaml
@@ -370,16 +370,30 @@ hosts:
    notes:
    - Only macOS in the fleet. Bootstrap uses launchd.
    age_pubkey: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs
-  ludo-mini:
-    kind: workstation
+  strong:
+    kind: proxmox-host
    os: linux
-    role: dev
+    role: hypervisor
    lan_ip: 192.168.178.181
-    mesh:
-      netbird:
-        fqdn: ludo-mini.netbird.selfhosted
    ssh:
-      user: dtoro
+      user: root
+    notes:
+    - Reformatted from Linux workstation ("ludo-mini" in this wiki, still
+      the machine's nickname) to Proxmox VE 9.2.3 on 2026-07-01. Renamed
+      the inventory/wiki identity from ludo-mini to strong on the same day
+      so it matches the OS/cluster hostname everywhere (bootstrap looks up
+      hosts/$(hostname).yaml, so a mismatch would break enrollment).
+    - Joined hubris's "Homelab" cluster same day. 2-node, no QDevice
+      tiebreaker yet — see hosts/hubris.md quorum note.
+    - Netbird not yet installed (fresh OS wiped prior enrollment); reachable
+      today only via the household LAN / existing Fritz static route to
+      192.168.8.0/24. Re-enroll in mesh as a follow-up if off-LAN access
+      to this host itself (not just its future guests) is needed.
+    - First step of the planned library-SSD migration — see
+      .hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md
+      (filename kept as-is, it's a historical planning doc). Only Phase 1
+      (Proxmox install + cluster join) is done; no physical
+      drive move, service migration, or GPU passthrough has happened yet.
  netbird-vps:
    kind: external
    os: linux