From c012a9124f951f9ffc5fa48c169a20e3e9bef7da Mon Sep 17 00:00:00 2001 From: dtoro Date: Wed, 1 Jul 2026 12:28:24 +0200 Subject: [PATCH] infra: join ludo-mini to hubris as Homelab cluster node "strong" Reformatted the ludo-mini workstation to Proxmox VE 9.2.3 and joined it to hubris's existing single-node "Homelab" cluster (2 nodes, no QDevice yet). Added a second NVMe as its own LVM-thin pool (ludo-lvm). Renamed the wiki/inventory identity from ludo-mini to strong to match the OS/ cluster hostname, since bootstrap's client-enrollment lookup depends on that match. Also regenerated hosts/grimmory.yaml, which was missing from git despite being referenced by inventory.yaml. Co-Authored-By: Claude Sonnet 5 --- .sops.yaml | 2 +- README.md | 3 +- hosts/grimmory.yaml | 26 +++++++++ hosts/hubris.md | 37 +++++++++++- hosts/ludo-mini.yaml | 20 ------- hosts/strong.md | 107 +++++++++++++++++++++++++++++++++++ hosts/strong.yaml | 32 +++++++++++ infrastructure/ssh-access.md | 28 +++++---- inventory.yaml | 28 ++++++--- 9 files changed, 242 insertions(+), 41 deletions(-) create mode 100644 hosts/grimmory.yaml delete mode 100644 hosts/ludo-mini.yaml create mode 100644 hosts/strong.md create mode 100644 hosts/strong.yaml diff --git a/.sops.yaml b/.sops.yaml index 85f8547..e89cfd7 100644 --- a/.sops.yaml +++ b/.sops.yaml @@ -41,7 +41,7 @@ creation_rules: - path_regex: ^secrets/gitea-tokens\.yaml$ # Workstations only. age: >- - # placeholder — fill with age_pubkey of: republic-laptop, mac-mini, ludo-mini, hubris + # placeholder — fill with age_pubkey of: republic-laptop, mac-mini, strong, hubris - path_regex: ^secrets/webhook-hmacs\.yaml$ # LXCs that run a webhook receiver. diff --git a/README.md b/README.md index c7a833d..8dafc87 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,8 @@ Living documentation for the **hubris** Proxmox homelab. Every node, every cross ## Map ### Hosts -- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77` +- [`hubris`](hosts/hubris.md) — Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77` — runs everything today +- [`strong`](hosts/strong.md) — Proxmox VE node (cluster hostname `strong`), `192.168.178.181` — 2nd member of the `Homelab` cluster as of 2026-07-01, hosts no guests yet ### VMs - [100 — `zimaos`](vms/100-zimaos.md) — ZimaOS 1.6.1, NAS frontend (evaluation) diff --git a/hosts/grimmory.yaml b/hosts/grimmory.yaml new file mode 100644 index 0000000..6529bf8 --- /dev/null +++ b/hosts/grimmory.yaml @@ -0,0 +1,26 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: grimmory +kind: lxc +os: linux +role: book-library +host: hubris +pve_id: 130 +lan_ip: 192.168.8.213 +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +mounts: +- /mnt/library +public_host: books.hubris.network +notes: +- Docker host for Grimmory (community fork of Booklore). Created 2026-06-29. +- Migrated from apps LXC 105; MariaDB data carried over (schema-compatible fork). +age_pubkey: age1uellsemnjrzgfg9fxw4jefpy05laxzggwnwhh6ny3wl7alyp6v8q0muxet +see_also: +- containers/130-grimmory.md +mcp_endpoint: https://mcp.hubris.network/mcp +secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/hubris.md b/hosts/hubris.md index 4bde448..41e8225 100644 --- a/hosts/hubris.md +++ b/hosts/hubris.md @@ -1,6 +1,9 @@ # `hubris` — Proxmox host -Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab. +Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's +workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node +`Homelab` cluster (see [Cluster](#cluster)); the second node is +[strong](strong.md), which hosts nothing yet. ## At a glance - **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`) @@ -22,6 +25,30 @@ Single-node Proxmox VE running 1 VM and 13 LXC containers. The whole homelab. `/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md). +## Cluster + +Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md) +(cluster/OS hostname `strong`), formed 2026-07-01. + +- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address). + strong reaches it via the existing Fritz!Box static route + (`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the + household LAN. Fine for a home cluster; not latency-isolated. +- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both + votes — if either node is down (reboot, maintenance, network hiccup), the + survivor's running guests keep working but `/etc/pve` goes read-only: + no start/stop/create/edit until quorum returns. Decided to skip a QDevice + for now; revisit if hubris's periodic reboots (BIOS/thermal work, see + Quirks below) make this painful in practice. +- **Storage:** `local` / `local-lvm` are the standard per-node default IDs + (every node has its own, not actually shared). The `library` lvmthin pool + is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since + it's a physical thinpool that only exists on this host's hardware. +- strong currently hosts no LXCs/VMs — it exists solely as a cluster + member so far. See [strong.md](strong.md) and the [library-SSD + migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) + for what comes next (physical drive move, service migration — not started). + ## Tenants ### VMs @@ -62,7 +89,7 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b ## Quirks -- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install. +- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe. - ZFS is **not** in use; storage is LVM-thin + ext4. - Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`. - Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29. @@ -73,6 +100,8 @@ See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/b - `root@hubris` (self, RSA) — local - `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22 +- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so + `pvecm add` could authenticate without a password prompt OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha. @@ -86,9 +115,13 @@ OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:220 - [Backups (disabled)](../infrastructure/backups.md) - [Operations cheatsheet](../operations/commands.md) - [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) +- [strong — Proxmox host](strong.md) ## Changelog +### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab") +User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed. + ### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md). diff --git a/hosts/ludo-mini.yaml b/hosts/ludo-mini.yaml deleted file mode 100644 index e29ff78..0000000 --- a/hosts/ludo-mini.yaml +++ /dev/null @@ -1,20 +0,0 @@ -# Generated by mcp/build_host_files.py from inventory.yaml. -# Do NOT edit by hand — your changes will be overwritten. -# Source of truth: ../inventory.yaml -name: ludo-mini -kind: workstation -os: linux -role: dev -lan_ip: 192.168.178.181 -mesh: - netbird: - fqdn: ludo-mini.netbird.selfhosted -mesh_globals: - primary: netbird - accepted: - - netbird - - tailscale -ssh: - user: dtoro -mcp_endpoint: https://mcp.hubris.network/mcp -secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/hosts/strong.md b/hosts/strong.md new file mode 100644 index 0000000..8a491f9 --- /dev/null +++ b/hosts/strong.md @@ -0,0 +1,107 @@ +# `strong` — Proxmox host + +Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly +a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to +Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet. + +## At a glance + +- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) +- **Naming:** the OS/cluster hostname is `strong` (leftover from install, + kept as-is). This wiki page and `inventory.yaml` also use `strong` as of + 2026-07-01 — earlier the same day the inventory entry was briefly named + `ludo-mini`, but that was renamed so the hostname bootstrap needs + (`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains + the machine's everyday nickname; some older docs (investigations, the + library-SSD migration plan) still refer to it that way — that's fine, + those are historical. +- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM. + - `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data` + (thinpool) 815G, 8G swap. + - `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an + existing APFS partition (previously used in a Mac) — wiped via + `pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned + into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin` + (handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one + step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool + `ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what + was actually created live; not worth renaming), restricted `nodes strong` + in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool. + Empty so far; this is separate from the [library-SSD migration + plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s + planned drive move from hubris (that hasn't happened) — this is + general-purpose VM/CT capacity. +- **Network:** `vmbr0` is bridged straight onto the household LAN — + `192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no + internal/uplink bridge split yet — `vmbr0` here plays the role hubris's + `vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`) + via the existing Fritz!Box static route through hubris; no new cabling + or routing was needed to join the cluster. +- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared + with hubris by convention, actually separate per-node volumes). Hubris's + `library` lvmthin pool is restricted to `nodes hubris` in + `/etc/pve/storage.cfg` and does not exist here. +- **Mesh:** Netbird not installed — fresh OS wiped whatever the old + workstation had. Reachable today only via LAN routing (confirmed DNS for + `*.hubris.network` already resolves correctly here via `192.168.8.2`). + Add to Netbird if off-LAN access to this host itself (distinct from any + future guests) is needed. +- **Homelab-context client enrollment:** not done yet — no + `/opt/homelab-context`, no age key, no `homelab` CLI. Separate from the + Proxmox cluster join. See [agent-enrollment.md](../operations/agent-enrollment.md). + +## Cluster membership + +Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on +2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full +cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice — +either node going down freezes management on the survivor). + +## SSH + +Root login via the same key set trusted on hubris (`root@hubris`, +`d.toro.v@pm.me`) — installed 2026-07-01 by appending to +`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys` +post cluster-join, so it's cluster-synced same as hubris). No password auth +needed going forward. + +## Related + +- [hubris — Proxmox host](hubris.md) +- [Library SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical) +- [Network](../infrastructure/network.md) +- [SSH access](../infrastructure/ssh-access.md) +- [Agent enrollment](../operations/agent-enrollment.md) + +## Changelog + +### 2026-07-01 — inventory identity renamed ludo-mini → strong +Discovered while starting client enrollment: `bootstrap.sh` looks up +`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not +`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster +member — Proxmox doesn't support in-place node rename, only leave+rejoin). +Renamed the wiki/inventory side instead: `inventory.yaml` key, this page +(`hosts/ludo-mini.md` → `hosts/strong.md`), README, ssh-access.md all now +say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname. + +### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm` +User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac +(arrived with an EFI + APFS partition table — confirmed disposable, wiped). +Used Proxmox's own disk-management API rather than raw LVM commands: +`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old +partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin +--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202 +--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox +storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content +rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction +pattern — this pool only physically exists here). Empty — no VM/CT disks +placed on it yet. + +### 2026-07-01 — Proxmox install; joined Homelab cluster +Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from +hubris's trusted set (root password used once, then discarded). Joined the +existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from +this node's side, using key-based SSH pre-authorized in both directions — +no interactive password prompt needed for the join itself. Cluster now 2 +nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname +as `strong` and skip a QDevice for now — both revisitable later. diff --git a/hosts/strong.yaml b/hosts/strong.yaml new file mode 100644 index 0000000..7a4db40 --- /dev/null +++ b/hosts/strong.yaml @@ -0,0 +1,32 @@ +# Generated by mcp/build_host_files.py from inventory.yaml. +# Do NOT edit by hand — your changes will be overwritten. +# Source of truth: ../inventory.yaml +name: strong +kind: proxmox-host +os: linux +role: hypervisor +lan_ip: 192.168.178.181 +mesh_globals: + primary: netbird + accepted: + - netbird + - tailscale +ssh: + user: root +notes: +- Reformatted from Linux workstation ("ludo-mini" in this wiki, still the machine's nickname) to Proxmox + VE 9.2.3 on 2026-07-01. Renamed the inventory/wiki identity from ludo-mini to strong on the same day + so it matches the OS/cluster hostname everywhere (bootstrap looks up hosts/$(hostname).yaml, so a mismatch + would break enrollment). +- "Joined hubris's \"Homelab\" cluster same day. 2-node, no QDevice tiebreaker yet \u2014 see hosts/hubris.md\ + \ quorum note." +- Netbird not yet installed (fresh OS wiped prior enrollment); reachable today only via the household + LAN / existing Fritz static route to 192.168.8.0/24. Re-enroll in mesh as a follow-up if off-LAN access + to this host itself (not just its future guests) is needed. +- "First step of the planned library-SSD migration \u2014 see .hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md\ + \ (filename kept as-is, it's a historical planning doc). Only Phase 1 (Proxmox install + cluster join)\ + \ is done; no physical drive move, service migration, or GPU passthrough has happened yet." +see_also: +- hosts/strong.md +mcp_endpoint: https://mcp.hubris.network/mcp +secrets_issuance_endpoint: https://secrets.hubris.network/issue diff --git a/infrastructure/ssh-access.md b/infrastructure/ssh-access.md index 3b80d08..44c05eb 100644 --- a/infrastructure/ssh-access.md +++ b/infrastructure/ssh-access.md @@ -97,19 +97,23 @@ When onboarding a new machine: ## Hosts -### Hubris (PVE host) +### Hubris + strong (PVE cluster: `Homelab`) -| Detail | Value | -|--------|-------| -| LAN IP | `192.168.8.77` | -| Netbird | `100.122.38.109` (FQDN: `proxmox-server.netbird.selfhosted`) | -| Netbird SSH port | `22022` (mesh-only, OIDC auth) | -| SSH user | `root` | -| Authorized keys | `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced) | +Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox +cluster-synced, so a key added on either node is authorized on both. -Authorized root keys currently deployed: +| Detail | hubris | strong | +|--------|--------|-----------| +| LAN IP | `192.168.8.77` | `192.168.178.181` | +| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) | +| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet | +| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a | +| SSH user | `root` | `root` | + +Authorized root keys currently deployed (cluster-wide): - `root@hubris` (self, RSA) - `d.toro.v@pm.me` (ed25519) — mac-mini +- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join ### LXCs @@ -138,7 +142,8 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`. |------|----|--------|--------------|----------| | mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` | | republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` | -| ludo-mini | Linux | `192.168.8.133` | `ludo-mini.netbird.selfhosted` | TBD | + +strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above. ### VPS (external) @@ -178,6 +183,9 @@ done ## Changelog +### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected +strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md). + ### 2026-06-02 — universal SSH reachability Replaced ad-hoc per-workstation SSH configs with inventory-generated diff --git a/inventory.yaml b/inventory.yaml index 670b3bc..af99d7f 100644 --- a/inventory.yaml +++ b/inventory.yaml @@ -370,16 +370,30 @@ hosts: notes: - Only macOS in the fleet. Bootstrap uses launchd. age_pubkey: age1z62ff2ak9zj5ctcvaxwyyhedwjvlwgm2dkn9nk3wrwk8fkavcpmsqwc2vs - ludo-mini: - kind: workstation + strong: + kind: proxmox-host os: linux - role: dev + role: hypervisor lan_ip: 192.168.178.181 - mesh: - netbird: - fqdn: ludo-mini.netbird.selfhosted ssh: - user: dtoro + user: root + notes: + - Reformatted from Linux workstation ("ludo-mini" in this wiki, still + the machine's nickname) to Proxmox VE 9.2.3 on 2026-07-01. Renamed + the inventory/wiki identity from ludo-mini to strong on the same day + so it matches the OS/cluster hostname everywhere (bootstrap looks up + hosts/$(hostname).yaml, so a mismatch would break enrollment). + - Joined hubris's "Homelab" cluster same day. 2-node, no QDevice + tiebreaker yet — see hosts/hubris.md quorum note. + - Netbird not yet installed (fresh OS wiped prior enrollment); reachable + today only via the household LAN / existing Fritz static route to + 192.168.8.0/24. Re-enroll in mesh as a follow-up if off-LAN access + to this host itself (not just its future guests) is needed. + - First step of the planned library-SSD migration — see + .hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md + (filename kept as-is, it's a historical planning doc). Only Phase 1 + (Proxmox install + cluster join) is done; no physical + drive move, service migration, or GPU passthrough has happened yet. netbird-vps: kind: external os: linux