dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion

Browse Source 
- Migrations 010 (content_hash) + 011 (search tsvector column)
- new: internal/knowledge/seed.go — knowledge seed ingest engine
- new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge
- wire knowledge ingest into oikos seed pipeline
- convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml
- archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/
- delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py
- remove empty knowledge/ directory tree
This commit is contained in:
authentik Default Admin
2026-07-07 20:22:30 +02:00
parent b2bfa26f64
commit 6b75f7302d
125 changed files with 7557 additions and 1938 deletions
Download Patch File Download Diff File Expand all files Collapse all files

162
archive/knowledge/hosts/hubris.md Normal file
View File

@@ -0,0 +1,162 @@
# `hubris` — Proxmox host
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
`Homelab` cluster (see [Cluster](#cluster)); the second node is
[strong](strong.md), which hosts nothing yet.
## At a glance
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
## Storage
| Pool | Type | Size | Use |
| ------------ | --------- | ----- | -------------------------------------- |
| `local` | dir | ~95G | ISOs, templates, `/etc`, configs |
| `local-lvm` | lvmthin | 856G | LXC/VM rootfs |
| `library` | lvmthin | 3.7T | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) |
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
## Cluster
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
(cluster/OS hostname `strong`), formed 2026-07-01.
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
strong reaches it via the existing Fritz!Box static route
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
household LAN. Fine for a home cluster; not latency-isolated.
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
votes — if either node is down (reboot, maintenance, network hiccup), the
survivor's running guests keep working but `/etc/pve` goes read-only:
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
Quirks below) make this painful in practice.
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
(every node has its own, not actually shared). The `library` lvmthin pool
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
it's a physical thinpool that only exists on this host's hardware.
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
member so far. See [strong.md](strong.md) and the [library-SSD
migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
for what comes next (physical drive move, service migration — not started).
## Tenants
### VMs
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
### LXC containers
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
## Boot-time tuning (load-bearing)
- **TCP BBR + `fq` qdisc** in `/etc/sysctl.d/99-bbr.conf`: `net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`, socket buffers up to 64 MiB (`rmem_max`/`wmem_max`/`tcp_rmem`/`tcp_wmem`). `tcp_bbr` module pinned to load at boot via `/etc/modules-load.d/bbr.conf`. Improves single-TCP-stream throughput on jittery / lossy links (notably WiFi clients downloading from [114 — nextcloud](../containers/114-nextcloud.md)). Container traffic crosses this kernel, so every guest inherits the change.
- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
## Network performance baseline (2026-05-14)
| Path | Throughput | Notes |
| ---- | ---------- | ----- |
| WAN (Ookla, from host) | ↓ 113.5 Mbit/s · ↑ 19.9 Mbit/s · 24.6 ms | Asymmetric residential link. 20 Mbit upload is the cap for public ingress (artifacto, blog via VPS traefik). |
| `eno1` link | 1000 Mbit/s, Full duplex | Negotiated correctly to router. |
| Host → LXC over `vmbr0` | **34.7 Gbit/s** | Single TCP stream, iperf3 10 s, 0 retransmits. |
| LXC → LXC over `vmbr0` | **34.8 Gbit/s** | Single TCP stream, iperf3 5 s, 0 retransmits. |
Intra-host throughput is memory-bandwidth bound — not a bottleneck for any current workload.
**eno1 error counters** (uptime 14 d 6 h at measurement time): `tx_packets` 99.99 M / `rx_packets` 96.08 M; `rx_errors` 1,619,203 (~1.7 % of RX); `align_errors` 10,919; `tx_errors` 20. The 1.6 M rx_errors in 14 days is the noteworthy datapoint — likely cable / switch-port. Re-check this counter in ~1 week and confirm it is not still climbing at the same rate; if it is, swap the patch cable first.
## Host services owned by external repos
| What | Repo | Path on host |
| --------------------------------- | -------------------------- | ----------------------------- |
| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor` | `/opt/claudio-monitor` |
| `backup-library` (restic) | `dtoro/backup-library` | `/opt/backup-library` *(disabled)* |
| `hubris-public-cert-sync` | (script + systemd unit) | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer |
See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
## Quirks
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
- ZFS is **not** in use; storage is LVM-thin + ext4.
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
## Authorized SSH keys (root)
- `root@hubris` (self, RSA) — local
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
`pvecm add` could authenticate without a password prompt
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
## Related
- [Containers index](../containers/index.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Caddy reverse proxy](../containers/121-caddy.md)
- [Mesh / Netbird](../infrastructure/mesh.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
- [strong — Proxmox host](strong.md)
## Changelog
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-05-14 — LXC 109 (syncthing) decommissioned
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
### 2026-05-14 — network performance baseline captured
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
### 2026-05-14 — LXC 116 (heaper) decommissioned
User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. Data subtree `/mnt/library/heaper` (224 MiB) retained for archaeology — see [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology) and [media-permissions](../infrastructure/media-permissions.md).
### 2026-05-13 — TCP BBR + fq qdisc enabled
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
### 2026-04-29 — relocated to better-ventilated spot
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
### 2026-04-28 — Phase 1 WiFi failover
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
### 2026-04-28 — wiki started
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
### 2026-04-23 — SSD cooling + thermal pads installed
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
### 2026-04-22 — drive removal A/B test
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-21 — crash-capture + RAS telemetry enabled
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
### 2026-04-21 — `cpu-epp.service` deployed
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).

9
archive/knowledge/hosts/index.md Normal file
View File

@@ -0,0 +1,9 @@
# Hosts
The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are
the generated `hosts/*.yaml` files at the repo root.
| Document | What it covers |
|----------|----------------|
| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. |
| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. |

159
archive/knowledge/hosts/strong.md Normal file
View File

@@ -0,0 +1,159 @@
# `strong` — Proxmox host
Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
## At a glance
- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) — hosts [7 LXCs](../containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05)
- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
2026-07-01 — earlier the same day the inventory entry was briefly named
`ludo-mini`, but that was renamed so the hostname bootstrap needs
(`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
the machine's everyday nickname; some older docs (investigations, the
library-SSD migration plan) still refer to it that way — that's fine,
those are historical.
- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
- `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
(thinpool) 815G, 8G swap.
- `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
existing APFS partition (previously used in a Mac) — wiped via
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
(handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
`ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
was actually created live; not worth renaming), restricted `nodes strong`
in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
Empty so far; this is separate from the [library-SSD migration
plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
planned drive move from hubris (that hasn't happened) — this is
general-purpose VM/CT capacity.
- **Network:** `vmbr0` is bridged straight onto the household LAN —
`192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
`vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
via the existing Fritz!Box static route through hubris; no new cabling
or routing was needed to join the cluster.
- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
with hubris by convention, actually separate per-node volumes). Hubris's
`library` lvmthin pool is restricted to `nodes hubris` in
`/etc/pve/storage.cfg` and does not exist here.
- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
workstation had. Reachable today only via LAN routing (confirmed DNS for
`*.hubris.network` already resolves correctly here via `192.168.8.2`).
Add to Netbird if off-LAN access to this host itself (distinct from any
future guests) is needed.
- **Homelab-context client enrollment:** done 2026-07-01 via
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
5-min sync timer are live; `homelab whoami` resolves correctly. See
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
## Cluster membership
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
either node going down freezes management on the survivor).
## SSH
Root login via the same key set trusted on hubris (`root@hubris`,
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
post cluster-join, so it's cluster-synced same as hubris). No password auth
needed going forward.
## Related
- [hubris — Proxmox host](hubris.md)
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
- [Network](../infrastructure/network.md)
- [SSH access](../infrastructure/ssh-access.md)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
## Changelog
### 2026-07-05 — Phase 2d: grimmory (130) migrated to strong
Migrated book library from hubris (192.168.8.213) to strong (192.168.8.247).
Rsync'd /books (2.6G) from hubris library SSD to ludo-lvm/media. Container
mounts /mnt/media_local → /mnt/library locally. Caddy backend for
books.hubris.network updated.
### 2026-07-05 — Phase 2: arriman (122) + jellyfin (101) migrated; library on ludo-lvm
Migrated arriman (→ 192.168.8.245) and jellyfin (→ 192.168.8.246) to strong.
Created 1.5T thin volume on ludo-lvm for media library (/mnt/media_local).
Rsync'd 363G of media data (movies, tv, anime, downloads, music) from hubris
library SSD to ludo-lvm. Both containers now mount /mnt/media_local directly
as local ext4 — no NFS cross-node dependency. Jellyfin gets Radeon 680M iGPU
(for VAAPI) + RX 7600 dGPU passthrough via dev0/dev1. Caddy backends updated
for media, jellyseerr, qbit, sab. Hubris freed 16 GiB RAM (8+8).
Dhcp scope narrowed to 192.168.8.100-239.
### 2026-07-05 — house (LXC 129) migrated to strong; DHCP scope narrowed
Migrated Yuvomi family planner from hubris (192.168.8.212) to strong
(192.168.8.244). Followed same restore pattern as elementsynapse (Phase 1b).
Discovered IP conflict: teddycloud (LXC 131) had 192.168.8.243 via DHCP
(scope was 192.168.8.241-254). Narrowed DHCP scope to 192.168.8.100-239,
gave teddycloud static IP 192.168.8.150. Caddy updated: house.hubris.network
→ 192.168.8.244:3000, teddy.hubris.network → 192.168.8.150:8443.
### 2026-07-05 — elementsynapse (LXC 118) migrated to strong
Migrated Matrix/Synapse + Element client from hubris (192.168.8.239) to a new
LXC on strong (192.168.8.242). Added vmbr1 on strong at 192.168.8.241/28
(portless internal bridge) for strong-hosted guests. Enabled IP forwarding,
proxy ARP on hubris vmbr0, and specific /32 routes for strong's guest subnet.
Caddy updated: element.hubris.network → 192.168.8.242:8080.
VPS traefik updated: matrix.hubris.network backend → 192.168.8.242:8008.
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
2. `sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
### 2026-07-01 — enrolled as a homelab-context client
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
### 2026-07-01 — inventory identity renamed ludo-mini → strong
Discovered while starting client enrollment: `bootstrap.sh` looks up
`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
member — Proxmox doesn't support in-place node rename, only leave+rejoin).
Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
(`hosts/ludo-mini.md` → `hosts/strong.md`), README, ssh-access.md all now
say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
Used Proxmox's own disk-management API rather than raw LVM commands:
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
pattern — this pool only physically exists here). Empty — no VM/CT disks
placed on it yet.
### 2026-07-01 — Proxmox install; joined Homelab cluster
Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
hubris's trusted set (root password used once, then discarded). Joined the
existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
this node's side, using key-based SSH pre-authorized in both directions —
no interactive password prompt needed for the join itself. Cluster now 2
nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
as `strong` and skip a QDevice for now — both revisitable later.