Bootstrap Homelab-Docs wiki

Initial documentation of the hubris Proxmox homelab as a cross-linked markdown wiki. Per-node pages, cross-cutting infrastructure pages, an investigation log, and an operations cheatsheet. Each node and topic ends with a Changelog section so changes can be tracked in-place going forward. Refreshed against live state on 2026-04-28 — 14 active LXCs (109 syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test state of the 2026-04-21 hubris crash-loop investigation.
2026-04-28 22:53:09 +02:00
commit f5cd320433
31 changed files with 2157 additions and 0 deletions
--- a/hosts/hubris.md
+++ b/hosts/hubris.md
@@ -0,0 +1,92 @@
+# `hubris` — Proxmox host
+
+Single-node Proxmox VE running 1 VM and 14 LXC containers (one of which is currently stopped). The whole homelab.
+
+## At a glance
+- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
+- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
+- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
+- **LAN:** `192.168.8.77/24` on bridge `vmbr0`, gateway `192.168.8.1`.
+- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
+- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
+
+## Storage
+
+| Pool         | Type      | Size  | Use                                    |
+| ------------ | --------- | ----- | -------------------------------------- |
+| `local`      | dir       | ~95G  | ISOs, templates, `/etc`, configs       |
+| `local-lvm`  | lvmthin   | 856G  | LXC/VM rootfs                          |
+| `library`    | lvmthin   | 3.7T  | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) |
+
+`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`, `syncthing`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
+
+## Tenants
+
+### VMs
+- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
+
+### LXC containers
+See [containers/index](../containers/index.md). 14 active, 1 stopped.
+
+## Boot-time tuning (load-bearing)
+
+- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
+- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
+- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
+
+## Host services owned by external repos
+
+| What                              | Repo                       | Path on host                  |
+| --------------------------------- | -------------------------- | ----------------------------- |
+| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor`   | `/opt/claudio-monitor`        |
+| `backup-library` (restic)         | `dtoro/backup-library`     | `/opt/backup-library` *(disabled)* |
+| `hubris-public-cert-sync`         | (script + systemd unit)    | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer |
+
+See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
+
+## Quirks
+
+- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install.
+- ZFS is **not** in use; storage is LVM-thin + ext4.
+- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
+- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23.
+- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
+- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
+
+## Authorized SSH keys (root)
+
+- `root@hubris` (self, RSA) — local
+- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
+
+OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
+
+## Related
+- [Containers index](../containers/index.md)
+- [DNS split-horizon](../infrastructure/dns.md)
+- [Caddy reverse proxy](../containers/121-caddy.md)
+- [Mesh / Netbird](../infrastructure/mesh.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+- [Monitoring](../infrastructure/monitoring.md)
+- [Backups (disabled)](../infrastructure/backups.md)
+- [Operations cheatsheet](../operations/commands.md)
+- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+
+## Changelog
+
+### 2026-04-28 — wiki started
+This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
+
+### 2026-04-23 — SSD cooling + thermal pads installed
+Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
+
+### 2026-04-22 — drive removal A/B test
+Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
+
+### 2026-04-22 — `cpu-epp.service` ordering bug fixed
+Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
+
+### 2026-04-21 — crash-capture + RAS telemetry enabled
+`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
+
+### 2026-04-21 — `cpu-epp.service` deployed
+Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md).