oikos/hosts/hubris.md

hubris — Proxmox host

Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node Homelab cluster (see Cluster); the second node is strong, which hosts nothing yet.

At a glance

• Role: Proxmox VE 9.1.2 hypervisor (kernel 6.14.11-4-pve)
• Hardware: GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for library LVM). 2× Realtek RTL8125 NICs (r8169).
• BIOS: 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See investigations.
• Uplink: vmbr1 (slave: eno1) → SODOLA switch → Fritz!Box 7590. DHCP-reserved 192.168.178.10/24, gateway 192.168.178.1.
• Homelab bridge: vmbr0 — portless internal bridge, 192.168.8.77/24 + 192.168.8.1/24 alias (LXC default gateway). All 16 LXCs and the HAOS VM are on vmbr0. Proxmox routes between vmbr0 and vmbr1; Fritz!Box has a static route 192.168.8.0/24 → 192.168.178.10.
• WiFi: disabled 2026-06-02 — wlp3s0 removed from /etc/network/interfaces, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
• Mesh: Netbird wt0 100.122.38.109/16. Resolver: 100.122.38.109 (the local netbird daemon, which forwards to LAN/upstream and learns *.hubris.network answers via that path). See mesh.
• UI: https://proxmox.hubris.network (via caddy) or https://192.168.8.77:8006.

Storage

Pool	Type	Size	Use
local	dir	~95G	ISOs, templates, /etc, configs
local-lvm	lvmthin	856G	LXC/VM rootfs
library	lvmthin	3.7T	Backs /mnt/library ext4 (mounted as /dev/mapper/library-library)

/mnt/library holds the shared media + data pool: anime, audiobooks, books, comics, documents, downloads, heaper, homecloud, images, marimo, movies, music, notes, podcasts, repos, roms, sophia. Bind-mounted into every container that needs it. Permissions standard: media GID 10000.

Cluster

Member of Homelab, a 2-node Proxmox cluster with strong (cluster/OS hostname strong), formed 2026-07-01.

• Corosync ring0: hubris's internal 192.168.8.77 (the vmbr0 address). strong reaches it via the existing Fritz!Box static route (192.168.8.0/24 → 192.168.178.10) — no dedicated corosync link, just the household LAN. Fine for a home cluster; not latency-isolated.
• Quorum: 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both votes — if either node is down (reboot, maintenance, network hiccup), the survivor's running guests keep working but /etc/pve goes read-only: no start/stop/create/edit until quorum returns. Decided to skip a QDevice for now; revisit if hubris's periodic reboots (BIOS/thermal work, see Quirks below) make this painful in practice.
• Storage: local / local-lvm are the standard per-node default IDs (every node has its own, not actually shared). The library lvmthin pool is explicitly restricted to nodes hubris in /etc/pve/storage.cfg since it's a physical thinpool that only exists on this host's hardware.
• strong currently hosts no LXCs/VMs — it exists solely as a cluster member so far. See strong.md and the library-SSD migration plan for what comes next (physical drive move, service migration — not started).

Tenants

VMs

• 108 — haos-16.3 — Home Assistant OS, 4 GiB / 32 GiB

LXC containers

See containers/index. 13 active (109 syncthing destroyed 2026-05-14).

Boot-time tuning (load-bearing)

• TCP BBR + fq qdisc in /etc/sysctl.d/99-bbr.conf: net.ipv4.tcp_congestion_control=bbr, net.core.default_qdisc=fq, socket buffers up to 64 MiB (rmem_max/wmem_max/tcp_rmem/tcp_wmem). tcp_bbr module pinned to load at boot via /etc/modules-load.d/bbr.conf. Improves single-TCP-stream throughput on jittery / lossy links (notably WiFi clients downloading from 114 — nextcloud). Container traffic crosses this kernel, so every guest inherits the change.
• cpu-epp.service (enabled) sets scaling_governor=powersave + EPP=balance_power at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In amd-pstate=active mode the governor MUST be powersave, not performance, for EPP to apply. Unit ordering: After=sysinit.target + Before=pve-guests.service so it runs before the guest fleet starts (the original After=multi-user.target left the hottest boot window on performance). Fixed 2026-04-22.
• Crash capture: /etc/sysctl.d/60-crash-capture.conf (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), /etc/modprobe.d/softdog.conf (soft_panic=1 soft_margin=60), /etc/systemd/system.conf.d/watchdog.conf (RuntimeWatchdogSec=15s). Pstore traces collected to /var/lib/systemd/pstore/ by systemd-pstore.service. Caveat: silent CPU lockups leave pstore empty.
• rasdaemon (Debian pkg) collects MCE / memory / PCIe AER / thermal events to /var/lib/rasdaemon/ras-mc_event.db. Query with ras-mc-ctl --summary / --errors. (mcelog is retired in Debian 13 — don't go looking for it.)

Network performance baseline (2026-05-14)

Path	Throughput	Notes
WAN (Ookla, from host)	↓ 113.5 Mbit/s · ↑ 19.9 Mbit/s · 24.6 ms	Asymmetric residential link. 20 Mbit upload is the cap for public ingress (artifacto, blog via VPS traefik).
eno1 link	1000 Mbit/s, Full duplex	Negotiated correctly to router.
Host → LXC over vmbr0	34.7 Gbit/s	Single TCP stream, iperf3 10 s, 0 retransmits.
LXC → LXC over vmbr0	34.8 Gbit/s	Single TCP stream, iperf3 5 s, 0 retransmits.

Intra-host throughput is memory-bandwidth bound — not a bottleneck for any current workload.

eno1 error counters (uptime 14 d 6 h at measurement time): tx_packets 99.99 M / rx_packets 96.08 M; rx_errors 1,619,203 (~1.7 % of RX); align_errors 10,919; tx_errors 20. The 1.6 M rx_errors in 14 days is the noteworthy datapoint — likely cable / switch-port. Re-check this counter in ~1 week and confirm it is not still climbing at the same rate; if it is, swap the patch cable first.

Host services owned by external repos

What	Repo	Path on host
claudio-monitor (5-min watchdog)	dtoro/claudio-monitor	/opt/claudio-monitor
backup-library (restic)	dtoro/backup-library	/opt/backup-library (disabled)
hubris-public-cert-sync	(script + systemd unit)	/usr/local/bin/hubris-public-cert-sync.sh, daily timer

See monitoring, backups, ingress.

Quirks

• /etc/pve is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
• ZFS is not in use; storage is LVM-thin + ext4.
• Two Realtek 8125 NICs use the in-tree r8169 driver, not the OOT r8125.
• Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
• Non-ECC RAM: silent memory faults are possible; suspect DIMMs if cpu-epp is on and crashes still happen.
• BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).

Authorized SSH keys (root)

• root@hubris (self, RSA) — local
• d.toro.v@pm.me (ed25519) — user's iMac, added 2026-04-22
• root@strong (RSA) — strong's cluster-join key, added 2026-07-01 so pvecm add could authenticate without a password prompt

OpenSSH on 0.0.0.0:22. Netbird's built-in SSH server is on 100.122.38.109:22022 and bypasses authorized_keys (OIDC/browser). See SSH access for the dual-server gotcha.

Related

• Containers index
• DNS split-horizon
• Caddy reverse proxy
• Mesh / Netbird
• Media permissions
• Monitoring
• Backups (disabled)
• Operations cheatsheet
• Investigation: 2026-04-21 crash loop
• strong — Proxmox host

Changelog

2026-07-01 — strong joined as a 2nd cluster node ("Homelab")

User reformatted strong (formerly a Linux dev workstation, 192.168.178.181) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is strong (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: root@hubris, d.toro.v@pm.me), then generated a keypair on strong and pre-authorized it here (root@strong) so pvecm add 192.168.8.77 --use_ssh 1 (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (192.168.8.77) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, no QDevice (explicit choice — see Cluster above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the library-SSD migration plan, nothing further from that plan has been executed.

2026-06-02 — Slate AX retired; SODOLA switch added; network restructured

Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: vmbr1 (eno1 → SODOLA → Fritz!Box) is the uplink at 192.168.178.10/24; vmbr0 is a portless internal bridge holding all LXCs/VMs with 192.168.8.1 as an alias (unchanged LXC gateway). Fritz!Box static route 192.168.8.0/24 → 192.168.178.10 enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (wlp3s0) also removed — was pointing at the Slate AX SSID, no longer useful. See network and migration plan.

2026-05-14 — LXC 109 (syncthing) decommissioned

User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). pct destroy 109 --purge cleaned vm-109-disk-0 on local-lvm and the /etc/pve/lxc/109.conf entry. Data subtree /mnt/library/syncthing was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in containers/index; references stripped from README, media-permissions, vms/100-zimaos, and containers/102-nfs-export.

2026-05-14 — network performance baseline captured

First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), eno1 1 Gb full-duplex negotiated, intra-host vmbr0 ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). iperf3 + speedtest-cli installed on host. Noted eno1 rx_errors at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k align_errors — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.

2026-05-14 — LXC 116 (heaper) decommissioned

User destroyed the heaper LXC. No 116.conf.bak left behind in /etc/pve/lxc/. Data subtree /mnt/library/heaper (224 MiB) retained for archaeology — see containers/index and media-permissions.

2026-05-13 — TCP BBR + fq qdisc enabled

/etc/sysctl.d/99-bbr.conf switches net.ipv4.tcp_congestion_control from cubic to bbr and net.core.default_qdisc from fq_codel to fq. Also bumps rmem_max/wmem_max to 64 MiB and widens tcp_rmem/tcp_wmem. tcp_bbr module pinned at boot via /etc/modules-load.d/bbr.conf. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.

2026-04-29 — relocated to better-ventilated spot

User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl 47.2 °C, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See investigation.

2026-04-28 — Phase 1 WiFi failover

Host now dual-homed: LAN 192.168.8.77 (primary) + WiFi 192.168.8.141 (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed wpasupplicant+iw; added wlp3s0 stanza to /etc/network/interfaces with wpa-conf; ARP isolation sysctls in post-up. Built wan-failover.service to remove the vmbr0 default route on eno1 carrier loss, since the bridge's carrier doesn't follow eno1 (the LXC veths keep it 1). LXC/VM guests are still LAN-only — Phase 2 will migrate them.

2026-04-28 — wiki started

This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel 6.14.11-4-pve, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, destroyed: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 *arr migration retention; the others removed since.

2026-04-23 — SSD cooling + thermal pads installed

Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See investigation.

2026-04-22 — drive removal A/B test

Removed external USB backup drive (Silicon Motion 090c:2320). Disabled the four backup-library*.timer units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. Result so far: 3+ days uptime — the drive looks like the primary contributor; cpu-epp remains as belt-and-suspenders thermal protection. See investigation.

2026-04-22 — cpu-epp.service ordering bug fixed

Was After=multi-user.target + WantedBy=multi-user.target — queued behind pve-guests.service, so the hottest boot window (20+ guests starting on performance) preceded EPP application. Now After=sysinit.target + Before=pve-guests.service.

2026-04-21 — crash-capture + RAS telemetry enabled

60-crash-capture.conf, softdog soft_panic=1, RuntimeWatchdog 15 s. rasdaemon installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.

2026-04-21 — cpu-epp.service deployed

Pinned governor=powersave, EPP=balance_power at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the crash-loop incident.