dtoro
b5c1247093
Problem: after the wiki-hq reorg, agent-instruction and human-doc domains were still scattered across the repo root, with three now-redundant stub files cluttering it. The organizing principle wasn't visible in the layout. Change — enforce three clear buckets: - .agents/ = how agents operate: OIKOS.md, HERMES.md (moved from root), shared/ conventions, domains/ schemas, skills/, and operations/ (operator cheatsheet + enrollment + hermes-agent, moved from root). - knowledge/ = what exists + evidence: wiki/, GLOSSARY.md, and sources/ now including investigations/ (incident records are evidence/sources). - root = substrate + two entry points (AGENTS.md, README.md), plus plans/ as its own design-intent domain. Moves: - investigations/ -> knowledge/sources/investigations/ (incl. archive/, index). - operations/ -> .agents/operations/. - HERMES.md -> .agents/HERMES.md. - Deleted unreferenced root stubs CAVEMAN.md, CONTRIBUTING.md, and OIKOS.md (its 7 remaining linkers repointed to .agents/OIKOS.md). Consumers updated: - inventory.yaml doc_page (agent-enrollment) + regenerated hosts/*.yaml + cards. - tools/setup-hermes-soul.sh and bootstrap.sh (x2) -> .agents/HERMES.md. - bin/homelab help string -> .agents/operations/hermes-agent.md. - knowledge/operations schemas, llm-wiki, page-templates, incident-investigation skill, AGENTS.md/README nav -> new investigations/operations paths. - All markdown links rewritten via the path-resolving mapper. Left in place (substrate/executable/separate-domain): hosts/, ledger/, tools/, plans/, oikos/, mcp/, secrets/, bin/, inventory.yaml. Verification: docs-lint at baseline (2 intentional cross-repo refs, no new breakage); gen-topology.py --check exit 0; build_host_files.py idempotent; all doc_page targets resolve; Hermes provisioning scripts point at the new path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
163 lines
17 KiB
Markdown
163 lines
17 KiB
Markdown
# `hubris` — Proxmox host
|
||
|
||
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
|
||
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
|
||
`Homelab` cluster (see [Cluster](#cluster)); the second node is
|
||
[strong](strong.md), which hosts nothing yet.
|
||
|
||
## At a glance
|
||
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
|
||
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
|
||
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
|
||
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
|
||
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
|
||
- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
|
||
- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
|
||
|
||
## Storage
|
||
|
||
| Pool | Type | Size | Use |
|
||
| ------------ | --------- | ----- | -------------------------------------- |
|
||
| `local` | dir | ~95G | ISOs, templates, `/etc`, configs |
|
||
| `local-lvm` | lvmthin | 856G | LXC/VM rootfs |
|
||
| `library` | lvmthin | 3.7T | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) |
|
||
|
||
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
|
||
|
||
## Cluster
|
||
|
||
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
|
||
(cluster/OS hostname `strong`), formed 2026-07-01.
|
||
|
||
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
|
||
strong reaches it via the existing Fritz!Box static route
|
||
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
|
||
household LAN. Fine for a home cluster; not latency-isolated.
|
||
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
|
||
votes — if either node is down (reboot, maintenance, network hiccup), the
|
||
survivor's running guests keep working but `/etc/pve` goes read-only:
|
||
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
|
||
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
|
||
Quirks below) make this painful in practice.
|
||
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
|
||
(every node has its own, not actually shared). The `library` lvmthin pool
|
||
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
|
||
it's a physical thinpool that only exists on this host's hardware.
|
||
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
|
||
member so far. See [strong.md](strong.md) and the [library-SSD
|
||
migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
|
||
for what comes next (physical drive move, service migration — not started).
|
||
|
||
## Tenants
|
||
|
||
### VMs
|
||
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
|
||
|
||
### LXC containers
|
||
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
|
||
|
||
## Boot-time tuning (load-bearing)
|
||
|
||
- **TCP BBR + `fq` qdisc** in `/etc/sysctl.d/99-bbr.conf`: `net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`, socket buffers up to 64 MiB (`rmem_max`/`wmem_max`/`tcp_rmem`/`tcp_wmem`). `tcp_bbr` module pinned to load at boot via `/etc/modules-load.d/bbr.conf`. Improves single-TCP-stream throughput on jittery / lossy links (notably WiFi clients downloading from [114 — nextcloud](../containers/114-nextcloud.md)). Container traffic crosses this kernel, so every guest inherits the change.
|
||
- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
|
||
- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
|
||
- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
|
||
|
||
## Network performance baseline (2026-05-14)
|
||
|
||
| Path | Throughput | Notes |
|
||
| ---- | ---------- | ----- |
|
||
| WAN (Ookla, from host) | ↓ 113.5 Mbit/s · ↑ 19.9 Mbit/s · 24.6 ms | Asymmetric residential link. 20 Mbit upload is the cap for public ingress (artifacto, blog via VPS traefik). |
|
||
| `eno1` link | 1000 Mbit/s, Full duplex | Negotiated correctly to router. |
|
||
| Host → LXC over `vmbr0` | **34.7 Gbit/s** | Single TCP stream, iperf3 10 s, 0 retransmits. |
|
||
| LXC → LXC over `vmbr0` | **34.8 Gbit/s** | Single TCP stream, iperf3 5 s, 0 retransmits. |
|
||
|
||
Intra-host throughput is memory-bandwidth bound — not a bottleneck for any current workload.
|
||
|
||
**eno1 error counters** (uptime 14 d 6 h at measurement time): `tx_packets` 99.99 M / `rx_packets` 96.08 M; `rx_errors` 1,619,203 (~1.7 % of RX); `align_errors` 10,919; `tx_errors` 20. The 1.6 M rx_errors in 14 days is the noteworthy datapoint — likely cable / switch-port. Re-check this counter in ~1 week and confirm it is not still climbing at the same rate; if it is, swap the patch cable first.
|
||
|
||
## Host services owned by external repos
|
||
|
||
| What | Repo | Path on host |
|
||
| --------------------------------- | -------------------------- | ----------------------------- |
|
||
| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor` | `/opt/claudio-monitor` |
|
||
| `backup-library` (restic) | `dtoro/backup-library` | `/opt/backup-library` *(disabled)* |
|
||
| `hubris-public-cert-sync` | (script + systemd unit) | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer |
|
||
|
||
See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
|
||
|
||
## Quirks
|
||
|
||
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
|
||
- ZFS is **not** in use; storage is LVM-thin + ext4.
|
||
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
|
||
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
|
||
- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
|
||
- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
|
||
|
||
## Authorized SSH keys (root)
|
||
|
||
- `root@hubris` (self, RSA) — local
|
||
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
|
||
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
|
||
`pvecm add` could authenticate without a password prompt
|
||
|
||
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
|
||
|
||
## Related
|
||
- [Containers index](../containers/index.md)
|
||
- [DNS split-horizon](../infrastructure/dns.md)
|
||
- [Caddy reverse proxy](../containers/121-caddy.md)
|
||
- [Mesh / Netbird](../infrastructure/mesh.md)
|
||
- [Media permissions](../infrastructure/media-permissions.md)
|
||
- [Monitoring](../infrastructure/monitoring.md)
|
||
- [Backups (disabled)](../infrastructure/backups.md)
|
||
- [Operations cheatsheet](../../../.agents/operations/commands.md)
|
||
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
|
||
- [strong — Proxmox host](strong.md)
|
||
|
||
## Changelog
|
||
|
||
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
|
||
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
|
||
|
||
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
|
||
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
|
||
|
||
### 2026-05-14 — LXC 109 (syncthing) decommissioned
|
||
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
|
||
|
||
### 2026-05-14 — network performance baseline captured
|
||
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
|
||
|
||
### 2026-05-14 — LXC 116 (heaper) decommissioned
|
||
User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. Data subtree `/mnt/library/heaper` (224 MiB) retained for archaeology — see [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology) and [media-permissions](../infrastructure/media-permissions.md).
|
||
|
||
### 2026-05-13 — TCP BBR + fq qdisc enabled
|
||
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
|
||
|
||
### 2026-04-29 — relocated to better-ventilated spot
|
||
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
|
||
|
||
### 2026-04-28 — Phase 1 WiFi failover
|
||
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
|
||
|
||
### 2026-04-28 — wiki started
|
||
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
|
||
|
||
### 2026-04-23 — SSD cooling + thermal pads installed
|
||
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
|
||
|
||
### 2026-04-22 — drive removal A/B test
|
||
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|
||
|
||
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
|
||
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
|
||
|
||
### 2026-04-21 — crash-capture + RAS telemetry enabled
|
||
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
|
||
|
||
### 2026-04-21 — `cpu-epp.service` deployed
|
||
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
|