docs: move narrative wiki under knowledge/wiki/ (phase 3)

Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-06 14:35:23 +02:00
parent bb5c0717a1
commit 8a6422bd7d
129 changed files with 249 additions and 264 deletions

View File

@@ -34,7 +34,7 @@ services_hosted:
- name: artifacto
backend: apps
url: https://artifacto.hubris.network
doc_page: containers/105-apps.md
doc_page: knowledge/wiki/containers/105-apps.md
config_repo: dtoro/Artifacto
- name: homelab_mcp
backend: apps
@@ -42,7 +42,7 @@ services_hosted:
systemd_unit: homelab-mcp
public_host: mcp.hubris.network
endpoint: https://mcp.hubris.network/mcp
doc_page: infrastructure/homelab-context.md
doc_page: knowledge/wiki/infrastructure/homelab-context.md
config_repo: dtoro/Homelab-Docs
note: MCP server. Read-only context + management. Reachable on the LAN via Caddy and from off-LAN via
Netbird (192.168.8.0/24 is a network resource routed through hubris).
@@ -58,7 +58,5 @@ services_hosted:
note: Issues per-client age private keys. Gated at source-IP layer (mesh + LAN subnets in MESH_SUBNETS).
risk_notes: "identity issuance \u2014 any change is security-sensitive; key operations are destructive-class"
age_pubkey: age1duyl8mkpgu80uv934dy8q7enqjms6yvdz264hme8uryuxmvvqesq6rusq0
see_also:
- containers/105-apps.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -29,12 +29,10 @@ services_hosted:
- name: arr_stack
backend: arriman
note: jellyseerr / qbit / sab on docker compose
doc_page: containers/122-arriman.md
doc_page: knowledge/wiki/containers/122-arriman.md
notes:
- Migrated from hubris to strong 2026-07-05 (Phase 2). Library on ludo-lvm.
- Contains homarr, radarr, sonarr, lidarr, sabnzbd, qbittorrent, bazarr, flaresolverr, prowlarr, jellyseerr
- qBittorrent auth subnet whitelist expanded to 192.168.8.0/24 (for seanime + Caddy access)
see_also:
- containers/122-arriman.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -16,7 +16,5 @@ mesh_globals:
- tailscale
notes:
- Runs Authentik outpost (reverse-proxy/SSO enforcement) for protected services
see_also:
- containers/106-auth-outpost.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -24,14 +24,12 @@ services_hosted:
backend: caddy
role: reverse-proxy
note: terminates all *.hubris.network
doc_page: containers/121-caddy.md
doc_page: knowledge/wiki/containers/121-caddy.md
config_repo: dtoro/caddy-conf
risk_notes: "wide blast radius \u2014 every *.hubris.network route rides on it (see oikos/policy.yaml\
\ service_overrides)"
notes:
- Terminates all *.hubris.network
- /etc/caddy is a git checkout of dtoro/caddy-conf
see_also:
- containers/121-caddy.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -20,12 +20,10 @@ services_hosted:
- name: dns
backend: dns
note: Technitium DNS, split-horizon zone
doc_page: containers/107-dns.md
doc_page: knowledge/wiki/containers/107-dns.md
risk_notes: "LAN-wide resolver \u2014 misconfig breaks name resolution for every client"
notes:
- Technitium DNS, split-horizon zone for *.hubris.network
- Primary DNS for 192.168.8.0/24 LAN (inventory.services.dns references this)
see_also:
- containers/107-dns.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -23,11 +23,9 @@ services_hosted:
- name: matrix
url: https://matrix.hubris.network
backend: elementsynapse
doc_page: containers/118-elementsynapse.md
doc_page: knowledge/wiki/containers/118-elementsynapse.md
risk_notes: "alert/approval channel for Oikos \u2014 outage silences agent escalation"
notes:
- Migrated from hubris to strong 2026-07-05 (Phase 1 of strong migration plan).
see_also:
- containers/118-elementsynapse.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -27,12 +27,10 @@ services_hosted:
url: https://git.hubris.network
backend: gitea
backend_url: http://192.168.8.121:3000
doc_page: containers/104-gitea.md
doc_page: knowledge/wiki/containers/104-gitea.md
config_repo: dtoro/gitea-customizations
risk_notes: hosts all config repos + deploy webhooks; outage blocks auto-deploy and sync
notes:
- Bare repos live at /mnt/library/repos/dtoro/*.git
see_also:
- containers/104-gitea.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -21,7 +21,5 @@ notes:
- Docker host for Grimmory (community fork of Booklore). Created 2026-06-29.
- Migrated from hubris to strong 2026-07-05 (Phase 2d). Books on ludo-lvm.
age_pubkey: age1uellsemnjrzgfg9fxw4jefpy05laxzggwnwhh6ny3wl7alyp6v8q0muxet
see_also:
- containers/130-grimmory.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -22,8 +22,6 @@ runs:
services_hosted:
- name: haos
backend: haos
doc_page: vms/108-haos.md
see_also:
- vms/108-haos.md
doc_page: knowledge/wiki/vms/108-haos.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -22,7 +22,5 @@ notes:
- 192.168.8.212 was the hubris IP before migration (briefly picked up by teddycloud via DHCP; teddycloud
has since been given a static IP, see hosts.teddycloud)
age_pubkey: age1s07zs83ehtlg8jtwvr75ltc3c4cdlemfwjuxrwjtwkqxkl9tpggsyrzn2h
see_also:
- containers/129-house.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -1,162 +0,0 @@
# `hubris` — Proxmox host
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
`Homelab` cluster (see [Cluster](#cluster)); the second node is
[strong](strong.md), which hosts nothing yet.
## At a glance
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
## Storage
| Pool | Type | Size | Use |
| ------------ | --------- | ----- | -------------------------------------- |
| `local` | dir | ~95G | ISOs, templates, `/etc`, configs |
| `local-lvm` | lvmthin | 856G | LXC/VM rootfs |
| `library` | lvmthin | 3.7T | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) |
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
## Cluster
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
(cluster/OS hostname `strong`), formed 2026-07-01.
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
strong reaches it via the existing Fritz!Box static route
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
household LAN. Fine for a home cluster; not latency-isolated.
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
votes — if either node is down (reboot, maintenance, network hiccup), the
survivor's running guests keep working but `/etc/pve` goes read-only:
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
Quirks below) make this painful in practice.
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
(every node has its own, not actually shared). The `library` lvmthin pool
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
it's a physical thinpool that only exists on this host's hardware.
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
member so far. See [strong.md](strong.md) and the [library-SSD
migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
for what comes next (physical drive move, service migration — not started).
## Tenants
### VMs
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
### LXC containers
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
## Boot-time tuning (load-bearing)
- **TCP BBR + `fq` qdisc** in `/etc/sysctl.d/99-bbr.conf`: `net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`, socket buffers up to 64 MiB (`rmem_max`/`wmem_max`/`tcp_rmem`/`tcp_wmem`). `tcp_bbr` module pinned to load at boot via `/etc/modules-load.d/bbr.conf`. Improves single-TCP-stream throughput on jittery / lossy links (notably WiFi clients downloading from [114 — nextcloud](../containers/114-nextcloud.md)). Container traffic crosses this kernel, so every guest inherits the change.
- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.12 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
## Network performance baseline (2026-05-14)
| Path | Throughput | Notes |
| ---- | ---------- | ----- |
| WAN (Ookla, from host) | ↓ 113.5 Mbit/s · ↑ 19.9 Mbit/s · 24.6 ms | Asymmetric residential link. 20 Mbit upload is the cap for public ingress (artifacto, blog via VPS traefik). |
| `eno1` link | 1000 Mbit/s, Full duplex | Negotiated correctly to router. |
| Host → LXC over `vmbr0` | **34.7 Gbit/s** | Single TCP stream, iperf3 10 s, 0 retransmits. |
| LXC → LXC over `vmbr0` | **34.8 Gbit/s** | Single TCP stream, iperf3 5 s, 0 retransmits. |
Intra-host throughput is memory-bandwidth bound — not a bottleneck for any current workload.
**eno1 error counters** (uptime 14 d 6 h at measurement time): `tx_packets` 99.99 M / `rx_packets` 96.08 M; `rx_errors` 1,619,203 (~1.7 % of RX); `align_errors` 10,919; `tx_errors` 20. The 1.6 M rx_errors in 14 days is the noteworthy datapoint — likely cable / switch-port. Re-check this counter in ~1 week and confirm it is not still climbing at the same rate; if it is, swap the patch cable first.
## Host services owned by external repos
| What | Repo | Path on host |
| --------------------------------- | -------------------------- | ----------------------------- |
| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor` | `/opt/claudio-monitor` |
| `backup-library` (restic) | `dtoro/backup-library` | `/opt/backup-library` *(disabled)* |
| `hubris-public-cert-sync` | (script + systemd unit) | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer |
See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
## Quirks
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
- ZFS is **not** in use; storage is LVM-thin + ext4.
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
## Authorized SSH keys (root)
- `root@hubris` (self, RSA) — local
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
`pvecm add` could authenticate without a password prompt
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
## Related
- [Containers index](../containers/index.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Caddy reverse proxy](../containers/121-caddy.md)
- [Mesh / Netbird](../infrastructure/mesh.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Operations cheatsheet](../operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
- [strong — Proxmox host](strong.md)
## Changelog
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../plans/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-05-14 — LXC 109 (syncthing) decommissioned
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
### 2026-05-14 — network performance baseline captured
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
### 2026-05-14 — LXC 116 (heaper) decommissioned
User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. Data subtree `/mnt/library/heaper` (224 MiB) retained for archaeology — see [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology) and [media-permissions](../infrastructure/media-permissions.md).
### 2026-05-13 — TCP BBR + fq qdisc enabled
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
### 2026-04-29 — relocated to better-ventilated spot
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 3435.5 °C, ACPI zone 4749 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 6061 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
### 2026-04-28 — Phase 1 WiFi failover
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
### 2026-04-28 — wiki started
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
### 2026-04-23 — SSD cooling + thermal pads installed
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 6061 °C, nvme1 3840 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
### 2026-04-22 — drive removal A/B test
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-21 — crash-capture + RAS telemetry enabled
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
### 2026-04-21 — `cpu-epp.service` deployed
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md).

View File

@@ -29,10 +29,8 @@ services_hosted:
url: https://proxmox.hubris.network
backend: hubris
port: 8006
doc_page: hosts/hubris.md
doc_page: knowledge/wiki/hosts/hubris.md
risk_notes: "hypervisor UI \u2014 changes here affect every guest on the node"
age_pubkey: age1xkklkvnk5z0fsnh6cfgv70hy9ksfy8rdprwerzw4yk3p4p7cxcqs2yvpz6
see_also:
- hosts/hubris.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -26,7 +26,7 @@ services_hosted:
- name: jellyfin
url: https://media.hubris.network
backend: jellyfin
doc_page: containers/101-jellyfin.md
doc_page: knowledge/wiki/containers/101-jellyfin.md
risk_notes: native Authentik OIDC via SSO-Auth plugin, no Caddy forward-auth gate; VAAPI transcode depends
on GPU passthrough on strong
notes:
@@ -35,7 +35,5 @@ notes:
- SSO-Auth plugin v4.0.0.4 with Authentik OIDC (no Caddy forward-auth gate)
- GPU passed via dev0+dev1: /dev/dri/renderD128 + card0
- Migrated from hubris to strong 2026-07-05 (Phase 2). Library on ludo-lvm.
see_also:
- containers/101-jellyfin.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -26,9 +26,7 @@ services_hosted:
- name: photos
url: https://photos.hubris.network
backend: mule-images
doc_page: containers/120-mule-images.md
doc_page: knowledge/wiki/containers/120-mule-images.md
config_repo: dtoro/mule-image
see_also:
- containers/120-mule-images.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -23,7 +23,7 @@ services_hosted:
- name: authentik
url: https://auth.hubris.network
backend: netbird-vps
doc_page: containers/106-auth-outpost.md
doc_page: knowledge/wiki/containers/106-auth-outpost.md
note: core runs on the VPS since 2026-05-31; LAN forward-auth outpost is auth-outpost (LXC 106) at 192.168.8.6:9000.
Previous backend value "authentik" referenced the retired embedded-outpost host (LXC 124).
risk_notes: "SSO provider \u2014 outage locks login to OIDC/forward-auth services"

View File

@@ -26,8 +26,6 @@ services_hosted:
- name: nextcloud
url: https://cloud.hubris.network
backend: nextcloud
doc_page: containers/114-nextcloud.md
see_also:
- containers/114-nextcloud.md
doc_page: knowledge/wiki/containers/114-nextcloud.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -14,7 +14,5 @@ mesh_globals:
accepted:
- netbird
- tailscale
see_also:
- containers/102-nfs-export.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -26,9 +26,7 @@ services_hosted:
- name: paperless
url: https://paperless.hubris.network
backend: paperless
doc_page: containers/103-paperless.md
doc_page: knowledge/wiki/containers/103-paperless.md
risk_notes: "document archive \u2014 treat data as irreplaceable; DB operations are destructive-class"
see_also:
- containers/103-paperless.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -22,7 +22,5 @@ notes:
- MariaDB sidecar at /opt/romm/docker-compose.yml.
- ROMs on ludo-lvm media volume at /mnt/media_local/roms.
- 1 core / 2 GiB RAM / 16 GiB rootfs (ludo-lvm).
see_also:
- containers/134-romm.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -25,7 +25,5 @@ notes:
- /anime mounted from strong ludo-lvm (/mnt/media_local/anime)
- Caddy: "https://seanime.hubris.network \u2192 192.168.8.248:43211"
- qBittorrent auth subnet whitelist expanded to 192.168.8.0/24 for seanime access
see_also:
- containers/133-seanime.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -19,7 +19,5 @@ mesh_globals:
- tailscale
mounts:
- /mnt/library
see_also:
- containers/119-sophia.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -1,159 +0,0 @@
# `strong` — Proxmox host
Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
## At a glance
- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) — hosts [7 LXCs](../containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05)
- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
2026-07-01 — earlier the same day the inventory entry was briefly named
`ludo-mini`, but that was renamed so the hostname bootstrap needs
(`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
the machine's everyday nickname; some older docs (investigations, the
library-SSD migration plan) still refer to it that way — that's fine,
those are historical.
- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
- `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
(thinpool) 815G, 8G swap.
- `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
existing APFS partition (previously used in a Mac) — wiped via
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
(handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
`ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
was actually created live; not worth renaming), restricted `nodes strong`
in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
Empty so far; this is separate from the [library-SSD migration
plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
planned drive move from hubris (that hasn't happened) — this is
general-purpose VM/CT capacity.
- **Network:** `vmbr0` is bridged straight onto the household LAN —
`192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
`vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
via the existing Fritz!Box static route through hubris; no new cabling
or routing was needed to join the cluster.
- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
with hubris by convention, actually separate per-node volumes). Hubris's
`library` lvmthin pool is restricted to `nodes hubris` in
`/etc/pve/storage.cfg` and does not exist here.
- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
workstation had. Reachable today only via LAN routing (confirmed DNS for
`*.hubris.network` already resolves correctly here via `192.168.8.2`).
Add to Netbird if off-LAN access to this host itself (distinct from any
future guests) is needed.
- **Homelab-context client enrollment:** done 2026-07-01 via
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
5-min sync timer are live; `homelab whoami` resolves correctly. See
[agent-enrollment.md](../operations/agent-enrollment.md).
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
["Granting a secret to a new client"](../operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
## Cluster membership
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
either node going down freezes management on the survivor).
## SSH
Root login via the same key set trusted on hubris (`root@hubris`,
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
post cluster-join, so it's cluster-synced same as hubris). No password auth
needed going forward.
## Related
- [hubris — Proxmox host](hubris.md)
- [Library SSD migration plan](../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
- [Network](../infrastructure/network.md)
- [SSH access](../infrastructure/ssh-access.md)
- [Agent enrollment](../operations/agent-enrollment.md)
## Changelog
### 2026-07-05 — Phase 2d: grimmory (130) migrated to strong
Migrated book library from hubris (192.168.8.213) to strong (192.168.8.247).
Rsync'd /books (2.6G) from hubris library SSD to ludo-lvm/media. Container
mounts /mnt/media_local → /mnt/library locally. Caddy backend for
books.hubris.network updated.
### 2026-07-05 — Phase 2: arriman (122) + jellyfin (101) migrated; library on ludo-lvm
Migrated arriman (→ 192.168.8.245) and jellyfin (→ 192.168.8.246) to strong.
Created 1.5T thin volume on ludo-lvm for media library (/mnt/media_local).
Rsync'd 363G of media data (movies, tv, anime, downloads, music) from hubris
library SSD to ludo-lvm. Both containers now mount /mnt/media_local directly
as local ext4 — no NFS cross-node dependency. Jellyfin gets Radeon 680M iGPU
(for VAAPI) + RX 7600 dGPU passthrough via dev0/dev1. Caddy backends updated
for media, jellyseerr, qbit, sab. Hubris freed 16 GiB RAM (8+8).
Dhcp scope narrowed to 192.168.8.100-239.
### 2026-07-05 — house (LXC 129) migrated to strong; DHCP scope narrowed
Migrated Yuvomi family planner from hubris (192.168.8.212) to strong
(192.168.8.244). Followed same restore pattern as elementsynapse (Phase 1b).
Discovered IP conflict: teddycloud (LXC 131) had 192.168.8.243 via DHCP
(scope was 192.168.8.241-254). Narrowed DHCP scope to 192.168.8.100-239,
gave teddycloud static IP 192.168.8.150. Caddy updated: house.hubris.network
→ 192.168.8.244:3000, teddy.hubris.network → 192.168.8.150:8443.
### 2026-07-05 — elementsynapse (LXC 118) migrated to strong
Migrated Matrix/Synapse + Element client from hubris (192.168.8.239) to a new
LXC on strong (192.168.8.242). Added vmbr1 on strong at 192.168.8.241/28
(portless internal bridge) for strong-hosted guests. Enabled IP forwarding,
proxy ARP on hubris vmbr0, and specific /32 routes for strong's guest subnet.
Caddy updated: element.hubris.network → 192.168.8.242:8080.
VPS traefik updated: matrix.hubris.network backend → 192.168.8.242:8008.
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
2. `sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
### 2026-07-01 — enrolled as a homelab-context client
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
### 2026-07-01 — inventory identity renamed ludo-mini → strong
Discovered while starting client enrollment: `bootstrap.sh` looks up
`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
member — Proxmox doesn't support in-place node rename, only leave+rejoin).
Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
(`hosts/ludo-mini.md``hosts/strong.md`), README, ssh-access.md all now
say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
Used Proxmox's own disk-management API rather than raw LVM commands:
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
pattern — this pool only physically exists here). Empty — no VM/CT disks
placed on it yet.
### 2026-07-01 — Proxmox install; joined Homelab cluster
Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
hubris's trusted set (root password used once, then discarded). Joined the
existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
this node's side, using key-based SSH pre-authorized in both directions —
no interactive password prompt needed for the join itself. Cluster now 2
nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
as `strong` and skip a QDevice for now — both revisitable later.

View File

@@ -28,7 +28,5 @@ notes:
\ (filename kept as-is, it's a historical planning doc). Only Phase 1 (Proxmox install + cluster join)\
\ is done; no physical drive move, service migration, or GPU passthrough has happened yet."
age_pubkey: age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4
see_also:
- hosts/strong.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -23,7 +23,7 @@ services_hosted:
- name: teddycloud
url: https://teddy.hubris.network
backend: teddycloud
doc_page: containers/131-teddycloud.md
doc_page: knowledge/wiki/containers/131-teddycloud.md
note: self-hosted TeddyCloud (Toniebox cloud reimplementation), docker compose
risk_notes: "no Caddy forward-auth gate (unlike sab.hubris.network on the same Caddyfile) \u2014 reachable\
\ to anyone on the LAN/mesh who can resolve teddy.hubris.network; undocumented in inventory.yaml until\
@@ -38,7 +38,5 @@ notes:
\ DHCP before that \u2014 see hosts/strong.md's 2026-07-05 changelog)."
- "No age_pubkey / homelab-context enrollment \u2014 not a homelab CLI client, just a docker-compose app\
\ container. Not a required follow-up unless it needs secrets."
see_also:
- containers/131-teddycloud.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -22,9 +22,7 @@ services_hosted:
backend: trmnl
url: https://trmnl.hubris.network
note: self-hosted middleware for TRMNL e-ink plugins (polled by TRMNL cloud)
doc_page: containers/128-trmnl.md
doc_page: knowledge/wiki/containers/128-trmnl.md
config_repo: dtoro/terminalito
see_also:
- containers/128-trmnl.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue

View File

@@ -21,8 +21,6 @@ services_hosted:
- name: zimaos
url: https://zimaos.hubris.network
backend: zimaos
doc_page: vms/100-zimaos.md
see_also:
- vms/100-zimaos.md
doc_page: knowledge/wiki/vms/100-zimaos.md
mcp_endpoint: https://mcp.hubris.network/mcp
secrets_issuance_endpoint: https://secrets.hubris.network/issue