From 04d1f39e7be5f756c3c16b91abe8ceb1257bb1fb Mon Sep 17 00:00:00 2001 From: root Date: Sun, 31 May 2026 00:05:55 +0200 Subject: [PATCH] mesh: document 2026-05-31 recovery + auth.hubris.network Traefik route Management crash-loop root cause and fix documented. Phase 6 (Authentik as Netbird IdP) marked complete and live. Co-Authored-By: Claude Sonnet 4.6 --- infrastructure/mesh.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/infrastructure/mesh.md b/infrastructure/mesh.md index 22416b2..475eadf 100644 --- a/infrastructure/mesh.md +++ b/infrastructure/mesh.md @@ -116,6 +116,23 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Changelog +### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik + +**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup. + +**Root cause:** The 2026-05-21 migration configured `management.json` with `OIDCConfigEndpoint: https://auth.hubris.network/...`. The public DNS for `auth.hubris.network` already pointed to `82.165.190.79` (VPS), but the VPS Traefik had no route for that host → management got EOF on every boot, crash-looped. + +**Fix:** +1. Added `auth-hubris` router to `/opt/traefik-dynamic.yaml` — `Host(auth.hubris.network)` with `certResolver: letsencrypt` → service `auth-authentik`. +2. Service backend: `https://192.168.8.175` (Caddy on hubris LAN) via the Netbird mesh. +3. ServersTransport `skip-verify` with `serverName: auth.hubris.network` + `insecureSkipVerify: true` so Traefik sends the correct SNI to Caddy. +4. Upgraded VPS Netbird client `0.69.0 → 0.71.3` via apt (`apt install netbird=0.71.3`). +5. Mesh fully recovered; management connected to peers within ~1 min. + +**Bootstrap note (if mesh is dark and management must restart):** If the WireGuard tunnels are fully dead AND management needs to restart (after e.g. a VPS reboot with no peer handshakes), the Traefik backend `192.168.8.175` will be unreachable and management will crash again. Recovery: temporarily open VPS port 22 via IONOS console (`nft insert rule inet hubris-fw input iifname "ens6" tcp dport 22 accept`), SSH in, run `socat TCP-LISTEN:8443,bind=172.30.0.1,reuseaddr,fork TCP:127.0.0.1:8443 &`, then from hubris `ssh -f -N -R 127.0.0.1:8443:192.168.8.175:443 root@82.165.190.79` — this bootstraps one management start, after which the mesh self-heals. + +**Architecture after this change:** `auth.hubris.network` is publicly accessible (HTTPS via VPS Traefik → Caddy on hubris → Authentik LXC 124). External devices authenticating to Netbird now hit this public path. Phase 6 (Authentik as Netbird IdP) is complete and live. + ### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.