dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

mesh: document 2026-05-31 recovery + auth.hubris.network Traefik route

Browse Source 
Management crash-loop root cause and fix documented.
Phase 6 (Authentik as Netbird IdP) marked complete and live.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
root
2026-05-31 00:05:55 +02:00
parent b34d362c19
commit 04d1f39e7b
1 changed files with 17 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
infrastructure/mesh.md
View File

@@ -116,6 +116,23 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.
**Root cause:** The 2026-05-21 migration configured `management.json` with `OIDCConfigEndpoint: https://auth.hubris.network/...`. The public DNS for `auth.hubris.network` already pointed to `82.165.190.79` (VPS), but the VPS Traefik had no route for that host → management got EOF on every boot, crash-looped.
**Fix:**
1. Added `auth-hubris` router to `/opt/traefik-dynamic.yaml` — `Host(auth.hubris.network)` with `certResolver: letsencrypt` → service `auth-authentik`.
2. Service backend: `https://192.168.8.175` (Caddy on hubris LAN) via the Netbird mesh.
3. ServersTransport `skip-verify` with `serverName: auth.hubris.network` + `insecureSkipVerify: true` so Traefik sends the correct SNI to Caddy.
4. Upgraded VPS Netbird client `0.69.0 → 0.71.3` via apt (`apt install netbird=0.71.3`).
5. Mesh fully recovered; management connected to peers within ~1 min.
**Bootstrap note (if mesh is dark and management must restart):** If the WireGuard tunnels are fully dead AND management needs to restart (after e.g. a VPS reboot with no peer handshakes), the Traefik backend `192.168.8.175` will be unreachable and management will crash again. Recovery: temporarily open VPS port 22 via IONOS console (`nft insert rule inet hubris-fw input iifname "ens6" tcp dport 22 accept`), SSH in, run `socat TCP-LISTEN:8443,bind=172.30.0.1,reuseaddr,fork TCP:127.0.0.1:8443 &`, then from hubris `ssh -f -N -R 127.0.0.1:8443:192.168.8.175:443 root@82.165.190.79` — this bootstraps one management start, after which the mesh self-heals.
**Architecture after this change:** `auth.hubris.network` is publicly accessible (HTTPS via VPS Traefik → Caddy on hubris → Authentik LXC 124). External devices authenticating to Netbird now hit this public path. Phase 6 (Authentik as Netbird IdP) is complete and live.
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.