From ebdd4c857a84dc499983c08e7bea7624cfa55c43 Mon Sep 17 00:00:00 2001 From: Claudio Date: Sun, 10 May 2026 20:56:09 +0200 Subject: [PATCH] mesh: document Netbird external-STUN fix (IONOS blocks UDP 3478) --- infrastructure/mesh.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/infrastructure/mesh.md b/infrastructure/mesh.md index 5d1dbb0..d507242 100644 --- a/infrastructure/mesh.md +++ b/infrastructure/mesh.md @@ -8,6 +8,24 @@ The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target e - **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration. - **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service. +## ICE / STUN — must use external STUN, not embedded + +The bundled netbird-server image runs an embedded STUN listener on UDP 3478. **IONOS drops inbound UDP 3478 to the VPS upstream of the host firewall** (verified 2026-05-10 via `tcpdump -ni any udp port 3478`: 0 packets captured during external probes from hubris). Without a reachable STUN server, the management API hands peers a STUN URI nothing can talk to → no srflx candidates → ICE always fails → every peer falls back to the websocket relay (`rels://netbird.hubris.network:443`). All cross-NAT traffic is then bottlenecked by the relay/VPS bandwidth (observed ~366 kB/s for Nextcloud uploads). + +**Fix**: in `/opt/config.yaml` on the VPS, declare external STUN servers under `server:` — this disables the embedded STUN automatically: + +```yaml +server: + stuns: + - uri: "stun:stun.l.google.com:19302" + - uri: "stun:stun1.l.google.com:19302" + - uri: "stun:stun.cloudflare.com:3478" +``` + +`docker restart netbird-server`, then `netbird down && netbird up` on each peer to force a resync. Verify with `netbird status -d` — `Connection type:` should flip from `Relayed` to `P2P` for peers that aren't behind double-NAT/CGNAT. + +If a peer is *still* relayed after this, it's a NAT-symmetry problem on its side, not a config bug — would need TURN to fix. + ## Consequence — every LXC wired to Authentik needs an internal override Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`. @@ -81,6 +99,9 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man ## Changelog +### 2026-05-10 — ICE direct p2p restored (external STUN swap) +All peers were `Connection type: Relayed` because the embedded STUN on the VPS was unreachable from outside (IONOS drops UDP 3478 inbound). Swapped to Google + Cloudflare STUN in `/opt/config.yaml`. After `netbird down/up`, peers now report `P2P` with srflx/host candidates. Backup of pre-change config at `/opt/config.yaml.bak-20260510-185050`. Fixes a real-world Nextcloud upload bottleneck (~366 kB/s through the relay → LAN-direct on same-network peers). + ### 2026-04-28 — wiki entry created Initial documentation.