dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

wiki: document 2026-05-21 netbird vanilla migration

Browse Source 
Updates to four pages reflecting the combined → vanilla mgmt+signal+relay+coturn
cutover and the IONOS-3478-firewall-exception discovery:

* infrastructure/mesh.md — rewrites the ICE/STUN section to cover the new
  TURN endpoint, the IONOS upstream TCP-3478 filtering (load-bearing,
  undocumented before today), and the verification probe. New changelog
  entry covering the migration outcome + Device Code Stage gap.

* infrastructure/vps-hardening.md — "At a glance" lists the new 6-service
  docker stack + host coturn. Firewall section notes the new
  `iifname ens6 tcp dport 3478 accept` rule plus the IONOS upstream
  exception. New changelog entry.

* containers/124-authentik.md — replaces the "Netbird IdP integration —
  DEFERRED" section with the LANDED state: Provider details (Public
  client type — Confidential breaks PKCE on the dashboard SPA), the
  first-time owner-promotion sqlite recipe, the missing Device Code
  Stage gap + workaround (setup-keys), and a note that the old 2026-04-22
  pre-work Provider/App is now obsolete and safe to delete. Updated
  changelog (Phase 6 landed).

* operations/agent-enrollment.md — new "Getting onto Netbird" subsection
  explaining the setup-key path (currently the only working flow until
  Device Code Stage lands) and why direct OIDC from the public internet
  fails (auth.hubris.network is mesh-only-reachable). Prerequisites table
  row updated to point at the new section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-05-21 13:48:01 +02:00
parent 7a062bdb6b
commit b42a986cc0
4 changed files with 118 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
infrastructure/mesh.md
View File

@@ -8,23 +8,40 @@ The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target e
- **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
## ICE / STUN — must use external STUN, not embedded
## ICE / STUN / TURN
The bundled netbird-server image runs an embedded STUN listener on UDP 3478. **IONOS drops inbound UDP 3478 to the VPS upstream of the host firewall** (verified 2026-05-10 via `tcpdump -ni any udp port 3478`: 0 packets captured during external probes from hubris). Without a reachable STUN server, the management API hands peers a STUN URI nothing can talk to → no srflx candidates → ICE always fails → every peer falls back to the websocket relay (`rels://netbird.hubris.network:443`). All cross-NAT traffic is then bottlenecked by the relay/VPS bandwidth (observed ~366 kB/s for Nextcloud uploads).
**Today** (post-2026-05-21 migration):
**Fix**: in `/opt/config.yaml` on the VPS, declare external STUN servers under `server:` — this disables the embedded STUN automatically:
- External STUN servers (Google + Cloudflare) declared under `server.Stuns` in `/opt/management.json`. Embedded STUN is no longer in use.
- coturn runs on the VPS host (apt-installed, systemd-managed). TURN URL `turn:netbird.hubris.network:3478?transport=tcp` is advertised to peers via mgmt's `TURNConfig.Turns` block. Long-term credentials at `user=netbird:<pwd from /root/turn-pass.txt>`.
- For non-symmetric peers, ICE picks direct `srflx/srflx` (P2P). For symmetric-NAT peers, ICE falls through to TURN-relay before falling back to the WSS relay (`rels://netbird.hubris.network:443`).
```yaml
server:
stuns:
- uri: "stun:stun.l.google.com:19302"
- uri: "stun:stun1.l.google.com:19302"
- uri: "stun:stun.cloudflare.com:3478"
**IONOS port-3478 caveat** (load-bearing — undocumented before 2026-05-21):
IONOS upstream filters STUN-class traffic on port 3478 **for both UDP and TCP** by default. The UDP block was already known (verified 2026-05-10 with `tcpdump -ni any udp port 3478`). The TCP block was discovered 2026-05-21 when TURN-allocate retries timed out from a symmetric-NAT laptop: TCP handshake completed (carried by kernel-only SYN exchange) but data packets never reached the VPS's `ens6` interface.
**Resolution**: operator must add an inbound firewall exception in the IONOS dashboard (Cloud Panel → Networks → Firewall policy on the VPS) for inbound TCP 3478 (and UDP 3478 if STUN-via-VPS-listener is ever needed; we use external STUN so this is optional). Without that exception, coturn is unreachable from outside even though host nftables + listener look fine.
**Verifying TURN works** end-to-end from an outside peer:
```python
# python3
import socket, struct, secrets
s = socket.create_connection(("netbird.hubris.network", 3478), timeout=10)
tid = secrets.token_bytes(12)
attrs = struct.pack("!HHI", 0x0019, 4, 17 << 24) # REQUESTED-TRANSPORT = UDP
msg = struct.pack("!HHI", 0x0003, len(attrs), 0x2112A442) + tid + attrs # Allocate
s.sendall(msg)
print(s.recv(4096).hex()) # expect 120 bytes starting 0x0113 (Allocate Error = auth challenge)
```
`docker restart netbird-server`, then `netbird down && netbird up` on each peer to force a resync. Verify with `netbird status -d` — `Connection type:` should flip from `Relayed` to `P2P` for peers that aren't behind double-NAT/CGNAT.
A 120-byte `0x0113`-typed response means coturn is reachable and responding. An indefinite TCP timeout (after `connected from ...` prints) means the IONOS rule was reverted or narrowed.
If a peer is *still* relayed after this, it's a NAT-symmetry problem on its side, not a config bug — would need TURN to fix.
**If a peer is still on the WSS relay after this** — verify in `netbird status -d`: `Connection type: P2P` with srflx/srflx is the cone-NAT happy path; `Connection type: P2P` with relay/srflx (or relay/prflx) means the peer's network is symmetric-NAT and it's using TURN. Falling back to `Relayed (rels://...)` only happens if TURN allocate also fails — investigate the IONOS exception first.
**Old combined-server note (history, kept for context):**
Pre-migration, the bundled `netbirdio/netbird-server` combined image silently ignored both `server.turns:` and top-level `TURNConfig:` YAML, so TURN was non-functional. That's why the architectural migration to the canonical multi-container stack (`mgmt + signal + relay + coturn`) happened. See the 2026-05-21 changelog entry below and the [Netbird-combined-no-TURN finding](https://github.com/netbirdio/netbird/issues) in homelab memory for the full discovery.
## Consequence — every LXC wired to Authentik needs an internal override
@@ -99,6 +116,17 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. Device Code Stage is NOT yet configured in Authentik → `netbird up` interactive auth flow returns an empty consent screen; new peers must use `--setup-key` until the stage is added.
Open follow-up: TURN-over-TLS on TCP 5349 (cert via certbot or extract Traefik's acme.json) for hostile-middlebox networks; plain TCP 3478 is sufficient for current usage.
### 2026-05-10 — ICE direct p2p restored (external STUN swap)
All peers were `Connection type: Relayed` because the embedded STUN on the VPS was unreachable from outside (IONOS drops UDP 3478 inbound). Swapped to Google + Cloudflare STUN in `/opt/config.yaml`. After `netbird down/up`, peers now report `P2P` with srflx/host candidates. Backup of pre-change config at `/opt/config.yaml.bak-20260510-185050`. Fixes a real-world Nextcloud upload bottleneck (~366 kB/s through the relay → LAN-direct on same-network peers).