wiki: document 2026-05-21 netbird vanilla migration

Updates to four pages reflecting the combined → vanilla mgmt+signal+relay+coturn
cutover and the IONOS-3478-firewall-exception discovery:

* infrastructure/mesh.md — rewrites the ICE/STUN section to cover the new
  TURN endpoint, the IONOS upstream TCP-3478 filtering (load-bearing,
  undocumented before today), and the verification probe. New changelog
  entry covering the migration outcome + Device Code Stage gap.

* infrastructure/vps-hardening.md — "At a glance" lists the new 6-service
  docker stack + host coturn. Firewall section notes the new
  `iifname ens6 tcp dport 3478 accept` rule plus the IONOS upstream
  exception. New changelog entry.

* containers/124-authentik.md — replaces the "Netbird IdP integration —
  DEFERRED" section with the LANDED state: Provider details (Public
  client type — Confidential breaks PKCE on the dashboard SPA), the
  first-time owner-promotion sqlite recipe, the missing Device Code
  Stage gap + workaround (setup-keys), and a note that the old 2026-04-22
  pre-work Provider/App is now obsolete and safe to delete. Updated
  changelog (Phase 6 landed).

* operations/agent-enrollment.md — new "Getting onto Netbird" subsection
  explaining the setup-key path (currently the only working flow until
  Device Code Stage lands) and why direct OIDC from the public internet
  fails (auth.hubris.network is mesh-only-reachable). Prerequisites table
  row updated to point at the new section.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

containers/124-authentik.md

@@ -100,16 +100,52 @@ API paths to bypass per app:
 | [Artifacto (105)](105-apps.md)            | Forward-auth + gateway-secret auto-login | Public `/p/*` paths bypass |
 | [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. |
 
-## Netbird IdP integration — DEFERRED
+## Netbird IdP integration — LANDED 2026-05-21
 
-The `netbirdio/netbird-server` combined image has no config knobs for external OIDC. Verified in `combined/cmd/config.go` on main (v0.69.0): `AuthConfig` only exposes issuer + redirect URIs; `ToManagementConfig()` hardcodes `AuthAudience="netbird-dashboard"`, `UserIDClaim="sub"`, always calls `buildEmbeddedIdPConfig()`. Code comment: `"embedded IdP is always enabled in combined server"`.
+The combined `netbirdio/netbird-server` image was replaced with the canonical vanilla stack (`mgmt + signal + relay + dashboard` 0.71.3) on the VPS so external OIDC actually works. Authentik is now the netbird dashboard's IdP. Full migration context in [mesh.md changelog](../infrastructure/mesh.md#changelog).
 
-To wire Authentik into Netbird login/user-sync, must migrate to the legacy split stack (`netbirdio/management` + `signal` + `dashboard` + coturn) with the richer `management.json` schema. Sqlite management data should migrate but needs verification; peers stay connected via wireguard keys.
+**Active provider & app:**
+- Provider `NetBird` (OAuth2/OpenID), **Client type: `Public`** (PKCE-only — `Confidential` would break the dashboard SPA's token exchange).
+- Client ID: `netbird-dashboard`. Client Secret is in `/opt/management.json` `PKCEAuthorizationFlow.ProviderConfig.ClientSecret` on the VPS (TODO: sops-encrypt as `secrets/netbird-authentik-oidc.yaml`).
+- Application `NetBird`, slug `netbird`, launch URL `https://netbird.hubris.network/`.
+- Redirect URIs: `https://netbird.hubris.network/peers`, `/nb-auth`, `/nb-silent-auth`, plus `https://netbird.hubris.network/` for post-logout.
+- Scopes enabled on the provider: `openid`, `profile`, `email`.
+- Discovery URL: `https://auth.hubris.network/application/o/netbird/.well-known/openid-configuration` — netbird mgmt fetches this on startup; logs `loaded OIDC configuration from the provided IDP configuration endpoint`.
 
-**Pre-work already in place (keep for re-use):**
+**Login flow:** netbird dashboard PKCE → Authentik authorize → redirect back to `/nb-auth` → JS token exchange at Authentik's `/token` endpoint → mgmt validates the bearer against Authentik's JWKS.
-- Provider `Provider for Netbird` + App `netbird`. Client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`. Redirect URIs for `https://netbird.hubris.network/{nb-auth,nb-silent-auth}` and `http://localhost:53000/`. Scopes: `openid profile email offline_access goauthentik.io/api`.
+
-- Service account `netbird-service` in `authentik Admins` group, non-expiring API token `netbird-service-api`.
+**First-time owner promotion gotcha** (write-down for future operators):
-- Netbird mgmt host (`82.165.190.79`) is now a peer on its own mesh. See [mesh](../infrastructure/mesh.md).
+
+When a new Authentik user logs in for the first time against an account that already has peers, netbird mgmt adds them as `role=user, blocked=1, pending_approval=1`. The OLD account-owner (the one in `store.db` from before the IdP swap) can't be reached anymore, so there's no admin to approve. Recovery is a direct sqlite update on `mgmt_data`:
+
+```
+docker stop netbird-mgmt
+sqlite3 /var/lib/docker/volumes/opt_mgmt_data/_data/store.db \
+  "UPDATE users SET role='owner', blocked=0, pending_approval=0 WHERE id='<new authentik sub>';"
+docker start netbird-mgmt
+```
+
+The Authentik sub-claim is the value of the `id` column on the newly-created user row (look for `role=user, blocked=1, pending_approval=1`).
+
+### KNOWN MISSING — Device Code Stage
+
+Authentik does NOT yet have an `OAuth2DeviceCodeStage` configured for this provider. Symptom: `netbird up` (without `--setup-key`) prints a device-code URL → user opens it → blank consent screen → device-code expires after 60s → CLI errors out. The token endpoint itself works (we verified with a direct device-code POST); only the consent UI is missing.
+
+**Workaround**: every new peer must use `netbird up --setup-key <KEY>`. Setup keys are generated from the dashboard at `https://netbird.hubris.network/setup-keys` by an already-enrolled operator.
+
+**Proper fix** (TODO): Authentik admin → Flows & Stages → Stages → Create → `OAuth2 Device Code Stage` + matching `OAuth2 Device Code Finish Stage`. Then either edit the provider's Authorization flow to include those stages, or create a dedicated device-code flow and bind the provider to it.
+
+### Self-service onboarding (not yet — future-session)
+
+`auth.hubris.network` is only reachable from inside the netbird mesh (split-horizon DNS). A brand-new client that isn't on the mesh yet can't OIDC-login → setup-key is the only path. To enable self-service onboarding via Authentik from the public internet:
+- Add a Traefik route on the VPS for `auth.hubris.network` that forwards via the netbird-routed `192.168.8.0/24` to LXC 124.
+- DNS already points `auth.hubris.network → 82.165.190.79` (IONOS wildcard).
+
+Tracked in homelab memory as a queued follow-up.
+
+### Old pre-work to remove
+
+The previous `Provider for Netbird` + `netbird` app from 2026-04-22 (client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`, service account `netbird-service`) is now obsolete — replaced by `netbird-dashboard` above. Safe to delete from Authentik admin UI; nothing currently uses the old client ID. The service account + API token can also be removed unless we wire IdpManagerConfig in mgmt later (currently `ManagerType: none`).
 
 ## DNS responsibility
 
@@ -123,11 +159,14 @@ dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` +
 
 ## Changelog
 
+### 2026-05-21 — Netbird IdP swap landed (Phase 6 done)
+VPS migrated from combined netbird-server to vanilla mgmt+signal+relay+dashboard 0.71.3 (see [mesh.md](../infrastructure/mesh.md)), enabling Authentik as the dashboard IdP via PKCE. New Provider/App = `netbird-dashboard`, replacing the deferred pre-work. Device Code Stage still missing — interactive `netbird up` fails consent; setup-keys are the workaround until that's added.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 
 ### 2026-04-22 — Phase 6 (Netbird IdP swap) deferred
-Combined netbird-server image can't take an external IdP. Pre-work in Authentik kept for later. Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
+Combined netbird-server image couldn't take an external IdP. Pre-work in Authentik kept for later (now superseded by 2026-05-21 above). Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
 
 ### 2026-04-22 — Artifacto, mulita, WriteFreely, Portainer wired
 Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto.

infrastructure/mesh.md

@@ -8,23 +8,40 @@ The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target e
 - **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
 - **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
 
-## ICE / STUN — must use external STUN, not embedded
+## ICE / STUN / TURN
 
-The bundled netbird-server image runs an embedded STUN listener on UDP 3478. **IONOS drops inbound UDP 3478 to the VPS upstream of the host firewall** (verified 2026-05-10 via `tcpdump -ni any udp port 3478`: 0 packets captured during external probes from hubris). Without a reachable STUN server, the management API hands peers a STUN URI nothing can talk to → no srflx candidates → ICE always fails → every peer falls back to the websocket relay (`rels://netbird.hubris.network:443`). All cross-NAT traffic is then bottlenecked by the relay/VPS bandwidth (observed ~366 kB/s for Nextcloud uploads).
+**Today** (post-2026-05-21 migration):
 
-**Fix**: in `/opt/config.yaml` on the VPS, declare external STUN servers under `server:` — this disables the embedded STUN automatically:
+- External STUN servers (Google + Cloudflare) declared under `server.Stuns` in `/opt/management.json`. Embedded STUN is no longer in use.
+- coturn runs on the VPS host (apt-installed, systemd-managed). TURN URL `turn:netbird.hubris.network:3478?transport=tcp` is advertised to peers via mgmt's `TURNConfig.Turns` block. Long-term credentials at `user=netbird:<pwd from /root/turn-pass.txt>`.
+- For non-symmetric peers, ICE picks direct `srflx/srflx` (P2P). For symmetric-NAT peers, ICE falls through to TURN-relay before falling back to the WSS relay (`rels://netbird.hubris.network:443`).
 
-```yaml
+**IONOS port-3478 caveat** (load-bearing — undocumented before 2026-05-21):
-server:
+
-  stuns:
+IONOS upstream filters STUN-class traffic on port 3478 **for both UDP and TCP** by default. The UDP block was already known (verified 2026-05-10 with `tcpdump -ni any udp port 3478`). The TCP block was discovered 2026-05-21 when TURN-allocate retries timed out from a symmetric-NAT laptop: TCP handshake completed (carried by kernel-only SYN exchange) but data packets never reached the VPS's `ens6` interface.
-    - uri: "stun:stun.l.google.com:19302"
+
-    - uri: "stun:stun1.l.google.com:19302"
+**Resolution**: operator must add an inbound firewall exception in the IONOS dashboard (Cloud Panel → Networks → Firewall policy on the VPS) for inbound TCP 3478 (and UDP 3478 if STUN-via-VPS-listener is ever needed; we use external STUN so this is optional). Without that exception, coturn is unreachable from outside even though host nftables + listener look fine.
-    - uri: "stun:stun.cloudflare.com:3478"
+
+**Verifying TURN works** end-to-end from an outside peer:
+
+```python
+# python3
+import socket, struct, secrets
+s = socket.create_connection(("netbird.hubris.network", 3478), timeout=10)
+tid = secrets.token_bytes(12)
+attrs = struct.pack("!HHI", 0x0019, 4, 17 << 24)   # REQUESTED-TRANSPORT = UDP
+msg = struct.pack("!HHI", 0x0003, len(attrs), 0x2112A442) + tid + attrs  # Allocate
+s.sendall(msg)
+print(s.recv(4096).hex())  # expect 120 bytes starting 0x0113 (Allocate Error = auth challenge)
 ```
 
-`docker restart netbird-server`, then `netbird down && netbird up` on each peer to force a resync. Verify with `netbird status -d` — `Connection type:` should flip from `Relayed` to `P2P` for peers that aren't behind double-NAT/CGNAT.
+A 120-byte `0x0113`-typed response means coturn is reachable and responding. An indefinite TCP timeout (after `connected from ...` prints) means the IONOS rule was reverted or narrowed.
 
-If a peer is *still* relayed after this, it's a NAT-symmetry problem on its side, not a config bug — would need TURN to fix.
+**If a peer is still on the WSS relay after this** — verify in `netbird status -d`: `Connection type: P2P` with srflx/srflx is the cone-NAT happy path; `Connection type: P2P` with relay/srflx (or relay/prflx) means the peer's network is symmetric-NAT and it's using TURN. Falling back to `Relayed (rels://...)` only happens if TURN allocate also fails — investigate the IONOS exception first.
+
+**Old combined-server note (history, kept for context):**
+
+Pre-migration, the bundled `netbirdio/netbird-server` combined image silently ignored both `server.turns:` and top-level `TURNConfig:` YAML, so TURN was non-functional. That's why the architectural migration to the canonical multi-container stack (`mgmt + signal + relay + coturn`) happened. See the 2026-05-21 changelog entry below and the [Netbird-combined-no-TURN finding](https://github.com/netbirdio/netbird/issues) in homelab memory for the full discovery.
 
 ## Consequence — every LXC wired to Authentik needs an internal override
 
@@ -99,6 +116,17 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
 
 ## Changelog
 
+### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
+The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
+
+Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/124-authentik.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
+
+Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
+
+The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. Device Code Stage is NOT yet configured in Authentik → `netbird up` interactive auth flow returns an empty consent screen; new peers must use `--setup-key` until the stage is added.
+
+Open follow-up: TURN-over-TLS on TCP 5349 (cert via certbot or extract Traefik's acme.json) for hostile-middlebox networks; plain TCP 3478 is sufficient for current usage.
+
 ### 2026-05-10 — ICE direct p2p restored (external STUN swap)
 All peers were `Connection type: Relayed` because the embedded STUN on the VPS was unreachable from outside (IONOS drops UDP 3478 inbound). Swapped to Google + Cloudflare STUN in `/opt/config.yaml`. After `netbird down/up`, peers now report `P2P` with srflx/host candidates. Backup of pre-change config at `/opt/config.yaml.bak-20260510-185050`. Fixes a real-world Nextcloud upload bottleneck (~366 kB/s through the relay → LAN-direct on same-network peers).
 

infrastructure/vps-hardening.md

@@ -8,6 +8,8 @@ IONOS VPS that runs the Netbird control plane and the [public ingress traefik](i
 - **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)).
 - **Public:** `82.165.190.79` (`ens6`).
 - **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`.
+- **Docker stack** at `/opt/docker-compose.yml`: `traefik` (TLS/ACME) + `dashboard` + `mgmt` + `signal` + `relay` + `proxy` — netbird-mgmt 0.71.3 vanilla deploy since 2026-05-21 (see [mesh.md changelog](mesh.md#changelog)).
+- **Host services (outside docker):** `coturn` (TURN-TCP on :3478, long-term creds at `/root/turn-pass.txt`, used by mgmt's `TURNConfig`).
 
 ## SSH
 
@@ -20,8 +22,9 @@ IONOS VPS that runs the Netbird control plane and the [public ingress traefik](i
 Config at `/etc/nftables.conf`, service enabled.
 
 - Public iface `ens6`. Wireguard iface `wt0`.
-- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6. Everything else drops.
+- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6, **TCP 3478** (coturn TURN-TCP, added 2026-05-21). Everything else drops.
 - `wt0` fully accepted in INPUT. `lo` accepted.
+- **IONOS upstream firewall** also gates inbound traffic before it reaches `ens6`. Open ports today: TCP 80/443 (traefik), UDP 51820 (netbird-proxy), TCP 3478 (coturn, added 2026-05-21). UDP 3478 is dropped by IONOS upstream regardless of local nftables. See [mesh.md ICE/STUN](mesh.md) for the STUN/TURN port matrix.
 - **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below.
 - Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops.
 - Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too.
@@ -77,6 +80,9 @@ Ordered by preference:
 
 ## Changelog
 
+### 2026-05-21 — netbird stack migrated combined → vanilla; coturn added
+Replaced the `netbirdio/netbird-server` combined image with the canonical `mgmt + signal + relay + dashboard` containers (0.71.3). Added host-side `coturn` for external TURN, with nftables rule `iifname "ens6" tcp dport 3478 accept` and an IONOS upstream firewall exception. Authentik on LXC 124 now provides OIDC for the netbird dashboard. Full context in [mesh.md changelog](mesh.md#changelog).
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 

operations/agent-enrollment.md

@@ -14,7 +14,7 @@ operational reference is here.
 | --- | --- | --- |
 | Hostname matches an entry in `inventory.yaml` | The bootstrap looks up `hosts/$(hostname).yaml`. | `hostname` (Linux) / `scutil --get LocalHostName` (macOS) |
 | OS is Linux or macOS | bootstrap detects via `uname -s` | `uname -s` |
-| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets | `netbird status` / `tailscale status` |
+| On the mesh (Netbird or Tailscale) **or** on the LAN | issuance is gated to mesh + LAN subnets. **For Netbird: use a setup-key, not interactive auth** — see "Getting onto Netbird" below. | `netbird status` / `tailscale status` |
 | `git`, `python3`, `python3-yaml`, `age`, `sops` | bootstrap preflight; `homelab` CLI imports yaml | See per-OS commands below |
 | Can resolve `*.hubris.network` | bootstrap calls `https://secrets.hubris.network/issue` and writes `https://mcp.hubris.network/sse` | `dig +short mcp.hubris.network` (should return `192.168.8.175`) |
 
@@ -28,6 +28,30 @@ hostname doesn't match any inventory entry. Two fixes:
 - **Rename the inventory entry**: edit `inventory.yaml` on hubris,
   regenerate `hosts/*.yaml`, push. The next sync (≤5 min) propagates.
 
+### Getting onto Netbird
+
+The bootstrap *requires* the new client to already be a connected mesh peer (its `netbird status` shows `Management: Connected`). Two paths to get there:
+
+**Path A — setup-key (currently the only working path, as of 2026-05-21):**
+
+1. From an **already-enrolled** machine (e.g. an existing workstation or hubris), log into the dashboard at `https://netbird.hubris.network/`. The dashboard auth flow goes via Authentik on [LXC 124](../containers/124-authentik.md), which is only reachable from inside the mesh — that's why this step has to be done from a mesh peer.
+2. **Setup Keys** → Create → set reusable + expiry as appropriate → copy the key.
+3. On the new client (after installing netbird per [netbird docs](https://docs.netbird.io/how-to/installation)):
+   ```bash
+   sudo netbird up --setup-key <KEY> \
+                   --management-url https://netbird.hubris.network
+   ```
+   Optionally pass `--ssh-jwt-cache-ttl=86400` here too (one less SSO per day for ssh into mesh peers).
+4. Confirm: `netbird status` shows `Management: Connected`, `Signal: Connected`, peer IP `100.122.x.x/16`. Then proceed to "Install dependencies" and "Run the bootstrap" below.
+
+**Path B — interactive OIDC ("`netbird up` without setup-key"):**
+
+Currently **broken** because Authentik's OAuth2 Device Code Stage isn't configured yet — the device-code URL renders a blank consent screen, and the code expires after 60s. See [124-authentik.md "KNOWN MISSING — Device Code Stage"](../containers/124-authentik.md) for the fix. Until then, stick with Path A.
+
+**Why we can't OIDC-login from the public internet:**
+
+`auth.hubris.network` resolves publicly to the VPS (`82.165.190.79`), but Traefik on the VPS doesn't currently route that hostname — only `netbird.hubris.network` is exposed. A new client off-mesh hitting `auth.hubris.network` gets a Traefik default 404. Solving this needs a VPS-side Traefik route forwarding `auth.hubris.network` via the netbird-routed `192.168.8.0/24` to LXC 124. Tracked as a future-session improvement.
+
 ### DNS prerequisite
 
 `*.hubris.network` resolves via the split-horizon dnsmasq on LXC 124