oikos/knowledge/sources/investigations/archive/2026-05-31-authentik-vps-migration.md

2026-05-31 — Authentik migrated from LXC 124 to the VPS

Summary

The NetBird management server (on the VPS) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down because mgmt couldn't start. A classic bootstrap deadlock: mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.

Resolved by moving Authentik off LXC 124 onto the VPS itself, so auth.hubris.network resolves to a container co-located with netbird-mgmt — no mesh dependency. A depends_on: condition: service_healthy on the mgmt service makes the deadlock structurally impossible to recur.

The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.

Timeline

Trigger

netbird status on clients showed Management: Disconnected; netbird.hubris.network unreachable. VPS docker logs netbird-mgmt:

Error: failed reading provided config file: /etc/netbird/management.json:
OIDC configuration request returned status 504 with response: Gateway Timeout

The 504 came from VPS traefik trying to reach the old auth-authentik backend (https://192.168.8.175, home Caddy) over the down mesh.

Fix — Phase A (restore the mesh)

1. Stood up Authentik on the VPS in /opt/docker-compose.yml: authentik-postgres, authentik-redis, authentik-server, authentik-worker (image ghcr.io/goauthentik/server:2026.5.2), on a dedicated auth Docker network (172.30.1.0/24). Traefik joined both netbird and auth networks.
2. Removed the auth-authentik → 192.168.8.175 backend from /opt/traefik-dynamic.yaml; Authentik now routed via Docker provider labels.
3. Added depends_on: authentik-server: condition: service_healthy to the mgmt service.
4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.

Fix — Phase B (full data migration)

Once the mesh was back, the homelab DB was reachable:

1. ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik' → gzip.
2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the postgres DB to drop authentik), restored the dump.
3. Swapped AUTHENTIK_SECRET_KEY in /opt/authentik.env to the homelab's key — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
4. Restarted Authentik. All users / apps / passwords / groups present.

Scripts left on the VPS: /opt/migrate-authentik.sh (re-runnable), /opt/backup-authentik.sh (daily cron, 14-day retention).

Fix — DNS cutover (the long tail)

The real reason the browser kept hitting the old Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had address=/auth.hubris.network/192.168.8.175, and the NetBird per-client resolver (100.122.255.254) cached it. Symptom: the login page console logged authentik(early): version 2026.2.2 (the old LXC) instead of 2026.5.2 (the VPS).

• Updated dnsmasq: address=/auth.hubris.network/82.165.190.79, systemctl restart dnsmasq.
• netbird down && netbird up does not reliably flush the NetBird resolver cache. Immediate per-client fixes: /etc/hosts override (mac), sudo resolvectl flush-caches (Linux), sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder (mac).

Issues hit during the migration (and fixes)

1. Redirect URI error. The restored DB had redirect URIs in REGEX matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to STRICT exact matching (Django ORM, RedirectURIMatchingMode.STRICT). Set all four: http://localhost:53000/ (CLI), https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}.
2. Only the password field showed (no username). NetBird passes login_hint=<email> in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3. "Request has been denied. Unknown error." Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (Reputation.objects.all().delete() — see 124-authentik), and broken default expression policies. The restored DB carried 8 default policies authored in old return-style syntax incompatible with 2026.5.x's eval context; ak apply_blueprints re-applied the current defaults.
4. Browser ran stale frontend JS. Console showed version 2026.2.2 while the backend was 2026.5.2 — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5. WebAuthn devices dead post-migration. Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.

Architecture delta

	Before	After
Authentik host	LXC 124 192.168.8.180	VPS 82.165.190.79, auth Docker net 172.30.1.0/24
Version	2026.2.2	2026.5.2
auth.hubris.network (LAN)	dnsmasq → 192.168.8.175 (Caddy)	dnsmasq → 82.165.190.79 (VPS traefik)
auth.hubris.network (public)	IONOS wildcard → VPS → mesh → LXC 124	IONOS wildcard → VPS → local container
Reachability	mesh-only (split-horizon)	public (auth UI now exposed on the open internet)
Redis	none (docs said 2026.x dropped it)	present (this deploy uses Redis)
DB engine	sqlite? (StoreConfig in mgmt) / Postgres for Authentik	Postgres, separate from netbird

Security note: the Authentik admin UI is now publicly reachable. Mitigated with a traefik admin-allowlist ipAllowList middleware on /if/admin/ (currently 5.61.168.0/24). Login/flow endpoints are public by necessity. The auth Docker network isolates Postgres/Redis from the netbird containers.

Lessons learned

1. Never put the IdP behind the VPN it authenticates. The circular dependency is invisible until a restart, then it's total.
2. depends_on: condition: service_healthy is the structural fix. Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
3. Migrating an Authentik DB requires carrying its AUTHENTIK_SECRET_KEY. It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
4. Default expression policies break across major Authentik versions. Always run ak apply_blueprints after restoring an older DB into a newer Authentik.
5. Split-horizon DNS + a VPN resolver cache = an invisible stale answer. The NetBird resolver (100.122.255.254) outlives netbird down/up. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
6. Redirect URIs: prefer STRICT over REGEX in Authentik 2026.x.
7. Frontend/backend version skew shows as "Unknown error." Confirm which Authentik the browser actually loaded (version line in the console) before chasing backend logs.
8. WebAuthn/passkeys do not survive host migrations — delete and re-enroll.

Forward-auth outpost — RESOLVED 2026-06-01

Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's embedded outpost (Caddy → 192.168.8.180:9000) — split-brain against the frozen DB. Pointing Caddy at https://auth.hubris.network instead fails: VPS Traefik rewrites X-Forwarded-Host → outpost can't match the app → 404 (tested + reverted).

Fixed with a dedicated LAN outpost (106 — auth-outpost, 192.168.8.6): goauthentik/proxy connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost hubris-lan-outpost carries the 3 proxy providers. Verified with 124-Authentik stopped. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).

2026-06-05 — identification stage skip: broken "Trust me" reputation policy

The default authentication flow was skipping the identification stage (showing only a password field with pending_user: ""), making login impossible. Root cause: a "Trust me" reputation policy (threshold: -3, negate: true) attached to the identification stage binding (#10). The negate: true + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's ReevaluateMarker skipped the stage entirely.

Fix: Deleted from Postgres:

DELETE FROM authentik_policies_policybinding WHERE policy_id = '<uuid>';
DELETE FROM authentik_policies_reputation_reputationpolicy WHERE policy_ptr_id = '<uuid>';
DELETE FROM authentik_policies_policy WHERE policy_uuid = '<uuid>';

The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).

Note: This is different from the 2026-05-31 issue where NetBird's login_hint OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.

Outstanding

• NetBird client version skew. Mac client is 0.68.3 vs mgmt 0.71.3; NetBird-SSH cert exchange fails between them. Upgrade all clients toward 0.71.x.
• republic-laptop regular SSH publickey auth still rejected (correct authorized_keys/perms/SELinux context, yet denied — sshd_config AuthorizedKeysFile suspect). DNS not yet flushed there either.
• muli-laptop needs netbird down && netbird up + resolvectl flush-caches.
• VPS port 22 opened for this repair; close once remote access is otherwise stable.
• Decommission LXC 124 Authentik after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
• Reconcile 124-authentik provider notes — docs describe a Public/PKCE provider; the migrated DB carries the Confidential netbird-dashboard client. Verify which is live and correct the page.
• sops-encrypt the VPS secrets (/opt/authentik.env) into the secrets/ tree.

Related

• 124 — authentik
• DNS split-horizon
• Public ingress (VPS traefik)
• Mesh migration