dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

authentik: migrate from LXC 124 to VPS; eliminate mesh bootstrap deadlock

Browse Source 
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).

- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-05-31 23:17:00 +02:00
parent 04d1f39e7b
commit 8dbba323c1
6 changed files with 118 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
containers/124-authentik.md
View File

@@ -1,6 +1,8 @@
# 124 — `authentik`
Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box".
> ⚠️ **MIGRATED 2026-05-31 — Authentik now runs on the [VPS](../infrastructure/ingress.md) (`82.165.190.79`), not this LXC.** This page's Authentik sections describe the *legacy* instance. The live IdP is the VPS stack (`/opt/docker-compose.yml`, image `2026.5.2`, Postgres + Redis on the `auth` Docker net). `auth.hubris.network` resolves to the VPS now. See [2026-05-31 migration investigation](../investigations/2026-05-31-authentik-vps-migration.md). **dnsmasq is unaffected — it still runs here.** LXC 124 Authentik is kept for a ~2-week dual-run, then decommissioned.
Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box" (DNS only, post-migration).
## At a glance
- **Hostname:** `authentik`
@@ -170,6 +172,9 @@ dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` +
## Changelog
### 2026-05-31 — Authentik migrated to the VPS (this LXC is now legacy)
Resolved a bootstrap deadlock — netbird-mgmt on the VPS couldn't start because it fetches Authentik's OIDC discovery on boot, but Authentik was mesh-only and the mesh was down because mgmt was down. Moved the whole Authentik stack onto the VPS (image `2026.2.2` → `2026.5.2`, now with Redis, on a dedicated `auth` Docker network), migrated the full Postgres DB (users/apps/passwords/groups), and added `depends_on: service_healthy` so the deadlock can't recur. dnsmasq stays on this LXC; its `auth.hubris.network` entry now points to `82.165.190.79`. Along the way: switched redirect URIs to `STRICT`, ran `ak apply_blueprints` to fix old `return`-syntax expression policies, and deleted dead WebAuthn devices. Full writeup: [investigation](../investigations/2026-05-31-authentik-vps-migration.md).
### 2026-05-21 — Netbird IdP swap landed (Phase 6 done)
VPS migrated from combined netbird-server to vanilla mgmt+signal+relay+dashboard 0.71.3 (see [mesh.md](../infrastructure/mesh.md)), enabling Authentik as the dashboard IdP via PKCE. New Provider/App = `netbird-dashboard`, replacing the deferred pre-work. Device Code Stage still missing — interactive `netbird up` fails consent; setup-keys are the workaround until that's added.

7
infrastructure/dns.md
View File

@@ -11,10 +11,10 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
## Live entries (as of 2026-04-28)
## Live entries (as of 2026-05-31)
```
address=/auth.hubris.network/192.168.8.175
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
@@ -79,6 +79,9 @@ Either:
## Changelog
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.

14
infrastructure/ingress.md
View File

@@ -45,6 +45,17 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
- Traefik reaches it over the `auth` Docker network (`172.30.1.0/24`); Postgres/Redis on that net are isolated from the netbird containers.
- Admin UI is IP-gated: an `admin-allowlist` ipAllowList middleware on `PathPrefix(/if/admin/)` (currently `5.61.168.0/24`). Login/flow endpoints stay public.
No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `auth`.
## Recipe — exposing another service
1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
@@ -74,6 +85,9 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
## Changelog
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
### 2026-04-28 — wiki entry created
Initial documentation.

3
infrastructure/mesh.md
View File

@@ -116,6 +116,9 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.

89
investigations/2026-05-31-authentik-vps-migration.md Normal file
View File

@@ -0,0 +1,89 @@
# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
## Summary
The NetBird management server (on the [VPS](../infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
Resolved by moving Authentik off [LXC 124](../containers/124-authentik.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
## Timeline
### Trigger
`netbird status` on clients showed `Management: Disconnected`; `netbird.hubris.network` unreachable. VPS `docker logs netbird-mgmt`:
```
Error: failed reading provided config file: /etc/netbird/management.json:
OIDC configuration request returned status 504 with response: Gateway Timeout
```
The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
### Fix — Phase A (restore the mesh)
1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
2. Removed the `auth-authentik` → `192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
### Fix — Phase B (full data migration)
Once the mesh was back, the homelab DB was reachable:
1. `ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik'` → gzip.
2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
3. **Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
4. Restarted Authentik. All users / apps / passwords / groups present.
Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
### Fix — DNS cutover (the long tail)
The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
- Updated dnsmasq: `address=/auth.hubris.network/82.165.190.79`, `systemctl restart dnsmasq`.
- `netbird down && netbird up` does **not** reliably flush the NetBird resolver cache. Immediate per-client fixes: `/etc/hosts` override (mac), `sudo resolvectl flush-caches` (Linux), `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` (mac).
## Issues hit during the migration (and fixes)
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../containers/124-authentik.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
## Architecture delta
| | Before | After |
|---|---|---|
| Authentik host | [LXC 124](../containers/124-authentik.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Version | `2026.2.2` | `2026.5.2` |
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
| Reachability | mesh-only (split-horizon) | **public** (auth UI now exposed on the open internet) |
| Redis | none (docs said 2026.x dropped it) | present (this deploy uses Redis) |
| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
## Lessons learned
1. **Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
2. **`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
3. **Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
4. **Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
5. **Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
6. **Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
7. **Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
8. **WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
## Outstanding
- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config` `AuthorizedKeysFile` suspect). DNS not yet flushed there either.
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
- **Reconcile [124-authentik](../containers/124-authentik.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
## Related
- [124 — authentik](../containers/124-authentik.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md)
- [Mesh migration](../infrastructure/mesh.md)

1
investigations/index.md
View File

@@ -6,6 +6,7 @@ Time-stamped incident reports and experiments. One entry per incident; the entry
| Date | Title | Status |
| ------------ | ------------------------------------------------------------------ | ------------- |
| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](2026-05-31-authentik-vps-migration.md) | Resolved; LXC 124 Authentik in ~2-week dual-run before decommission |
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) |
## Conventions