dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

authentik: migrate from LXC 124 to VPS; eliminate mesh bootstrap deadlock

Browse Source 
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).

- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-05-31 23:17:00 +02:00
parent 04d1f39e7b
commit 8dbba323c1
6 changed files with 118 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
infrastructure/dns.md
View File

@@ -11,10 +11,10 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
## Live entries (as of 2026-04-28)
## Live entries (as of 2026-05-31)
```
address=/auth.hubris.network/192.168.8.175
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
@@ -79,6 +79,9 @@ Either:
## Changelog
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../investigations/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.

14
infrastructure/ingress.md
View File

@@ -45,6 +45,17 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
Since 2026-05-31 [Authentik runs on the VPS itself](../investigations/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
- Traefik reaches it over the `auth` Docker network (`172.30.1.0/24`); Postgres/Redis on that net are isolated from the netbird containers.
- Admin UI is IP-gated: an `admin-allowlist` ipAllowList middleware on `PathPrefix(/if/admin/)` (currently `5.61.168.0/24`). Login/flow endpoints stay public.
No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `auth`.
## Recipe — exposing another service
1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
@@ -74,6 +85,9 @@ Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Cadd
## Changelog
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
Authentik migrated onto the VPS ([investigation](../investigations/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
### 2026-04-28 — wiki entry created
Initial documentation.

3
infrastructure/mesh.md
View File

@@ -116,6 +116,9 @@ Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-man
## Changelog
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../investigations/2026-05-31-authentik-vps-migration.md).
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.