> ⚠️ **MIGRATED 2026-05-31 — Authentik now runs on the [VPS](../infrastructure/ingress.md) (`82.165.190.79`), not this LXC.** This page's Authentik sections describe the *legacy* instance. The live IdP is the VPS stack (`/opt/docker-compose.yml`, image `2026.5.2`, Postgres + Redis on the `auth` Docker net). `auth.hubris.network` resolves to the VPS now. See [2026-05-31 migration investigation](../investigations/2026-05-31-authentik-vps-migration.md). **dnsmasq is unaffected — it still runs here.** LXC 124 Authentik is kept for a ~2-week dual-run, then decommissioned.
Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box" (DNS only, post-migration).
## At a glance
- **Hostname:** `authentik`
- **IP:** `192.168.8.180` (statically configured — the only LXC with a static IP)
- **Container DNS (in `/etc/pve/lxc/124.conf`):** `192.168.8.1 1.1.1.1` (router DNS plus a fallback added 2026-04-21 because router DNS flakes intermittently — Authentik is the resolver itself for the *rest* of the LAN, but its own LXC uses upstream).
-`.env` mode 600, **untracked**, holds `AUTHENTIK_SECRET_KEY` and `PG_PASS`.
-`AUTHENTIK_TAG=2026.2.2` — pinned. Don't let it drift to `:latest`. Telemetry / update-check / error-reporting disabled.
- Ports: 9000 (http), 9443 (https) on the LXC.
- Embedded outpost lives at `/outpost.goauthentik.io/*` on the Authentik host — the forward-auth endpoint Caddy points at.
- Stack is **not** git-tracked yet. If/when wiring auto-deploy: mirror the `mule-image` pattern (webhook receiver outside the app repo at `/opt/authentik-deploy/`). Repo `dtoro/authentik-conf` is reserved but not created.
## Forward-auth pattern (every gated app)
- **One Proxy Provider per app.** Authentik enforces a UNIQUE constraint `application.provider_id`, so one Provider = one Application. "Domain-level" only means they share the cookie domain. Each provider in "Forward auth (domain level)" mode, External host `https://auth.hubris.network`, Cookie domain `hubris.network`. First one was `hubris-forward-auth` (Paperless).
- **Authentication flow:** MUST be `default-authentication-flow` (NOT `default-source-authentication` — that's for IdP federation; gives `FlowNonApplicableException` + 404 on the authorize endpoint).
Caddy snippet `(authentik)` lives at the top of `/etc/caddy/Caddyfile`. Points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid hairpin TLS round-trip stripping `X-Forwarded-Host`. Must explicitly set `header_up X-Forwarded-Host {host}` in the forward-auth block. Used by gated sites with `import authentik`.
### Per-app username override pattern (Authentik)
Used when the app's local user ID doesn't match the user's Authentik username (e.g., Nextcloud's `admin` ≠ Authentik's `dtoro`).
1. On the Authentik user: add attribute `<app>_uid: <target_local_username>` (YAML, Directory → Users → Edit → Attributes).
- **Use a custom claim key** (e.g. `nc_uid`), not `preferred_username` — the default `profile` scope mapping emits `preferred_username` and will overwrite yours depending on ordering.
3. Attach the new scope to the provider (Providers → app → Scopes).
4. On the app side, point its OIDC UID-mapping setting at the custom claim.
### Bypass forward-auth for API paths (mobile apps)
If the app has its own token-based API auth and a mobile client, API paths must bypass forward-auth — mobile apps can't follow the browser login redirect. Pattern in the Caddyfile site block:
- [qBittorrent](122-arriman.md): `/api/*` (session cookie from `/api/v2/auth/login`)
- [SABnzbd](122-arriman.md): `/api?*` (apikey query param) — match `/api*` for query-string APIs
- Homarr: no mobile client
- [Portainer](105-apps.md): mobile uses same session auth as web; no bypass typically needed
### Backend trust of Authentik headers (skip the app's own login after SSO)
- [**Paperless**](103-paperless.md): `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Restart `paperless-webserver paperless-task-queue paperless-scheduler paperless-consumer`. Django auto-creates matching users on first SSO login; promote to superuser via existing admin UI.
- Apps without header-auth support: users log in twice (SSO + app login). Acceptable but degraded UX.
| [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. |
## Netbird IdP integration — LANDED 2026-05-21
The combined `netbirdio/netbird-server` image was replaced with the canonical vanilla stack (`mgmt + signal + relay + dashboard` 0.71.3) on the VPS so external OIDC actually works. Authentik is now the netbird dashboard's IdP. Full migration context in [mesh.md changelog](../infrastructure/mesh.md#changelog).
**Active provider & app:**
- Provider `NetBird` (OAuth2/OpenID), **Client type: `Public`** (PKCE-only — `Confidential` would break the dashboard SPA's token exchange).
- Client ID: `netbird-dashboard`. Client Secret is in `/opt/management.json` `PKCEAuthorizationFlow.ProviderConfig.ClientSecret` on the VPS (TODO: sops-encrypt as `secrets/netbird-authentik-oidc.yaml`).
- Redirect URIs: `https://netbird.hubris.network/peers`, `/nb-auth`, `/nb-silent-auth`, plus `https://netbird.hubris.network/` for post-logout.
- Scopes enabled on the provider: `openid`, `profile`, `email`.
- Discovery URL: `https://auth.hubris.network/application/o/netbird/.well-known/openid-configuration` — netbird mgmt fetches this on startup; logs `loaded OIDC configuration from the provided IDP configuration endpoint`.
**Login flow:** netbird dashboard PKCE → Authentik authorize → redirect back to `/nb-auth` → JS token exchange at Authentik's `/token` endpoint → mgmt validates the bearer against Authentik's JWKS.
**First-time owner promotion gotcha** (write-down for future operators):
When a new Authentik user logs in for the first time against an account that already has peers, netbird mgmt adds them as `role=user, blocked=1, pending_approval=1`. The OLD account-owner (the one in `store.db` from before the IdP swap) can't be reached anymore, so there's no admin to approve. Recovery is a direct sqlite update on `mgmt_data`:
2. **Brand** (System → Brands → edit the brand serving `auth.hubris.network`): set **Device code flow** field to `default-device-code-flow`.
3. No provider-side change is required — Authentik 2026.x routes `/device` via the brand's device-code flow, not via the OAuth2/OpenID provider's `Authorization flow`.
**Why this matters for the lab**: Authentik 2026.x doesn't ship a default device-code flow. Without this configuration, the URL `https://auth.hubris.network/device` renders blank (the `/device` endpoint is unrouted), so `netbird up` device-codes expire without consent → only `--setup-key` works for onboarding. The above unblocks interactive onboarding.
**Verifying** from a browser tab: visit `https://auth.hubris.network/device`. You should see a form with one **Code** input + Continue button. Then `netbird up` (no setup-key) end-to-end:
- CLI prints `verification_uri_complete: https://auth.hubris.network/device?code=...`
- Open URL → identification (skipped if logged in) → password (re-auth check) → consent ("Authorize NetBird?") → Continue
`auth.hubris.network` is only reachable from inside the netbird mesh (split-horizon DNS). A brand-new client that isn't on the mesh yet can't OIDC-login → setup-key is the only path. To enable self-service onboarding via Authentik from the public internet:
- Add a Traefik route on the VPS for `auth.hubris.network` that forwards via the netbird-routed `192.168.8.0/24` to LXC 124.
- DNS already points `auth.hubris.network → 82.165.190.79` (IONOS wildcard).
Tracked in homelab memory as a queued follow-up.
### Old pre-work to remove
The previous `Provider for Netbird` + `netbird` app from 2026-04-22 (client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`, service account `netbird-service`) is now obsolete — replaced by `netbird-dashboard` above. Safe to delete from Authentik admin UI; nothing currently uses the old client ID. The service account + API token can also be removed unless we wire IdpManagerConfig in mgmt later (currently `ManagerType: none`).
## DNS responsibility
dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` + `127.0.0.1:53`, serving every `*.hubris.network` subdomain → `192.168.8.175`. **There is no wildcard** — every site needs an explicit `address=` entry. See [DNS split-horizon](../infrastructure/dns.md).
## Related
- [DNS split-horizon](../infrastructure/dns.md)
- [Caddy (121)](121-caddy.md)
- [Mesh migration](../infrastructure/mesh.md)
- Every gated app under [containers/index](index.md)
## Changelog
### 2026-06-01 — RETIRED (shut down)
dnsmasq relocated to [dns (107)](107-dns.md) Technitium at `192.168.8.2`. With both its services gone (Authentik → VPS, dnsmasq → 107), LXC 124 was verified idle and **shut down** (`pct stop`). Kept (not destroyed) with the `pre-arch-migration` snapshot for ≥30-day rollback, then `pct destroy`.
### 2026-06-01 — forward-auth outpost cut over to LXC 106; Authentik stopped on 124
The embedded outpost that Caddy called at `192.168.8.180:9000` was replaced by a dedicated LAN outpost on [106 — auth-outpost](106-auth-outpost.md) (`192.168.8.6:9000`), connected to the VPS core. Authentik (`server/worker/postgresql`) was **stopped** on this LXC; forward-auth apps verified working without it. **dnsmasq still runs here** — 124 stays up as a DNS-only box until DNS is relocated (architecture-migration Phase 2). Once DNS moves, 124 can be fully retired.
### 2026-05-31 — Authentik migrated to the VPS (this LXC is now legacy)
Resolved a bootstrap deadlock — netbird-mgmt on the VPS couldn't start because it fetches Authentik's OIDC discovery on boot, but Authentik was mesh-only and the mesh was down because mgmt was down. Moved the whole Authentik stack onto the VPS (image `2026.2.2` → `2026.5.2`, now with Redis, on a dedicated `auth` Docker network), migrated the full Postgres DB (users/apps/passwords/groups), and added `depends_on: service_healthy` so the deadlock can't recur. dnsmasq stays on this LXC; its `auth.hubris.network` entry now points to `82.165.190.79`. Along the way: switched redirect URIs to `STRICT`, ran `ak apply_blueprints` to fix old `return`-syntax expression policies, and deleted dead WebAuthn devices. Full writeup: [investigation](../investigations/2026-05-31-authentik-vps-migration.md).
VPS migrated from combined netbird-server to vanilla mgmt+signal+relay+dashboard 0.71.3 (see [mesh.md](../infrastructure/mesh.md)), enabling Authentik as the dashboard IdP via PKCE. New Provider/App = `netbird-dashboard`, replacing the deferred pre-work. Device Code Stage still missing — interactive `netbird up` fails consent; setup-keys are the workaround until that's added.
Combined netbird-server image couldn't take an external IdP. Pre-work in Authentik kept for later (now superseded by 2026-05-21 above). Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto.
### 2026-04-21 — deployed; Phases 1–5 complete
LXC 124 provisioned, stack at `/opt/authentik`, public URL via Caddy, Paperless + Booklore + Nextcloud + Home Assistant wired. dnsmasq for split-horizon DNS lives on the same LXC.
@@ -47,7 +47,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
> Each owns its own clone on LXC 105. They don't conflict because each
> deploy.sh only touches its own service unit + venv.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
@@ -7,11 +7,11 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
## Components
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md),`192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at`192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
## Live entries (as of 2026-05-31)
## Live entries (as of 2026-06-04)
```
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
@@ -41,24 +41,20 @@ Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that point
## Why split-horizon
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
## The gotcha that cost a debug session (2026-04-22)
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
## Recipe — adding a new subdomain
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md):
```
address=/<new>.hubris.network/192.168.8.175
```
3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path.
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124).
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
## Public path — what does and doesn't follow the LAN map
@@ -76,21 +72,17 @@ Either:
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
## Changelog
### 2026-06-04 — Caddy: proxy auth.hubris.network → VPS; fix stale /etc/hosts across fleet
Authentik had migrated to the VPS on 2026-05-31, but:
- Caddy's `auth.hubris.network` block still proxied to dead `192.168.8.6:9000`
- Caddy's `authentik` forward-auth snippet and `sso.hubris.network` block also referenced the stale IP
- Gitea's `/etc/hosts` had both the VPS IP and the old Caddy IP, causing Gitea to fail on OAuth2 token exchange with `404` then `issuer mismatch`
**Fixed:**
- Caddy Caddyfile: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxy to VPS (`82.165.190.79:443`) with proper Host header stripping
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
## Why this shape
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
