cleanup: destroy LXC 124, remove stale docs, update DNS refs to Technitium
This commit is contained in:
@@ -1,197 +0,0 @@
|
|||||||
# 124 — `authentik`
|
|
||||||
|
|
||||||
> ⚠️ **MIGRATED 2026-05-31 — Authentik now runs on the [VPS](../infrastructure/ingress.md) (`82.165.190.79`), not this LXC.** This page's Authentik sections describe the *legacy* instance. The live IdP is the VPS stack (`/opt/docker-compose.yml`, image `2026.5.2`, Postgres + Redis on the `auth` Docker net). `auth.hubris.network` resolves to the VPS now. See [2026-05-31 migration investigation](../investigations/2026-05-31-authentik-vps-migration.md). **dnsmasq is unaffected — it still runs here.** LXC 124 Authentik is kept for a ~2-week dual-run, then decommissioned.
|
|
||||||
|
|
||||||
Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box" (DNS only, post-migration).
|
|
||||||
|
|
||||||
## At a glance
|
|
||||||
- **Hostname:** `authentik`
|
|
||||||
- **IP:** `192.168.8.180` (statically configured — the only LXC with a static IP)
|
|
||||||
- **Privilege:** privileged
|
|
||||||
- **Resources:** 2 cores / 4 GiB RAM / 20 GiB rootfs
|
|
||||||
- **Mounts:** none from `/mnt/library`
|
|
||||||
- **Public hostname:** [`auth.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `:9000`
|
|
||||||
- **Container DNS (in `/etc/pve/lxc/124.conf`):** `192.168.8.1 1.1.1.1` (router DNS plus a fallback added 2026-04-21 because router DNS flakes intermittently — Authentik is the resolver itself for the *rest* of the LAN, but its own LXC uses upstream).
|
|
||||||
|
|
||||||
## Authentik stack (`/opt/authentik/`)
|
|
||||||
|
|
||||||
Upstream `docker-compose.yml` + `.env`. Services: `postgresql` (16-alpine), `server`, `worker`. Authentik 2026.x dropped the Redis dependency.
|
|
||||||
|
|
||||||
- `.env` mode 600, **untracked**, holds `AUTHENTIK_SECRET_KEY` and `PG_PASS`.
|
|
||||||
- `AUTHENTIK_TAG=2026.2.2` — pinned. Don't let it drift to `:latest`. Telemetry / update-check / error-reporting disabled.
|
|
||||||
- Ports: 9000 (http), 9443 (https) on the LXC.
|
|
||||||
- Embedded outpost lives at `/outpost.goauthentik.io/*` on the Authentik host — the forward-auth endpoint Caddy points at.
|
|
||||||
- Stack is **not** git-tracked yet. If/when wiring auto-deploy: mirror the `mule-image` pattern (webhook receiver outside the app repo at `/opt/authentik-deploy/`). Repo `dtoro/authentik-conf` is reserved but not created.
|
|
||||||
|
|
||||||
## Forward-auth pattern (every gated app)
|
|
||||||
|
|
||||||
- **One Proxy Provider per app.** Authentik enforces a UNIQUE constraint `application.provider_id`, so one Provider = one Application. "Domain-level" only means they share the cookie domain. Each provider in "Forward auth (domain level)" mode, External host `https://auth.hubris.network`, Cookie domain `hubris.network`. First one was `hubris-forward-auth` (Paperless).
|
|
||||||
- **Authentication flow:** MUST be `default-authentication-flow` (NOT `default-source-authentication` — that's for IdP federation; gives `FlowNonApplicableException` + 404 on the authorize endpoint).
|
|
||||||
- **Authorization flow:** `default-provider-authorization-implicit-consent` (or explicit).
|
|
||||||
- **Application Launch URL** MUST be the full public URL `https://<sub>.hubris.network/` — outpost matches incoming `X-Forwarded-Host` against it.
|
|
||||||
- Each Application MUST have at least one **policy/group/user binding** — zero bindings = outpost returns 404 on access.
|
|
||||||
- **Restart Authentik after binding new apps to the outpost:**
|
|
||||||
```
|
|
||||||
pct exec 124 -- docker compose -f /opt/authentik/docker-compose.yml restart server worker
|
|
||||||
```
|
|
||||||
|
|
||||||
Caddy snippet `(authentik)` lives at the top of `/etc/caddy/Caddyfile`. Points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid hairpin TLS round-trip stripping `X-Forwarded-Host`. Must explicitly set `header_up X-Forwarded-Host {host}` in the forward-auth block. Used by gated sites with `import authentik`.
|
|
||||||
|
|
||||||
### Per-app username override pattern (Authentik)
|
|
||||||
|
|
||||||
Used when the app's local user ID doesn't match the user's Authentik username (e.g., Nextcloud's `admin` ≠ Authentik's `dtoro`).
|
|
||||||
|
|
||||||
1. On the Authentik user: add attribute `<app>_uid: <target_local_username>` (YAML, Directory → Users → Edit → Attributes).
|
|
||||||
2. Customization → Property Mappings → Create → **Scope Mapping** (not SAML):
|
|
||||||
- Name: `<app>-uid-override`, Scope name: `<app>-uid`, Expression:
|
|
||||||
```python
|
|
||||||
return {"nc_uid": user.attributes.get("<app>_uid", user.username)}
|
|
||||||
```
|
|
||||||
- **Use a custom claim key** (e.g. `nc_uid`), not `preferred_username` — the default `profile` scope mapping emits `preferred_username` and will overwrite yours depending on ordering.
|
|
||||||
3. Attach the new scope to the provider (Providers → app → Scopes).
|
|
||||||
4. On the app side, point its OIDC UID-mapping setting at the custom claim.
|
|
||||||
|
|
||||||
For Nextcloud:
|
|
||||||
```
|
|
||||||
occ user_oidc:provider <name> --mapping-uid=nc_uid
|
|
||||||
occ user_oidc:provider <name> --scope="openid profile email <app>-uid"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Bypass forward-auth for API paths (mobile apps)
|
|
||||||
|
|
||||||
If the app has its own token-based API auth and a mobile client, API paths must bypass forward-auth — mobile apps can't follow the browser login redirect. Pattern in the Caddyfile site block:
|
|
||||||
|
|
||||||
```
|
|
||||||
paperless.hubris.network {
|
|
||||||
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
|
|
||||||
@api path /api/*
|
|
||||||
handle @api {
|
|
||||||
reverse_proxy 192.168.8.130:8000
|
|
||||||
}
|
|
||||||
handle {
|
|
||||||
import authentik
|
|
||||||
reverse_proxy 192.168.8.130:8000
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
API paths to bypass per app:
|
|
||||||
- [Paperless](103-paperless.md): `/api/*` (Bearer)
|
|
||||||
- [Sonarr / Radarr / Lidarr / etc.](122-arriman.md): `/api/*` (X-Api-Key)
|
|
||||||
- [qBittorrent](122-arriman.md): `/api/*` (session cookie from `/api/v2/auth/login`)
|
|
||||||
- [SABnzbd](122-arriman.md): `/api?*` (apikey query param) — match `/api*` for query-string APIs
|
|
||||||
- Homarr: no mobile client
|
|
||||||
- [Portainer](105-apps.md): mobile uses same session auth as web; no bypass typically needed
|
|
||||||
|
|
||||||
### Backend trust of Authentik headers (skip the app's own login after SSO)
|
|
||||||
|
|
||||||
- [**Paperless**](103-paperless.md): `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Restart `paperless-webserver paperless-task-queue paperless-scheduler paperless-consumer`. Django auto-creates matching users on first SSO login; promote to superuser via existing admin UI.
|
|
||||||
- Apps without header-auth support: users log in twice (SSO + app login). Acceptable but degraded UX.
|
|
||||||
|
|
||||||
## Per-app integration map
|
|
||||||
|
|
||||||
| App | Type | Notes |
|
|
||||||
| ----------------------------------------- | ---------------- | ----- |
|
|
||||||
| [Paperless (103)](103-paperless.md) | Forward-auth + REMOTE_USER | `/api/*` bypass |
|
|
||||||
| [Nextcloud (114)](114-nextcloud.md) | Native OIDC | `nc_uid` override; local dnsmasq required (Guzzle bypasses `/etc/hosts`) |
|
|
||||||
| [mulita (120)](120-mule-images.md) | Native OIDC | `extra_hosts` override in compose |
|
|
||||||
| [Booklore (105)](105-apps.md) | Native OIDC | Redirect URI `/oauth2-callback`; `extra_hosts` |
|
|
||||||
| [Portainer (105)](105-apps.md) | Native OAuth2 | `portainer_uid` custom claim; `--trusted-origins` flag |
|
|
||||||
| [WriteFreely (105)](105-apps.md) | Native OIDC | `[oauth.generic]` block; `extra_hosts` |
|
|
||||||
| [qBittorrent (122)](122-arriman.md) | Forward-auth via IP whitelist | Reverse-proxy support enabled in qBit |
|
|
||||||
| [Artifacto (105)](105-apps.md) | Forward-auth + gateway-secret auto-login | Public `/p/*` paths bypass |
|
|
||||||
| [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. |
|
|
||||||
|
|
||||||
## Netbird IdP integration — LANDED 2026-05-21
|
|
||||||
|
|
||||||
The combined `netbirdio/netbird-server` image was replaced with the canonical vanilla stack (`mgmt + signal + relay + dashboard` 0.71.3) on the VPS so external OIDC actually works. Authentik is now the netbird dashboard's IdP. Full migration context in [mesh.md changelog](../infrastructure/mesh.md#changelog).
|
|
||||||
|
|
||||||
**Active provider & app:**
|
|
||||||
- Provider `NetBird` (OAuth2/OpenID), **Client type: `Public`** (PKCE-only — `Confidential` would break the dashboard SPA's token exchange).
|
|
||||||
- Client ID: `netbird-dashboard`. Client Secret is in `/opt/management.json` `PKCEAuthorizationFlow.ProviderConfig.ClientSecret` on the VPS (TODO: sops-encrypt as `secrets/netbird-authentik-oidc.yaml`).
|
|
||||||
- Application `NetBird`, slug `netbird`, launch URL `https://netbird.hubris.network/`.
|
|
||||||
- Redirect URIs: `https://netbird.hubris.network/peers`, `/nb-auth`, `/nb-silent-auth`, plus `https://netbird.hubris.network/` for post-logout.
|
|
||||||
- Scopes enabled on the provider: `openid`, `profile`, `email`.
|
|
||||||
- Discovery URL: `https://auth.hubris.network/application/o/netbird/.well-known/openid-configuration` — netbird mgmt fetches this on startup; logs `loaded OIDC configuration from the provided IDP configuration endpoint`.
|
|
||||||
|
|
||||||
**Login flow:** netbird dashboard PKCE → Authentik authorize → redirect back to `/nb-auth` → JS token exchange at Authentik's `/token` endpoint → mgmt validates the bearer against Authentik's JWKS.
|
|
||||||
|
|
||||||
**First-time owner promotion gotcha** (write-down for future operators):
|
|
||||||
|
|
||||||
When a new Authentik user logs in for the first time against an account that already has peers, netbird mgmt adds them as `role=user, blocked=1, pending_approval=1`. The OLD account-owner (the one in `store.db` from before the IdP swap) can't be reached anymore, so there's no admin to approve. Recovery is a direct sqlite update on `mgmt_data`:
|
|
||||||
|
|
||||||
```
|
|
||||||
docker stop netbird-mgmt
|
|
||||||
sqlite3 /var/lib/docker/volumes/opt_mgmt_data/_data/store.db \
|
|
||||||
"UPDATE users SET role='owner', blocked=0, pending_approval=0 WHERE id='<new authentik sub>';"
|
|
||||||
docker start netbird-mgmt
|
|
||||||
```
|
|
||||||
|
|
||||||
The Authentik sub-claim is the value of the `id` column on the newly-created user row (look for `role=user, blocked=1, pending_approval=1`).
|
|
||||||
|
|
||||||
### Device Code grant — configured (2026-05-21)
|
|
||||||
|
|
||||||
`netbird up` (interactive, without `--setup-key`) works against Authentik. The recipe:
|
|
||||||
|
|
||||||
1. **Flow** `default-device-code-flow` (designation: `Stage Configuration`) with 4 stage bindings in order:
|
|
||||||
- 10: `default-authentication-identification` (username/email lookup)
|
|
||||||
- 20: `default-authentication-password` (password validation)
|
|
||||||
- 30: `default-authentication-login` (attach authenticated user to session)
|
|
||||||
- 40: `default-provider-authorization-explicit-consent`'s Consent Stage (`default-provider-authorization-consent`) — the "Authorize NetBird?" approval
|
|
||||||
2. **Brand** (System → Brands → edit the brand serving `auth.hubris.network`): set **Device code flow** field to `default-device-code-flow`.
|
|
||||||
3. No provider-side change is required — Authentik 2026.x routes `/device` via the brand's device-code flow, not via the OAuth2/OpenID provider's `Authorization flow`.
|
|
||||||
|
|
||||||
**Why this matters for the lab**: Authentik 2026.x doesn't ship a default device-code flow. Without this configuration, the URL `https://auth.hubris.network/device` renders blank (the `/device` endpoint is unrouted), so `netbird up` device-codes expire without consent → only `--setup-key` works for onboarding. The above unblocks interactive onboarding.
|
|
||||||
|
|
||||||
**Verifying** from a browser tab: visit `https://auth.hubris.network/device`. You should see a form with one **Code** input + Continue button. Then `netbird up` (no setup-key) end-to-end:
|
|
||||||
- CLI prints `verification_uri_complete: https://auth.hubris.network/device?code=...`
|
|
||||||
- Open URL → identification (skipped if logged in) → password (re-auth check) → consent ("Authorize NetBird?") → Continue
|
|
||||||
- CLI completes registration with `Connected`
|
|
||||||
|
|
||||||
### Self-service onboarding (not yet — future-session)
|
|
||||||
|
|
||||||
`auth.hubris.network` is only reachable from inside the netbird mesh (split-horizon DNS). A brand-new client that isn't on the mesh yet can't OIDC-login → setup-key is the only path. To enable self-service onboarding via Authentik from the public internet:
|
|
||||||
- Add a Traefik route on the VPS for `auth.hubris.network` that forwards via the netbird-routed `192.168.8.0/24` to LXC 124.
|
|
||||||
- DNS already points `auth.hubris.network → 82.165.190.79` (IONOS wildcard).
|
|
||||||
|
|
||||||
Tracked in homelab memory as a queued follow-up.
|
|
||||||
|
|
||||||
### Old pre-work to remove
|
|
||||||
|
|
||||||
The previous `Provider for Netbird` + `netbird` app from 2026-04-22 (client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`, service account `netbird-service`) is now obsolete — replaced by `netbird-dashboard` above. Safe to delete from Authentik admin UI; nothing currently uses the old client ID. The service account + API token can also be removed unless we wire IdpManagerConfig in mgmt later (currently `ManagerType: none`).
|
|
||||||
|
|
||||||
## DNS responsibility
|
|
||||||
|
|
||||||
dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` + `127.0.0.1:53`, serving every `*.hubris.network` subdomain → `192.168.8.175`. **There is no wildcard** — every site needs an explicit `address=` entry. See [DNS split-horizon](../infrastructure/dns.md).
|
|
||||||
|
|
||||||
## Related
|
|
||||||
- [DNS split-horizon](../infrastructure/dns.md)
|
|
||||||
- [Caddy (121)](121-caddy.md)
|
|
||||||
- [Mesh migration](../infrastructure/mesh.md)
|
|
||||||
- Every gated app under [containers/index](index.md)
|
|
||||||
|
|
||||||
## Changelog
|
|
||||||
|
|
||||||
### 2026-06-01 — RETIRED (shut down)
|
|
||||||
dnsmasq relocated to [dns (107)](107-dns.md) Technitium at `192.168.8.2`. With both its services gone (Authentik → VPS, dnsmasq → 107), LXC 124 was verified idle and **shut down** (`pct stop`). Kept (not destroyed) with the `pre-arch-migration` snapshot for ≥30-day rollback, then `pct destroy`.
|
|
||||||
|
|
||||||
### 2026-06-01 — forward-auth outpost cut over to LXC 106; Authentik stopped on 124
|
|
||||||
The embedded outpost that Caddy called at `192.168.8.180:9000` was replaced by a dedicated LAN outpost on [106 — auth-outpost](106-auth-outpost.md) (`192.168.8.6:9000`), connected to the VPS core. Authentik (`server/worker/postgresql`) was **stopped** on this LXC; forward-auth apps verified working without it. **dnsmasq still runs here** — 124 stays up as a DNS-only box until DNS is relocated (architecture-migration Phase 2). Once DNS moves, 124 can be fully retired.
|
|
||||||
|
|
||||||
### 2026-05-31 — Authentik migrated to the VPS (this LXC is now legacy)
|
|
||||||
Resolved a bootstrap deadlock — netbird-mgmt on the VPS couldn't start because it fetches Authentik's OIDC discovery on boot, but Authentik was mesh-only and the mesh was down because mgmt was down. Moved the whole Authentik stack onto the VPS (image `2026.2.2` → `2026.5.2`, now with Redis, on a dedicated `auth` Docker network), migrated the full Postgres DB (users/apps/passwords/groups), and added `depends_on: service_healthy` so the deadlock can't recur. dnsmasq stays on this LXC; its `auth.hubris.network` entry now points to `82.165.190.79`. Along the way: switched redirect URIs to `STRICT`, ran `ak apply_blueprints` to fix old `return`-syntax expression policies, and deleted dead WebAuthn devices. Full writeup: [investigation](../investigations/2026-05-31-authentik-vps-migration.md).
|
|
||||||
|
|
||||||
### 2026-05-21 — Netbird IdP swap landed (Phase 6 done)
|
|
||||||
VPS migrated from combined netbird-server to vanilla mgmt+signal+relay+dashboard 0.71.3 (see [mesh.md](../infrastructure/mesh.md)), enabling Authentik as the dashboard IdP via PKCE. New Provider/App = `netbird-dashboard`, replacing the deferred pre-work. Device Code Stage still missing — interactive `netbird up` fails consent; setup-keys are the workaround until that's added.
|
|
||||||
|
|
||||||
### 2026-04-28 — wiki entry created
|
|
||||||
Initial documentation.
|
|
||||||
|
|
||||||
### 2026-04-22 — Phase 6 (Netbird IdP swap) deferred
|
|
||||||
Combined netbird-server image couldn't take an external IdP. Pre-work in Authentik kept for later (now superseded by 2026-05-21 above). Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
|
|
||||||
|
|
||||||
### 2026-04-22 — Artifacto, mulita, WriteFreely, Portainer wired
|
|
||||||
Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto.
|
|
||||||
|
|
||||||
### 2026-04-21 — deployed; Phases 1–5 complete
|
|
||||||
LXC 124 provisioned, stack at `/opt/authentik`, public URL via Caddy, Paperless + Booklore + Nextcloud + Home Assistant wired. dnsmasq for split-horizon DNS lives on the same LXC.
|
|
||||||
@@ -47,7 +47,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
|||||||
> Each owns its own clone on LXC 105. They don't conflict because each
|
> Each owns its own clone on LXC 105. They don't conflict because each
|
||||||
> deploy.sh only touches its own service unit + venv.
|
> deploy.sh only touches its own service unit + venv.
|
||||||
|
|
||||||
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
|
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../hosts/netbird-vps.md). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
|
||||||
|
|
||||||
## When you change a tracked config
|
## When you change a tracked config
|
||||||
|
|
||||||
|
|||||||
@@ -7,11 +7,11 @@ There is **no wildcard on the LAN side**. Every subdomain needs an explicit entr
|
|||||||
## Components
|
## Components
|
||||||
|
|
||||||
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
|
||||||
- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md), `192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`).
|
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/124-authentik.md) (decommissioned 2026-06-04).
|
||||||
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
|
||||||
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
|
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
|
||||||
|
|
||||||
## Live entries (as of 2026-05-31)
|
## Live entries (as of 2026-06-04)
|
||||||
|
|
||||||
```
|
```
|
||||||
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
|
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
|
||||||
@@ -41,24 +41,20 @@ Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that point
|
|||||||
|
|
||||||
## Why split-horizon
|
## Why split-horizon
|
||||||
|
|
||||||
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
|
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
|
||||||
|
|
||||||
## The gotcha that cost a debug session (2026-04-22)
|
## The gotcha that cost a debug session (2026-04-22)
|
||||||
|
|
||||||
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
|
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
|
||||||
|
|
||||||
## Recipe — adding a new subdomain
|
## Recipe — adding a new subdomain
|
||||||
|
|
||||||
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
|
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
|
||||||
2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md):
|
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
|
||||||
```
|
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
|
||||||
address=/<new>.hubris.network/192.168.8.175
|
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
||||||
```
|
|
||||||
3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path.
|
|
||||||
4. Verify: `dig @192.168.8.180 +short <new>.hubris.network` → `192.168.8.175`.
|
|
||||||
5. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
|
|
||||||
|
|
||||||
> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124).
|
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
|
||||||
|
|
||||||
## Public path — what does and doesn't follow the LAN map
|
## Public path — what does and doesn't follow the LAN map
|
||||||
|
|
||||||
@@ -76,21 +72,17 @@ Either:
|
|||||||
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
|
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
|
||||||
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
|
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
|
||||||
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
|
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
|
||||||
- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq
|
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
|
||||||
|
|
||||||
## Changelog
|
## Changelog
|
||||||
|
|
||||||
### 2026-06-04 — Caddy: proxy auth.hubris.network → VPS; fix stale /etc/hosts across fleet
|
### 2026-06-04 — LXC 124 destroyed; Caddy proxy fixes; inventory/docs cleanup
|
||||||
Authentik had migrated to the VPS on 2026-05-31, but:
|
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
|
||||||
- Caddy's `auth.hubris.network` block still proxied to dead `192.168.8.6:9000`
|
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
|
||||||
- Caddy's `authentik` forward-auth snippet and `sso.hubris.network` block also referenced the stale IP
|
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
|
||||||
- Gitea's `/etc/hosts` had both the VPS IP and the old Caddy IP, causing Gitea to fail on OAuth2 token exchange with `404` then `issuer mismatch`
|
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
|
||||||
|
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
|
||||||
**Fixed:**
|
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
|
||||||
- Caddy Caddyfile: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxy to VPS (`82.165.190.79:443`) with proper Host header stripping
|
|
||||||
- Gitea (LXC 104), Jellyfin (101), Paperless (103), Apps (105), Nextcloud (114), ElementSynapse (118), Mule-Images (120), ArrStack (122), Plato (126): /etc/hosts → `192.168.8.175` only (Caddy proxy)
|
|
||||||
- Caddy LXC (121) /etc/hosts: self-reference removed, points to VPS
|
|
||||||
- All stale `192.168.8.6` references eliminated from Caddyfile
|
|
||||||
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
|
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
|
||||||
|
|
||||||
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
|
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ Public client
|
|||||||
[Home backend on 192.168.8.x]
|
[Home backend on 192.168.8.x]
|
||||||
```
|
```
|
||||||
|
|
||||||
Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
|
LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
|
||||||
|
|
||||||
## Why this shape
|
## Why this shape
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user