Files
oikos/infrastructure/ingress.md
dtoro 8dbba323c1 authentik: migrate from LXC 124 to VPS; eliminate mesh bootstrap deadlock
Moved Authentik (2026.2.2 -> 2026.5.2, +Redis, dedicated auth Docker net)
off LXC 124 onto the VPS so netbird-mgmt's OIDC dependency no longer requires
the mesh it authenticates. depends_on: service_healthy makes the deadlock
structurally impossible. Full Postgres DB migrated (users/apps/passwords/groups).

- investigations/2026-05-31-authentik-vps-migration.md: full writeup + lessons
- 124-authentik: migration banner + changelog (now legacy; dnsmasq stays)
- dns: auth.hubris.network -> 82.165.190.79; NetBird resolver cache gotcha
- ingress: auth served by local container via Docker-provider labels (not cert-mirror)
- mesh: follow-up entry superseding the morning band-aid; bootstrap note obsolete

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-05-31 23:17:00 +02:00

7.2 KiB

Public ingress — VPS traefik + cert mirror

How home services reach the open internet without exposing the home network. Two-stage pattern: traefik on the IONOS VPS terminates TLS at the public edge, then reverse-proxies over the netbird mesh to home Caddy / direct backends.

The shape

Public client
    │  *.hubris.network → 82.165.190.79 (IONOS wildcard)
    ▼
[VPS traefik]   priority 1: HostSNI(*) → netbird-proxy:8443  ← netbird control plane
                priority 10: per-host HTTP routers           ← home services
    │  HTTP over netbird mesh
    ▼
[Home backend on 192.168.8.x]

Mesh clients see dnsmasq on LXC 124192.168.8.175 → home Caddy (121), unchanged. The two paths are independent.

Why this shape

  • VPS traefik already has a HostSNI(*) TCP passthrough at priority 1 so netbird's own ingress (netbird.hubris.network, future *.proxy.hubris.network) is unaffected.
  • Per-hostname HTTP file-provider routers at priority 10 win over the passthrough for the listed hosts and let traefik terminate TLS itself for those.
  • Traefik's own ACME (letsencrypt resolver) fails on this box: HostSNI(*) grabs TLS-ALPN-01 challenges before traefik's allowACMEByPass=true can respond. Solution: home Caddy obtains certs via IONOS DNS-01 (no conflict) and the VPS mirrors the result over.

Components

On the VPS (82.165.190.79)

  • /opt/traefik-dynamic.yaml — file-watched dynamic config. One http.routers.<name>-public + one http.services.<name>-public per exposed service, plus one entry in the top-level tls.certificates list per hostname.
  • Let's-Encrypt-effective directory: /letsencrypt/ inside the traefik container, backed by the host-side docker volume opt_netbird_traefik_letsencrypt.
  • Backups before edits: cp /opt/traefik-dynamic.yaml /opt/traefik-dynamic.yaml.bak.$(date +%s) — several bak files live alongside.

On the PVE host (192.168.8.77)

  • /usr/local/bin/hubris-public-cert-sync.sh — runs daily via hubris-public-cert-sync.timer. Maps source hostname → VPS cert filenames in a bash assoc array. For each mapping: pct pull cert+key from Caddy (121)'s store, diff against the VPS copy, scp only on change.
  • Filenames are stable per host so the dynamic.yaml never needs editing on renewal — traefik file-watches and hot-reloads the cert.

Services currently exposed

Hostname Path scope Backend Middlewares Cert files on VPS
artifacto.hubris.network /p/*, /static/*, /healthz 192.168.8.205:3100 artifacto-strip-sso + artifacto-ratelimit (50 rps / 100 burst) fullchain.crt / privkey.key
blog.hubris.network whole host 192.168.8.205:8080 blog-ratelimit (100 rps / 200 burst) blog.fullchain.crt / blog.privkey.key

artifacto-strip-sso blanks inbound X-Authentik-* and X-Artifacto-Gateway so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See Artifacto on apps (105).

auth.hubris.network — different pattern (local container, not cert-mirror)

Since 2026-05-31 Authentik runs on the VPS itself, so auth.hubris.network is served by a local Docker container, not proxied to a home backend. It therefore does not use the file-provider + cert-mirror pattern above:

  • Routed via traefik Docker provider labels on the authentik-server service (/opt/docker-compose.yml), not traefik-dynamic.yaml.
  • TLS via traefik's own letsencrypt resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
  • Traefik reaches it over the auth Docker network (172.30.1.0/24); Postgres/Redis on that net are isolated from the netbird containers.
  • Admin UI is IP-gated: an admin-allowlist ipAllowList middleware on PathPrefix(/if/admin/) (currently 5.61.168.0/24). Login/flow endpoints stay public.

No cert-mirror entry and no hubris-public-cert-sync.sh mapping is needed for auth.

Recipe — exposing another service

  1. Ensure home Caddy on LXC 121 already serves the hostname (cert exists at /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/).
  2. Add an entry to HOSTS in /usr/local/bin/hubris-public-cert-sync.sh mapping the hostname → VPS filenames. Run once: systemctl start hubris-public-cert-sync.service. Confirm the cert landed.
  3. Edit /opt/traefik-dynamic.yaml on the VPS:
    • Add to tls.certificates: paths /letsencrypt/<crt> and /letsencrypt/<key>.
    • Add http.routers.<host>-public: rule: 'Host(\`)'(or with path matchers if scope-gating),entryPoints: [websecure], priority: 10, tls: {}, service: -public, middlewares: [...]`.
    • Add a ratelimit middleware under http.middlewares if wanted.
    • Add http.services.<host>-public.loadBalancer.servers[0].url: 'http://<lan-ip>:<port>'.
  4. Verify:
    ssh root@100.122.165.149 'curl -skI --resolve <host>:443:127.0.0.1 https://<host>/'   # 2xx/3xx
    curl -skI --resolve <host>:443:<vps-public-ip> https://<host>/                          # same
    
  5. No DNS edit needed — the IONOS wildcard already points at the VPS.

What does NOT follow this pattern

  • netbird.hubris.network (and any future *.proxy.hubris.network) uses the netbird-proxy / HostSNI passthrough path. Netbird handles its own cert via ACME cleanly because it is the passthrough target.

Changelog

2026-05-31 — auth.hubris.network now served locally on the VPS

Authentik migrated onto the VPS (investigation). Unlike the home-backed services above, auth is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no traefik-dynamic.yaml router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (auth, 172.30.1.0/24) to reach it while keeping its DB/Redis isolated from the netbird stack.

2026-04-28 — wiki entry created

Initial documentation.

2026-04-23 — blog.hubris.network exposed

WriteFreely on apps (105). Whole host is public.

2026-04-22 — pattern established with Artifacto

First service through the file-provider router. IONOS wildcard moved to the VPS this day. Cert mirror script + timer deployed on the PVE host.