diff --git a/infrastructure/references/cert-sync-and-traefik-config.md b/infrastructure/references/cert-sync-and-traefik-config.md new file mode 100644 index 0000000..4a63457 --- /dev/null +++ b/infrastructure/references/cert-sync-and-traefik-config.md @@ -0,0 +1,351 @@ +# Current cert sync script + traefik dynamic config + +Snapshot of the two artifacts that control public service exposure as of +2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP +(192.168.8.206) to new strong IP (192.168.8.246). + +## hubris-public-cert-sync.sh (PVE host, `/usr/local/bin/`) + +```bash +#!/bin/bash +# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames +# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume +# and hot-reloads. +# +# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01 +# challenges before traefik's allowACMEByPass can respond, so traefik can't +# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict); +# we just mirror what it already has. +# +# Runs daily via hubris-public-cert-sync.timer. + +set -euo pipefail + +CADDY_LXC=121 +CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory +VPS_HOST=root@100.122.165.149 +VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data + +# Map: source hostname -> "crt_filename key_filename" on the VPS. +# Stable names so traefik dynamic.yaml doesn't need edits on renewal. +declare -A HOSTS=( + [artifacto.hubris.network]="fullchain.crt privkey.key" + [blog.hubris.network]="blog.fullchain.crt blog.privkey.key" + [trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key" + [sso.hubris.network]="sso.fullchain.crt sso.privkey.key" + [media.hubris.network]="media.fullchain.crt media.privkey.key" + [paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key" +) + +TMP=$(mktemp -d) +trap 'rm -rf "$TMP"' EXIT + +for host in "${!HOSTS[@]}"; do + read -r crt_name key_name <<< "${HOSTS[$host]}" + pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name" + pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name" + + if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then + echo "hubris-public-cert-sync: $host unchanged" + continue + fi + + scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/" + echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))" +done +``` + +### Adding a new host + +1. Caddy must already have the cert (verify `pct exec 121 -- ls "$CADDY_BASE/$host/"`) +2. Add a line to the HOSTS array: `[new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"` +3. `systemctl start hubris-public-cert-sync.service` to sync immediately +4. Verify certs landed: `ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"` +5. Add matching `tls.certificates` entry in traefik dynamic config + +--- + +## traefik-dynamic.yaml (VPS, `/opt/`) + +```yaml +tcp: + serversTransports: + pp-v2: + proxyProtocol: + version: 2 + +tls: + certificates: + - certFile: /letsencrypt/fullchain.crt + keyFile: /letsencrypt/privkey.key + - certFile: /letsencrypt/blog.fullchain.crt + keyFile: /letsencrypt/blog.privkey.key + - certFile: /letsencrypt/trmnl.fullchain.crt + keyFile: /letsencrypt/trmnl.privkey.key + - certFile: /letsencrypt/sso.fullchain.crt + keyFile: /letsencrypt/sso.privkey.key + - certFile: /letsencrypt/media.fullchain.crt + keyFile: /letsencrypt/media.privkey.key + - certFile: /letsencrypt/paperless.fullchain.crt + keyFile: /letsencrypt/paperless.privkey.key + +http: + routers: + artifacto-public: + rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - artifacto-strip-sso + - artifacto-ratelimit + service: artifacto-public + + blog-public: + rule: 'Host(`blog.hubris.network`)' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - blog-ratelimit + service: blog-public + + trmnl-public: + rule: 'Host(`trmnl.hubris.network`)' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - trmnl-ratelimit + service: trmnl-public + + matrix-public: + rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - matrix-ratelimit + service: matrix-public + + matrix-wellknown: + rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))' + entryPoints: + - websecure + priority: 20 + tls: + certResolver: letsencrypt + service: matrix-wellknown-svc + + house-public: + rule: Host(`house.hubris.network`) + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - house-ratelimit + service: house-public + + sso-public: + rule: 'Host(`sso.hubris.network`)' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - sso-ratelimit + service: sso-public + + media-public: + rule: 'Host(`media.hubris.network`)' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - media-ratelimit + service: media-public + + paperless-api-public: + rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)' + entryPoints: + - websecure + priority: 20 + tls: + certResolver: letsencrypt + middlewares: + - paperless-ratelimit + service: paperless-public + + paperless-public: + rule: 'Host(`paperless.hubris.network`)' + entryPoints: + - websecure + priority: 10 + tls: + certResolver: letsencrypt + middlewares: + - authentik-forwardauth + - paperless-ratelimit + service: paperless-public + + middlewares: + artifacto-strip-sso: + headers: + customRequestHeaders: + X-Artifacto-Gateway: "" + X-Authentik-Username: "" + X-Authentik-Groups: "" + X-Authentik-Email: "" + X-Authentik-Name: "" + X-Authentik-Uid: "" + X-Authentik-Jwt: "" + X-Authentik-Meta-Jwks: "" + X-Authentik-Meta-Outpost: "" + X-Authentik-Meta-Provider: "" + X-Authentik-Meta-App: "" + X-Authentik-Meta-Version: "" + + admin-allowlist: + ipAllowList: + sourceRange: + - "5.61.168.0/24" + + artifacto-ratelimit: + rateLimit: + average: 50 + period: 1s + burst: 100 + blog-ratelimit: + rateLimit: + average: 100 + period: 1s + burst: 200 + trmnl-ratelimit: + rateLimit: + average: 20 + period: 1s + burst: 40 + matrix-ratelimit: + rateLimit: + average: 30 + period: 1s + burst: 60 + house-ratelimit: + rateLimit: + average: 30 + period: 1s + burst: 60 + sso-ratelimit: + rateLimit: + average: 30 + period: 1s + burst: 60 + media-ratelimit: + rateLimit: + average: 30 + period: 1s + burst: 60 + paperless-ratelimit: + rateLimit: + average: 20 + period: 1s + burst: 40 + authentik-forwardauth: + forwardAuth: + address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik" + trustForwardHeader: true + authResponseHeaders: + - X-authentik-username + - X-authentik-groups + - X-authentik-email + - X-authentik-name + - X-authentik-uid + - X-authentik-jwt + - X-authentik-meta-jwks + - X-authentik-meta-outpost + - X-authentik-meta-provider + - X-authentik-meta-app + - X-authentik-meta-version + + services: + artifacto-public: + loadBalancer: + servers: + - url: 'http://192.168.8.205:3100' + blog-public: + loadBalancer: + servers: + - url: 'http://192.168.8.205:8080' + trmnl-public: + loadBalancer: + servers: + - url: 'http://192.168.8.211:9851' + matrix-public: + loadBalancer: + servers: + - url: 'http://192.168.8.242:8008' + matrix-wellknown-svc: + loadBalancer: + servers: + - url: 'http://matrix-wellknown:80' + house-public: + loadBalancer: + servers: + - url: 'http://192.168.8.244:3000' + sso-public: + loadBalancer: + servers: + - url: 'http://192.168.8.6:9000' + media-public: + loadBalancer: + servers: + - url: 'http://192.168.8.246:8096' + paperless-public: + loadBalancer: + servers: + - url: 'http://192.168.8.130:8000' +``` + +### Adding a new service — four blocks needed + +1. **Router** — `http.routers.-public` with `tls: {}` (not + `certResolver`) +2. **Middleware** — rate limit, one per service +3. **Service** — `http.services.-public` with the backend URL +4. **tls.certificates** — add a new `- certFile/keryFile` pair matching the + cert sync HOSTS entry + +### Key file naming convention + +| Cert name | Host | Convention | +|-----------|------|------------| +| `fullchain.crt` + `privkey.key` | `artifacto.hubris.network` | First service — no prefix | +| `blog.fullchain.crt` + `blog.privkey.key` | `blog.hubris.network` | `{nickname}.fullchain.crt` | +| `trmnl.fullchain.crt` + `trmnl.privkey.key` | `trmnl.hubris.network` | `{nickname}.fullchain.crt` | +| `sso.fullchain.crt` + `sso.privkey.key` | `sso.hubris.network` | `{nickname}.fullchain.crt` | +| `media.fullchain.crt` + `media.privkey.key` | `media.hubris.network` | `{nickname}.fullchain.crt` | +| `paperless.fullchain.crt` + `paperless.privkey.key` | `paperless.hubris.network` | `{nickname}.fullchain.crt` | + +### ⚠️ Critical — keep backends in sync after LXC migrations + +When moving an LXC between Proxmox nodes, update **both**: +1. **Caddy** (`/etc/caddy/Caddyfile` on LXC 121) +2. **VPS traefik** (`/opt/traefik-dynamic.yaml` — via hubris bridge SSH) + +Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy +but **missed** in VPS traefik — old IP `192.168.8.206` remained. This caused +Bad Gateway for off-LAN users. Use Python-based editing (see +`references/traefik-config-editing.md`) for accurate surgical fixes. \ No newline at end of file