oikos/infrastructure/references/cert-sync-and-traefik-config.md

Current cert sync script + traefik dynamic config

Snapshot of the two artifacts that control public service exposure as of 2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP (192.168.8.206) to new strong IP (192.168.8.246).

hubris-public-cert-sync.sh (PVE host, /usr/local/bin/)

#!/bin/bash
# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames
# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume
# and hot-reloads.
#
# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01
# challenges before traefik's allowACMEByPass can respond, so traefik can't
# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict);
# we just mirror what it already has.
#
# Runs daily via hubris-public-cert-sync.timer.

set -euo pipefail

CADDY_LXC=121
CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
VPS_HOST=root@100.122.165.149
VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data

# Map: source hostname -> "crt_filename key_filename" on the VPS.
# Stable names so traefik dynamic.yaml doesn't need edits on renewal.
declare -A HOSTS=(
  [artifacto.hubris.network]="fullchain.crt privkey.key"
  [blog.hubris.network]="blog.fullchain.crt blog.privkey.key"
  [trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key"
  [sso.hubris.network]="sso.fullchain.crt sso.privkey.key"
  [media.hubris.network]="media.fullchain.crt media.privkey.key"
  [paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key"
)

TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT

for host in "${!HOSTS[@]}"; do
  read -r crt_name key_name <<< "${HOSTS[$host]}"
  pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name"
  pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name"

  if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then
    echo "hubris-public-cert-sync: $host unchanged"
    continue
  fi

  scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/"
  echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))"
done

Adding a new host

1. Caddy must already have the cert (verify pct exec 121 -- ls "$CADDY_BASE/$host/")
2. Add a line to the HOSTS array: [new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"
3. systemctl start hubris-public-cert-sync.service to sync immediately
4. Verify certs landed: ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"
5. Add matching tls.certificates entry in traefik dynamic config

---

traefik-dynamic.yaml (VPS, /opt/)

tcp:
  serversTransports:
    pp-v2:
      proxyProtocol:
        version: 2

tls:
  certificates:
    - certFile: /letsencrypt/fullchain.crt
      keyFile: /letsencrypt/privkey.key
    - certFile: /letsencrypt/blog.fullchain.crt
      keyFile: /letsencrypt/blog.privkey.key
    - certFile: /letsencrypt/trmnl.fullchain.crt
      keyFile: /letsencrypt/trmnl.privkey.key
    - certFile: /letsencrypt/sso.fullchain.crt
      keyFile: /letsencrypt/sso.privkey.key
    - certFile: /letsencrypt/media.fullchain.crt
      keyFile: /letsencrypt/media.privkey.key
    - certFile: /letsencrypt/paperless.fullchain.crt
      keyFile: /letsencrypt/paperless.privkey.key

http:
  routers:
    artifacto-public:
      rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - artifacto-strip-sso
        - artifacto-ratelimit
      service: artifacto-public

    blog-public:
      rule: 'Host(`blog.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - blog-ratelimit
      service: blog-public

    trmnl-public:
      rule: 'Host(`trmnl.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - trmnl-ratelimit
      service: trmnl-public

    matrix-public:
      rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - matrix-ratelimit
      service: matrix-public

    matrix-wellknown:
      rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))'
      entryPoints:
        - websecure
      priority: 20
      tls:
        certResolver: letsencrypt
      service: matrix-wellknown-svc

    house-public:
      rule: Host(`house.hubris.network`)
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - house-ratelimit
      service: house-public

    sso-public:
      rule: 'Host(`sso.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - sso-ratelimit
      service: sso-public

    media-public:
      rule: 'Host(`media.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - media-ratelimit
      service: media-public

    paperless-api-public:
      rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)'
      entryPoints:
        - websecure
      priority: 20
      tls:
        certResolver: letsencrypt
      middlewares:
        - paperless-ratelimit
      service: paperless-public

    paperless-public:
      rule: 'Host(`paperless.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - authentik-forwardauth
        - paperless-ratelimit
      service: paperless-public

  middlewares:
    artifacto-strip-sso:
      headers:
        customRequestHeaders:
          X-Artifacto-Gateway: ""
          X-Authentik-Username: ""
          X-Authentik-Groups: ""
          X-Authentik-Email: ""
          X-Authentik-Name: ""
          X-Authentik-Uid: ""
          X-Authentik-Jwt: ""
          X-Authentik-Meta-Jwks: ""
          X-Authentik-Meta-Outpost: ""
          X-Authentik-Meta-Provider: ""
          X-Authentik-Meta-App: ""
          X-Authentik-Meta-Version: ""

    admin-allowlist:
      ipAllowList:
        sourceRange:
          - "5.61.168.0/24"

    artifacto-ratelimit:
      rateLimit:
        average: 50
        period: 1s
        burst: 100
    blog-ratelimit:
      rateLimit:
        average: 100
        period: 1s
        burst: 200
    trmnl-ratelimit:
      rateLimit:
        average: 20
        period: 1s
        burst: 40
    matrix-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    house-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    sso-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    media-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    paperless-ratelimit:
      rateLimit:
        average: 20
        period: 1s
        burst: 40
    authentik-forwardauth:
      forwardAuth:
        address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik"
        trustForwardHeader: true
        authResponseHeaders:
          - X-authentik-username
          - X-authentik-groups
          - X-authentik-email
          - X-authentik-name
          - X-authentik-uid
          - X-authentik-jwt
          - X-authentik-meta-jwks
          - X-authentik-meta-outpost
          - X-authentik-meta-provider
          - X-authentik-meta-app
          - X-authentik-meta-version

  services:
    artifacto-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.205:3100'
    blog-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.205:8080'
    trmnl-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.211:9851'
    matrix-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.242:8008'
    matrix-wellknown-svc:
      loadBalancer:
        servers:
          - url: 'http://matrix-wellknown:80'
    house-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.244:3000'
    sso-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.6:9000'
    media-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.246:8096'
    paperless-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.130:8000'

Adding a new service — four blocks needed

1. Router — http.routers.<name>-public with tls: {} (not certResolver)
2. Middleware — rate limit, one per service
3. Service — http.services.<name>-public with the backend URL
4. tls.certificates — add a new - certFile/keryFile pair matching the cert sync HOSTS entry

Key file naming convention

Cert name	Host	Convention
fullchain.crt + privkey.key	artifacto.hubris.network	First service — no prefix
blog.fullchain.crt + blog.privkey.key	blog.hubris.network	{nickname}.fullchain.crt
trmnl.fullchain.crt + trmnl.privkey.key	trmnl.hubris.network	{nickname}.fullchain.crt
sso.fullchain.crt + sso.privkey.key	sso.hubris.network	{nickname}.fullchain.crt
media.fullchain.crt + media.privkey.key	media.hubris.network	{nickname}.fullchain.crt
paperless.fullchain.crt + paperless.privkey.key	paperless.hubris.network	{nickname}.fullchain.crt

⚠️ Critical — keep backends in sync after LXC migrations

When moving an LXC between Proxmox nodes, update both:

1. Caddy (/etc/caddy/Caddyfile on LXC 121)
2. VPS traefik (/opt/traefik-dynamic.yaml — via hubris bridge SSH)

Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy but missed in VPS traefik — old IP 192.168.8.206 remained. This caused Bad Gateway for off-LAN users. Use Python-based editing (see references/traefik-config-editing.md) for accurate surgical fixes.