oikos/archive/knowledge/references/cert-sync-and-traefik-config.md

# Current cert sync script + traefik dynamic config

Snapshot of the two artifacts that control public service exposure as of
2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP
(192.168.8.206) to new strong IP (192.168.8.246).

## hubris-public-cert-sync.sh (PVE host, `/usr/local/bin/`)

```bash
#!/bin/bash
# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames
# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume
# and hot-reloads.
#
# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01
# challenges before traefik's allowACMEByPass can respond, so traefik can't
# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict);
# we just mirror what it already has.
#
# Runs daily via hubris-public-cert-sync.timer.

set -euo pipefail

CADDY_LXC=121
CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
VPS_HOST=root@100.122.165.149
VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data

# Map: source hostname -> "crt_filename key_filename" on the VPS.
# Stable names so traefik dynamic.yaml doesn't need edits on renewal.
declare -A HOSTS=(
  [artifacto.hubris.network]="fullchain.crt privkey.key"
  [blog.hubris.network]="blog.fullchain.crt blog.privkey.key"
  [trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key"
  [sso.hubris.network]="sso.fullchain.crt sso.privkey.key"
  [media.hubris.network]="media.fullchain.crt media.privkey.key"
  [paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key"
)

TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT

for host in "${!HOSTS[@]}"; do
  read -r crt_name key_name <<< "${HOSTS[$host]}"
  pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name"
  pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name"

  if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then
    echo "hubris-public-cert-sync: $host unchanged"
    continue
  fi

  scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/"
  echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))"
done
```

### Adding a new host

1. Caddy must already have the cert (verify `pct exec 121 -- ls "$CADDY_BASE/$host/"`)
2. Add a line to the HOSTS array: `[new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"`
3. `systemctl start hubris-public-cert-sync.service` to sync immediately
4. Verify certs landed: `ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"`
5. Add matching `tls.certificates` entry in traefik dynamic config

---

## traefik-dynamic.yaml (VPS, `/opt/`)

```yaml
tcp:
  serversTransports:
    pp-v2:
      proxyProtocol:
        version: 2

tls:
  certificates:
    - certFile: /letsencrypt/fullchain.crt
      keyFile: /letsencrypt/privkey.key
    - certFile: /letsencrypt/blog.fullchain.crt
      keyFile: /letsencrypt/blog.privkey.key
    - certFile: /letsencrypt/trmnl.fullchain.crt
      keyFile: /letsencrypt/trmnl.privkey.key
    - certFile: /letsencrypt/sso.fullchain.crt
      keyFile: /letsencrypt/sso.privkey.key
    - certFile: /letsencrypt/media.fullchain.crt
      keyFile: /letsencrypt/media.privkey.key
    - certFile: /letsencrypt/paperless.fullchain.crt
      keyFile: /letsencrypt/paperless.privkey.key

http:
  routers:
    artifacto-public:
      rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - artifacto-strip-sso
        - artifacto-ratelimit
      service: artifacto-public

    blog-public:
      rule: 'Host(`blog.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - blog-ratelimit
      service: blog-public

    trmnl-public:
      rule: 'Host(`trmnl.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - trmnl-ratelimit
      service: trmnl-public

    matrix-public:
      rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - matrix-ratelimit
      service: matrix-public

    matrix-wellknown:
      rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))'
      entryPoints:
        - websecure
      priority: 20
      tls:
        certResolver: letsencrypt
      service: matrix-wellknown-svc

    house-public:
      rule: Host(`house.hubris.network`)
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - house-ratelimit
      service: house-public

    sso-public:
      rule: 'Host(`sso.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - sso-ratelimit
      service: sso-public

    media-public:
      rule: 'Host(`media.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - media-ratelimit
      service: media-public

    paperless-api-public:
      rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)'
      entryPoints:
        - websecure
      priority: 20
      tls:
        certResolver: letsencrypt
      middlewares:
        - paperless-ratelimit
      service: paperless-public

    paperless-public:
      rule: 'Host(`paperless.hubris.network`)'
      entryPoints:
        - websecure
      priority: 10
      tls:
        certResolver: letsencrypt
      middlewares:
        - authentik-forwardauth
        - paperless-ratelimit
      service: paperless-public

  middlewares:
    artifacto-strip-sso:
      headers:
        customRequestHeaders:
          X-Artifacto-Gateway: ""
          X-Authentik-Username: ""
          X-Authentik-Groups: ""
          X-Authentik-Email: ""
          X-Authentik-Name: ""
          X-Authentik-Uid: ""
          X-Authentik-Jwt: ""
          X-Authentik-Meta-Jwks: ""
          X-Authentik-Meta-Outpost: ""
          X-Authentik-Meta-Provider: ""
          X-Authentik-Meta-App: ""
          X-Authentik-Meta-Version: ""

    admin-allowlist:
      ipAllowList:
        sourceRange:
          - "5.61.168.0/24"

    artifacto-ratelimit:
      rateLimit:
        average: 50
        period: 1s
        burst: 100
    blog-ratelimit:
      rateLimit:
        average: 100
        period: 1s
        burst: 200
    trmnl-ratelimit:
      rateLimit:
        average: 20
        period: 1s
        burst: 40
    matrix-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    house-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    sso-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    media-ratelimit:
      rateLimit:
        average: 30
        period: 1s
        burst: 60
    paperless-ratelimit:
      rateLimit:
        average: 20
        period: 1s
        burst: 40
    authentik-forwardauth:
      forwardAuth:
        address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik"
        trustForwardHeader: true
        authResponseHeaders:
          - X-authentik-username
          - X-authentik-groups
          - X-authentik-email
          - X-authentik-name
          - X-authentik-uid
          - X-authentik-jwt
          - X-authentik-meta-jwks
          - X-authentik-meta-outpost
          - X-authentik-meta-provider
          - X-authentik-meta-app
          - X-authentik-meta-version

  services:
    artifacto-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.205:3100'
    blog-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.205:8080'
    trmnl-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.211:9851'
    matrix-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.242:8008'
    matrix-wellknown-svc:
      loadBalancer:
        servers:
          - url: 'http://matrix-wellknown:80'
    house-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.244:3000'
    sso-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.6:9000'
    media-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.246:8096'
    paperless-public:
      loadBalancer:
        servers:
          - url: 'http://192.168.8.130:8000'
```

### Adding a new service — four blocks needed

1. **Router** — `http.routers.<name>-public` with `tls: {}` (not
   `certResolver`)
2. **Middleware** — rate limit, one per service
3. **Service** — `http.services.<name>-public` with the backend URL
4. **tls.certificates** — add a new `- certFile/keryFile` pair matching the
   cert sync HOSTS entry

### Key file naming convention

| Cert name | Host | Convention |
|-----------|------|------------|
| `fullchain.crt` + `privkey.key` | `artifacto.hubris.network` | First service — no prefix |
| `blog.fullchain.crt` + `blog.privkey.key` | `blog.hubris.network` | `{nickname}.fullchain.crt` |
| `trmnl.fullchain.crt` + `trmnl.privkey.key` | `trmnl.hubris.network` | `{nickname}.fullchain.crt` |
| `sso.fullchain.crt` + `sso.privkey.key` | `sso.hubris.network` | `{nickname}.fullchain.crt` |
| `media.fullchain.crt` + `media.privkey.key` | `media.hubris.network` | `{nickname}.fullchain.crt` |
| `paperless.fullchain.crt` + `paperless.privkey.key` | `paperless.hubris.network` | `{nickname}.fullchain.crt` |

### ⚠️ Critical — keep backends in sync after LXC migrations

When moving an LXC between Proxmox nodes, update **both**:
1. **Caddy** (`/etc/caddy/Caddyfile` on LXC 121)
2. **VPS traefik** (`/opt/traefik-dynamic.yaml` — via hubris bridge SSH)

Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy
but **missed** in VPS traefik — old IP `192.168.8.206` remained. This caused
Bad Gateway for off-LAN users. Use Python-based editing (see
`references/traefik-config-editing.md`) for accurate surgical fixes.