oikos/containers/121-caddy.md

121 — caddy

The reverse proxy. Terminates TLS for every *.hubris.network hostname on the LAN/mesh and forwards to the right backend.

At a glance

• Hostname: caddy
• IP: 192.168.8.175
• Privilege: unprivileged
• Resources: 1 core / 512 MiB RAM / 6 GiB rootfs
• Mounts: none from /mnt/library
• Config: /etc/caddy/Caddyfile is a git checkout of dtoro/caddy-conf
• Cert source: Let's Encrypt DNS-01 via IONOS API (IONOS_AUTH_API_TOKEN).

Sites currently served (live as of 2026-04-28)

• artifacto.hubris.network → apps (105) :3100
• auth.hubris.network → authentik (124) :9000
• blog.hubris.network → apps (105) :8080
• books.hubris.network → apps (105) :6060
• cloud.hubris.network → nextcloud (114) :443
• docker.hubris.network → apps (105) :9443
• git.hubris.network → gitea (104) :3000 (+ handle_path /_plantuml/* → apps :8079)
• home.hubris.network → haos VM (108) 192.168.8.101:8123
• jellyseerr.hubris.network → arriman (122) :5056
• matrix.hubris.network → elementsynapse (118) :8008
• media.hubris.network → jellyfin (101) :8096
• paperless.hubris.network → paperless (103) :8000
• photos.hubris.network → mule-images (120) :3000
• proxmox.hubris.network → hubris host :8006
• qbit.hubris.network → arriman (122) :8080
• sab.hubris.network → arriman (122) :8082 (Authentik forward-auth)

Reminder: Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in DNS split-horizon too.

Snippet: (authentik) forward-auth

A snippet at the top of the Caddyfile (used as import authentik in any site block) wires forward-auth to the embedded Authentik outpost. It points at http://192.168.8.180:9000 directly (NOT https://auth.hubris.network) to avoid Caddy-to-self round-tripping that strips X-Forwarded-Host. The forward-auth block must explicitly set header_up X-Forwarded-Host {host}. See Authentik.

For apps with mobile clients, /api/* (or equivalent) bypasses forward-auth — see the per-app gotchas in Authentik.

Caddy environment

Untracked secrets in /etc/caddy/caddy.env:

• IONOS_AUTH_API_TOKEN — DNS-01 ACME
• ARTIFACTO_GATEWAY_SECRET — must match SSO_GATEWAY_SECRET on apps (105). Rotate together.

Auto-deploy

Push to dtoro/caddy-conf master → gitea webhook → http://192.168.8.175:9797/deploy → caddy-deploy-webhook.service validates HMAC against /etc/caddy-deploy/secret → runs /etc/caddy/scripts/deploy.sh:

1. git pull
2. caddy validate
3. systemctl reload caddy

Gitea webhook id 2 on dtoro/caddy-conf. Receiver, deploy script, install scripts, systemd unit all live in the repo under scripts/. The unit has ReadWritePaths=/etc/caddy — load-bearing (ProtectSystem=full would otherwise block git pull). Git credentials for the root-run service are at /etc/caddy-deploy/git-credentials (mode 600).

Systemd unit changes inside the repo do NOT auto-reinstall — you have to re-run scripts/webhook/install.sh manually after editing the unit.

Related

• DNS split-horizon — must add entry for every new subdomain
• Authentik (124) — forward-auth + IdP
• Auto-deploy
• Public ingress (VPS traefik) — mirrors Caddy's certs to the VPS for public exposure
• Gitea (104) — webhook source
• All container pages list their Caddy backend.

Changelog

2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082

sab.hubris.network now uses import authentik inside a handle block. SABnzbd own auth disabled, local_ranges set for transparent proxy. Port bumped from :8081 to :8082 (fix from 2026-06-04) now documented.

2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added

Symptom: All *.hubris.network hosts except photos and auth (VPS-hosted) returned tlsv1 alert internal error or timeout. Only 3 site blocks (photos, prism, photos2) remained in the Caddyfile.

Root cause: The Caddyfile was manually edited directly on LXC 121 (not via the dtoro/caddy-conf git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.

Fix:

• Restored Caddyfile from origin/master (git checkout --force origin/master -- Caddyfile)
• systemctl reload caddy

Permanent safeguards added to /etc/caddy/scripts/deploy.sh:

• Site-count guard: refuses to reload if fewer than 20 *.hubris.network blocks detected
• Dirty-tree auto-stash: stashes local changes before git pull --ff-only so the webhook doesn't fail on local edits
• Auto-backup: saves Caddyfile.bak.<timestamp> before any modifications, keeps last 5

Also: elementsynapse LXC 118 found to have DHCP-overridden static IP (actual .244 vs config .239) during incident investigation — fixed.

2026-06-02 — caddy.service unit missing; recreated

After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (caddy_1:2.11.3-hubris1_amd64) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at /lib/systemd/system/caddy.service with standard Caddy service config + EnvironmentFile=/etc/caddy/caddy.env (already present in caddy.service.d/override.conf). Risk: the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the caddy-conf repo or rebuild the hubris1 package to include it.

2026-04-28 — wiki entry created

Initial documentation. 16 active sites at this date.

2026-04-22 — Artifacto + Authentik gateway-secret pattern wired

Caddy site artifacto.hubris.network: forward-auth on admin paths, header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}, public matchers /p/*, /static/*, /healthz. caddy.env extended with the gateway secret.

2026-04-21 — (authentik) snippet added

Top-of-file snippet used by every gated site. Targets http://192.168.8.180:9000 directly to dodge hairpin TLS round-trip.

2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)

For the arriman (122) docker stack. Caddy + certs work; needed corresponding DNS split-horizon entries to be added to land cleanly.

2026-04-20 — auto-deploy pipeline shipped

Repo dtoro/caddy-conf. Webhook receiver at :9797, validate + reload on push to master.