dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
Files
3a35289f4629180984395c7d3d413384c5164fed
oikos/knowledge/wiki/containers/121-caddy.md
dtoro 4efddb8bed docs: fix pre-existing broken links surfaced by docs-lint 
Problem: docs-lint (added in the wiki-hq reorg) surfaced 126 broken relative
links that predated this session — a container rename, incident/plan docs
that moved into archive/done subfolders without their inbound links being
updated, and a handful of relative-depth bugs in files nested under
containers/archive/ and plans/done/.

Fixes applied, by category:
- 124-authentik.md -> 106-auth-outpost.md (container was renamed; ~40 refs).
- investigations/{2026-04-21-hubris-crash-loop,2026-05-31-authentik-vps-migration}.md
  -> archive/ prefix (both moved to investigations/archive/ previously).
- plans/{2026-06-01-slate-ax-to-sodola-migration,2026-06-04_130000-deprecate-claudio-bot,
  2026-06-25-yuvomi-deployment}.md -> plans/done/ prefix.
- Depth bugs in files nested one level deeper than their siblings assumed
  (investigations/archive/*, knowledge/wiki/containers/archive/*,
  plans/done/*) — corrected relative-path depth.
- Destroyed containers with no surviving page (126-plato) delinked to the
  containers/index.md archaeology row instead of a 404.
- ludo-mini.yaml -> strong.yaml (host was renamed, same physical machine).
- netbird-vps.md (no narrative page exists) -> netbird-vps.yaml (substrate
  record, matching the existing convention for hosts without a wiki page).
- runbook-dpkg-interrupted.md refs -> .agents/skills/runbook-dpkg-interrupted/SKILL.md
  (missed in the phase-4 runbook move because the referencing files used a
  bare filename, not a runbooks/ prefix).
- One dangling forward-reference to a never-written investigation delinked
  to the actual incident record it was describing.

Left alone: two links in knowledge/wiki/containers/101-jellyfin.md into
devops/homelab-authentik-admin/ — an intentional reference to a sibling repo,
not present in this checkout.

Verification: broken-link count 126 -> 2 (real remainder is the cross-repo
reference above); gen-topology.py --check still exit 0; build_host_files.py
still idempotent; all inventory.yaml doc_page targets still resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 17:53:35 +02:00

7.2 KiB
Raw Blame History

121 — caddy

The reverse proxy. Terminates TLS for every *.hubris.network hostname on the LAN/mesh and forwards to the right backend.

At a glance

  • Hostname: caddy
  • IP: 192.168.8.175
  • Privilege: unprivileged
  • Resources: 1 core / 512 MiB RAM / 6 GiB rootfs
  • Mounts: none from /mnt/library
  • Config: /etc/caddy/Caddyfile is a git checkout of dtoro/caddy-conf
  • Cert source: Let's Encrypt DNS-01 via IONOS API (IONOS_AUTH_API_TOKEN).

Sites currently served (live as of 2026-07-06)

Reminder: Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in DNS split-horizon too.

Snippet: (authentik) forward-auth

A snippet at the top of the Caddyfile (used as import authentik in any site block) wires forward-auth to the embedded Authentik outpost. It points at http://192.168.8.180:9000 directly (NOT https://auth.hubris.network) to avoid Caddy-to-self round-tripping that strips X-Forwarded-Host. The forward-auth block must explicitly set header_up X-Forwarded-Host {host}. See Authentik.

For apps with mobile clients, /api/* (or equivalent) bypasses forward-auth — see the per-app gotchas in Authentik.

Caddy environment

Untracked secrets in /etc/caddy/caddy.env:

  • IONOS_AUTH_API_TOKEN — DNS-01 ACME
  • ARTIFACTO_GATEWAY_SECRET — must match SSO_GATEWAY_SECRET on apps (105). Rotate together.

Auto-deploy

Push to dtoro/caddy-conf master → gitea webhook → http://192.168.8.175:9797/deploy → caddy-deploy-webhook.service validates HMAC against /etc/caddy-deploy/secret → runs /etc/caddy/scripts/deploy.sh:

  1. git pull
  2. caddy validate
  3. systemctl reload caddy

Gitea webhook id 2 on dtoro/caddy-conf. Receiver, deploy script, install scripts, systemd unit all live in the repo under scripts/. The unit has ReadWritePaths=/etc/caddy — load-bearing (ProtectSystem=full would otherwise block git pull). Git credentials for the root-run service are at /etc/caddy-deploy/git-credentials (mode 600).

Systemd unit changes inside the repo do NOT auto-reinstall — you have to re-run scripts/webhook/install.sh manually after editing the unit.

Related

Changelog

2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082

sab.hubris.network now uses import authentik inside a handle block. SABnzbd own auth disabled, local_ranges set for transparent proxy. Port bumped from :8081 to :8082 (fix from 2026-06-04) now documented.

2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added

Symptom: All *.hubris.network hosts except photos and auth (VPS-hosted) returned tlsv1 alert internal error or timeout. Only 3 site blocks (photos, prism, photos2) remained in the Caddyfile.

Root cause: The Caddyfile was manually edited directly on LXC 121 (not via the dtoro/caddy-conf git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.

Fix:

  • Restored Caddyfile from origin/master (git checkout --force origin/master -- Caddyfile)
  • systemctl reload caddy

Permanent safeguards added to /etc/caddy/scripts/deploy.sh:

  • Site-count guard: refuses to reload if fewer than 20 *.hubris.network blocks detected
  • Dirty-tree auto-stash: stashes local changes before git pull --ff-only so the webhook doesn't fail on local edits
  • Auto-backup: saves Caddyfile.bak.<timestamp> before any modifications, keeps last 5

Also: elementsynapse LXC 118 found to have DHCP-overridden static IP (actual .244 vs config .239) during incident investigation — fixed.

2026-06-02 — caddy.service unit missing; recreated

After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (caddy_1:2.11.3-hubris1_amd64) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at /lib/systemd/system/caddy.service with standard Caddy service config + EnvironmentFile=/etc/caddy/caddy.env (already present in caddy.service.d/override.conf). Risk: the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the caddy-conf repo or rebuild the hubris1 package to include it.

2026-04-28 — wiki entry created

Initial documentation. 16 active sites at this date.

2026-04-22 — Artifacto + Authentik gateway-secret pattern wired

Caddy site artifacto.hubris.network: forward-auth on admin paths, header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}, public matchers /p/*, /static/*, /healthz. caddy.env extended with the gateway secret.

2026-04-21 — (authentik) snippet added

Top-of-file snippet used by every gated site. Targets http://192.168.8.180:9000 directly to dodge hairpin TLS round-trip.

2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)

For the arriman (122) docker stack. Caddy + certs work; needed corresponding DNS split-horizon entries to be added to land cleanly.

2026-04-20 — auto-deploy pipeline shipped

Repo dtoro/caddy-conf. Webhook receiver at :9797, validate + reload on push to master.

Reference in New Issue View Git Blame Copy Permalink