Files

dtoro 8a6422bd7d docs: move narrative wiki under knowledge/wiki/ (phase 3)

Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-06 14:35:23 +02:00

7.2 KiB

Raw Blame History

121 — `caddy`

The reverse proxy. Terminates TLS for every *.hubris.network hostname on the LAN/mesh and forwards to the right backend.

At a glance

Hostname: caddy
IP: 192.168.8.175
Privilege: unprivileged
Resources: 1 core / 512 MiB RAM / 6 GiB rootfs
Mounts: none from /mnt/library
Config: /etc/caddy/Caddyfile is a git checkout of dtoro/caddy-conf
Cert source: Let's Encrypt DNS-01 via IONOS API (IONOS_AUTH_API_TOKEN).

Sites currently served (live as of 2026-07-06)

artifacto.hubris.network → apps (105) :3100
blog.hubris.network → apps (105) :8080
books.hubris.network → grimmory (130) :6060
cloud.hubris.network → nextcloud (114) :443
docker.hubris.network → apps (105) :9443
git.hubris.network → gitea (104) :3000 (+ handle_path /_plantuml/* → apps :8079)
home.hubris.network → haos VM (108) 192.168.8.101:8123
house.hubris.network → house (129) :3000
jellyseerr.hubris.network → arriman (122) :5056
matrix.hubris.network → elementsynapse (118) :8008
media.hubris.network → jellyfin (101) :8096
paperless.hubris.network → paperless (103) :8000
photos.hubris.network → mule-images (120) :3000
proxmox.hubris.network → hubris host :8006
qbit.hubris.network → arriman (122) :8080
roms.hubris.network → romm (134) :80
sab.hubris.network → arriman (122) :8082 (Authentik forward-auth)
teddy.hubris.network → LXC 131 192.168.8.150:8443
trmnl.hubris.network → trmnl (128) :9851

Reminder: Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in DNS split-horizon too.

Snippet: `(authentik)` forward-auth

A snippet at the top of the Caddyfile (used as import authentik in any site block) wires forward-auth to the embedded Authentik outpost. It points at http://192.168.8.180:9000 directly (NOT https://auth.hubris.network) to avoid Caddy-to-self round-tripping that strips X-Forwarded-Host. The forward-auth block must explicitly set header_up X-Forwarded-Host {host}. See Authentik.

For apps with mobile clients, /api/* (or equivalent) bypasses forward-auth — see the per-app gotchas in Authentik.

Caddy environment

Untracked secrets in /etc/caddy/caddy.env:

IONOS_AUTH_API_TOKEN — DNS-01 ACME
ARTIFACTO_GATEWAY_SECRET — must match SSO_GATEWAY_SECRET on apps (105). Rotate together.

Auto-deploy

Push to dtoro/caddy-conf master → gitea webhook → http://192.168.8.175:9797/deploy → caddy-deploy-webhook.service validates HMAC against /etc/caddy-deploy/secret → runs /etc/caddy/scripts/deploy.sh:

git pull
caddy validate
systemctl reload caddy

Gitea webhook id 2 on dtoro/caddy-conf. Receiver, deploy script, install scripts, systemd unit all live in the repo under scripts/. The unit has ReadWritePaths=/etc/caddy — load-bearing (ProtectSystem=full would otherwise block git pull). Git credentials for the root-run service are at /etc/caddy-deploy/git-credentials (mode 600).

Systemd unit changes inside the repo do NOT auto-reinstall — you have to re-run scripts/webhook/install.sh manually after editing the unit.

DNS split-horizon — must add entry for every new subdomain
Authentik (124) — forward-auth + IdP
Auto-deploy
Public ingress (VPS traefik) — mirrors Caddy's certs to the VPS for public exposure
Gitea (104) — webhook source
All container pages list their Caddy backend.

Changelog

2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082

sab.hubris.network now uses import authentik inside a handle block. SABnzbd own auth disabled, local_ranges set for transparent proxy. Port bumped from :8081 to :8082 (fix from 2026-06-04) now documented.

2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added

Symptom: All *.hubris.network hosts except photos and auth (VPS-hosted) returned tlsv1 alert internal error or timeout. Only 3 site blocks (photos, prism, photos2) remained in the Caddyfile.

Root cause: The Caddyfile was manually edited directly on LXC 121 (not via the dtoro/caddy-conf git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.

Fix:

Restored Caddyfile from origin/master (git checkout --force origin/master -- Caddyfile)
systemctl reload caddy

Permanent safeguards added to /etc/caddy/scripts/deploy.sh:

Site-count guard: refuses to reload if fewer than 20 *.hubris.network blocks detected
Dirty-tree auto-stash: stashes local changes before git pull --ff-only so the webhook doesn't fail on local edits
Auto-backup: saves Caddyfile.bak.<timestamp> before any modifications, keeps last 5

Also: elementsynapse LXC 118 found to have DHCP-overridden static IP (actual .244 vs config .239) during incident investigation — fixed.

2026-06-02 — caddy.service unit missing; recreated

After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (caddy_1:2.11.3-hubris1_amd64) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at /lib/systemd/system/caddy.service with standard Caddy service config + EnvironmentFile=/etc/caddy/caddy.env (already present in caddy.service.d/override.conf). Risk: the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the caddy-conf repo or rebuild the hubris1 package to include it.

2026-04-28 — wiki entry created

Initial documentation. 16 active sites at this date.

2026-04-22 — Artifacto + Authentik gateway-secret pattern wired

Caddy site artifacto.hubris.network: forward-auth on admin paths, header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}, public matchers /p/*, /static/*, /healthz. caddy.env extended with the gateway secret.

2026-04-21 — `(authentik)` snippet added

Top-of-file snippet used by every gated site. Targets http://192.168.8.180:9000 directly to dodge hairpin TLS round-trip.

2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)

For the arriman (122) docker stack. Caddy + certs work; needed corresponding DNS split-horizon entries to be added to land cleanly.

2026-04-20 — auto-deploy pipeline shipped

Repo dtoro/caddy-conf. Webhook receiver at :9797, validate + reload on push to master.

7.2 KiB Raw Blame History

121 — caddy