docs: move narrative wiki under knowledge/wiki/ (phase 3)
Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.
Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
outbound + between-moved-files), including .hermes/, runbooks, operations,
investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
embed doc_page); update oikos/gen-topology.py output path, candidate doc
paths, and footer links; update code-comment doc paths.
Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.
Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
topology regen fixed one, introduced none). Remaining are pre-existing refs
to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
106
knowledge/wiki/containers/121-caddy.md
Normal file
106
knowledge/wiki/containers/121-caddy.md
Normal file
@@ -0,0 +1,106 @@
|
||||
# 121 — `caddy`
|
||||
|
||||
The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the LAN/mesh and forwards to the right backend.
|
||||
|
||||
## At a glance
|
||||
- **Hostname:** `caddy`
|
||||
- **IP:** `192.168.8.175`
|
||||
- **Privilege:** **unprivileged**
|
||||
- **Resources:** 1 core / 512 MiB RAM / 6 GiB rootfs
|
||||
- **Mounts:** none from `/mnt/library`
|
||||
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
|
||||
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
|
||||
|
||||
## Sites currently served (live as of 2026-07-06)
|
||||
|
||||
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
|
||||
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
|
||||
- `books.hubris.network` → [grimmory (130)](130-grimmory.md) `:6060`
|
||||
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
|
||||
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
|
||||
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
|
||||
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
|
||||
- `house.hubris.network` → [house (129)](129-house.md) `:3000`
|
||||
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
|
||||
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
|
||||
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
|
||||
- `paperless.hubris.network` → [paperless (103)](103-paperless.md) `:8000`
|
||||
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
|
||||
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
|
||||
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
|
||||
- `roms.hubris.network` → [romm (134)](134-romm.md) `:80`
|
||||
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
|
||||
- `teddy.hubris.network` → LXC 131 `192.168.8.150:8443`
|
||||
- `trmnl.hubris.network` → [trmnl (128)](128-trmnl.md) `:9851`
|
||||
|
||||
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
|
||||
|
||||
## Snippet: `(authentik)` forward-auth
|
||||
|
||||
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup).
|
||||
|
||||
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md).
|
||||
|
||||
## Caddy environment
|
||||
|
||||
Untracked secrets in `/etc/caddy/caddy.env`:
|
||||
- `IONOS_AUTH_API_TOKEN` — DNS-01 ACME
|
||||
- `ARTIFACTO_GATEWAY_SECRET` — must match `SSO_GATEWAY_SECRET` on [apps (105)](105-apps.md). Rotate together.
|
||||
|
||||
## Auto-deploy
|
||||
|
||||
Push to `dtoro/caddy-conf` master → gitea webhook → `http://192.168.8.175:9797/deploy` → `caddy-deploy-webhook.service` validates HMAC against `/etc/caddy-deploy/secret` → runs `/etc/caddy/scripts/deploy.sh`:
|
||||
1. `git pull`
|
||||
2. `caddy validate`
|
||||
3. `systemctl reload caddy`
|
||||
|
||||
Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scripts, systemd unit all live in the repo under `scripts/`. The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). Git credentials for the root-run service are at `/etc/caddy-deploy/git-credentials` (mode 600).
|
||||
|
||||
> **Systemd unit changes inside the repo do NOT auto-reinstall** — you have to re-run `scripts/webhook/install.sh` manually after editing the unit.
|
||||
|
||||
## Related
|
||||
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
|
||||
- [Authentik (124)](124-authentik.md) — forward-auth + IdP
|
||||
- [Auto-deploy](../infrastructure/auto-deploy.md)
|
||||
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
|
||||
- [Gitea (104)](104-gitea.md) — webhook source
|
||||
- All container pages list their Caddy backend.
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082
|
||||
`sab.hubris.network` now uses `import authentik` inside a `handle` block. SABnzbd own auth disabled, `local_ranges` set for transparent proxy. Port bumped from `:8081` to `:8082` (fix from 2026-06-04) now documented.
|
||||
|
||||
### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
|
||||
**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
|
||||
|
||||
**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
|
||||
|
||||
**Fix:**
|
||||
- Restored Caddyfile from `origin/master` (`git checkout --force origin/master -- Caddyfile`)
|
||||
- `systemctl reload caddy`
|
||||
|
||||
**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
|
||||
- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
|
||||
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
|
||||
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
|
||||
|
||||
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
|
||||
|
||||
### 2026-06-02 — caddy.service unit missing; recreated
|
||||
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation. 16 active sites at this date.
|
||||
|
||||
### 2026-04-22 — Artifacto + Authentik gateway-secret pattern wired
|
||||
Caddy site `artifacto.hubris.network`: forward-auth on admin paths, `header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}`, public matchers `/p/*`, `/static/*`, `/healthz`. `caddy.env` extended with the gateway secret.
|
||||
|
||||
### 2026-04-21 — `(authentik)` snippet added
|
||||
Top-of-file snippet used by every gated site. Targets `http://192.168.8.180:9000` directly to dodge hairpin TLS round-trip.
|
||||
|
||||
### 2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)
|
||||
For the [arriman (122)](122-arriman.md) docker stack. Caddy + certs work; needed corresponding [DNS split-horizon](../infrastructure/dns.md) entries to be added to land cleanly.
|
||||
|
||||
### 2026-04-20 — auto-deploy pipeline shipped
|
||||
Repo `dtoro/caddy-conf`. Webhook receiver at `:9797`, validate + reload on push to master.
|
||||
Reference in New Issue
Block a user