Bootstrap Homelab-Docs wiki

Initial documentation of the hubris Proxmox homelab as a cross-linked markdown wiki. Per-node pages, cross-cutting infrastructure pages, an investigation log, and an operations cheatsheet. Each node and topic ends with a Changelog section so changes can be tracked in-place going forward. Refreshed against live state on 2026-04-28 — 14 active LXCs (109 syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test state of the 2026-04-21 hubris crash-loop investigation.
2026-04-28 22:53:09 +02:00
commit f5cd320433
31 changed files with 2157 additions and 0 deletions
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -0,0 +1,98 @@
+# Contributing to the Homelab Wiki
+
+## Voice
+
+Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks.
+
+## Page templates
+
+### Container page (`containers/<id>-<name>.md`)
+
+```markdown
+# <id> — `<name>`
+
+One-sentence purpose.
+
+## At a glance
+- **Hostname:** `<name>`
+- **IP:** `192.168.8.x`
+- **Privilege:** privileged | unprivileged
+- **Resources:** N cores / M GiB RAM / D GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library` (if any)
+- **Public hostname:** `<sub>.hubris.network` (if proxied)
+
+## Role
+What it does, what it talks to.
+
+## Service / port map
+| Service | Listen | Notes |
+
+## Storage / config paths
+
+## Auto-deploy
+(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md)
+
+## Related
+- [Caddy](121-caddy.md) (if proxied)
+- [DNS](../infrastructure/dns.md) (if has subdomain)
+- [Authentik](124-authentik.md) (if SSO)
+- ...
+
+## Changelog
+### YYYY-MM-DD — short title
+What changed, why, link to investigation if any.
+```
+
+### Cross-cutting page (`infrastructure/<topic>.md`)
+
+```markdown
+# <Topic>
+
+One-sentence summary.
+
+## Why
+Design rationale — what it replaces, what it solves.
+
+## Components
+Where it runs, what files matter.
+
+## How to apply / use
+Recipes.
+
+## Gotchas
+
+## Related
+Links to nodes that host or depend on this.
+
+## Changelog
+```
+
+### Investigation (`investigations/YYYY-MM-DD-slug.md`)
+
+```markdown
+# YYYY-MM-DD — <title>
+
+## Summary
+1-3 sentences.
+
+## Timeline
+
+## Root cause
+
+## Mitigations applied
+
+## Open questions
+```
+
+## Linking discipline
+
+- Every container page links to every cross-cutting page it participates in.
+- Every cross-cutting page lists the nodes that participate.
+- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog.
+
+## Changelog hygiene
+
+- Reverse-chronological (newest first).
+- One entry per discrete change, even if you make several in one day.
+- If a change spans nodes, repeat the entry on each affected page (different perspective is fine).
+- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them.
--- a/README.md
+++ b/README.md
@@ -0,0 +1,74 @@
+# Homelab Wiki — `hubris`
+
+Living documentation for the **hubris** Proxmox homelab. Every node, every cross-cutting system, and every meaningful incident is its own page; pages are linked so you can start anywhere and walk the graph.
+
+> Last refreshed against live state: **2026-04-28**.
+
+## Map
+
+### Hosts
+- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77`
+
+### VMs
+- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS
+
+### LXC containers
+See the full table in [`containers/index.md`](containers/index.md). Quick links:
+
+| ID  | Name             | IP              | Role                                          |
+| --- | ---------------- | --------------- | --------------------------------------------- |
+| 101 | [jellyfin](containers/101-jellyfin.md)             | 192.168.8.206 | Media server                                  |
+| 103 | [paperless](containers/103-paperless.md)           | 192.168.8.130 | Document mgmt                                 |
+| 104 | [gitea](containers/104-gitea.md)                   | 192.168.8.121 | Git server                                    |
+| 105 | [apps](containers/105-apps.md)                     | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) |
+| 109 | [syncthing](containers/109-syncthing.md)           | (stopped)     | File sync                                     |
+| 114 | [nextcloud](containers/114-nextcloud.md)           | 192.168.8.224 | Personal cloud                                |
+| 116 | [heaper](containers/116-heaper.md)                 | 192.168.8.116 | Heaper                                        |
+| 118 | [elementsynapse](containers/118-elementsynapse.md) | 192.168.8.239 | Matrix Synapse                                |
+| 119 | [sophia](containers/119-sophia.md)                 | 192.168.8.157 | Sophia                                        |
+| 120 | [mule-images](containers/120-mule-images.md)       | 192.168.8.136 | Mule-image / mulita photos                    |
+| 121 | [caddy](containers/121-caddy.md)                   | 192.168.8.175 | Reverse proxy                                 |
+| 122 | [arriman](containers/122-arriman.md)               | 192.168.8.132 | Docker host (\*arr stack)                     |
+| 123 | [claudio-bot](containers/123-claudio-bot.md)       | 192.168.8.230 | Matrix control plane                          |
+| 124 | [authentik](containers/124-authentik.md)           | 192.168.8.180 | SSO + split-horizon DNS                       |
+
+### Cross-cutting infrastructure
+- [DNS — split-horizon](infrastructure/dns.md)
+- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
+- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
+- [Monitoring — claudio-monitor](infrastructure/monitoring.md)
+- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
+- [SSH access](infrastructure/ssh-access.md)
+- [Backups — restic on external drive (disabled)](infrastructure/backups.md)
+- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md)
+- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md)
+
+### Investigations
+Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md).
+
+### Operations
+- [Command cheatsheet](operations/commands.md)
+
+## Conventions
+
+- **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format:
+  ```
+  ### YYYY-MM-DD — short title
+  one or two lines on what changed and why.
+  ```
+- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
+- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
+- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image, claudio-bot) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
+- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
+
+## Maintaining this wiki
+
+When you change a node:
+1. Update the relevant page (config snapshot, ports, mounts).
+2. Add a changelog entry at the bottom of that page.
+3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry.
+4. If it's an incident, add an entry to [`investigations/`](investigations/index.md).
+
+## See also
+
+- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone
--- a/containers/101-jellyfin.md
+++ b/containers/101-jellyfin.md
@@ -0,0 +1,34 @@
+# 101 — `jellyfin`
+
+Media server: serves the movies / TV / anime / music / audiobooks / podcasts libraries from `/mnt/library` to LAN clients.
+
+## At a glance
+- **Hostname:** `jellyfin`
+- **IP:** `192.168.8.206`
+- **Privilege:** **unprivileged** + idmap (so it can write to the `media` group on `/mnt/library`)
+- **Resources:** 2 cores / 4 GiB RAM / 16 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096`
+
+## Service / port map
+
+| Service  | Listen | Notes |
+| -------- | ------ | ----- |
+| jellyfin | `:8096` | HTTP (caddy terminates TLS) |
+
+## Permissions
+Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; idmap block in `/etc/pve/lxc/101.conf` maps in-container GID 10000 to host GID 10000.
+
+## Related
+- [Caddy reverse proxy](121-caddy.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads
+- [DNS split-horizon](../infrastructure/dns.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation. No config changes.
+
+### 2026-04-20 — joined the `media` GID 10000 standard
+Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.
--- a/containers/103-paperless.md
+++ b/containers/103-paperless.md
@@ -0,0 +1,41 @@
+# 103 — `paperless`
+
+Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/documents` and serves OCR'd indexed copies.
+
+## At a glance
+- **Hostname:** `paperless`
+- **IP:** `192.168.8.130`
+- **Privilege:** privileged
+- **Resources:** 2 cores / 3 GiB RAM / 8 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library` (consumes `documents/`)
+- **Public hostname:** [`paperless.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8000`
+
+## Service / port map
+
+| Service             | Listen  | Notes                            |
+| ------------------- | ------- | -------------------------------- |
+| paperless-webserver | `:8000` | HTTP                             |
+| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
+
+## Auth
+Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
+
+## Storage
+- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
+
+## Known issues
+- Disk usage was 86.9% at last claudio-monitor reading on 2026-04-21. Monitor or grow rootfs.
+
+## Related
+- [Authentik](124-authentik.md)
+- [Caddy](121-caddy.md)
+- [DNS](../infrastructure/dns.md)
+- [Monitoring](../infrastructure/monitoring.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-21 — wired into Authentik forward-auth
+Cookie-domain-level proxy provider, embedded outpost binding. `/api/*` bypass added in Caddy site block to keep mobile clients working.
--- a/containers/104-gitea.md
+++ b/containers/104-gitea.md
@@ -0,0 +1,58 @@
+# 104 — `gitea`
+
+Self-hosted git server. Source of truth for all `dtoro/*` repositories that auto-deploy across the lab.
+
+## At a glance
+- **Hostname:** `gitea`
+- **IP:** `192.168.8.121`
+- **Privilege:** privileged
+- **Resources:** 1 core / 1 GiB RAM / 8 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library` (under `/mnt/library/repos`, owner UID 102 / GID 105 — don't touch from other containers)
+- **Public hostname:** [`git.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000`
+- **Version:** 1.25.4 (last verified)
+
+## Service / port map
+
+| Service | Listen   | Notes                          |
+| ------- | -------- | ------------------------------ |
+| gitea   | `:3000`  | HTTP web + git over HTTP       |
+| gitea SSH | `:22`  | git over SSH                   |
+| `gitea-deploy-webhook` | `127.0.0.1:9797` | Receives push events for the customizations repo |
+
+## Customizations
+`/var/lib/gitea/custom` is a checkout of [`dtoro/gitea-customizations`](https://git.hubris.network/dtoro/gitea-customizations).
+- `templates/custom/footer.tmpl` injects:
+  - **Online3DViewer** v0.18.0 (vendored under `public/assets/o3dv/`) for inline 3D preview of CAD raw-file links: 3dm, 3ds, 3mf, amf, bim, brep, dae, fbx, fcstd, glb, gltf, ifc, igs, iges, stp, step, stl, obj, off, ply, wrl.
+  - **PlantUML client JS** (vanilla, vendored from `gitea.com/davidsvantesson/plantuml-code-highlight`). Browser calls `https://git.hubris.network/_plantuml/img/<encoded>` — caddy `handle_path /_plantuml/*` rewrites and proxies to the PlantUML server on [apps (105)](105-apps.md) `:8079`.
+- `home.tmpl` is the anonymous landing page (Homesauce hero + sign-in/explore buttons).
+- O3DV binary assets are *not* tracked — `deploy.sh` fetches them on first run.
+
+## Auto-deploy
+Push to `dtoro/gitea-customizations` master → gitea webhook → `http://127.0.0.1:9797/deploy` (loopback) → `gitea-deploy-webhook.service` validates HMAC against `/etc/gitea-deploy/secret` → runs `/opt/gitea-customizations/deploy.sh`. Then `systemctl restart gitea` if templates changed.
+
+`app.ini` `[webhook] ALLOWED_HOST_LIST` includes `127.0.0.1`, `192.168.8.175`, `192.168.8.205`, `192.168.8.230`, `192.168.8.136`, `192.168.8.77` — every webhook target across the lab. **Don't strip these when editing app.ini.** See [auto-deploy](../infrastructure/auto-deploy.md) for the full list.
+
+Git creds for the deploy service: `/etc/gitea-deploy/git-credentials` (mode 600), referenced via `credential.helper = store --file=/etc/gitea-deploy/git-credentials` in the repo's `.git/config` (because `ProtectHome=true` blocks `/root`).
+
+## DNS workaround
+LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (router DNS resolves the public IONOS A record, which is a dead end for LAN clients — see [DNS](../infrastructure/dns.md)). Belt-and-suspenders systemd unit `hubris-hosts-override.service` re-applies it idempotently. Override line is **outside** the `# --- BEGIN PVE ---` markers (Proxmox rewrites inside).
+
+## Related
+- [Caddy](121-caddy.md)
+- [DNS](../infrastructure/dns.md)
+- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
+- [PlantUML server lives on apps (105)](105-apps.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-22 — `app.ini` ALLOWED_HOST_LIST extended for Artifacto webhook
+Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
+
+### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
+For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
+
+### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
+`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.
--- a/containers/105-apps.md
+++ b/containers/105-apps.md
@@ -0,0 +1,88 @@
+# 105 — `apps`
+
+Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog).
+
+## At a glance
+- **Hostname:** `apps`
+- **IP:** `192.168.8.205`
+- **Privilege:** privileged (so container UID = host UID for bind-mount sanity)
+- **Resources:** 2 cores / 4 GiB RAM / 30 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Public hostnames:** see table below
+
+## Public hostname → service map
+
+| Hostname                          | Container        | Backend port | Notes |
+| --------------------------------- | ---------------- | ------------ | ----- |
+| `docker.hubris.network`           | Portainer        | `:9443`      | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
+| `books.hubris.network`            | Booklore         | `:6060`      | Native OIDC. Redirect URI `/oauth2-callback`. |
+| `artifacto.hubris.network`        | Artifacto        | `:3100`      | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
+| `blog.hubris.network`             | WriteFreely      | `:8080`      | Native OIDC via `[oauth.generic]`. |
+| `git.hubris.network/_plantuml/*`  | PlantUML server  | `:8079`      | Same-origin route from [gitea (104)](104-gitea.md). |
+
+## Stacks
+
+### Artifacto (`/opt/artifacto/`)
+Self-hosted HTML-artifact share (Go + SQLite). Repo `dtoro/Artifacto` is the working tree itself — `git push origin main` triggers rebuild.
+- Compose: tracked `docker-compose.yml` binds `./data:/data` and `127.0.0.1:3100:3000`. `docker-compose.override.yml` uses the `!override` YAML tag on `volumes:` and `ports:` to redirect `data` to `/mnt/library/documents/artifacts`.
+- Data dir: `/mnt/library/documents/artifacts` — owner **UID 10001 : GID 10001** (matches the `app` user baked into the Dockerfile), mode 750. Parent `/mnt/library/documents` stays `www-data:www-data 750`; bind targets the child directly so no traversal needed.
+- `.env` (untracked) holds `ADMIN_PASSWORD`, `SESSION_SECRET` (`openssl rand -hex 32`), `BASE_URL=https://artifacto.hubris.network`, `SSO_GATEWAY_SECRET` (must match `ARTIFACTO_GATEWAY_SECRET` in `/etc/caddy/caddy.env` on [caddy](121-caddy.md)). Don't `git checkout .env`.
+- Auth: Authentik forward-auth on the admin surface (`/`, `/login`, `/a/*`, `/api/*`). `X-Authentik-Username` is honored only when the `X-Artifacto-Gateway` header matches `SSO_GATEWAY_SECRET` — prevents direct LAN connections from spoofing identity.
+- Public exposure: VPS traefik gates path scope to share URLs; admin surface returns 404 at the edge. See [ingress](../infrastructure/ingress.md).
+
+#### Auto-deploy pipeline (Artifacto)
+Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webhook.py`, secret in `/etc/artifacto-deploy/secret`, systemd unit `artifacto-deploy-webhook.service`. Listens on `0.0.0.0:9798`, validates `X-Gitea-Signature` HMAC-SHA256, filters to `refs/heads/main`, runs deploy in a daemon thread (returns 202 immediately). Gitea webhook id 7 on `dtoro/Artifacto`.
+
+- Logs: `pct exec 105 -- journalctl -u artifacto-deploy-webhook -f`
+- Manual deploy: `pct exec 105 -- /opt/artifacto-deploy/deploy.sh`
+- Health: `pct exec 105 -- curl -s http://127.0.0.1:9798/health` → `ok`
+- Slug length bumped 5 → 8 chars (in `internal/store/artifacts.go:41`) once the endpoint went public.
+
+### PlantUML (`/opt/plantuml/`)
+`plantuml/plantuml-server:jetty` listening on `192.168.8.205:8079`, context path `/plantuml` (`BASE_URL=plantuml`). Reached only via [gitea](104-gitea.md)'s caddy `handle_path /_plantuml/*` rewrite — no DNS record of its own. Same-origin from gitea so the embedded markdown previewer can call it without CORS.
+
+### Portainer
+Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
+
+### Booklore
+Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` (NOT `/api/oidc`). Container needs `extra_hosts: auth.hubris.network:192.168.8.175`. **Edit via Portainer UI** if it's a Portainer-managed stack.
+
+> ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`. Running `docker compose up -d <svc>` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning.
+
+### WriteFreely (`/opt/writefreely-docker/`)
+Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts.
+
+## Permissions
+- LXC is privileged → in-container UID == host UID.
+- Member of [media GID 10000](../infrastructure/media-permissions.md). Adding `media` to the LXC alone is *not* enough for inner Docker containers — each container needs `--group-add 10000` or `user: "<uid>:10000"` to write into `/mnt/library`.
+
+## Related
+- [Gitea (104)](104-gitea.md) — uses the PlantUML server
+- [Caddy (121)](121-caddy.md)
+- [Authentik (124)](124-authentik.md)
+- [DNS](../infrastructure/dns.md)
+- [Auto-deploy](../infrastructure/auto-deploy.md)
+- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-22 — Artifacto deployed + Authentik gateway-secret auto-login wired
+Repo `dtoro/Artifacto`. Compose, data dir, `.env`, webhook (id 7), Caddyfile site block, gateway-secret pattern, public-internet share exposure on the VPS — all set up that day. Slug length bumped 5 → 8 for brute-force resistance.
+
+### 2026-04-22 — Booklore mariadb data wiped (lesson, not a feature)
+`docker compose up -d` from the host shell on a Portainer-managed stack recreated services and re-initialized the bind-mounted `./mariadb/config`. Recovered by reseeding from snapshot. Rule recorded: edit Portainer-managed stacks via the Portainer UI.
+
+### 2026-04-22 — Portainer wired into Authentik (native OAuth2)
+Custom-claim `portainer_uid` scope. CLI added `--trusted-origins docker.hubris.network`.
+
+### 2026-04-22 — `app.ini` host whitelist on gitea extended
+Added `192.168.8.205` so the Artifacto webhook delivery wouldn't be rejected. See [gitea (104)](104-gitea.md).
+
+### 2026-04-21 — Booklore + WriteFreely wired into Authentik OIDC
+Native OIDC for both. `extra_hosts` override added inside each container so DNS resolves `auth.hubris.network` to the Caddy IP.
+
+### 2026-04-20 — PlantUML server deployed
+Used by [gitea (104)](104-gitea.md)'s footer.tmpl injection. Same-origin via caddy `handle_path /_plantuml/*` rewrite — no separate hostname.
--- a/containers/109-syncthing.md
+++ b/containers/109-syncthing.md
@@ -0,0 +1,26 @@
+# 109 — `syncthing`
+
+Cross-device file sync into `/mnt/library/syncthing`.
+
+## At a glance
+- **Hostname:** `syncthing`
+- **IP:** (none — currently stopped)
+- **Privilege:** **unprivileged** + idmap (for the `media` GID 10000 standard)
+- **Resources:** 2 cores / 2 GiB RAM / 8 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Status:** **stopped** (was running prior to 2026-04-21).
+
+## Permissions quirk
+The syncthing systemd unit runs with `User=root`, and systemd skips `initgroups()` for `User=root`. Drop-in adds `SupplementaryGroups=media` to `syncthing@.service` so it actually picks up GID 10000. See [media permissions](../infrastructure/media-permissions.md).
+
+## Related
+- [Media permissions](../infrastructure/media-permissions.md)
+- [Hubris host](../hosts/hubris.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation. Container is currently stopped — reason not yet documented; investigate before re-enabling or destroying.
+
+### 2026-04-20 — joined the `media` GID 10000 standard
+Idmap block + drop-in `SupplementaryGroups=media` on `syncthing@.service`. Config backup: `/root/109.conf.bak.*`.
--- a/containers/114-nextcloud.md
+++ b/containers/114-nextcloud.md
@@ -0,0 +1,48 @@
+# 114 — `nextcloud`
+
+Personal cloud / file collaboration. Source-of-truth for the photo libraries surfaced by [mulita (120)](120-mule-images.md).
+
+## At a glance
+- **Hostname:** `nextcloud`
+- **IP:** `192.168.8.224`
+- **Privilege:** privileged
+- **Resources:** 4 cores / 6 GiB RAM / 25 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library` (Nextcloud user files under `/mnt/library/homecloud`)
+- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
+
+## Auth
+Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern.
+
+Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
+
+## DNS workaround (load-bearing)
+Nextcloud's PHP HTTP client (`OC\Http\Client\DnsPinMiddleware`) calls `dns_get_record()` directly — it bypasses `/etc/hosts`. So the Authentik OIDC discovery URL needs **real DNS** answering with `192.168.8.175` for `auth.hubris.network`. Recipe applied:
+- Local `dnsmasq` listening on `127.0.0.1:53`, `/etc/dnsmasq.d/hubris-internal.conf` with `address=/auth.hubris.network/192.168.8.175`, `server=192.168.8.1`, `server=1.1.1.1`, `interface=lo`, `bind-interfaces`, `no-hosts`, `no-resolv`.
+- `pct set 114 --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"` so the LXC starts with the right resolver order.
+- `/etc/hosts` override kept as belt-and-suspenders.
+
+Also needs `allow_local_remote_servers=true` in `config.php`.
+
+> Once internal DNS at the router level handles `*.hubris.network`, this LXC-local dnsmasq can go away. See [DNS](../infrastructure/dns.md).
+
+## Storage
+Files at `/mnt/library/homecloud`. Owned by Nextcloud's own permission model — **deliberately not on the `media` group**, NC manages it itself and would complain. See [media permissions](../infrastructure/media-permissions.md).
+
+## Related
+- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
+- [Authentik (124)](124-authentik.md)
+- [Caddy (121)](121-caddy.md)
+- [DNS](../infrastructure/dns.md)
+- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-26 — Nextcloud-rooted libraries shipped (mulita)
+[mulita (120)](120-mule-images.md) now reads photos from `/nextcloud-users/<nc_user>/files/...` and writes mutations back via WebDAV. Pre-migration DB dump kept at `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump`.
+
+### 2026-04-21 — wired into Authentik (native OIDC + nc_uid override)
+Local `dnsmasq` added on this LXC because Guzzle bypasses `/etc/hosts`. Resolver order set to `127.0.0.1 192.168.8.1 1.1.1.1`. Username override `dtoro → admin`.
--- a/containers/116-heaper.md
+++ b/containers/116-heaper.md
@@ -0,0 +1,23 @@
+# 116 — `heaper`
+
+Single-purpose container for the "heaper" workload. Mounts `/mnt/library` and reads/writes under `/mnt/library/heaper`.
+
+## At a glance
+- **Hostname:** `heaper`
+- **IP:** `192.168.8.116`
+- **Privilege:** privileged
+- **Resources:** 2 cores / 2 GiB RAM / 10 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Public hostname:** none
+
+## Permissions
+Member of the [media GID 10000](../infrastructure/media-permissions.md) standard via `www-data` (the data subtree at `/mnt/library/heaper` is single-container, so it stays out of the shared setgid group treatment, but the LXC has the standard group hookup).
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Stub. Detail what heaper actually does the next time we touch this LXC.
--- a/containers/118-elementsynapse.md
+++ b/containers/118-elementsynapse.md
@@ -0,0 +1,32 @@
+# 118 — `elementsynapse`
+
+Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
+
+## At a glance
+- **Hostname:** `elementsynapse`
+- **IP:** `192.168.8.239`
+- **Privilege:** **unprivileged**
+- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs
+- **Mounts:** none from `/mnt/library`
+- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008`
+- **`server_name`:** `avispero` (intentionally different from the public URL).
+
+## Service / port map
+
+| Service        | Listen  | Notes                            |
+| -------------- | ------- | -------------------------------- |
+| synapse        | `:8008` | Matrix client-server / federation HTTP. Caddy terminates TLS for `matrix.hubris.network`. |
+
+## Known issues
+- Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21 — the Matrix DB grows steadily. Watch or grow rootfs.
+
+## Related
+- [claudio-bot (123)](123-claudio-bot.md) — connects directly to `192.168.8.239:8008` (avoids hairpin-NAT TLS issue on the public URL)
+- [Caddy](121-caddy.md)
+- [DNS](../infrastructure/dns.md)
+- [Monitoring](../infrastructure/monitoring.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
--- a/containers/119-sophia.md
+++ b/containers/119-sophia.md
@@ -0,0 +1,23 @@
+# 119 — `sophia`
+
+Single-container workload "sophia". Reads/writes under `/mnt/library/sophia`.
+
+## At a glance
+- **Hostname:** `sophia`
+- **IP:** `192.168.8.157`
+- **Privilege:** privileged
+- **Resources:** 2 cores / 1 GiB RAM / 10 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Public hostname:** none
+
+## Permissions
+LXC has `www-data` in the [media GID 10000 group](../infrastructure/media-permissions.md). The `/mnt/library/sophia` subtree itself is single-container and not on the setgid shared group.
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Stub. Document what runs inside next time we work on this LXC.
--- a/containers/120-mule-images.md
+++ b/containers/120-mule-images.md
@@ -0,0 +1,93 @@
+# 120 — `mule-images`
+
+Hosts `mule-image` / "mulita" — the photos app at `photos.hubris.network`. Auto-deploys from `dtoro/mule-image` on `git push origin main`.
+
+## At a glance
+- **Hostname:** `mule-images`
+- **IP:** `192.168.8.136`
+- **Privilege:** privileged
+- **Resources:** 4 cores / 8 GiB RAM / 60 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000` (frontend)
+
+## Stack (`/opt/mule-image`)
+
+`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose at `/opt/mule-image/docker-compose.yml`. Services:
+
+| Service           | Port   | Notes |
+| ----------------- | ------ | ----- |
+| frontend          | 3000   | Reverse-proxied by Caddy |
+| backend           | 8001   | FastAPI |
+| worker-vision     | —      | ML scan worker |
+| worker-light      | —      | Lightweight worker |
+| worker-watcher    | —      | FS watcher |
+| db                | (pg)   | pgvector |
+| redis             | (rd)   | queue |
+
+`.env` is **untracked** — `git checkout .env` will wipe it. Holds:
+- `PHOTO_DIRS=/mnt/library/images/`
+- `NEXTCLOUD_USERS_HOST_PATH=/mnt/library/homecloud`
+- `NEXTCLOUD_BASE_URL=https://cloud.hubris.network`
+- OIDC client secret + scopes
+- `SECRET_KEY` (generated)
+
+## Nextcloud-rooted libraries (since 2026-04-26)
+
+Photo libraries live under each user's Nextcloud `files/` tree, NOT in `/mnt/library/images/*`.
+- `/mnt/library/homecloud` is bind-mounted into `backend`, `worker-light`, `worker-watcher`, `worker-vision` as `/nextcloud-users`. Each NC user is `/nextcloud-users/<nc_user>/files/`.
+- Reads use that bind directly.
+- Mutations (upload, delete, rename, move) dispatch through `services/nextcloud_dav.py` (HTTP Basic auth, per-user app password Fernet-encrypted in `users.nextcloud_app_password_enc`) so Nextcloud's `oc_filecache`, trashbin, comments, and desktop-sync clients stay coherent.
+- Photo copy + cross-system moves return 501 with a "use Nextcloud's web UI" hint — defer until needed.
+- `users.nextcloud_username` overrides the default OIDC `preferred_username`. **`dtoro` (mule-image) maps to `admin` (Nextcloud)** — don't assume username equality.
+- Surviving SourceRoots in DB: `Photos` → `/nextcloud-users/admin/files/Photos`; `Memories` → `/nextcloud-users/admin/files/Memories` (both owned by `dtoro`). User `muli` has `nextcloud_username=muli` backfilled but no SourceRoot yet.
+- Pre-migration DB dump: `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump` (11 MB) on the host.
+
+## Authentication (since 2026-04-22)
+
+Native OIDC via Authentik. Code in `backend/app/auth_oidc.py`, routes `/api/v1/auth/oidc/{login,callback}`. Authentik side:
+- OAuth2/OIDC Provider, client ID `fCuHew48ONTskDjUKnMTZjFbVXuHwvQqTScQRNQ1`
+- App slug `mule-image`
+- Redirect URI: `https://photos.hubris.network/api/v1/auth/oidc/callback`
+
+Backend container needs `extra_hosts: auth.hubris.network:192.168.8.175` via `docker-compose.override.yml` (gitignored). Otherwise Authlib's metadata fetch fails with `SSL: CERTIFICATE_VERIFY_FAILED: self-signed certificate` (it ends up at a random public host because LXC DNS resolves the public IONOS A record).
+
+Caddyfile stays plain `reverse_proxy 192.168.8.136:3000` — no forward-auth, no `/api/*` bypass needed.
+
+## Auto-deploy
+
+Push to `dtoro/mule-image` `main` → gitea webhook → `http://192.168.8.136:9797/deploy` → `mule-deploy-webhook.service`:
+- Validates HMAC against `/etc/mule-deploy/secret`
+- Filters to `refs/heads/main`
+- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
+- `git pull --ff-only` + `docker compose up -d --build` + `docker image prune -f`
+
+Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`, unit at `/etc/systemd/system/mule-deploy-webhook.service`. Same shape as the Caddy + Artifacto pipelines. Gitea webhook id 6.
+
+`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
+
+Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`.
+Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`.
+
+For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials` (mode 600) — same token as `/etc/caddy-deploy/git-credentials` on [caddy](121-caddy.md).
+
+## Related
+- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
+- [Authentik (124)](124-authentik.md)
+- [Caddy (121)](121-caddy.md)
+- [DNS](../infrastructure/dns.md)
+- [Auto-deploy](../infrastructure/auto-deploy.md)
+- [Gitea (104)](104-gitea.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-26 — Nextcloud-rooted libraries shipped
+Bind `/mnt/library/homecloud` into the workers, reads via filesystem, writes via WebDAV. `users.nextcloud_username` override field added; `dtoro → admin` mapping. Surviving SourceRoots cleaned up to NC paths.
+
+### 2026-04-22 — native OIDC via Authentik
+Authlib-based code in `backend/app/auth_oidc.py`. `extra_hosts` override for `auth.hubris.network` in compose override (gitignored).
+
+### 2026-04-21 — auto-deploy pipeline shipped
+Webhook receiver at `:9797`, async deploy returning 202. Mirrors caddy-conf / gitea-customizations.
--- a/containers/121-caddy.md
+++ b/containers/121-caddy.md
@@ -0,0 +1,81 @@
+# 121 — `caddy`
+
+The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the LAN/mesh and forwards to the right backend.
+
+## At a glance
+- **Hostname:** `caddy`
+- **IP:** `192.168.8.175`
+- **Privilege:** **unprivileged**
+- **Resources:** 1 core / 512 MiB RAM / 6 GiB rootfs
+- **Mounts:** none from `/mnt/library`
+- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
+- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
+
+## Sites currently served (live as of 2026-04-28)
+
+- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
+- `auth.hubris.network` → [authentik (124)](124-authentik.md) `:9000`
+- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
+- `books.hubris.network` → [apps (105)](105-apps.md) `:6060`
+- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
+- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
+- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
+- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
+- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
+- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
+- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
+- `paperless.hubris.network` → [paperless (103)](103-paperless.md) `:8000`
+- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
+- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
+- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
+- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8081`
+
+> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
+
+## Snippet: `(authentik)` forward-auth
+
+A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup).
+
+For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md).
+
+## Caddy environment
+
+Untracked secrets in `/etc/caddy/caddy.env`:
+- `IONOS_AUTH_API_TOKEN` — DNS-01 ACME
+- `ARTIFACTO_GATEWAY_SECRET` — must match `SSO_GATEWAY_SECRET` on [apps (105)](105-apps.md). Rotate together.
+
+## Auto-deploy
+
+Push to `dtoro/caddy-conf` master → gitea webhook → `http://192.168.8.175:9797/deploy` → `caddy-deploy-webhook.service` validates HMAC against `/etc/caddy-deploy/secret` → runs `/etc/caddy/scripts/deploy.sh`:
+1. `git pull`
+2. `caddy validate`
+3. `systemctl reload caddy`
+
+Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scripts, systemd unit all live in the repo under `scripts/`. The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). Git credentials for the root-run service are at `/etc/caddy-deploy/git-credentials` (mode 600).
+
+> **Systemd unit changes inside the repo do NOT auto-reinstall** — you have to re-run `scripts/webhook/install.sh` manually after editing the unit.
+
+## Related
+- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
+- [Authentik (124)](124-authentik.md) — forward-auth + IdP
+- [Auto-deploy](../infrastructure/auto-deploy.md)
+- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
+- [Gitea (104)](104-gitea.md) — webhook source
+- All container pages list their Caddy backend.
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation. 16 active sites at this date.
+
+### 2026-04-22 — Artifacto + Authentik gateway-secret pattern wired
+Caddy site `artifacto.hubris.network`: forward-auth on admin paths, `header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}`, public matchers `/p/*`, `/static/*`, `/healthz`. `caddy.env` extended with the gateway secret.
+
+### 2026-04-21 — `(authentik)` snippet added
+Top-of-file snippet used by every gated site. Targets `http://192.168.8.180:9000` directly to dodge hairpin TLS round-trip.
+
+### 2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)
+For the [arriman (122)](122-arriman.md) docker stack. Caddy + certs work; needed corresponding [DNS split-horizon](../infrastructure/dns.md) entries to be added to land cleanly.
+
+### 2026-04-20 — auto-deploy pipeline shipped
+Repo `dtoro/caddy-conf`. Webhook receiver at `:9797`, validate + reload on push to master.
--- a/containers/122-arriman.md
+++ b/containers/122-arriman.md
@@ -0,0 +1,111 @@
+# 122 — `arriman`
+
+Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr) compose. Replaced the old yunohost-based [LXC 100](index.md#recently-destroyed-kept-for-archaeology) on 2026-04-21.
+
+## At a glance
+- **Hostname:** `arriman`
+- **IP:** `192.168.8.132`
+- **Privilege:** privileged
+- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
+- **Mounts:** `/mnt/library` ↔ `/mnt/library`
+- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
+
+## Compose
+
+`/home/arr/ezarr/docker-compose.yml`. Network `ezarr_default` `172.18.0.0/16`. Service users all run with primary GID 10000 (host `media` group) inside their containers — see [media permissions](../infrastructure/media-permissions.md). Per-app `/config/<app>-config/` dirs bind to `/config` inside.
+
+`.env` contents:
+- `DOWNLOADS_DIR=/mnt/library/downloads/`
+- `MEDIA_DIR=/mnt/library/`
+- `MEDIACENTER_GID=10000` ← **must be 10000, not 13000.** s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. Without this, qBit errors every torrent with "Permission denied".
+- `SONARR_UID=13001`, `RADARR_UID=13002`, `LIDARR_UID=13003`, `PROWLARR_UID=13006`, `JELLYSEERR_UID=13012`, `BAZARR_UID=13013`, `QBITTORRENT_UID`, `SABNZBD_UID`
+
+## Service / port map
+
+| Service       | Host:Container | Public hostname                      |
+| ------------- | -------------- | ------------------------------------ |
+| sonarr        | `8989:8989`    | direct only                          |
+| radarr        | `7878:7878`    | direct only                          |
+| lidarr        | `8686:8686`    | direct only                          |
+| prowlarr      | `9696:9696`    | direct only                          |
+| bazarr        | `6767:6767`    | direct only                          |
+| jellyseerr    | `5056:5055`    | [`jellyseerr.hubris.network`](../infrastructure/dns.md) |
+| qbittorrent   | `8080:8080`    | [`qbit.hubris.network`](../infrastructure/dns.md)       |
+| sabnzbd       | `8081:8080`    | [`sab.hubris.network`](../infrastructure/dns.md)        |
+| flaresolverr  | `8191:8191`    | internal only                        |
+
+Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `ezarr_default` using docker service names.
+
+## Categories (qBit + SAB + *arr)
+
+| Category | qBit savePath               | SAB dir | Consumer                          |
+| -------- | --------------------------- | ------- | --------------------------------- |
+| tv       | `/data/torrents/tv`         | tv      | Sonarr                            |
+| movies   | `/data/torrents/movies`     | movies  | Radarr                            |
+| music    | `/data/torrents/music`      | music   | Lidarr                            |
+| anime    | `/data/torrents/anime`      | anime   | Sonarr (2nd download client, root `/data/media/anime`) |
+| books / audiobooks / comics / podcasts | `/data/torrents/<cat>` | `<cat>` | (reserved, none in use yet) |
+
+Path mapping: host `/mnt/library/<cat>` ↔ container `/data/media/<cat>`. Downloads: host `/mnt/library/downloads/<torrents|usenet>/<cat>` ↔ container `/data/torrents/<cat>` and `/data/usenet/<cat>`.
+
+## Auth (qBit reverse-proxy + Authentik forward-auth)
+
+qBit auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
+- `WebUI\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
+- `WebUI\ReverseProxySupportEnabled=true`
+- `WebUI\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
+
+> **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`.
+
+Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy.
+
+SABnzbd `host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
+
+## Credentials
+
+Backups of all migration-time secrets at `/mnt/library/backup/arr-migration-2026-04-21/state/`:
+- `lxc100-apikey-{sonarr,radarr,prowlarr,lidarr}.txt`
+- `qbit-creds.txt`, `sab-creds.txt`
+- `lxc100-jellyseerr-settings.json`
+
+\*arr API keys were preserved from the yunohost DB — don't rotate unless needed.
+
+qBittorrent: `admin` / `CDyjxMKvjfGupCKzXFkjsKw4nRrXMRL`.
+
+## Migration gotchas (apply if ever re-importing)
+
+- Imported \*arr `config.xml`: must have `<BindAddress>*</BindAddress>`, `<Port>{ezarr-port}</Port>`, `<UrlBase></UrlBase>`. Otherwise app listens only on 127.0.0.1, wrong port, wrong path.
+- Imported DB paths must be rewritten `/mnt/library/` → `/data/media/` (RootFolders / Series / Movies / Artists tables). Without this, scans and imports fail silently.
+- Bazarr `config/config.yaml` (yaml, not ini) — `ip: 0.0.0.0`, `port: 6767`, empty top-level `base_url`; sonarr/radarr connection sections normalized to docker service names.
+- Staging dir for migrations: `/mnt/library/backup/stage/<app>/` (visible to both old LXC 100 and arriman because both bind-mount the same host path).
+
+## Permissions
+
+Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has `www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr` in the `media` group. Each docker container also receives GID 10000 as its primary group via `MEDIACENTER_GID=10000`.
+
+## Rollback material (kept until 2026-04-28+)
+
+`/mnt/library/backup/arr-migration-2026-04-21/`:
+- `tars/` — per-subtree tars of yunohost app data (granular restore)
+- `snapshots/` — vzdumps of LXC 100 (pre-migration) and LXC 122 (pre-import), plus post-stop vzdumps of LXC 100 + 106
+- `stage/<app>-preimport.tar.gz` — original empty ezarr `/config/<app>-config/` dirs
+- `state/` — service dumps, listening ports, API keys, jellyseerr settings at migration time
+- `DESTROY_AFTER_2026-04-28.md` — destroy checklist + post-destroy restore commands
+
+## Related
+- [Caddy (121)](121-caddy.md)
+- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass
+- [DNS](../infrastructure/dns.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+- [Hubris host](../hosts/hubris.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-26 — `MEDIACENTER_GID` flipped 13000 → 10000
+qBit was erroring every torrent with "Permission denied" because s6-setuidgid only honors the primary PGID and `group_add:` doesn't propagate. Setting the primary GID to 10000 (the host `media` group) fixed it.
+
+### 2026-04-21 — \*arr stack migrated yunohost → docker-compose
+Old LXC 100 stopped; ezarr stack stood up here; data and DBs imported with path rewrites. flaresolverr folded in from old LXC 106. Three Caddy sites added (jellyseerr, qbit, sab) → see [caddy](121-caddy.md). Rollback artifacts under `/mnt/library/backup/arr-migration-2026-04-21/`.
--- a/containers/123-claudio-bot.md
+++ b/containers/123-claudio-bot.md
@@ -0,0 +1,79 @@
+# 123 — `claudio-bot`
+
+Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications.
+
+## At a glance
+- **Hostname:** `claudio-bot`
+- **IP:** `192.168.8.230`
+- **Privilege:** **unprivileged**
+- **Resources:** 1 core / 512 MiB RAM / 8 GiB rootfs
+- **Mounts:** none from `/mnt/library`
+- **Public hostname:** none
+
+## Stack
+
+Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`).
+
+Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed).
+
+## Configuration
+
+Secrets at `/etc/claudio-bot/` (mode 600):
+- `config.yaml`
+- `matrix.token`
+- `anthropic.key`
+- `lmstudio.key`
+- `ipc.token`
+
+Deploy git creds at `/etc/claudio-deploy/git-credentials`.
+
+### LLM backend (pluggable)
+
+`llm.backend` in `config.yaml`:
+- `anthropic` — Claude API via `anthropic.key`
+- `lmstudio` — OpenAI-compat HTTP via `lmstudio.key` (bearer)
+
+Currently set to `lmstudio` → `google/gemma-4-e4b` on the Mac mini at `192.168.8.174:1234` (since 2026-04-25). Switch back with `backend:` + `systemctl restart claudio-bot`. Original config saved at `/etc/claudio-bot/config.yaml.bak`. The LM Studio agent in `bot_core/lmstudio.py` translates Anthropic → OpenAI tool definitions.
+
+## IPC
+
+`http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by:
+- [claudio-monitor on hubris](../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`)
+- The (currently disabled) [restic backup wrapper](../infrastructure/backups.md) (token in `/etc/restic/bot.token`)
+
+> Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together.
+
+## Plugins
+
+Module under `plugins/<name>.py` exposing a `Plugin` class; add `<name>` to `plugins:` in `config.yaml`. Plugins can register slash commands, Claude tools, and `on_notify` / `on_proposal_response` hooks.
+
+Active plugins:
+- `system` — `ping`, `help`, `status`, `list_plugins`
+- `backup` — ingests `/notify` from the backup wrapper (currently silent — backups disabled)
+- `monitor` — `/monitor status`, `/monitor history [N]`, `/monitor clear <key>`. Tools: `get_alerts`, `get_monitor_history` for NL queries via Claude.
+
+## Auto-deploy
+
+Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf.
+
+`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.230`.
+
+## Related
+- [elementsynapse (118)](118-elementsynapse.md)
+- [Monitoring (claudio-monitor)](../infrastructure/monitoring.md)
+- [Backups (disabled)](../infrastructure/backups.md)
+- [Auto-deploy](../infrastructure/auto-deploy.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-25 — LLM backend switched to LM Studio
+`backend: lmstudio` → `google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field.
+
+### 2026-04-21 — `monitor` plugin added
+Receives events from [claudio-monitor](../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`.
+
+### 2026-04-20 — claudio-bot deployed
+LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server.
--- a/containers/124-authentik.md
+++ b/containers/124-authentik.md
@@ -0,0 +1,136 @@
+# 124 — `authentik`
+
+Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box".
+
+## At a glance
+- **Hostname:** `authentik`
+- **IP:** `192.168.8.180` (statically configured — the only LXC with a static IP)
+- **Privilege:** privileged
+- **Resources:** 2 cores / 4 GiB RAM / 20 GiB rootfs
+- **Mounts:** none from `/mnt/library`
+- **Public hostname:** [`auth.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `:9000`
+- **Container DNS (in `/etc/pve/lxc/124.conf`):** `192.168.8.1 1.1.1.1` (router DNS plus a fallback added 2026-04-21 because router DNS flakes intermittently — Authentik is the resolver itself for the *rest* of the LAN, but its own LXC uses upstream).
+
+## Authentik stack (`/opt/authentik/`)
+
+Upstream `docker-compose.yml` + `.env`. Services: `postgresql` (16-alpine), `server`, `worker`. Authentik 2026.x dropped the Redis dependency.
+
+- `.env` mode 600, **untracked**, holds `AUTHENTIK_SECRET_KEY` and `PG_PASS`.
+- `AUTHENTIK_TAG=2026.2.2` — pinned. Don't let it drift to `:latest`. Telemetry / update-check / error-reporting disabled.
+- Ports: 9000 (http), 9443 (https) on the LXC.
+- Embedded outpost lives at `/outpost.goauthentik.io/*` on the Authentik host — the forward-auth endpoint Caddy points at.
+- Stack is **not** git-tracked yet. If/when wiring auto-deploy: mirror the `mule-image` pattern (webhook receiver outside the app repo at `/opt/authentik-deploy/`). Repo `dtoro/authentik-conf` is reserved but not created.
+
+## Forward-auth pattern (every gated app)
+
+- **One Proxy Provider per app.** Authentik enforces a UNIQUE constraint `application.provider_id`, so one Provider = one Application. "Domain-level" only means they share the cookie domain. Each provider in "Forward auth (domain level)" mode, External host `https://auth.hubris.network`, Cookie domain `hubris.network`. First one was `hubris-forward-auth` (Paperless).
+- **Authentication flow:** MUST be `default-authentication-flow` (NOT `default-source-authentication` — that's for IdP federation; gives `FlowNonApplicableException` + 404 on the authorize endpoint).
+- **Authorization flow:** `default-provider-authorization-implicit-consent` (or explicit).
+- **Application Launch URL** MUST be the full public URL `https://<sub>.hubris.network/` — outpost matches incoming `X-Forwarded-Host` against it.
+- Each Application MUST have at least one **policy/group/user binding** — zero bindings = outpost returns 404 on access.
+- **Restart Authentik after binding new apps to the outpost:**
+  ```
+  pct exec 124 -- docker compose -f /opt/authentik/docker-compose.yml restart server worker
+  ```
+
+Caddy snippet `(authentik)` lives at the top of `/etc/caddy/Caddyfile`. Points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid hairpin TLS round-trip stripping `X-Forwarded-Host`. Must explicitly set `header_up X-Forwarded-Host {host}` in the forward-auth block. Used by gated sites with `import authentik`.
+
+### Per-app username override pattern (Authentik)
+
+Used when the app's local user ID doesn't match the user's Authentik username (e.g., Nextcloud's `admin` ≠ Authentik's `dtoro`).
+
+1. On the Authentik user: add attribute `<app>_uid: <target_local_username>` (YAML, Directory → Users → Edit → Attributes).
+2. Customization → Property Mappings → Create → **Scope Mapping** (not SAML):
+   - Name: `<app>-uid-override`, Scope name: `<app>-uid`, Expression:
+     ```python
+     return {"nc_uid": user.attributes.get("<app>_uid", user.username)}
+     ```
+   - **Use a custom claim key** (e.g. `nc_uid`), not `preferred_username` — the default `profile` scope mapping emits `preferred_username` and will overwrite yours depending on ordering.
+3. Attach the new scope to the provider (Providers → app → Scopes).
+4. On the app side, point its OIDC UID-mapping setting at the custom claim.
+
+For Nextcloud:
+```
+occ user_oidc:provider <name> --mapping-uid=nc_uid
+occ user_oidc:provider <name> --scope="openid profile email <app>-uid"
+```
+
+### Bypass forward-auth for API paths (mobile apps)
+
+If the app has its own token-based API auth and a mobile client, API paths must bypass forward-auth — mobile apps can't follow the browser login redirect. Pattern in the Caddyfile site block:
+
+```
+paperless.hubris.network {
+    tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
+    @api path /api/*
+    handle @api {
+        reverse_proxy 192.168.8.130:8000
+    }
+    handle {
+        import authentik
+        reverse_proxy 192.168.8.130:8000
+    }
+}
+```
+
+API paths to bypass per app:
+- [Paperless](103-paperless.md): `/api/*` (Bearer)
+- [Sonarr / Radarr / Lidarr / etc.](122-arriman.md): `/api/*` (X-Api-Key)
+- [qBittorrent](122-arriman.md): `/api/*` (session cookie from `/api/v2/auth/login`)
+- [SABnzbd](122-arriman.md): `/api?*` (apikey query param) — match `/api*` for query-string APIs
+- Homarr: no mobile client
+- [Portainer](105-apps.md): mobile uses same session auth as web; no bypass typically needed
+
+### Backend trust of Authentik headers (skip the app's own login after SSO)
+
+- [**Paperless**](103-paperless.md): `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Restart `paperless-webserver paperless-task-queue paperless-scheduler paperless-consumer`. Django auto-creates matching users on first SSO login; promote to superuser via existing admin UI.
+- Apps without header-auth support: users log in twice (SSO + app login). Acceptable but degraded UX.
+
+## Per-app integration map
+
+| App                                       | Type             | Notes |
+| ----------------------------------------- | ---------------- | ----- |
+| [Paperless (103)](103-paperless.md)       | Forward-auth + REMOTE_USER | `/api/*` bypass |
+| [Nextcloud (114)](114-nextcloud.md)       | Native OIDC      | `nc_uid` override; local dnsmasq required (Guzzle bypasses `/etc/hosts`) |
+| [mulita (120)](120-mule-images.md)        | Native OIDC      | `extra_hosts` override in compose |
+| [Booklore (105)](105-apps.md)             | Native OIDC      | Redirect URI `/oauth2-callback`; `extra_hosts` |
+| [Portainer (105)](105-apps.md)            | Native OAuth2    | `portainer_uid` custom claim; `--trusted-origins` flag |
+| [WriteFreely (105)](105-apps.md)          | Native OIDC      | `[oauth.generic]` block; `extra_hosts` |
+| [qBittorrent (122)](122-arriman.md)       | Forward-auth via IP whitelist | Reverse-proxy support enabled in qBit |
+| [Artifacto (105)](105-apps.md)            | Forward-auth + gateway-secret auto-login | Public `/p/*` paths bypass |
+| [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. |
+
+## Netbird IdP integration — DEFERRED
+
+The `netbirdio/netbird-server` combined image has no config knobs for external OIDC. Verified in `combined/cmd/config.go` on main (v0.69.0): `AuthConfig` only exposes issuer + redirect URIs; `ToManagementConfig()` hardcodes `AuthAudience="netbird-dashboard"`, `UserIDClaim="sub"`, always calls `buildEmbeddedIdPConfig()`. Code comment: `"embedded IdP is always enabled in combined server"`.
+
+To wire Authentik into Netbird login/user-sync, must migrate to the legacy split stack (`netbirdio/management` + `signal` + `dashboard` + coturn) with the richer `management.json` schema. Sqlite management data should migrate but needs verification; peers stay connected via wireguard keys.
+
+**Pre-work already in place (keep for re-use):**
+- Provider `Provider for Netbird` + App `netbird`. Client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`. Redirect URIs for `https://netbird.hubris.network/{nb-auth,nb-silent-auth}` and `http://localhost:53000/`. Scopes: `openid profile email offline_access goauthentik.io/api`.
+- Service account `netbird-service` in `authentik Admins` group, non-expiring API token `netbird-service-api`.
+- Netbird mgmt host (`82.165.190.79`) is now a peer on its own mesh. See [mesh](../infrastructure/mesh.md).
+
+## DNS responsibility
+
+dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` + `127.0.0.1:53`, serving every `*.hubris.network` subdomain → `192.168.8.175`. **There is no wildcard** — every site needs an explicit `address=` entry. See [DNS split-horizon](../infrastructure/dns.md).
+
+## Related
+- [DNS split-horizon](../infrastructure/dns.md)
+- [Caddy (121)](121-caddy.md)
+- [Mesh migration](../infrastructure/mesh.md)
+- Every gated app under [containers/index](index.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-22 — Phase 6 (Netbird IdP swap) deferred
+Combined netbird-server image can't take an external IdP. Pre-work in Authentik kept for later. Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access.
+
+### 2026-04-22 — Artifacto, mulita, WriteFreely, Portainer wired
+Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto.
+
+### 2026-04-21 — deployed; Phases 1–5 complete
+LXC 124 provisioned, stack at `/opt/authentik`, public URL via Caddy, Paperless + Booklore + Nextcloud + Home Assistant wired. dnsmasq for split-horizon DNS lives on the same LXC.
--- a/containers/index.md
+++ b/containers/index.md
@@ -0,0 +1,47 @@
+# LXC containers — index
+
+All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per-container page.
+
+| ID  | Name             | IP              | Priv | Cores | RAM   | Disk  | Mounts                | Public hostname                       | Status   |
+| --- | ---------------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- |
+| 101 | [jellyfin](101-jellyfin.md)             | 192.168.8.206 | unpriv (idmap) | 2 | 4 GiB  | 16 GiB | `/mnt/library`        | `media.hubris.network`                | running  |
+| 103 | [paperless](103-paperless.md)           | 192.168.8.130 | priv | 2 | 3 GiB  | 8 GiB  | `/mnt/library`        | `paperless.hubris.network`            | running  |
+| 104 | [gitea](104-gitea.md)                   | 192.168.8.121 | priv | 1 | 1 GiB  | 8 GiB  | `/mnt/library`        | `git.hubris.network`                  | running  |
+| 105 | [apps](105-apps.md)                     | 192.168.8.205 | priv | 2 | 4 GiB  | 30 GiB | `/mnt/library`        | `docker` / `books` / `artifacto` / `blog` | running |
+| 109 | [syncthing](109-syncthing.md)           | (stopped)     | unpriv (idmap) | 2 | 2 GiB  | 8 GiB  | `/mnt/library`        | —                                     | **stopped** |
+| 114 | [nextcloud](114-nextcloud.md)           | 192.168.8.224 | priv | 4 | 6 GiB  | 25 GiB | `/mnt/library`        | `cloud.hubris.network`                | running  |
+| 116 | [heaper](116-heaper.md)                 | 192.168.8.116 | priv | 2 | 2 GiB  | 10 GiB | `/mnt/library`        | —                                     | running  |
+| 118 | [elementsynapse](118-elementsynapse.md) | 192.168.8.239 | unpriv | 1 | 2 GiB  | 8 GiB  | —                     | `matrix.hubris.network`               | running  |
+| 119 | [sophia](119-sophia.md)                 | 192.168.8.157 | priv | 2 | 1 GiB  | 10 GiB | `/mnt/library`        | —                                     | running  |
+| 120 | [mule-images](120-mule-images.md)       | 192.168.8.136 | priv | 4 | 8 GiB  | 60 GiB | `/mnt/library`        | `photos.hubris.network`               | running  |
+| 121 | [caddy](121-caddy.md)                   | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB  | —                     | (terminates all `*.hubris.network`)   | running  |
+| 122 | [arriman](122-arriman.md)               | 192.168.8.132 | priv | 4 | 8 GiB  | 24 GiB | `/mnt/library`        | `jellyseerr` / `qbit` / `sab`         | running  |
+| 123 | [claudio-bot](123-claudio-bot.md)       | 192.168.8.230 | unpriv | 1 | 512 MiB | 8 GiB | —                     | —                                     | running  |
+| 124 | [authentik](124-authentik.md)           | 192.168.8.180 | priv | 2 | 4 GiB  | 20 GiB | —                     | `auth.hubris.network`                 | running  |
+
+## Recently destroyed (kept for archaeology)
+
+| ID  | Name             | Destroyed       | Reason                                        |
+| --- | ---------------- | --------------- | --------------------------------------------- |
+| 100 | arr (yunohost)   | ~2026-04-28     | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
+| 106 | flaresolverr     | ~2026-04-28     | Folded into the arriman docker compose       |
+| 107 | marimo           | between 2026-04-21 and 2026-04-28 | Decommissioned                |
+| 110 | photoprism       | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
+| 111 | karakeep         | between 2026-04-21 and 2026-04-28 | Decommissioned                |
+| 112 | immich           | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
+| 115 | reticulum        | between 2026-04-21 and 2026-04-28 | Decommissioned                |
+
+> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
+
+## Conventions
+
+- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. IPs are stable via the LAN router's DHCP reservations.
+- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
+- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library`. Containers that don't mount `/mnt/library` don't need it.
+- Most containers are privileged. Unprivileged ones (`101`, `109`, `118`, `121`, `123`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+- [Caddy](121-caddy.md) — terminates every public hostname
+- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain
--- a/hosts/hubris.md
+++ b/hosts/hubris.md
@@ -0,0 +1,92 @@
+# `hubris` — Proxmox host
+
+Single-node Proxmox VE running 1 VM and 14 LXC containers (one of which is currently stopped). The whole homelab.
+
+## At a glance
+- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
+- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
+- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md).
+- **LAN:** `192.168.8.77/24` on bridge `vmbr0`, gateway `192.168.8.1`.
+- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
+- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
+
+## Storage
+
+| Pool         | Type      | Size  | Use                                    |
+| ------------ | --------- | ----- | -------------------------------------- |
+| `local`      | dir       | ~95G  | ISOs, templates, `/etc`, configs       |
+| `local-lvm`  | lvmthin   | 856G  | LXC/VM rootfs                          |
+| `library`    | lvmthin   | 3.7T  | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) |
+
+`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`, `syncthing`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
+
+## Tenants
+
+### VMs
+- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
+
+### LXC containers
+See [containers/index](../containers/index.md). 14 active, 1 stopped.
+
+## Boot-time tuning (load-bearing)
+
+- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
+- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
+- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
+
+## Host services owned by external repos
+
+| What                              | Repo                       | Path on host                  |
+| --------------------------------- | -------------------------- | ----------------------------- |
+| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor`   | `/opt/claudio-monitor`        |
+| `backup-library` (restic)         | `dtoro/backup-library`     | `/opt/backup-library` *(disabled)* |
+| `hubris-public-cert-sync`         | (script + systemd unit)    | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer |
+
+See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
+
+## Quirks
+
+- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install.
+- ZFS is **not** in use; storage is LVM-thin + ext4.
+- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
+- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23.
+- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
+- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
+
+## Authorized SSH keys (root)
+
+- `root@hubris` (self, RSA) — local
+- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
+
+OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
+
+## Related
+- [Containers index](../containers/index.md)
+- [DNS split-horizon](../infrastructure/dns.md)
+- [Caddy reverse proxy](../containers/121-caddy.md)
+- [Mesh / Netbird](../infrastructure/mesh.md)
+- [Media permissions](../infrastructure/media-permissions.md)
+- [Monitoring](../infrastructure/monitoring.md)
+- [Backups (disabled)](../infrastructure/backups.md)
+- [Operations cheatsheet](../operations/commands.md)
+- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+
+## Changelog
+
+### 2026-04-28 — wiki started
+This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
+
+### 2026-04-23 — SSD cooling + thermal pads installed
+Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
+
+### 2026-04-22 — drive removal A/B test
+Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
+
+### 2026-04-22 — `cpu-epp.service` ordering bug fixed
+Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
+
+### 2026-04-21 — crash-capture + RAS telemetry enabled
+`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
+
+### 2026-04-21 — `cpu-epp.service` deployed
+Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md).
--- a/infrastructure/auto-deploy.md
+++ b/infrastructure/auto-deploy.md
@@ -0,0 +1,93 @@
+# Auto-deploy — gitea-webhook pipelines
+
+Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes.
+
+## Two shapes
+
+### Shape A — checkout IS the working tree (config repos)
+
+`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap.
+
+### Shape B — receiver outside the app repo (compose stacks)
+
+The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.
+
+## Common
+
+- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`.
+- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded.
+- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently:
+  - `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
+  - `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
+  - `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
+  - `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
+  - `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
+  - `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
+
+  **Don't strip these when editing app.ini.**
+
+- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`.
+
+## Pipelines
+
+| Repo                            | Target                                       | Shape | Receiver                              | Webhook id | Reload action |
+| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- |
+| `dtoro/caddy-conf`              | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` |
+| `dtoro/gitea-customizations`    | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
+| `dtoro/mule-image`              | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
+| `dtoro/Artifacto`               | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
+| `dtoro/claudio-bot`             | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
+| `dtoro/backup-library`          | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy`  | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
+
+> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern.
+
+## When you change a tracked config
+
+Always commit + push. Local-only edits drift. Common ones:
+
+- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys)
+- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
+- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
+- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
+- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys)
+- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
+
+## Per-pipeline notes / gotchas
+
+### caddy-conf
+- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits.
+- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`).
+
+### gitea-customizations
+- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
+- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
+
+### mule-image / Artifacto
+- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
+- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
+- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`.
+- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`.
+- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated.
+
+### backup-library
+- Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
+- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
+
+## Related
+- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
+- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
+- [Backups (disabled)](backups.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation. Six active pipelines.
+
+### 2026-04-22 — Artifacto pipeline added
+Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
+
+### 2026-04-21 — mule-image + claudio-bot pipelines added
+Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
+
+### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped
+Initial three. Set the conventions everything else follows.
--- a/infrastructure/backups.md
+++ b/infrastructure/backups.md
@@ -0,0 +1,115 @@
+# Backups — restic on external drive (DISABLED)
+
+Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md).
+
+## Status
+
+**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
+- `backup-library@homecloud.timer`
+- `backup-library@images.timer`
+- `backup-library@small.timer`
+- `backup-library-check.timer`
+
+Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
+
+**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
+
+**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
+
+## Design
+
+Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.
+
+Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.
+
+## Components
+
+- **Repo:** `dtoro/backup-library`
+- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
+- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
+- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
+- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.
+
+## Mount-on-demand
+
+`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
+
+## UAS blacklist
+
+`/etc/modprobe.d/usb-storage-quirks.conf`:
+```
+options usb-storage quirks=090c:2320:u
+```
+Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.
+
+## Schedule
+
+Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
+
+| Timer                              | When           | Include list                         | Approx size |
+| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
+| `backup-library@homecloud.timer`   | day 1 / month  | `/etc/restic/include-homecloud.list` | ~315 G       |
+| `backup-library@images.timer`      | day 10 / month | `/etc/restic/include-images.list`    | ~103 G       |
+| `backup-library@small.timer`       | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |
+
+Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.
+
+Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).
+
+Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.
+
+## Thermal caps
+
+Baked into the systemd units:
+- `IOReadBandwidthMax=/mnt/library 50M`
+- `IOWriteBandwidthMax=/mnt/backup 30M`
+- `--read-concurrency=1` on restic.
+
+## Wrapper
+
+`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.
+
+## Notifications
+
+POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.
+
+`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
+
+## Recovery
+
+Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.
+
+## Known SPOF
+
+Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented.
+
+## Drive history
+
+The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
+
+After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md).
+
+## Thermal monitoring
+
+Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- [claudio-bot (123)](../containers/123-claudio-bot.md)
+- [Monitoring](monitoring.md)
+- [Auto-deploy](auto-deploy.md)
+- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation. Status remains DISABLED.
+
+### 2026-04-22 — DISABLED
+Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
+
+### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
+Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
+
+### 2026-04-20 — deployed; redesigned for thermal-gentleness
+Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.
--- a/infrastructure/dns.md
+++ b/infrastructure/dns.md
@@ -0,0 +1,86 @@
+# DNS — split-horizon `*.hubris.network`
+
+LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).
+
+There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.
+
+## Components
+
+- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
+- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md), `192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`).
+- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
+- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
+
+## Live entries (as of 2026-04-28)
+
+```
+address=/auth.hubris.network/192.168.8.175
+address=/git.hubris.network/192.168.8.175
+address=/media.hubris.network/192.168.8.175
+address=/paperless.hubris.network/192.168.8.175
+address=/books.hubris.network/192.168.8.175
+address=/home.hubris.network/192.168.8.175
+address=/cloud.hubris.network/192.168.8.175
+address=/matrix.hubris.network/192.168.8.175
+address=/proxmox.hubris.network/192.168.8.175
+address=/docker.hubris.network/192.168.8.175
+address=/jellyseerr.hubris.network/192.168.8.175
+address=/qbit.hubris.network/192.168.8.175
+address=/sab.hubris.network/192.168.8.175
+address=/blog.hubris.network/192.168.8.175
+address=/photos.hubris.network/192.168.8.175
+address=/artifacto.hubris.network/192.168.8.175
+```
+
+## Why split-horizon
+
+The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy.
+
+## The gotcha that cost a debug session (2026-04-22)
+
+Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
+
+## Recipe — adding a new subdomain
+
+1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
+2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md):
+   ```
+   address=/<new>.hubris.network/192.168.8.175
+   ```
+3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path.
+4. Verify: `dig @192.168.8.180 +short <new>.hubris.network` → `192.168.8.175`.
+5. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
+
+> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124).
+
+## Public path — what does and doesn't follow the LAN map
+
+- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
+- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
+- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.
+
+## Long-term plan
+
+Either:
+- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
+- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.
+
+## Related
+- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
+- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
+- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
+- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation. 16 active entries.
+
+### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
+Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.
+
+### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
+Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.
+
+### 2026-04-21 — dnsmasq stood up on LXC 124
+Co-located with Authentik. Initial entries cover everything routed through Caddy.
--- a/infrastructure/ingress.md
+++ b/infrastructure/ingress.md
@@ -0,0 +1,84 @@
+# Public ingress — VPS traefik + cert mirror
+
+How home services reach the open internet without exposing the home network. Two-stage pattern: traefik on the IONOS VPS terminates TLS at the public edge, then reverse-proxies over the netbird mesh to home Caddy / direct backends.
+
+## The shape
+
+```
+Public client
+    │  *.hubris.network → 82.165.190.79 (IONOS wildcard)
+    ▼
+[VPS traefik]   priority 1: HostSNI(*) → netbird-proxy:8443  ← netbird control plane
+                priority 10: per-host HTTP routers           ← home services
+    │  HTTP over netbird mesh
+    ▼
+[Home backend on 192.168.8.x]
+```
+
+Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
+
+## Why this shape
+
+- VPS traefik already has a HostSNI(*) TCP passthrough at priority 1 so netbird's own ingress (`netbird.hubris.network`, future `*.proxy.hubris.network`) is unaffected.
+- Per-hostname HTTP file-provider routers at priority 10 win over the passthrough for the listed hosts and let traefik terminate TLS itself for those.
+- Traefik's own ACME (`letsencrypt` resolver) fails on this box: HostSNI(*) grabs TLS-ALPN-01 challenges before traefik's `allowACMEByPass=true` can respond. Solution: home Caddy obtains certs via IONOS DNS-01 (no conflict) and the VPS *mirrors* the result over.
+
+## Components
+
+### On the VPS (`82.165.190.79`)
+
+- `/opt/traefik-dynamic.yaml` — file-watched dynamic config. One `http.routers.<name>-public` + one `http.services.<name>-public` per exposed service, plus one entry in the top-level `tls.certificates` list per hostname.
+- Let's-Encrypt-effective directory: `/letsencrypt/` inside the traefik container, backed by the host-side docker volume `opt_netbird_traefik_letsencrypt`.
+- Backups before edits: `cp /opt/traefik-dynamic.yaml /opt/traefik-dynamic.yaml.bak.$(date +%s)` — several bak files live alongside.
+
+### On the PVE host (`192.168.8.77`)
+
+- `/usr/local/bin/hubris-public-cert-sync.sh` — runs daily via `hubris-public-cert-sync.timer`. Maps source hostname → VPS cert filenames in a bash assoc array. For each mapping: `pct pull` cert+key from [Caddy (121)](../containers/121-caddy.md)'s store, diff against the VPS copy, scp only on change.
+- Filenames are stable per host so the dynamic.yaml never needs editing on renewal — traefik file-watches and hot-reloads the cert.
+
+## Services currently exposed
+
+| Hostname                       | Path scope                       | Backend                          | Middlewares                                  | Cert files on VPS                          |
+| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
+| `artifacto.hubris.network`     | `/p/*`, `/static/*`, `/healthz`  | `192.168.8.205:3100`             | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key`           |
+| `blog.hubris.network`          | whole host                       | `192.168.8.205:8080`             | `blog-ratelimit` (100 rps / 200 burst)       | `blog.fullchain.crt` / `blog.privkey.key` |
+
+`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
+
+## Recipe — exposing another service
+
+1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
+2. Add an entry to `HOSTS` in `/usr/local/bin/hubris-public-cert-sync.sh` mapping the hostname → VPS filenames. Run once: `systemctl start hubris-public-cert-sync.service`. Confirm the cert landed.
+3. Edit `/opt/traefik-dynamic.yaml` on the VPS:
+   - Add to `tls.certificates`: paths `/letsencrypt/<crt>` and `/letsencrypt/<key>`.
+   - Add `http.routers.<host>-public`: `rule: 'Host(\`<host>\`)'` (or with path matchers if scope-gating), `entryPoints: [websecure]`, `priority: 10`, `tls: {}`, `service: <host>-public`, `middlewares: [...]`.
+   - Add a ratelimit middleware under `http.middlewares` if wanted.
+   - Add `http.services.<host>-public.loadBalancer.servers[0].url: 'http://<lan-ip>:<port>'`.
+4. Verify:
+   ```
+   ssh root@100.122.165.149 'curl -skI --resolve <host>:443:127.0.0.1 https://<host>/'   # 2xx/3xx
+   curl -skI --resolve <host>:443:<vps-public-ip> https://<host>/                          # same
+   ```
+5. **No DNS edit needed** — the IONOS wildcard already points at the VPS.
+
+## What does NOT follow this pattern
+
+- `netbird.hubris.network` (and any future `*.proxy.hubris.network`) uses the netbird-proxy / HostSNI passthrough path. Netbird handles its own cert via ACME cleanly because it *is* the passthrough target.
+
+## Related
+- [DNS split-horizon](dns.md)
+- [Caddy (121)](../containers/121-caddy.md) — cert source, internal counterpart
+- [Mesh migration](mesh.md) — netbird is the transport between VPS and home
+- [VPS hardening](vps-hardening.md) — fail2ban / nftables that the access logs feed
+- [Artifacto on apps (105)](../containers/105-apps.md) — first publicly-exposed service
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-23 — `blog.hubris.network` exposed
+WriteFreely on [apps (105)](../containers/105-apps.md). Whole host is public.
+
+### 2026-04-22 — pattern established with Artifacto
+First service through the file-provider router. IONOS wildcard moved to the VPS this day. Cert mirror script + timer deployed on the PVE host.
--- a/infrastructure/media-permissions.md
+++ b/infrastructure/media-permissions.md
@@ -0,0 +1,95 @@
+# Media permissions — `media` GID 10000
+
+Standard for any LXC reading/writing `/mnt/library` on [hubris](../hosts/hubris.md). Applied 2026-04-20.
+
+## Standard
+
+Every LXC that mounts `/mnt/library` participates in a shared `media` group with **GID 10000**. Shared subtrees are owned by that group with the setgid bit (`drwxrwsr-x`, mode `2775`), so new files auto-inherit the right group regardless of which container wrote them.
+
+## Why
+
+`/mnt/library` is a cross-container storage pool. \*arr writes, jellyfin reads, syncthing syncs, photoprism / immich / mulita scan, paperless ingests. Without a shared group, each container sees files as `nobody:nogroup` (unprivileged) or `www-data` (privileged 1:1) and the permission web collapses into one-off chmods. GID 10000 bridges privileged and unprivileged containers.
+
+## Onboarding a new LXC
+
+1. `pct set <id> -mp0 /mnt/library,mp=/mnt/library` (if not already mounted).
+2. Inside the container:
+   ```
+   groupadd -g 10000 media
+   usermod -aG media <service-user>   # for every user that needs library access
+   ```
+3. If the container is **unprivileged** (check `pct config <id> | grep unprivileged`), append this idmap block to `/etc/pve/lxc/<id>.conf` (back up first):
+   ```
+   lxc.idmap: u 0 100000 65536
+   lxc.idmap: g 0 100000 10000
+   lxc.idmap: g 10000 10000 1
+   lxc.idmap: g 10001 110001 55535
+   ```
+   Then `pct stop <id> && pct start <id>`.
+4. For systemd services running with `User=root` (not typical — but [syncthing (109)](../containers/109-syncthing.md) is one), add a drop-in with `SupplementaryGroups=media`. Systemd skips `initgroups()` for `User=root`.
+5. `pct exec` sessions don't get supplementary groups (no initgroups). Use `sudo -i` or `su - <user>` inside the container to verify membership interactively. Real services use `initgroups` and work correctly.
+
+## State snapshot
+
+### Host
+
+- Group `media` GID 10000 exists.
+- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
+- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
+  - `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
+  - `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / booklore
+  - `downloads` — \*arr stack output
+  - `images` — photoprism / immich / mulita
+  - `roms` — emu frontends
+  - `syncthing` — cross-device sync
+- Container-specific subtrees intentionally **not** migrated (keep their own owner:group):
+  - `documents` (paperless, `www-data:www-data 750`)
+  - `homecloud` (nextcloud — its own permission model, easy to break)
+  - `marimo` (marimo venv) — *LXC since destroyed; review whether subtree still serves a purpose*
+  - `notes`, `heaper`, `sophia` (single-container use)
+  - `repos` (owner UID 102 GID 105 from inside [gitea](../containers/104-gitea.md) — don't touch)
+
+### LXCs with media-group membership
+
+| ID  | Name                                          | Priv | Media-group members                           |
+| --- | --------------------------------------------- | ---- | --------------------------------------------- |
+| 101 | [jellyfin](../containers/101-jellyfin.md)     | **unpriv + idmap** | jellyfin                          |
+| 103 | [paperless](../containers/103-paperless.md)   | priv | www-data                                      |
+| 104 | [gitea](../containers/104-gitea.md)           | priv | www-data, gitea                               |
+| 105 | [apps](../containers/105-apps.md)             | priv | www-data                                      |
+| 109 | [syncthing](../containers/109-syncthing.md)   | **unpriv + idmap** | root (via `SupplementaryGroups=media` drop-in on `syncthing@.service`) |
+| 114 | [nextcloud](../containers/114-nextcloud.md)   | priv | www-data                                      |
+| 116 | [heaper](../containers/116-heaper.md)         | priv | www-data                                      |
+| 119 | [sophia](../containers/119-sophia.md)         | priv | www-data                                      |
+| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data                                    |
+| 122 | [arriman](../containers/122-arriman.md)       | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
+
+> Some entries from the older snapshot — 100 (arr-yunohost), 107 (marimo), 110 (photoprism), 112 (immich) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
+
+Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*`.
+
+## Gotchas
+
+- **[apps (105)](../containers/105-apps.md) is a Docker host.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000` or `user: "<uid>:10000"` in compose. Booklore, audiobookshelf-in-docker, etc. need this per-container.
+- **`pct exec` does NOT run initgroups.** So `pct exec 109 -- id` shows only the primary group. For interactive verification, use `pct exec 109 -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
+- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
+- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.
+- **`/etc/subgid`** must retain both `root:100000:65536` AND `root:10000:1`. Dropping the second breaks startup of any unprivileged LXC with the idmap block.
+- **\*arr "Set Permissions" options** can override the setgid inheritance by explicitly chown'ing files. Leave those off, or set the group to `media`. Relevant to Sonarr/Radarr/qBittorrent on [arriman (122)](../containers/122-arriman.md).
+- **Nextcloud** files under `/mnt/library/homecloud` are deliberately NOT in the media group. NC manages its own permission model. See [nextcloud (114)](../containers/114-nextcloud.md).
+- **\*arr stack on arriman** required `MEDIACENTER_GID=10000` (not 13000) in `.env` because s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. See [arriman (122)](../containers/122-arriman.md#changelog).
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- All container pages list whether they're in the standard
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-26 — `MEDIACENTER_GID` fix on [arriman (122)](../containers/122-arriman.md)
+qBit was erroring every torrent with "Permission denied" because `MEDIACENTER_GID=13000` was set as a supplementary GID via `group_add:`. Changed to 10000 (primary GID); fix described above is now standard.
+
+### 2026-04-20 — standard rolled out
+GID 10000 hostgroup, idmap blocks for unprivileged LXCs, setgid 2775 on shared subtrees, `media` membership for service users in every participating LXC.
--- a/infrastructure/mesh.md
+++ b/infrastructure/mesh.md
@@ -0,0 +1,94 @@
+# Mesh — Tailscale → Netbird migration
+
+The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target end-state. In-progress as of 2026-04-21.
+
+## Current state
+
+- **PVE host** uses Netbird (`wt0`, `100.122.38.109/16`). Its resolver is the local netbird daemon, which forwards to LAN/upstream — so the PVE host gets `*.hubris.network → 192.168.8.175` via the system resolver chain.
+- **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
+- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
+
+## Consequence — every LXC wired to Authentik needs an internal override
+
+Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`.
+
+Two techniques. Pick by HTTP-client behavior.
+
+### A) `/etc/hosts` override
+
+Works for libc `getaddrinfo` clients: curl, wget, most Go/Python/Ruby apps, gitea.
+
+- Place the line **outside** the `# --- BEGIN PVE ---` / `# --- END PVE ---` markers. Proxmox rewrites everything inside that block on every container start.
+- Belt-and-suspenders: `/etc/systemd/system/hubris-hosts-override.service` (oneshot, enabled, idempotent).
+
+### B) Local dnsmasq
+
+Required for clients that bypass `/etc/hosts`. **Nextcloud (PHP Guzzle + `OC\Http\Client\DnsPinMiddleware`) is one** — uses `dns_get_record()`, not `getaddrinfo`. Likely candidates for the same treatment: Python/Ruby/Java apps that do explicit DNS resolution before connecting.
+
+Recipe:
+```
+apt install dnsmasq
+
+cat > /etc/dnsmasq.d/hubris-internal.conf <<EOF
+address=/auth.hubris.network/192.168.8.175
+server=192.168.8.1
+server=1.1.1.1
+interface=lo
+bind-interfaces
+no-hosts
+no-resolv
+EOF
+
+# Set LXC default nameservers and live resolv.conf
+pct set <id> --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"
+# Then update /etc/resolv.conf inside the LXC too.
+```
+
+### Known overrides applied
+
+| LXC                                        | Technique                                | Notes |
+| ------------------------------------------ | ---------------------------------------- | ----- |
+| [104 (gitea)](../containers/104-gitea.md)  | `/etc/hosts` + `hubris-hosts-override.service` | Standard |
+| [114 (nextcloud)](../containers/114-nextcloud.md) | local dnsmasq @ 127.0.0.1 + resolv.conf reorder | Guzzle bypasses /etc/hosts |
+| [105 (apps)](../containers/105-apps.md), inner containers | `extra_hosts:` in compose | Booklore, mulita, WriteFreely each ship with this |
+
+## Adding new LXCs
+
+- Don't add new LXCs to Tailscale; add them to Netbird. Tailscale is being decommissioned on hubris.
+- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. If nameserver is `192.168.8.1` or `100.100.100.100`, add the hosts override. If it's the netbird daemon IP, skip.
+
+## Long-term fix
+
+Either:
+- Split-horizon DNS at LAN/router level so `*.hubris.network → 192.168.8.175` for every LAN client. Eliminates all per-LXC overrides.
+- Once Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain on the PVE host.
+
+## CRITICAL — never `docker compose up` Portainer-managed stacks
+
+[apps (105)](../containers/105-apps.md) runs multiple stacks deployed via the Portainer UI (`/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`). Running `docker compose up -d <svc>` from the host shell on these triggers a recreate of OTHER services in the stack — labels drift, compose treats the project as reconciling, containers get destroyed and remade. **This wiped Booklore's mariadb data on 2026-04-22** (bind mount `./mariadb/config` re-initialized fresh).
+
+Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-managed stacks:
+1. Edit the compose in Portainer UI → **Stacks → <stack> → Editor → Update stack**. Portainer handles the recreate cleanly with its own state tracking.
+2. Do NOT edit `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/docker-compose.yml` directly.
+3. For data-preserving DNS overrides on stateful services, prefer Caddy-side handling (forward-auth) or the app's built-in OIDC over container-level `extra_hosts` when possible.
+
+## Related
+- [DNS split-horizon](dns.md)
+- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides
+- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
+- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
+- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-22 — Booklore mariadb data wiped (lesson recorded)
+The "never docker compose up Portainer-managed stacks" rule comes from this. See [apps (105)](../containers/105-apps.md#changelog).
+
+### 2026-04-22 — netbird mgmt host joined its own mesh
+`82.165.190.79` is now a peer (`100.122.165.149`). LAN access + split-horizon DNS via PVE peer. See [VPS hardening](vps-hardening.md).
+
+### 2026-04-21 — overrides applied to gitea (104) and nextcloud (114)
+Two techniques documented; Nextcloud forced the dnsmasq route because of Guzzle.
--- a/infrastructure/monitoring.md
+++ b/infrastructure/monitoring.md
@@ -0,0 +1,75 @@
+# Monitoring — `claudio-monitor`
+
+Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
+
+## Why
+
+After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
+
+## Components
+
+- **Repo:** `dtoro/claudio-monitor`
+- **Checkout:** `/opt/claudio-monitor` on the PVE host
+- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
+- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
+- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
+- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
+- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
+- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
+
+## Per-LXC data
+
+A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
+
+## Matrix-side plugin
+
+`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
+- `/monitor status`
+- `/monitor history [N]`
+- `/monitor clear <key>`
+
+Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
+
+Events posted to claudio-bot `/notify`:
+```json
+{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
+```
+`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
+
+## Current thresholds & active alerts
+
+Active warnings at the time monitor was deployed (2026-04-21):
+- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
+- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
+
+## Testing
+
+- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
+- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
+- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
+
+## Not yet deployed (tracked in plan file)
+
+- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
+- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
+
+## Related repos
+
+- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
+- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
+- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
+
+## Related pages
+- [Hubris host](../hosts/hubris.md)
+- [claudio-bot (123)](../containers/123-claudio-bot.md)
+- [HAOS VM (108)](../vms/108-haos.md)
+- [Backups (disabled)](backups.md)
+- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
+General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.
--- a/infrastructure/ssh-access.md
+++ b/infrastructure/ssh-access.md
@@ -0,0 +1,45 @@
+# SSH access
+
+How to reach hubris and the VPS over SSH, and the dual-server gotcha.
+
+## Hubris
+
+Two SSH endpoints — easy to hit the wrong one.
+
+| Server         | Listen                       | Auth                              | Notes |
+| -------------- | ---------------------------- | --------------------------------- | ----- |
+| OpenSSH        | `0.0.0.0:22`                 | `authorized_keys` at `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced; symlinked from `/root/.ssh/authorized_keys`) | Standard. |
+| Netbird SSH    | `100.122.38.109:22022`       | OIDC / browser auth — bypasses `authorized_keys` | If a client lands here it'll open a browser tab to authenticate, then sometimes hang. Force port 22 or use the LAN IP. |
+
+### Authorized root keys
+- `root@hubris` (self, RSA) — original.
+- `d.toro.v@pm.me` (ed25519) — user's iMac (`mac-mini.netbird.selfhosted`, LAN `192.168.8.174`), added 2026-04-22.
+
+### Notes
+- Password auth is enabled on hubris but the root password is **not** the one the user expects. Prefer key flows; don't try `ssh-copy-id` blind.
+- Off-LAN access from the iMac uses the LAN path. As of 2026-04-22 the iMac's Netbird tunnel to hubris was P2P healthy but no packets were captured on `wt0`; needs revisit if remote access becomes critical.
+
+## VPS (`82.165.190.79` / `100.122.165.149`)
+
+- **Mesh-only.** Public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`.
+- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
+- `ListenAddress` is still 0.0.0.0; gating is firewall-layer.
+- Authorized root keys: PVE (`root@hubris`), iMac (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer before disabling its access paths.
+
+See [VPS hardening](vps-hardening.md) for the firewall + fail2ban rules and recovery paths.
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- [Mesh migration](mesh.md)
+- [VPS hardening](vps-hardening.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-23 — VPS SSH hardened to mesh-only
+Public `:22` blocked at nftables. Key-only sshd. See [VPS hardening](vps-hardening.md).
+
+### 2026-04-22 — iMac key authorized on hubris
+`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.
--- a/infrastructure/vps-hardening.md
+++ b/infrastructure/vps-hardening.md
@@ -0,0 +1,84 @@
+# VPS hardening — `82.165.190.79` / `100.122.165.149`
+
+IONOS VPS that runs the Netbird control plane and the [public ingress traefik](ingress.md). Hardened 2026-04-23 from its stock-Plesk state.
+
+## At a glance
+- **Hostname:** `inspiring-ramanujan.82-165-190-79.plesk.page`
+- **OS:** Debian 13
+- **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)).
+- **Public:** `82.165.190.79` (`ens6`).
+- **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`.
+
+## SSH
+
+- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
+- **Mesh-only**: public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. `ListenAddress` itself is still `0.0.0.0` — gating is firewall-layer.
+- Authorized root keys: PVE (`root@hubris`), Mac Mini (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer **before** disabling its access paths.
+
+## Firewall — nftables (`inet hubris-fw`)
+
+Config at `/etc/nftables.conf`, service enabled.
+
+- Public iface `ens6`. Wireguard iface `wt0`.
+- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6. Everything else drops.
+- `wt0` fully accepted in INPUT. `lo` accepted.
+- **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below.
+- Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops.
+- Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too.
+
+## fail2ban
+
+- **Jail `traefik-4xx`** tails `/var/log/traefik/access.log` (bind-mounted from container). Filter at `/etc/fail2ban/filter.d/traefik-4xx.conf` matches 401/403/404/429 from `blog-public@file` or `artifacto-public@file` routers only — netbird-grpc traffic isn't considered.
+- Tunables: `findtime=600, maxretry=30, bantime=3600`.
+- **Action** at `/etc/fail2ban/action.d/nft-hubris.conf` adds/removes elements from `inet hubris-fw banned4` with per-element timeout.
+
+### CRITICAL invariant — wireguard / fail2ban
+
+**Bans must never affect `wt0` or wireguard UDP.** The FORWARD chain explicitly `accept`s the following *before* the ban check:
+- `udp 51820` (wireguard)
+- `udp 3478` (STUN)
+- `ct state established,related`
+
+The INPUT ban rule is scoped to `iifname "ens6"`.
+
+Violating this takes the mesh down for every home device (they share one public IP) and the only recovery is IONOS console → `nft flush set inet hubris-fw banned4`.
+
+## Traefik access log
+
+- Written to `/var/log/traefik/access.log` on the host via a bind mount added to `/opt/docker-compose.yml` (traefik volumes include `/var/log/traefik:/logs`) plus `--accesslog.filepath=/logs/access.log`.
+- CLF format. Real client IP arrives correctly because docker userland-proxy is off — see [public ingress](ingress.md).
+
+## Plesk / mail / FTP / Dr.Web
+
+Stopped and disabled (not uninstalled). All of:
+`dovecot`, `dovecot.socket`, `postfix`, `postfix@-`, `pc-remote`, `xinetd`, `plesk-task-manager`, `plesk-web-socket`, `sw-cp-server`, `sw-engine`, `plesk-repaird`, `plesk-repaird.socket`, `drwebd`.
+
+`psa.service` is masked (was a one-shot boot bootstrap). `/etc/cron.d/plesk-backup-manager-task` renamed to `.disabled`.
+
+Reverse: `systemctl unmask psa; systemctl enable --now <svc>`.
+
+## Auto-patching
+
+- `unattended-upgrades` enabled (stock).
+- Drop-in at `/etc/apt/apt.conf.d/52hubris-reboot.conf` sets auto-reboot at **04:00 UTC** when `/var/run/reboot-required` is set.
+- Runs inside the stock `apt-daily-upgrade.timer`.
+
+## Recovery paths
+
+Ordered by preference:
+1. **SSH via mesh** — primary. Any mesh peer with an authorized key.
+2. **IONOS web console** (my.ionos.com → VPS → Console) — uses the system password, not SSH keys. Bypasses any firewall misconfig.
+3. **IONOS rescue mode** — boot rescue, mount rootfs, edit `/etc/nftables.conf` or `/etc/ssh/sshd_config.d/10-hubris-hardening.conf` to a known-good state, reboot.
+
+## Related
+- [Public ingress (VPS traefik)](ingress.md)
+- [Mesh migration](mesh.md) — VPS as a mesh peer
+- [SSH access](ssh-access.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-23 — hardened
+nftables firewall, mesh-only SSH, fail2ban traefik jail, Plesk disabled, auto-reboot 04:00 UTC, wireguard/fail2ban invariant established.
--- a/investigations/2026-04-21-hubris-crash-loop.md
+++ b/investigations/2026-04-21-hubris-crash-loop.md
@@ -0,0 +1,86 @@
+# 2026-04-21 — Hubris crash loop (thermal + USB drive)
+
+## Summary
+
+[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
+
+## Timeline
+
+### 2026-04-19 — drive attached
+External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
+
+### 2026-04-19 → 2026-04-21 — first crashes
+Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
+
+### 2026-04-21 — crash cadence escalates
+23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once `cpu-epp.service` started applying `EPP=balance_power`.
+
+### 2026-04-21 — first set of mitigations applied
+
+- **`cpu-epp.service`** deployed. Sets `scaling_governor=powersave` + EPP=`balance_power` at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle.
+- **Crash capture**: `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`).
+- **`rasdaemon`** installed (Debian pkg; mcelog is retired) — logs MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query `ras-mc-ctl --summary`.
+- **UAS blacklist** for the drive: `/etc/modprobe.d/usb-storage-quirks.conf` → `options usb-storage quirks=090c:2320:u`. Forces BOT instead of UAS for the SMI bridge.
+- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
+
+### 2026-04-22 — recurrence after 30 h 37 m
+Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
+
+### 2026-04-22 — `cpu-epp.service` design bug fixed
+Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
+
+### 2026-04-22 — drive removed (A/B test)
+User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
+
+### 2026-04-23 — SSD cooling + thermal pads installed
+Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
+
+#### 2026-04-23 thermal-pad verdict
+Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. **The load-bearing signal is the lifetime warning-time counter, not absolute sensor1** — watch that going forward, not sensor1 threshold.
+
+### 2026-04-28 — A/B test passing so far
+3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established.
+
+## Root cause
+
+Two-contributor:
+1. **Thermal**: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under `performance` governor. Mitigated by `cpu-epp.service`. In `amd-pstate=active` mode the governor MUST be `powersave` for EPP values to take effect — `performance` silently ignores them.
+2. **Storage**: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far).
+
+Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so `fwupdmgr` can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+.
+
+## Diagnostic notes
+
+- **Pure hardware hangs leave no trace.** `pstore` stays empty across reboots despite `panic_on_oops/hardlockup/softlockup/rcu` + softdog `soft_panic=1` being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic.
+- **First check** if hubris crash-loops again with the same silent-cutoff signature: `cpu-epp.service` is active and `/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` is `balance_power`.
+- **Non-ECC RAM means memory faults are silent.** If `cpu-epp` is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out `pcie_aspm=off pci=nomsi` workarounds in `/etc/default/grub`.
+- **Grub escalation** (`pcie_aspm=off pcie_port_pm=off pci=nomsi`) **deliberately NOT applied** yet to keep the drive-removal test clean. Enable only if crashes continue without the drive.
+- `mcelog` is not in Debian 13 repos; `rasdaemon` is the replacement.
+
+## BIOS update path
+
+Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a `.exe` flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email `service@gmktec.com` with the S/N to confirm a newer BIOS / AGESA level actually exists. `fwupd` is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically.
+
+## Mitigations applied
+
+| Mitigation                                | Status   | Notes                                                |
+| ----------------------------------------- | -------- | ---------------------------------------------------- |
+| `cpu-epp.service` (powersave + balance_power) | Active   | Unit ordering fixed 2026-04-22 to run before guests |
+| Crash capture (sysctl + softdog + watchdog) | Active   | Catches everything but pure silicon hangs           |
+| `rasdaemon`                               | Active   | Persistent telemetry for MCE / memory / PCIe AER    |
+| UAS blacklist for `090c:2320`             | Active (drive currently absent) | `/etc/modprobe.d/usb-storage-quirks.conf` |
+| Mount-on-demand (`backup-usb.sh`)         | Active (drive currently absent) | Backup unit `ExecStartPre`/`ExecStopPost` hooks |
+| Drive physically removed                  | Active   | A/B test in progress                                 |
+| Thermal pads on NVMes                     | Active 2026-04-23 | Steady-state composite 47 °C / sensor1 60 °C |
+| `pcie_aspm=off pci=nomsi`                 | NOT applied | Reserved for if crashes recur without the drive    |
+
+## Affected nodes
+- [Hubris host](../hosts/hubris.md)
+- [Backups (disabled)](../infrastructure/backups.md)
+- [Monitoring](../infrastructure/monitoring.md)
+
+## Open questions
+- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
+- If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?)
+- Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation.
+- Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed.
--- a/investigations/index.md
+++ b/investigations/index.md
@@ -0,0 +1,16 @@
+# Investigations
+
+Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
+
+## Index
+
+| Date         | Title                                                              | Status        |
+| ------------ | ------------------------------------------------------------------ | ------------- |
+| 2026-04-21   | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) |
+
+## Conventions
+
+- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
+- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
+- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
+- Link back from every node's changelog that's affected.
--- a/operations/commands.md
+++ b/operations/commands.md
@@ -0,0 +1,56 @@
+# Operations cheatsheet
+
+Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`.
+
+## Proxmox CLI
+
+| Command | Use |
+| --- | --- |
+| `pct list` / `qm list` | List LXC containers / VMs |
+| `pct config <id>` / `qm config <id>` | Container / VM config |
+| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../infrastructure/media-permissions.md)) |
+| `pct enter <id>` | Shell into a container |
+| `pct start <id>` / `pct stop <id>` | Boot / halt a container |
+| `pvesm status` | Storage pools status |
+| `pvesh get /nodes --output-format json` | Node summary as JSON |
+| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status |
+| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by [claudio-monitor](../infrastructure/monitoring.md)) |
+| `pveversion` | PVE version |
+| `journalctl -u pve-cluster -n 100` | PVE service logs |
+
+## Storage
+
+- Shared mount: `/mnt/library` (ext4 on lvmthin `library`).
+- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data`
+- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../infrastructure/media-permissions.md) for the GID-10000 onboarding recipe.
+
+## Reverse proxy
+
+- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). Tracked in `dtoro/caddy-conf`. Edits should be committed + pushed — see [auto-deploy](../infrastructure/auto-deploy.md).
+- Hot reload: `pct exec 121 -- systemctl reload caddy`.
+- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`.
+
+## DNS
+
+- Split-horizon entries: `/etc/dnsmasq.d/hubris-split.conf` on [LXC 124](../containers/124-authentik.md). Hard restart on edit: `pct exec 124 -- systemctl restart dnsmasq`. SIGHUP isn't reliable.
+- Verify: `dig @192.168.8.180 +short <host>.hubris.network`.
+- See [DNS](../infrastructure/dns.md).
+
+## Web access
+
+- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI
+
+## Telemetry quick checks
+
+- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot
+- `ras-mc-ctl --errors` — full event log
+- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power`
+- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave`
+- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md))
+
+## Related
+- [Hubris host](../hosts/hubris.md)
+- [Containers index](../containers/index.md)
+- [DNS](../infrastructure/dns.md)
+- [Monitoring](../infrastructure/monitoring.md)
+- [Auto-deploy](../infrastructure/auto-deploy.md)
--- a/vms/108-haos.md
+++ b/vms/108-haos.md
@@ -0,0 +1,44 @@
+# 108 — `haos-16.3`
+
+Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC, hence the qm tenant).
+
+## At a glance
+- **Type:** QEMU VM
+- **HAOS version:** 16.3 (last verified)
+- **IP:** `192.168.8.101`
+- **Resources:** 4 GiB RAM, 32 GiB boot disk
+- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123`
+
+## Auth
+
+Native OIDC via the HACS integration `christiaangoossens/hass-oidc-auth` (repo `https://github.com/christiaangoossens/hass-oidc-auth`).
+
+Key gotchas:
+- HAOS containers don't honor the Network-panel DNS. Set Supervisor DNS via:
+  ```
+  ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
+  ```
+  so OIDC discovery resolves internally to [authentik (124)](124-authentik.md).
+- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
+- YAML config:
+  - `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
+  - `features.default_redirect: true` — skip the welcome-splash so users land on the normal HA login page.
+- Run `ha core restart` after each config change — HA caches DNS and OIDC discovery across frontend reloads.
+
+## Telemetry
+
+HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
+
+## Related
+- [Authentik (124)](124-authentik.md)
+- [Caddy (121)](121-caddy.md)
+- [DNS](../infrastructure/dns.md)
+- [Monitoring](../infrastructure/monitoring.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-21 — wired into Authentik via HACS hass-oidc-auth
+Supervisor DNS pointed at LXC 124 dnsmasq; YAML features for auto-linking + default redirect.