commit f5cd320433999082ddf2a0b2ea3aaf267978e619 Author: dtoro Date: Tue Apr 28 22:53:09 2026 +0200 Bootstrap Homelab-Docs wiki Initial documentation of the hubris Proxmox homelab as a cross-linked markdown wiki. Per-node pages, cross-cutting infrastructure pages, an investigation log, and an operations cheatsheet. Each node and topic ends with a Changelog section so changes can be tracked in-place going forward. Refreshed against live state on 2026-04-28 — 14 active LXCs (109 syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test state of the 2026-04-21 hubris crash-loop investigation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..30fb393 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,98 @@ +# Contributing to the Homelab Wiki + +## Voice + +Concise, technical, sysadmin-to-sysadmin. No marketing prose, no exclamation marks. + +## Page templates + +### Container page (`containers/-.md`) + +```markdown +# — `` + +One-sentence purpose. + +## At a glance +- **Hostname:** `` +- **IP:** `192.168.8.x` +- **Privilege:** privileged | unprivileged +- **Resources:** N cores / M GiB RAM / D GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` (if any) +- **Public hostname:** `.hubris.network` (if proxied) + +## Role +What it does, what it talks to. + +## Service / port map +| Service | Listen | Notes | + +## Storage / config paths + +## Auto-deploy +(if any) — link to [auto-deploy](../infrastructure/auto-deploy.md) + +## Related +- [Caddy](121-caddy.md) (if proxied) +- [DNS](../infrastructure/dns.md) (if has subdomain) +- [Authentik](124-authentik.md) (if SSO) +- ... + +## Changelog +### YYYY-MM-DD — short title +What changed, why, link to investigation if any. +``` + +### Cross-cutting page (`infrastructure/.md`) + +```markdown +# + +One-sentence summary. + +## Why +Design rationale — what it replaces, what it solves. + +## Components +Where it runs, what files matter. + +## How to apply / use +Recipes. + +## Gotchas + +## Related +Links to nodes that host or depend on this. + +## Changelog +``` + +### Investigation (`investigations/YYYY-MM-DD-slug.md`) + +```markdown +# YYYY-MM-DD — + +## Summary +1-3 sentences. + +## Timeline + +## Root cause + +## Mitigations applied + +## Open questions +``` + +## Linking discipline + +- Every container page links to every cross-cutting page it participates in. +- Every cross-cutting page lists the nodes that participate. +- Every investigation links to the nodes it implicates *and* gets back-linked from each node's changelog. + +## Changelog hygiene + +- Reverse-chronological (newest first). +- One entry per discrete change, even if you make several in one day. +- If a change spans nodes, repeat the entry on each affected page (different perspective is fine). +- Don't rewrite history — entries are append-only. Mistakes get a follow-up entry that supersedes them. diff --git a/README.md b/README.md new file mode 100644 index 0000000..cf8a5ac --- /dev/null +++ b/README.md @@ -0,0 +1,74 @@ +# Homelab Wiki — `hubris` + +Living documentation for the **hubris** Proxmox homelab. Every node, every cross-cutting system, and every meaningful incident is its own page; pages are linked so you can start anywhere and walk the graph. + +> Last refreshed against live state: **2026-04-28**. + +## Map + +### Hosts +- [`hubris`](hosts/hubris.md) — single Proxmox VE node, GMKtec NucBox M6 Ultra, `192.168.8.77` + +### VMs +- [108 — `haos-16.3`](vms/108-haos.md) — Home Assistant OS + +### LXC containers +See the full table in [`containers/index.md`](containers/index.md). Quick links: + +| ID | Name | IP | Role | +| --- | ---------------- | --------------- | --------------------------------------------- | +| 101 | [jellyfin](containers/101-jellyfin.md) | 192.168.8.206 | Media server | +| 103 | [paperless](containers/103-paperless.md) | 192.168.8.130 | Document mgmt | +| 104 | [gitea](containers/104-gitea.md) | 192.168.8.121 | Git server | +| 105 | [apps](containers/105-apps.md) | 192.168.8.205 | Docker host (Artifacto / Booklore / PlantUML / Portainer / WriteFreely) | +| 109 | [syncthing](containers/109-syncthing.md) | (stopped) | File sync | +| 114 | [nextcloud](containers/114-nextcloud.md) | 192.168.8.224 | Personal cloud | +| 116 | [heaper](containers/116-heaper.md) | 192.168.8.116 | Heaper | +| 118 | [elementsynapse](containers/118-elementsynapse.md) | 192.168.8.239 | Matrix Synapse | +| 119 | [sophia](containers/119-sophia.md) | 192.168.8.157 | Sophia | +| 120 | [mule-images](containers/120-mule-images.md) | 192.168.8.136 | Mule-image / mulita photos | +| 121 | [caddy](containers/121-caddy.md) | 192.168.8.175 | Reverse proxy | +| 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) | +| 123 | [claudio-bot](containers/123-claudio-bot.md) | 192.168.8.230 | Matrix control plane | +| 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS | + +### Cross-cutting infrastructure +- [DNS — split-horizon](infrastructure/dns.md) +- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md) +- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md) +- [Monitoring — claudio-monitor](infrastructure/monitoring.md) +- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md) +- [SSH access](infrastructure/ssh-access.md) +- [Backups — restic on external drive (disabled)](infrastructure/backups.md) +- [Auto-deploy — gitea-webhook pipelines](infrastructure/auto-deploy.md) +- [VPS hardening — IONOS / netbird control plane](infrastructure/vps-hardening.md) + +### Investigations +Time-stamped incident notes / experiments in [`investigations/`](investigations/index.md). + +### Operations +- [Command cheatsheet](operations/commands.md) + +## Conventions + +- **Each node page** ends with a `## Changelog` section. Reverse-chronological. Entry format: + ``` + ### YYYY-MM-DD — short title + one or two lines on what changed and why. + ``` +- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug. +- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog. +- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image, claudio-bot) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local. +- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not. + +## Maintaining this wiki + +When you change a node: +1. Update the relevant page (config snapshot, ports, mounts). +2. Add a changelog entry at the bottom of that page. +3. If the change touches a cross-cutting system (DNS, Caddy, Authentik, mesh), update *that* page too and link it from the changelog entry. +4. If it's an incident, add an entry to [`investigations/`](investigations/index.md). + +## See also + +- [`CONTRIBUTING.md`](CONTRIBUTING.md) — page templates and tone diff --git a/containers/101-jellyfin.md b/containers/101-jellyfin.md new file mode 100644 index 0000000..b374805 --- /dev/null +++ b/containers/101-jellyfin.md @@ -0,0 +1,34 @@ +# 101 — `jellyfin` + +Media server: serves the movies / TV / anime / music / audiobooks / podcasts libraries from `/mnt/library` to LAN clients. + +## At a glance +- **Hostname:** `jellyfin` +- **IP:** `192.168.8.206` +- **Privilege:** **unprivileged** + idmap (so it can write to the `media` group on `/mnt/library`) +- **Resources:** 2 cores / 4 GiB RAM / 16 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096` + +## Service / port map + +| Service | Listen | Notes | +| -------- | ------ | ----- | +| jellyfin | `:8096` | HTTP (caddy terminates TLS) | + +## Permissions +Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; idmap block in `/etc/pve/lxc/101.conf` maps in-container GID 10000 to host GID 10000. + +## Related +- [Caddy reverse proxy](121-caddy.md) +- [Media permissions](../infrastructure/media-permissions.md) +- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads +- [DNS split-horizon](../infrastructure/dns.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. No config changes. + +### 2026-04-20 — joined the `media` GID 10000 standard +Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`. diff --git a/containers/103-paperless.md b/containers/103-paperless.md new file mode 100644 index 0000000..c08aac3 --- /dev/null +++ b/containers/103-paperless.md @@ -0,0 +1,41 @@ +# 103 — `paperless` + +Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/documents` and serves OCR'd indexed copies. + +## At a glance +- **Hostname:** `paperless` +- **IP:** `192.168.8.130` +- **Privilege:** privileged +- **Resources:** 2 cores / 3 GiB RAM / 8 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` (consumes `documents/`) +- **Public hostname:** [`paperless.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8000` + +## Service / port map + +| Service | Listen | Notes | +| ------------------- | ------- | -------------------------------- | +| paperless-webserver | `:8000` | HTTP | +| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers | + +## Auth +Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login. + +## Storage +- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design). + +## Known issues +- Disk usage was 86.9% at last claudio-monitor reading on 2026-04-21. Monitor or grow rootfs. + +## Related +- [Authentik](124-authentik.md) +- [Caddy](121-caddy.md) +- [DNS](../infrastructure/dns.md) +- [Monitoring](../infrastructure/monitoring.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-21 — wired into Authentik forward-auth +Cookie-domain-level proxy provider, embedded outpost binding. `/api/*` bypass added in Caddy site block to keep mobile clients working. diff --git a/containers/104-gitea.md b/containers/104-gitea.md new file mode 100644 index 0000000..09e75f8 --- /dev/null +++ b/containers/104-gitea.md @@ -0,0 +1,58 @@ +# 104 — `gitea` + +Self-hosted git server. Source of truth for all `dtoro/*` repositories that auto-deploy across the lab. + +## At a glance +- **Hostname:** `gitea` +- **IP:** `192.168.8.121` +- **Privilege:** privileged +- **Resources:** 1 core / 1 GiB RAM / 8 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` (under `/mnt/library/repos`, owner UID 102 / GID 105 — don't touch from other containers) +- **Public hostname:** [`git.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000` +- **Version:** 1.25.4 (last verified) + +## Service / port map + +| Service | Listen | Notes | +| ------- | -------- | ------------------------------ | +| gitea | `:3000` | HTTP web + git over HTTP | +| gitea SSH | `:22` | git over SSH | +| `gitea-deploy-webhook` | `127.0.0.1:9797` | Receives push events for the customizations repo | + +## Customizations +`/var/lib/gitea/custom` is a checkout of [`dtoro/gitea-customizations`](https://git.hubris.network/dtoro/gitea-customizations). +- `templates/custom/footer.tmpl` injects: + - **Online3DViewer** v0.18.0 (vendored under `public/assets/o3dv/`) for inline 3D preview of CAD raw-file links: 3dm, 3ds, 3mf, amf, bim, brep, dae, fbx, fcstd, glb, gltf, ifc, igs, iges, stp, step, stl, obj, off, ply, wrl. + - **PlantUML client JS** (vanilla, vendored from `gitea.com/davidsvantesson/plantuml-code-highlight`). Browser calls `https://git.hubris.network/_plantuml/img/<encoded>` — caddy `handle_path /_plantuml/*` rewrites and proxies to the PlantUML server on [apps (105)](105-apps.md) `:8079`. +- `home.tmpl` is the anonymous landing page (Homesauce hero + sign-in/explore buttons). +- O3DV binary assets are *not* tracked — `deploy.sh` fetches them on first run. + +## Auto-deploy +Push to `dtoro/gitea-customizations` master → gitea webhook → `http://127.0.0.1:9797/deploy` (loopback) → `gitea-deploy-webhook.service` validates HMAC against `/etc/gitea-deploy/secret` → runs `/opt/gitea-customizations/deploy.sh`. Then `systemctl restart gitea` if templates changed. + +`app.ini` `[webhook] ALLOWED_HOST_LIST` includes `127.0.0.1`, `192.168.8.175`, `192.168.8.205`, `192.168.8.230`, `192.168.8.136`, `192.168.8.77` — every webhook target across the lab. **Don't strip these when editing app.ini.** See [auto-deploy](../infrastructure/auto-deploy.md) for the full list. + +Git creds for the deploy service: `/etc/gitea-deploy/git-credentials` (mode 600), referenced via `credential.helper = store --file=/etc/gitea-deploy/git-credentials` in the repo's `.git/config` (because `ProtectHome=true` blocks `/root`). + +## DNS workaround +LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (router DNS resolves the public IONOS A record, which is a dead end for LAN clients — see [DNS](../infrastructure/dns.md)). Belt-and-suspenders systemd unit `hubris-hosts-override.service` re-applies it idempotently. Override line is **outside** the `# --- BEGIN PVE ---` markers (Proxmox rewrites inside). + +## Related +- [Caddy](121-caddy.md) +- [DNS](../infrastructure/dns.md) +- [Auto-deploy pipelines](../infrastructure/auto-deploy.md) +- [PlantUML server lives on apps (105)](105-apps.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-22 — `app.ini` ALLOWED_HOST_LIST extended for Artifacto webhook +Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md). + +### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added +For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency. + +### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped +`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`. diff --git a/containers/105-apps.md b/containers/105-apps.md new file mode 100644 index 0000000..ef9b409 --- /dev/null +++ b/containers/105-apps.md @@ -0,0 +1,88 @@ +# 105 — `apps` + +Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, Booklore, PlantUML server, Portainer (and historically WriteFreely / blog). + +## At a glance +- **Hostname:** `apps` +- **IP:** `192.168.8.205` +- **Privilege:** privileged (so container UID = host UID for bind-mount sanity) +- **Resources:** 2 cores / 4 GiB RAM / 30 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Public hostnames:** see table below + +## Public hostname → service map + +| Hostname | Container | Backend port | Notes | +| --------------------------------- | ---------------- | ------------ | ----- | +| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). | +| `books.hubris.network` | Booklore | `:6060` | Native OIDC. Redirect URI `/oauth2-callback`. | +| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). | +| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. | +| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). | + +## Stacks + +### Artifacto (`/opt/artifacto/`) +Self-hosted HTML-artifact share (Go + SQLite). Repo `dtoro/Artifacto` is the working tree itself — `git push origin main` triggers rebuild. +- Compose: tracked `docker-compose.yml` binds `./data:/data` and `127.0.0.1:3100:3000`. `docker-compose.override.yml` uses the `!override` YAML tag on `volumes:` and `ports:` to redirect `data` to `/mnt/library/documents/artifacts`. +- Data dir: `/mnt/library/documents/artifacts` — owner **UID 10001 : GID 10001** (matches the `app` user baked into the Dockerfile), mode 750. Parent `/mnt/library/documents` stays `www-data:www-data 750`; bind targets the child directly so no traversal needed. +- `.env` (untracked) holds `ADMIN_PASSWORD`, `SESSION_SECRET` (`openssl rand -hex 32`), `BASE_URL=https://artifacto.hubris.network`, `SSO_GATEWAY_SECRET` (must match `ARTIFACTO_GATEWAY_SECRET` in `/etc/caddy/caddy.env` on [caddy](121-caddy.md)). Don't `git checkout .env`. +- Auth: Authentik forward-auth on the admin surface (`/`, `/login`, `/a/*`, `/api/*`). `X-Authentik-Username` is honored only when the `X-Artifacto-Gateway` header matches `SSO_GATEWAY_SECRET` — prevents direct LAN connections from spoofing identity. +- Public exposure: VPS traefik gates path scope to share URLs; admin surface returns 404 at the edge. See [ingress](../infrastructure/ingress.md). + +#### Auto-deploy pipeline (Artifacto) +Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webhook.py`, secret in `/etc/artifacto-deploy/secret`, systemd unit `artifacto-deploy-webhook.service`. Listens on `0.0.0.0:9798`, validates `X-Gitea-Signature` HMAC-SHA256, filters to `refs/heads/main`, runs deploy in a daemon thread (returns 202 immediately). Gitea webhook id 7 on `dtoro/Artifacto`. + +- Logs: `pct exec 105 -- journalctl -u artifacto-deploy-webhook -f` +- Manual deploy: `pct exec 105 -- /opt/artifacto-deploy/deploy.sh` +- Health: `pct exec 105 -- curl -s http://127.0.0.1:9798/health` → `ok` +- Slug length bumped 5 → 8 chars (in `internal/store/artifacts.go:41`) once the endpoint went public. + +### PlantUML (`/opt/plantuml/`) +`plantuml/plantuml-server:jetty` listening on `192.168.8.205:8079`, context path `/plantuml` (`BASE_URL=plantuml`). Reached only via [gitea](104-gitea.md)'s caddy `handle_path /_plantuml/*` rewrite — no DNS record of its own. Same-origin from gitea so the embedded markdown previewer can call it without CORS. + +### Portainer +Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`). + +### Booklore +Native OIDC via Authentik (Settings → OIDC). Redirect URI `/oauth2-callback` (NOT `/api/oidc`). Container needs `extra_hosts: auth.hubris.network:192.168.8.175`. **Edit via Portainer UI** if it's a Portainer-managed stack. + +> ⚠️ **Never `docker compose up` Portainer-managed stacks from the host shell.** Portainer's compose state lives at `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`. Running `docker compose up -d <svc>` from the host triggers recreates of OTHER services in the stack and silently destroys bind-mounted data. **This wiped Booklore's mariadb data on 2026-04-22.** Use the Portainer UI editor for compose changes. See [mesh migration](../infrastructure/mesh.md#critical-never-docker-compose-up-portainer-managed-stacks) for the full warning. + +### WriteFreely (`/opt/writefreely-docker/`) +Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts. + +## Permissions +- LXC is privileged → in-container UID == host UID. +- Member of [media GID 10000](../infrastructure/media-permissions.md). Adding `media` to the LXC alone is *not* enough for inner Docker containers — each container needs `--group-add 10000` or `user: "<uid>:10000"` to write into `/mnt/library`. + +## Related +- [Gitea (104)](104-gitea.md) — uses the PlantUML server +- [Caddy (121)](121-caddy.md) +- [Authentik (124)](124-authentik.md) +- [DNS](../infrastructure/dns.md) +- [Auto-deploy](../infrastructure/auto-deploy.md) +- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-22 — Artifacto deployed + Authentik gateway-secret auto-login wired +Repo `dtoro/Artifacto`. Compose, data dir, `.env`, webhook (id 7), Caddyfile site block, gateway-secret pattern, public-internet share exposure on the VPS — all set up that day. Slug length bumped 5 → 8 for brute-force resistance. + +### 2026-04-22 — Booklore mariadb data wiped (lesson, not a feature) +`docker compose up -d` from the host shell on a Portainer-managed stack recreated services and re-initialized the bind-mounted `./mariadb/config`. Recovered by reseeding from snapshot. Rule recorded: edit Portainer-managed stacks via the Portainer UI. + +### 2026-04-22 — Portainer wired into Authentik (native OAuth2) +Custom-claim `portainer_uid` scope. CLI added `--trusted-origins docker.hubris.network`. + +### 2026-04-22 — `app.ini` host whitelist on gitea extended +Added `192.168.8.205` so the Artifacto webhook delivery wouldn't be rejected. See [gitea (104)](104-gitea.md). + +### 2026-04-21 — Booklore + WriteFreely wired into Authentik OIDC +Native OIDC for both. `extra_hosts` override added inside each container so DNS resolves `auth.hubris.network` to the Caddy IP. + +### 2026-04-20 — PlantUML server deployed +Used by [gitea (104)](104-gitea.md)'s footer.tmpl injection. Same-origin via caddy `handle_path /_plantuml/*` rewrite — no separate hostname. diff --git a/containers/109-syncthing.md b/containers/109-syncthing.md new file mode 100644 index 0000000..bdad580 --- /dev/null +++ b/containers/109-syncthing.md @@ -0,0 +1,26 @@ +# 109 — `syncthing` + +Cross-device file sync into `/mnt/library/syncthing`. + +## At a glance +- **Hostname:** `syncthing` +- **IP:** (none — currently stopped) +- **Privilege:** **unprivileged** + idmap (for the `media` GID 10000 standard) +- **Resources:** 2 cores / 2 GiB RAM / 8 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Status:** **stopped** (was running prior to 2026-04-21). + +## Permissions quirk +The syncthing systemd unit runs with `User=root`, and systemd skips `initgroups()` for `User=root`. Drop-in adds `SupplementaryGroups=media` to `syncthing@.service` so it actually picks up GID 10000. See [media permissions](../infrastructure/media-permissions.md). + +## Related +- [Media permissions](../infrastructure/media-permissions.md) +- [Hubris host](../hosts/hubris.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. Container is currently stopped — reason not yet documented; investigate before re-enabling or destroying. + +### 2026-04-20 — joined the `media` GID 10000 standard +Idmap block + drop-in `SupplementaryGroups=media` on `syncthing@.service`. Config backup: `/root/109.conf.bak.*`. diff --git a/containers/114-nextcloud.md b/containers/114-nextcloud.md new file mode 100644 index 0000000..97cf24a --- /dev/null +++ b/containers/114-nextcloud.md @@ -0,0 +1,48 @@ +# 114 — `nextcloud` + +Personal cloud / file collaboration. Source-of-truth for the photo libraries surfaced by [mulita (120)](120-mule-images.md). + +## At a glance +- **Hostname:** `nextcloud` +- **IP:** `192.168.8.224` +- **Privilege:** privileged +- **Resources:** 4 cores / 6 GiB RAM / 25 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` (Nextcloud user files under `/mnt/library/homecloud`) +- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) + +## Auth +Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern. + +Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on). + +## DNS workaround (load-bearing) +Nextcloud's PHP HTTP client (`OC\Http\Client\DnsPinMiddleware`) calls `dns_get_record()` directly — it bypasses `/etc/hosts`. So the Authentik OIDC discovery URL needs **real DNS** answering with `192.168.8.175` for `auth.hubris.network`. Recipe applied: +- Local `dnsmasq` listening on `127.0.0.1:53`, `/etc/dnsmasq.d/hubris-internal.conf` with `address=/auth.hubris.network/192.168.8.175`, `server=192.168.8.1`, `server=1.1.1.1`, `interface=lo`, `bind-interfaces`, `no-hosts`, `no-resolv`. +- `pct set 114 --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"` so the LXC starts with the right resolver order. +- `/etc/hosts` override kept as belt-and-suspenders. + +Also needs `allow_local_remote_servers=true` in `config.php`. + +> Once internal DNS at the router level handles `*.hubris.network`, this LXC-local dnsmasq can go away. See [DNS](../infrastructure/dns.md). + +## Storage +Files at `/mnt/library/homecloud`. Owned by Nextcloud's own permission model — **deliberately not on the `media` group**, NC manages it itself and would complain. See [media permissions](../infrastructure/media-permissions.md). + +## Related +- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV +- [Authentik (124)](124-authentik.md) +- [Caddy (121)](121-caddy.md) +- [DNS](../infrastructure/dns.md) +- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md) +- [Media permissions](../infrastructure/media-permissions.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-26 — Nextcloud-rooted libraries shipped (mulita) +[mulita (120)](120-mule-images.md) now reads photos from `/nextcloud-users/<nc_user>/files/...` and writes mutations back via WebDAV. Pre-migration DB dump kept at `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump`. + +### 2026-04-21 — wired into Authentik (native OIDC + nc_uid override) +Local `dnsmasq` added on this LXC because Guzzle bypasses `/etc/hosts`. Resolver order set to `127.0.0.1 192.168.8.1 1.1.1.1`. Username override `dtoro → admin`. diff --git a/containers/116-heaper.md b/containers/116-heaper.md new file mode 100644 index 0000000..8bf8119 --- /dev/null +++ b/containers/116-heaper.md @@ -0,0 +1,23 @@ +# 116 — `heaper` + +Single-purpose container for the "heaper" workload. Mounts `/mnt/library` and reads/writes under `/mnt/library/heaper`. + +## At a glance +- **Hostname:** `heaper` +- **IP:** `192.168.8.116` +- **Privilege:** privileged +- **Resources:** 2 cores / 2 GiB RAM / 10 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Public hostname:** none + +## Permissions +Member of the [media GID 10000](../infrastructure/media-permissions.md) standard via `www-data` (the data subtree at `/mnt/library/heaper` is single-container, so it stays out of the shared setgid group treatment, but the LXC has the standard group hookup). + +## Related +- [Hubris host](../hosts/hubris.md) +- [Media permissions](../infrastructure/media-permissions.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Stub. Detail what heaper actually does the next time we touch this LXC. diff --git a/containers/118-elementsynapse.md b/containers/118-elementsynapse.md new file mode 100644 index 0000000..22512ef --- /dev/null +++ b/containers/118-elementsynapse.md @@ -0,0 +1,32 @@ +# 118 — `elementsynapse` + +Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`. + +## At a glance +- **Hostname:** `elementsynapse` +- **IP:** `192.168.8.239` +- **Privilege:** **unprivileged** +- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs +- **Mounts:** none from `/mnt/library` +- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008` +- **`server_name`:** `avispero` (intentionally different from the public URL). + +## Service / port map + +| Service | Listen | Notes | +| -------------- | ------- | -------------------------------- | +| synapse | `:8008` | Matrix client-server / federation HTTP. Caddy terminates TLS for `matrix.hubris.network`. | + +## Known issues +- Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21 — the Matrix DB grows steadily. Watch or grow rootfs. + +## Related +- [claudio-bot (123)](123-claudio-bot.md) — connects directly to `192.168.8.239:8008` (avoids hairpin-NAT TLS issue on the public URL) +- [Caddy](121-caddy.md) +- [DNS](../infrastructure/dns.md) +- [Monitoring](../infrastructure/monitoring.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. diff --git a/containers/119-sophia.md b/containers/119-sophia.md new file mode 100644 index 0000000..03b6a02 --- /dev/null +++ b/containers/119-sophia.md @@ -0,0 +1,23 @@ +# 119 — `sophia` + +Single-container workload "sophia". Reads/writes under `/mnt/library/sophia`. + +## At a glance +- **Hostname:** `sophia` +- **IP:** `192.168.8.157` +- **Privilege:** privileged +- **Resources:** 2 cores / 1 GiB RAM / 10 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Public hostname:** none + +## Permissions +LXC has `www-data` in the [media GID 10000 group](../infrastructure/media-permissions.md). The `/mnt/library/sophia` subtree itself is single-container and not on the setgid shared group. + +## Related +- [Hubris host](../hosts/hubris.md) +- [Media permissions](../infrastructure/media-permissions.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Stub. Document what runs inside next time we work on this LXC. diff --git a/containers/120-mule-images.md b/containers/120-mule-images.md new file mode 100644 index 0000000..793ae62 --- /dev/null +++ b/containers/120-mule-images.md @@ -0,0 +1,93 @@ +# 120 — `mule-images` + +Hosts `mule-image` / "mulita" — the photos app at `photos.hubris.network`. Auto-deploys from `dtoro/mule-image` on `git push origin main`. + +## At a glance +- **Hostname:** `mule-images` +- **IP:** `192.168.8.136` +- **Privilege:** privileged +- **Resources:** 4 cores / 8 GiB RAM / 60 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000` (frontend) + +## Stack (`/opt/mule-image`) + +`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose at `/opt/mule-image/docker-compose.yml`. Services: + +| Service | Port | Notes | +| ----------------- | ------ | ----- | +| frontend | 3000 | Reverse-proxied by Caddy | +| backend | 8001 | FastAPI | +| worker-vision | — | ML scan worker | +| worker-light | — | Lightweight worker | +| worker-watcher | — | FS watcher | +| db | (pg) | pgvector | +| redis | (rd) | queue | + +`.env` is **untracked** — `git checkout .env` will wipe it. Holds: +- `PHOTO_DIRS=/mnt/library/images/` +- `NEXTCLOUD_USERS_HOST_PATH=/mnt/library/homecloud` +- `NEXTCLOUD_BASE_URL=https://cloud.hubris.network` +- OIDC client secret + scopes +- `SECRET_KEY` (generated) + +## Nextcloud-rooted libraries (since 2026-04-26) + +Photo libraries live under each user's Nextcloud `files/` tree, NOT in `/mnt/library/images/*`. +- `/mnt/library/homecloud` is bind-mounted into `backend`, `worker-light`, `worker-watcher`, `worker-vision` as `/nextcloud-users`. Each NC user is `/nextcloud-users/<nc_user>/files/`. +- Reads use that bind directly. +- Mutations (upload, delete, rename, move) dispatch through `services/nextcloud_dav.py` (HTTP Basic auth, per-user app password Fernet-encrypted in `users.nextcloud_app_password_enc`) so Nextcloud's `oc_filecache`, trashbin, comments, and desktop-sync clients stay coherent. +- Photo copy + cross-system moves return 501 with a "use Nextcloud's web UI" hint — defer until needed. +- `users.nextcloud_username` overrides the default OIDC `preferred_username`. **`dtoro` (mule-image) maps to `admin` (Nextcloud)** — don't assume username equality. +- Surviving SourceRoots in DB: `Photos` → `/nextcloud-users/admin/files/Photos`; `Memories` → `/nextcloud-users/admin/files/Memories` (both owned by `dtoro`). User `muli` has `nextcloud_username=muli` backfilled but no SourceRoot yet. +- Pre-migration DB dump: `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump` (11 MB) on the host. + +## Authentication (since 2026-04-22) + +Native OIDC via Authentik. Code in `backend/app/auth_oidc.py`, routes `/api/v1/auth/oidc/{login,callback}`. Authentik side: +- OAuth2/OIDC Provider, client ID `fCuHew48ONTskDjUKnMTZjFbVXuHwvQqTScQRNQ1` +- App slug `mule-image` +- Redirect URI: `https://photos.hubris.network/api/v1/auth/oidc/callback` + +Backend container needs `extra_hosts: auth.hubris.network:192.168.8.175` via `docker-compose.override.yml` (gitignored). Otherwise Authlib's metadata fetch fails with `SSL: CERTIFICATE_VERIFY_FAILED: self-signed certificate` (it ends up at a random public host because LXC DNS resolves the public IONOS A record). + +Caddyfile stays plain `reverse_proxy 192.168.8.136:3000` — no forward-auth, no `/api/*` bypass needed. + +## Auto-deploy + +Push to `dtoro/mule-image` `main` → gitea webhook → `http://192.168.8.136:9797/deploy` → `mule-deploy-webhook.service`: +- Validates HMAC against `/etc/mule-deploy/secret` +- Filters to `refs/heads/main` +- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout) +- `git pull --ff-only` + `docker compose up -d --build` + `docker image prune -f` + +Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`, unit at `/etc/systemd/system/mule-deploy-webhook.service`. Same shape as the Caddy + Artifacto pipelines. Gitea webhook id 6. + +`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`. + +Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`. +Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`. + +For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials` (mode 600) — same token as `/etc/caddy-deploy/git-credentials` on [caddy](121-caddy.md). + +## Related +- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries +- [Authentik (124)](124-authentik.md) +- [Caddy (121)](121-caddy.md) +- [DNS](../infrastructure/dns.md) +- [Auto-deploy](../infrastructure/auto-deploy.md) +- [Gitea (104)](104-gitea.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-26 — Nextcloud-rooted libraries shipped +Bind `/mnt/library/homecloud` into the workers, reads via filesystem, writes via WebDAV. `users.nextcloud_username` override field added; `dtoro → admin` mapping. Surviving SourceRoots cleaned up to NC paths. + +### 2026-04-22 — native OIDC via Authentik +Authlib-based code in `backend/app/auth_oidc.py`. `extra_hosts` override for `auth.hubris.network` in compose override (gitignored). + +### 2026-04-21 — auto-deploy pipeline shipped +Webhook receiver at `:9797`, async deploy returning 202. Mirrors caddy-conf / gitea-customizations. diff --git a/containers/121-caddy.md b/containers/121-caddy.md new file mode 100644 index 0000000..bda0433 --- /dev/null +++ b/containers/121-caddy.md @@ -0,0 +1,81 @@ +# 121 — `caddy` + +The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the LAN/mesh and forwards to the right backend. + +## At a glance +- **Hostname:** `caddy` +- **IP:** `192.168.8.175` +- **Privilege:** **unprivileged** +- **Resources:** 1 core / 512 MiB RAM / 6 GiB rootfs +- **Mounts:** none from `/mnt/library` +- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy) +- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`). + +## Sites currently served (live as of 2026-04-28) + +- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100` +- `auth.hubris.network` → [authentik (124)](124-authentik.md) `:9000` +- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080` +- `books.hubris.network` → [apps (105)](105-apps.md) `:6060` +- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443` +- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443` +- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`) +- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123` +- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056` +- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008` +- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096` +- `paperless.hubris.network` → [paperless (103)](103-paperless.md) `:8000` +- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000` +- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006` +- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080` +- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8081` + +> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too. + +## Snippet: `(authentik)` forward-auth + +A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup). + +For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md). + +## Caddy environment + +Untracked secrets in `/etc/caddy/caddy.env`: +- `IONOS_AUTH_API_TOKEN` — DNS-01 ACME +- `ARTIFACTO_GATEWAY_SECRET` — must match `SSO_GATEWAY_SECRET` on [apps (105)](105-apps.md). Rotate together. + +## Auto-deploy + +Push to `dtoro/caddy-conf` master → gitea webhook → `http://192.168.8.175:9797/deploy` → `caddy-deploy-webhook.service` validates HMAC against `/etc/caddy-deploy/secret` → runs `/etc/caddy/scripts/deploy.sh`: +1. `git pull` +2. `caddy validate` +3. `systemctl reload caddy` + +Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scripts, systemd unit all live in the repo under `scripts/`. The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). Git credentials for the root-run service are at `/etc/caddy-deploy/git-credentials` (mode 600). + +> **Systemd unit changes inside the repo do NOT auto-reinstall** — you have to re-run `scripts/webhook/install.sh` manually after editing the unit. + +## Related +- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain +- [Authentik (124)](124-authentik.md) — forward-auth + IdP +- [Auto-deploy](../infrastructure/auto-deploy.md) +- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure +- [Gitea (104)](104-gitea.md) — webhook source +- All container pages list their Caddy backend. + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. 16 active sites at this date. + +### 2026-04-22 — Artifacto + Authentik gateway-secret pattern wired +Caddy site `artifacto.hubris.network`: forward-auth on admin paths, `header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}`, public matchers `/p/*`, `/static/*`, `/healthz`. `caddy.env` extended with the gateway secret. + +### 2026-04-21 — `(authentik)` snippet added +Top-of-file snippet used by every gated site. Targets `http://192.168.8.180:9000` directly to dodge hairpin TLS round-trip. + +### 2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132) +For the [arriman (122)](122-arriman.md) docker stack. Caddy + certs work; needed corresponding [DNS split-horizon](../infrastructure/dns.md) entries to be added to land cleanly. + +### 2026-04-20 — auto-deploy pipeline shipped +Repo `dtoro/caddy-conf`. Webhook receiver at `:9797`, validate + reload on push to master. diff --git a/containers/122-arriman.md b/containers/122-arriman.md new file mode 100644 index 0000000..04f9542 --- /dev/null +++ b/containers/122-arriman.md @@ -0,0 +1,111 @@ +# 122 — `arriman` + +Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr) compose. Replaced the old yunohost-based [LXC 100](index.md#recently-destroyed-kept-for-archaeology) on 2026-04-21. + +## At a glance +- **Hostname:** `arriman` +- **IP:** `192.168.8.132` +- **Privilege:** privileged +- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs +- **Mounts:** `/mnt/library` ↔ `/mnt/library` +- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below) + +## Compose + +`/home/arr/ezarr/docker-compose.yml`. Network `ezarr_default` `172.18.0.0/16`. Service users all run with primary GID 10000 (host `media` group) inside their containers — see [media permissions](../infrastructure/media-permissions.md). Per-app `/config/<app>-config/` dirs bind to `/config` inside. + +`.env` contents: +- `DOWNLOADS_DIR=/mnt/library/downloads/` +- `MEDIA_DIR=/mnt/library/` +- `MEDIACENTER_GID=10000` ← **must be 10000, not 13000.** s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. Without this, qBit errors every torrent with "Permission denied". +- `SONARR_UID=13001`, `RADARR_UID=13002`, `LIDARR_UID=13003`, `PROWLARR_UID=13006`, `JELLYSEERR_UID=13012`, `BAZARR_UID=13013`, `QBITTORRENT_UID`, `SABNZBD_UID` + +## Service / port map + +| Service | Host:Container | Public hostname | +| ------------- | -------------- | ------------------------------------ | +| sonarr | `8989:8989` | direct only | +| radarr | `7878:7878` | direct only | +| lidarr | `8686:8686` | direct only | +| prowlarr | `9696:9696` | direct only | +| bazarr | `6767:6767` | direct only | +| jellyseerr | `5056:5055` | [`jellyseerr.hubris.network`](../infrastructure/dns.md) | +| qbittorrent | `8080:8080` | [`qbit.hubris.network`](../infrastructure/dns.md) | +| sabnzbd | `8081:8080` | [`sab.hubris.network`](../infrastructure/dns.md) | +| flaresolverr | `8191:8191` | internal only | + +Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `ezarr_default` using docker service names. + +## Categories (qBit + SAB + *arr) + +| Category | qBit savePath | SAB dir | Consumer | +| -------- | --------------------------- | ------- | --------------------------------- | +| tv | `/data/torrents/tv` | tv | Sonarr | +| movies | `/data/torrents/movies` | movies | Radarr | +| music | `/data/torrents/music` | music | Lidarr | +| anime | `/data/torrents/anime` | anime | Sonarr (2nd download client, root `/data/media/anime`) | +| books / audiobooks / comics / podcasts | `/data/torrents/<cat>` | `<cat>` | (reserved, none in use yet) | + +Path mapping: host `/mnt/library/<cat>` ↔ container `/data/media/<cat>`. Downloads: host `/mnt/library/downloads/<torrents|usenet>/<cat>` ↔ container `/data/torrents/<cat>` and `/data/usenet/<cat>`. + +## Auth (qBit reverse-proxy + Authentik forward-auth) + +qBit auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines: +- `WebUI\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32` +- `WebUI\ReverseProxySupportEnabled=true` +- `WebUI\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16` + +> **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`. + +Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy. + +SABnzbd `host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host. + +## Credentials + +Backups of all migration-time secrets at `/mnt/library/backup/arr-migration-2026-04-21/state/`: +- `lxc100-apikey-{sonarr,radarr,prowlarr,lidarr}.txt` +- `qbit-creds.txt`, `sab-creds.txt` +- `lxc100-jellyseerr-settings.json` + +\*arr API keys were preserved from the yunohost DB — don't rotate unless needed. + +qBittorrent: `admin` / `CDyjxMKvjfGupCKzXFkjsKw4nRrXMRL`. + +## Migration gotchas (apply if ever re-importing) + +- Imported \*arr `config.xml`: must have `<BindAddress>*</BindAddress>`, `<Port>{ezarr-port}</Port>`, `<UrlBase></UrlBase>`. Otherwise app listens only on 127.0.0.1, wrong port, wrong path. +- Imported DB paths must be rewritten `/mnt/library/` → `/data/media/` (RootFolders / Series / Movies / Artists tables). Without this, scans and imports fail silently. +- Bazarr `config/config.yaml` (yaml, not ini) — `ip: 0.0.0.0`, `port: 6767`, empty top-level `base_url`; sonarr/radarr connection sections normalized to docker service names. +- Staging dir for migrations: `/mnt/library/backup/stage/<app>/` (visible to both old LXC 100 and arriman because both bind-mount the same host path). + +## Permissions + +Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has `www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr` in the `media` group. Each docker container also receives GID 10000 as its primary group via `MEDIACENTER_GID=10000`. + +## Rollback material (kept until 2026-04-28+) + +`/mnt/library/backup/arr-migration-2026-04-21/`: +- `tars/` — per-subtree tars of yunohost app data (granular restore) +- `snapshots/` — vzdumps of LXC 100 (pre-migration) and LXC 122 (pre-import), plus post-stop vzdumps of LXC 100 + 106 +- `stage/<app>-preimport.tar.gz` — original empty ezarr `/config/<app>-config/` dirs +- `state/` — service dumps, listening ports, API keys, jellyseerr settings at migration time +- `DESTROY_AFTER_2026-04-28.md` — destroy checklist + post-destroy restore commands + +## Related +- [Caddy (121)](121-caddy.md) +- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass +- [DNS](../infrastructure/dns.md) +- [Media permissions](../infrastructure/media-permissions.md) +- [Hubris host](../hosts/hubris.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-26 — `MEDIACENTER_GID` flipped 13000 → 10000 +qBit was erroring every torrent with "Permission denied" because s6-setuidgid only honors the primary PGID and `group_add:` doesn't propagate. Setting the primary GID to 10000 (the host `media` group) fixed it. + +### 2026-04-21 — \*arr stack migrated yunohost → docker-compose +Old LXC 100 stopped; ezarr stack stood up here; data and DBs imported with path rewrites. flaresolverr folded in from old LXC 106. Three Caddy sites added (jellyseerr, qbit, sab) → see [caddy](121-caddy.md). Rollback artifacts under `/mnt/library/backup/arr-migration-2026-04-21/`. diff --git a/containers/123-claudio-bot.md b/containers/123-claudio-bot.md new file mode 100644 index 0000000..92e65f9 --- /dev/null +++ b/containers/123-claudio-bot.md @@ -0,0 +1,79 @@ +# 123 — `claudio-bot` + +Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications. + +## At a glance +- **Hostname:** `claudio-bot` +- **IP:** `192.168.8.230` +- **Privilege:** **unprivileged** +- **Resources:** 1 core / 512 MiB RAM / 8 GiB rootfs +- **Mounts:** none from `/mnt/library` +- **Public hostname:** none + +## Stack + +Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`). + +Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed). + +## Configuration + +Secrets at `/etc/claudio-bot/` (mode 600): +- `config.yaml` +- `matrix.token` +- `anthropic.key` +- `lmstudio.key` +- `ipc.token` + +Deploy git creds at `/etc/claudio-deploy/git-credentials`. + +### LLM backend (pluggable) + +`llm.backend` in `config.yaml`: +- `anthropic` — Claude API via `anthropic.key` +- `lmstudio` — OpenAI-compat HTTP via `lmstudio.key` (bearer) + +Currently set to `lmstudio` → `google/gemma-4-e4b` on the Mac mini at `192.168.8.174:1234` (since 2026-04-25). Switch back with `backend:` + `systemctl restart claudio-bot`. Original config saved at `/etc/claudio-bot/config.yaml.bak`. The LM Studio agent in `bot_core/lmstudio.py` translates Anthropic → OpenAI tool definitions. + +## IPC + +`http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by: +- [claudio-monitor on hubris](../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`) +- The (currently disabled) [restic backup wrapper](../infrastructure/backups.md) (token in `/etc/restic/bot.token`) + +> Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together. + +## Plugins + +Module under `plugins/<name>.py` exposing a `Plugin` class; add `<name>` to `plugins:` in `config.yaml`. Plugins can register slash commands, Claude tools, and `on_notify` / `on_proposal_response` hooks. + +Active plugins: +- `system` — `ping`, `help`, `status`, `list_plugins` +- `backup` — ingests `/notify` from the backup wrapper (currently silent — backups disabled) +- `monitor` — `/monitor status`, `/monitor history [N]`, `/monitor clear <key>`. Tools: `get_alerts`, `get_monitor_history` for NL queries via Claude. + +## Auto-deploy + +Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf. + +`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.230`. + +## Related +- [elementsynapse (118)](118-elementsynapse.md) +- [Monitoring (claudio-monitor)](../infrastructure/monitoring.md) +- [Backups (disabled)](../infrastructure/backups.md) +- [Auto-deploy](../infrastructure/auto-deploy.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-25 — LLM backend switched to LM Studio +`backend: lmstudio` → `google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field. + +### 2026-04-21 — `monitor` plugin added +Receives events from [claudio-monitor](../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`. + +### 2026-04-20 — claudio-bot deployed +LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server. diff --git a/containers/124-authentik.md b/containers/124-authentik.md new file mode 100644 index 0000000..79551db --- /dev/null +++ b/containers/124-authentik.md @@ -0,0 +1,136 @@ +# 124 — `authentik` + +Central Identity Provider for the lab. Also runs the [split-horizon dnsmasq](../infrastructure/dns.md) — ergo "the SSO and DNS box". + +## At a glance +- **Hostname:** `authentik` +- **IP:** `192.168.8.180` (statically configured — the only LXC with a static IP) +- **Privilege:** privileged +- **Resources:** 2 cores / 4 GiB RAM / 20 GiB rootfs +- **Mounts:** none from `/mnt/library` +- **Public hostname:** [`auth.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `:9000` +- **Container DNS (in `/etc/pve/lxc/124.conf`):** `192.168.8.1 1.1.1.1` (router DNS plus a fallback added 2026-04-21 because router DNS flakes intermittently — Authentik is the resolver itself for the *rest* of the LAN, but its own LXC uses upstream). + +## Authentik stack (`/opt/authentik/`) + +Upstream `docker-compose.yml` + `.env`. Services: `postgresql` (16-alpine), `server`, `worker`. Authentik 2026.x dropped the Redis dependency. + +- `.env` mode 600, **untracked**, holds `AUTHENTIK_SECRET_KEY` and `PG_PASS`. +- `AUTHENTIK_TAG=2026.2.2` — pinned. Don't let it drift to `:latest`. Telemetry / update-check / error-reporting disabled. +- Ports: 9000 (http), 9443 (https) on the LXC. +- Embedded outpost lives at `/outpost.goauthentik.io/*` on the Authentik host — the forward-auth endpoint Caddy points at. +- Stack is **not** git-tracked yet. If/when wiring auto-deploy: mirror the `mule-image` pattern (webhook receiver outside the app repo at `/opt/authentik-deploy/`). Repo `dtoro/authentik-conf` is reserved but not created. + +## Forward-auth pattern (every gated app) + +- **One Proxy Provider per app.** Authentik enforces a UNIQUE constraint `application.provider_id`, so one Provider = one Application. "Domain-level" only means they share the cookie domain. Each provider in "Forward auth (domain level)" mode, External host `https://auth.hubris.network`, Cookie domain `hubris.network`. First one was `hubris-forward-auth` (Paperless). +- **Authentication flow:** MUST be `default-authentication-flow` (NOT `default-source-authentication` — that's for IdP federation; gives `FlowNonApplicableException` + 404 on the authorize endpoint). +- **Authorization flow:** `default-provider-authorization-implicit-consent` (or explicit). +- **Application Launch URL** MUST be the full public URL `https://<sub>.hubris.network/` — outpost matches incoming `X-Forwarded-Host` against it. +- Each Application MUST have at least one **policy/group/user binding** — zero bindings = outpost returns 404 on access. +- **Restart Authentik after binding new apps to the outpost:** + ``` + pct exec 124 -- docker compose -f /opt/authentik/docker-compose.yml restart server worker + ``` + +Caddy snippet `(authentik)` lives at the top of `/etc/caddy/Caddyfile`. Points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid hairpin TLS round-trip stripping `X-Forwarded-Host`. Must explicitly set `header_up X-Forwarded-Host {host}` in the forward-auth block. Used by gated sites with `import authentik`. + +### Per-app username override pattern (Authentik) + +Used when the app's local user ID doesn't match the user's Authentik username (e.g., Nextcloud's `admin` ≠ Authentik's `dtoro`). + +1. On the Authentik user: add attribute `<app>_uid: <target_local_username>` (YAML, Directory → Users → Edit → Attributes). +2. Customization → Property Mappings → Create → **Scope Mapping** (not SAML): + - Name: `<app>-uid-override`, Scope name: `<app>-uid`, Expression: + ```python + return {"nc_uid": user.attributes.get("<app>_uid", user.username)} + ``` + - **Use a custom claim key** (e.g. `nc_uid`), not `preferred_username` — the default `profile` scope mapping emits `preferred_username` and will overwrite yours depending on ordering. +3. Attach the new scope to the provider (Providers → app → Scopes). +4. On the app side, point its OIDC UID-mapping setting at the custom claim. + +For Nextcloud: +``` +occ user_oidc:provider <name> --mapping-uid=nc_uid +occ user_oidc:provider <name> --scope="openid profile email <app>-uid" +``` + +### Bypass forward-auth for API paths (mobile apps) + +If the app has its own token-based API auth and a mobile client, API paths must bypass forward-auth — mobile apps can't follow the browser login redirect. Pattern in the Caddyfile site block: + +``` +paperless.hubris.network { + tls { dns ionos {env.IONOS_AUTH_API_TOKEN} } + @api path /api/* + handle @api { + reverse_proxy 192.168.8.130:8000 + } + handle { + import authentik + reverse_proxy 192.168.8.130:8000 + } +} +``` + +API paths to bypass per app: +- [Paperless](103-paperless.md): `/api/*` (Bearer) +- [Sonarr / Radarr / Lidarr / etc.](122-arriman.md): `/api/*` (X-Api-Key) +- [qBittorrent](122-arriman.md): `/api/*` (session cookie from `/api/v2/auth/login`) +- [SABnzbd](122-arriman.md): `/api?*` (apikey query param) — match `/api*` for query-string APIs +- Homarr: no mobile client +- [Portainer](105-apps.md): mobile uses same session auth as web; no bypass typically needed + +### Backend trust of Authentik headers (skip the app's own login after SSO) + +- [**Paperless**](103-paperless.md): `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Restart `paperless-webserver paperless-task-queue paperless-scheduler paperless-consumer`. Django auto-creates matching users on first SSO login; promote to superuser via existing admin UI. +- Apps without header-auth support: users log in twice (SSO + app login). Acceptable but degraded UX. + +## Per-app integration map + +| App | Type | Notes | +| ----------------------------------------- | ---------------- | ----- | +| [Paperless (103)](103-paperless.md) | Forward-auth + REMOTE_USER | `/api/*` bypass | +| [Nextcloud (114)](114-nextcloud.md) | Native OIDC | `nc_uid` override; local dnsmasq required (Guzzle bypasses `/etc/hosts`) | +| [mulita (120)](120-mule-images.md) | Native OIDC | `extra_hosts` override in compose | +| [Booklore (105)](105-apps.md) | Native OIDC | Redirect URI `/oauth2-callback`; `extra_hosts` | +| [Portainer (105)](105-apps.md) | Native OAuth2 | `portainer_uid` custom claim; `--trusted-origins` flag | +| [WriteFreely (105)](105-apps.md) | Native OIDC | `[oauth.generic]` block; `extra_hosts` | +| [qBittorrent (122)](122-arriman.md) | Forward-auth via IP whitelist | Reverse-proxy support enabled in qBit | +| [Artifacto (105)](105-apps.md) | Forward-auth + gateway-secret auto-login | Public `/p/*` paths bypass | +| [Home Assistant VM (108)](../vms/108-haos.md) | HACS `christiaangoossens/hass-oidc-auth` | `automatic_user_linking: true`, `default_redirect: true`. Supervisor DNS via `ha dns options`. | + +## Netbird IdP integration — DEFERRED + +The `netbirdio/netbird-server` combined image has no config knobs for external OIDC. Verified in `combined/cmd/config.go` on main (v0.69.0): `AuthConfig` only exposes issuer + redirect URIs; `ToManagementConfig()` hardcodes `AuthAudience="netbird-dashboard"`, `UserIDClaim="sub"`, always calls `buildEmbeddedIdPConfig()`. Code comment: `"embedded IdP is always enabled in combined server"`. + +To wire Authentik into Netbird login/user-sync, must migrate to the legacy split stack (`netbirdio/management` + `signal` + `dashboard` + coturn) with the richer `management.json` schema. Sqlite management data should migrate but needs verification; peers stay connected via wireguard keys. + +**Pre-work already in place (keep for re-use):** +- Provider `Provider for Netbird` + App `netbird`. Client ID `xZwVTFCsxWdBM3uIGS15wAAcVvsJiTtWdxVCEela`. Redirect URIs for `https://netbird.hubris.network/{nb-auth,nb-silent-auth}` and `http://localhost:53000/`. Scopes: `openid profile email offline_access goauthentik.io/api`. +- Service account `netbird-service` in `authentik Admins` group, non-expiring API token `netbird-service-api`. +- Netbird mgmt host (`82.165.190.79`) is now a peer on its own mesh. See [mesh](../infrastructure/mesh.md). + +## DNS responsibility + +dnsmasq runs alongside Authentik on this LXC, listening on `192.168.8.180:53` + `127.0.0.1:53`, serving every `*.hubris.network` subdomain → `192.168.8.175`. **There is no wildcard** — every site needs an explicit `address=` entry. See [DNS split-horizon](../infrastructure/dns.md). + +## Related +- [DNS split-horizon](../infrastructure/dns.md) +- [Caddy (121)](121-caddy.md) +- [Mesh migration](../infrastructure/mesh.md) +- Every gated app under [containers/index](index.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-22 — Phase 6 (Netbird IdP swap) deferred +Combined netbird-server image can't take an external IdP. Pre-work in Authentik kept for later. Netbird mgmt host instead joined its own mesh as a peer (`100.122.165.149`) for split-horizon DNS access. + +### 2026-04-22 — Artifacto, mulita, WriteFreely, Portainer wired +Native OIDC for mulita / WriteFreely / Portainer; gateway-secret auto-login pattern for Artifacto. + +### 2026-04-21 — deployed; Phases 1–5 complete +LXC 124 provisioned, stack at `/opt/authentik`, public URL via Caddy, Paperless + Booklore + Nextcloud + Home Assistant wired. dnsmasq for split-horizon DNS lives on the same LXC. diff --git a/containers/index.md b/containers/index.md new file mode 100644 index 0000000..27505a4 --- /dev/null +++ b/containers/index.md @@ -0,0 +1,47 @@ +# LXC containers — index + +All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per-container page. + +| ID | Name | IP | Priv | Cores | RAM | Disk | Mounts | Public hostname | Status | +| --- | ---------------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- | +| 101 | [jellyfin](101-jellyfin.md) | 192.168.8.206 | unpriv (idmap) | 2 | 4 GiB | 16 GiB | `/mnt/library` | `media.hubris.network` | running | +| 103 | [paperless](103-paperless.md) | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running | +| 104 | [gitea](104-gitea.md) | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running | +| 105 | [apps](105-apps.md) | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `books` / `artifacto` / `blog` | running | +| 109 | [syncthing](109-syncthing.md) | (stopped) | unpriv (idmap) | 2 | 2 GiB | 8 GiB | `/mnt/library` | — | **stopped** | +| 114 | [nextcloud](114-nextcloud.md) | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running | +| 116 | [heaper](116-heaper.md) | 192.168.8.116 | priv | 2 | 2 GiB | 10 GiB | `/mnt/library` | — | running | +| 118 | [elementsynapse](118-elementsynapse.md) | 192.168.8.239 | unpriv | 1 | 2 GiB | 8 GiB | — | `matrix.hubris.network` | running | +| 119 | [sophia](119-sophia.md) | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running | +| 120 | [mule-images](120-mule-images.md) | 192.168.8.136 | priv | 4 | 8 GiB | 60 GiB | `/mnt/library` | `photos.hubris.network` | running | +| 121 | [caddy](121-caddy.md) | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running | +| 122 | [arriman](122-arriman.md) | 192.168.8.132 | priv | 4 | 8 GiB | 24 GiB | `/mnt/library` | `jellyseerr` / `qbit` / `sab` | running | +| 123 | [claudio-bot](123-claudio-bot.md) | 192.168.8.230 | unpriv | 1 | 512 MiB | 8 GiB | — | — | running | +| 124 | [authentik](124-authentik.md) | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running | + +## Recently destroyed (kept for archaeology) + +| ID | Name | Destroyed | Reason | +| --- | ---------------- | --------------- | --------------------------------------------- | +| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired | +| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose | +| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned | +| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) | +| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned | +| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) | +| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned | + +> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs. + +## Conventions + +- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. IPs are stable via the LAN router's DHCP reservations. +- `onboot=1` on every container — the host brings them up after `pve-guests.service`. +- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library`. Containers that don't mount `/mnt/library` don't need it. +- Most containers are privileged. Unprivileged ones (`101`, `109`, `118`, `121`, `123`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard. + +## Related +- [Hubris host](../hosts/hubris.md) +- [Media permissions](../infrastructure/media-permissions.md) +- [Caddy](121-caddy.md) — terminates every public hostname +- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain diff --git a/hosts/hubris.md b/hosts/hubris.md new file mode 100644 index 0000000..d0c504b --- /dev/null +++ b/hosts/hubris.md @@ -0,0 +1,92 @@ +# `hubris` — Proxmox host + +Single-node Proxmox VE running 1 VM and 14 LXC containers (one of which is currently stopped). The whole homelab. + +## At a glance +- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`) +- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`). +- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../investigations/2026-04-21-hubris-crash-loop.md). +- **LAN:** `192.168.8.77/24` on bridge `vmbr0`, gateway `192.168.8.1`. +- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md). +- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`. + +## Storage + +| Pool | Type | Size | Use | +| ------------ | --------- | ----- | -------------------------------------- | +| `local` | dir | ~95G | ISOs, templates, `/etc`, configs | +| `local-lvm` | lvmthin | 856G | LXC/VM rootfs | +| `library` | lvmthin | 3.7T | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) | + +`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`, `syncthing`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md). + +## Tenants + +### VMs +- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB + +### LXC containers +See [containers/index](../containers/index.md). 14 active, 1 stopped. + +## Boot-time tuning (load-bearing) + +- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22. +- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty. +- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.) + +## Host services owned by external repos + +| What | Repo | Path on host | +| --------------------------------- | -------------------------- | ----------------------------- | +| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor` | `/opt/claudio-monitor` | +| `backup-library` (restic) | `dtoro/backup-library` | `/opt/backup-library` *(disabled)* | +| `hubris-public-cert-sync` | (script + systemd unit) | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer | + +See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md). + +## Quirks + +- `/etc/pve` is fuse — normal for the Proxmox cluster filesystem, even on a single-node install. +- ZFS is **not** in use; storage is LVM-thin + ext4. +- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`. +- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23. +- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen. +- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher). + +## Authorized SSH keys (root) + +- `root@hubris` (self, RSA) — local +- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22 + +OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha. + +## Related +- [Containers index](../containers/index.md) +- [DNS split-horizon](../infrastructure/dns.md) +- [Caddy reverse proxy](../containers/121-caddy.md) +- [Mesh / Netbird](../infrastructure/mesh.md) +- [Media permissions](../infrastructure/media-permissions.md) +- [Monitoring](../infrastructure/monitoring.md) +- [Backups (disabled)](../infrastructure/backups.md) +- [Operations cheatsheet](../operations/commands.md) +- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) + +## Changelog + +### 2026-04-28 — wiki started +This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since. + +### 2026-04-23 — SSD cooling + thermal pads installed +Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict). + +### 2026-04-22 — drive removal A/B test +Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). + +### 2026-04-22 — `cpu-epp.service` ordering bug fixed +Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`. + +### 2026-04-21 — crash-capture + RAS telemetry enabled +`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else. + +### 2026-04-21 — `cpu-epp.service` deployed +Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../investigations/2026-04-21-hubris-crash-loop.md). diff --git a/infrastructure/auto-deploy.md b/infrastructure/auto-deploy.md new file mode 100644 index 0000000..a6091c9 --- /dev/null +++ b/infrastructure/auto-deploy.md @@ -0,0 +1,93 @@ +# Auto-deploy — gitea-webhook pipelines + +Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes. + +## Two shapes + +### Shape A — checkout IS the working tree (config repos) + +`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap. + +### Shape B — receiver outside the app repo (compose stacks) + +The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout. + +## Common + +- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`. +- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded. +- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently: + - `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md)) + - `192.168.8.175` ([caddy (121)](../containers/121-caddy.md)) + - `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto) + - `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md)) + - `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md)) + - `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library) + + **Don't strip these when editing app.ini.** + +- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`. + +## Pipelines + +| Repo | Target | Shape | Receiver | Webhook id | Reload action | +| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- | +| `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` | +| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed | +| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` | +| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` | +| `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` | +| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) | + +> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). `dtoro/authentik-conf` is reserved but the LXC stack is not git-tracked yet. The dnsmasq config on [authentik (124)](../containers/124-authentik.md) is also not tracked — if it gets a `dtoro/dnsmasq-conf`, mirror the caddy-conf pattern. + +## When you change a tracked config + +Always commit + push. Local-only edits drift. Common ones: + +- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys) +- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys) +- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys) +- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys) +- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys) +- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys) + +## Per-pipeline notes / gotchas + +### caddy-conf +- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits. +- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). + +### gitea-customizations +- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP. +- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run. + +### mule-image / Artifacto +- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`. +- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL. +- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`. +- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`. +- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated. + +### backup-library +- Currently the only deploy that targets the host directly (`192.168.8.77:9798`). +- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run. + +## Related +- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these +- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets +- [Backups (disabled)](backups.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. Six active pipelines. + +### 2026-04-22 — Artifacto pipeline added +Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended. + +### 2026-04-21 — mule-image + claudio-bot pipelines added +Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot. + +### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped +Initial three. Set the conventions everything else follows. diff --git a/infrastructure/backups.md b/infrastructure/backups.md new file mode 100644 index 0000000..1ca413c --- /dev/null +++ b/infrastructure/backups.md @@ -0,0 +1,115 @@ +# Backups — restic on external drive (DISABLED) + +Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../investigations/2026-04-21-hubris-crash-loop.md). + +## Status + +**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d: +- `backup-library@homecloud.timer` +- `backup-library@images.timer` +- `backup-library@small.timer` +- `backup-library-check.timer` + +Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver). + +**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). + +**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive. + +## Design + +Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger. + +Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`. + +## Components + +- **Repo:** `dtoro/backup-library` +- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md) +- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this. +- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.** +- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`. + +## Mount-on-demand + +`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge. + +## UAS blacklist + +`/etc/modprobe.d/usb-storage-quirks.conf`: +``` +options usb-storage quirks=090c:2320:u +``` +Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`. + +## Schedule + +Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown: + +| Timer | When | Include list | Approx size | +| ---------------------------------- | -------------- | ------------------------------------ | ----------- | +| `backup-library@homecloud.timer` | day 1 / month | `/etc/restic/include-homecloud.list` | ~315 G | +| `backup-library@images.timer` | day 10 / month | `/etc/restic/include-images.list` | ~103 G | +| `backup-library@small.timer` | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G | + +Snapshots tagged `chunk-<group>` so forget/prune treats each series independently. + +Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`). + +Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`. + +## Thermal caps + +Baked into the systemd units: +- `IOReadBandwidthMax=/mnt/library 50M` +- `IOWriteBandwidthMax=/mnt/backup 30M` +- `--read-concurrency=1` on restic. + +## Wrapper + +`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env. + +## Notifications + +POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together. + +`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify. + +## Recovery + +Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery. + +## Known SPOF + +Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) are not yet implemented. + +## Drive history + +The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again. + +After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../investigations/2026-04-21-hubris-crash-loop.md). + +## Thermal monitoring + +Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md). + +## Related +- [Hubris host](../hosts/hubris.md) +- [claudio-bot (123)](../containers/123-claudio-bot.md) +- [Monitoring](monitoring.md) +- [Auto-deploy](auto-deploy.md) +- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. Status remains DISABLED. + +### 2026-04-22 — DISABLED +Drive removed as the A/B test in the [crash investigation](../investigations/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized. + +### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive +Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups. + +### 2026-04-20 — deployed; redesigned for thermal-gentleness +Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21. diff --git a/infrastructure/dns.md b/infrastructure/dns.md new file mode 100644 index 0000000..c903a4b --- /dev/null +++ b/infrastructure/dns.md @@ -0,0 +1,86 @@ +# DNS — split-horizon `*.hubris.network` + +LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md). + +There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry. + +## Components + +- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22). +- **LAN authoritative for `hubris.network` records:** dnsmasq on [authentik (124)](../containers/124-authentik.md), `192.168.8.180:53` and `127.0.0.1:53`. Config at `/etc/dnsmasq.d/hubris-split.conf`. Forwards everything else to `1.1.1.1` and `8.8.8.8` (`no-resolv`, `server=...`). +- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns. +- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where. + +## Live entries (as of 2026-04-28) + +``` +address=/auth.hubris.network/192.168.8.175 +address=/git.hubris.network/192.168.8.175 +address=/media.hubris.network/192.168.8.175 +address=/paperless.hubris.network/192.168.8.175 +address=/books.hubris.network/192.168.8.175 +address=/home.hubris.network/192.168.8.175 +address=/cloud.hubris.network/192.168.8.175 +address=/matrix.hubris.network/192.168.8.175 +address=/proxmox.hubris.network/192.168.8.175 +address=/docker.hubris.network/192.168.8.175 +address=/jellyseerr.hubris.network/192.168.8.175 +address=/qbit.hubris.network/192.168.8.175 +address=/sab.hubris.network/192.168.8.175 +address=/blog.hubris.network/192.168.8.175 +address=/photos.hubris.network/192.168.8.175 +address=/artifacto.hubris.network/192.168.8.175 +``` + +## Why split-horizon + +The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The dnsmasq override on LXC 124 keeps LAN traffic on the home Caddy. + +## The gotcha that cost a debug session (2026-04-22) + +Creating a new Caddyfile site block is necessary but **not sufficient**. Without the LXC-124 dnsmasq entry, LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine. + +## Recipe — adding a new subdomain + +1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md). +2. Append to `/etc/dnsmasq.d/hubris-split.conf` on [authentik (124)](../containers/124-authentik.md): + ``` + address=/<new>.hubris.network/192.168.8.175 + ``` +3. `pct exec 124 -- systemctl restart dnsmasq` — reload/SIGHUP may not pick up the new `address=` line; hard restart is the reliable path. +4. Verify: `dig @192.168.8.180 +short <new>.hubris.network` → `192.168.8.175`. +5. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`. + +> The dnsmasq config is **not** tracked in git. If you stand up a `dtoro/dnsmasq-conf` repo, wire it like `caddy-conf` (webhook on LXC 124). + +## Public path — what does and doesn't follow the LAN map + +- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh. +- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404. +- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern. + +## Long-term plan + +Either: +- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides. +- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain. + +## Related +- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here +- [Ingress (VPS traefik)](ingress.md) — public-side counterpart +- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition +- [Authentik (124)](../containers/124-authentik.md) — host of the split-horizon dnsmasq + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. 16 active entries. + +### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79 +Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services. + +### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab) +Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day. + +### 2026-04-21 — dnsmasq stood up on LXC 124 +Co-located with Authentik. Initial entries cover everything routed through Caddy. diff --git a/infrastructure/ingress.md b/infrastructure/ingress.md new file mode 100644 index 0000000..2ef3a6d --- /dev/null +++ b/infrastructure/ingress.md @@ -0,0 +1,84 @@ +# Public ingress — VPS traefik + cert mirror + +How home services reach the open internet without exposing the home network. Two-stage pattern: traefik on the IONOS VPS terminates TLS at the public edge, then reverse-proxies over the netbird mesh to home Caddy / direct backends. + +## The shape + +``` +Public client + │ *.hubris.network → 82.165.190.79 (IONOS wildcard) + ▼ +[VPS traefik] priority 1: HostSNI(*) → netbird-proxy:8443 ← netbird control plane + priority 10: per-host HTTP routers ← home services + │ HTTP over netbird mesh + ▼ +[Home backend on 192.168.8.x] +``` + +Mesh clients see [dnsmasq on LXC 124](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent. + +## Why this shape + +- VPS traefik already has a HostSNI(*) TCP passthrough at priority 1 so netbird's own ingress (`netbird.hubris.network`, future `*.proxy.hubris.network`) is unaffected. +- Per-hostname HTTP file-provider routers at priority 10 win over the passthrough for the listed hosts and let traefik terminate TLS itself for those. +- Traefik's own ACME (`letsencrypt` resolver) fails on this box: HostSNI(*) grabs TLS-ALPN-01 challenges before traefik's `allowACMEByPass=true` can respond. Solution: home Caddy obtains certs via IONOS DNS-01 (no conflict) and the VPS *mirrors* the result over. + +## Components + +### On the VPS (`82.165.190.79`) + +- `/opt/traefik-dynamic.yaml` — file-watched dynamic config. One `http.routers.<name>-public` + one `http.services.<name>-public` per exposed service, plus one entry in the top-level `tls.certificates` list per hostname. +- Let's-Encrypt-effective directory: `/letsencrypt/` inside the traefik container, backed by the host-side docker volume `opt_netbird_traefik_letsencrypt`. +- Backups before edits: `cp /opt/traefik-dynamic.yaml /opt/traefik-dynamic.yaml.bak.$(date +%s)` — several bak files live alongside. + +### On the PVE host (`192.168.8.77`) + +- `/usr/local/bin/hubris-public-cert-sync.sh` — runs daily via `hubris-public-cert-sync.timer`. Maps source hostname → VPS cert filenames in a bash assoc array. For each mapping: `pct pull` cert+key from [Caddy (121)](../containers/121-caddy.md)'s store, diff against the VPS copy, scp only on change. +- Filenames are stable per host so the dynamic.yaml never needs editing on renewal — traefik file-watches and hot-reloads the cert. + +## Services currently exposed + +| Hostname | Path scope | Backend | Middlewares | Cert files on VPS | +| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ | +| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` | +| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` | + +`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md). + +## Recipe — exposing another service + +1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`). +2. Add an entry to `HOSTS` in `/usr/local/bin/hubris-public-cert-sync.sh` mapping the hostname → VPS filenames. Run once: `systemctl start hubris-public-cert-sync.service`. Confirm the cert landed. +3. Edit `/opt/traefik-dynamic.yaml` on the VPS: + - Add to `tls.certificates`: paths `/letsencrypt/<crt>` and `/letsencrypt/<key>`. + - Add `http.routers.<host>-public`: `rule: 'Host(\`<host>\`)'` (or with path matchers if scope-gating), `entryPoints: [websecure]`, `priority: 10`, `tls: {}`, `service: <host>-public`, `middlewares: [...]`. + - Add a ratelimit middleware under `http.middlewares` if wanted. + - Add `http.services.<host>-public.loadBalancer.servers[0].url: 'http://<lan-ip>:<port>'`. +4. Verify: + ``` + ssh root@100.122.165.149 'curl -skI --resolve <host>:443:127.0.0.1 https://<host>/' # 2xx/3xx + curl -skI --resolve <host>:443:<vps-public-ip> https://<host>/ # same + ``` +5. **No DNS edit needed** — the IONOS wildcard already points at the VPS. + +## What does NOT follow this pattern + +- `netbird.hubris.network` (and any future `*.proxy.hubris.network`) uses the netbird-proxy / HostSNI passthrough path. Netbird handles its own cert via ACME cleanly because it *is* the passthrough target. + +## Related +- [DNS split-horizon](dns.md) +- [Caddy (121)](../containers/121-caddy.md) — cert source, internal counterpart +- [Mesh migration](mesh.md) — netbird is the transport between VPS and home +- [VPS hardening](vps-hardening.md) — fail2ban / nftables that the access logs feed +- [Artifacto on apps (105)](../containers/105-apps.md) — first publicly-exposed service + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-23 — `blog.hubris.network` exposed +WriteFreely on [apps (105)](../containers/105-apps.md). Whole host is public. + +### 2026-04-22 — pattern established with Artifacto +First service through the file-provider router. IONOS wildcard moved to the VPS this day. Cert mirror script + timer deployed on the PVE host. diff --git a/infrastructure/media-permissions.md b/infrastructure/media-permissions.md new file mode 100644 index 0000000..192cea6 --- /dev/null +++ b/infrastructure/media-permissions.md @@ -0,0 +1,95 @@ +# Media permissions — `media` GID 10000 + +Standard for any LXC reading/writing `/mnt/library` on [hubris](../hosts/hubris.md). Applied 2026-04-20. + +## Standard + +Every LXC that mounts `/mnt/library` participates in a shared `media` group with **GID 10000**. Shared subtrees are owned by that group with the setgid bit (`drwxrwsr-x`, mode `2775`), so new files auto-inherit the right group regardless of which container wrote them. + +## Why + +`/mnt/library` is a cross-container storage pool. \*arr writes, jellyfin reads, syncthing syncs, photoprism / immich / mulita scan, paperless ingests. Without a shared group, each container sees files as `nobody:nogroup` (unprivileged) or `www-data` (privileged 1:1) and the permission web collapses into one-off chmods. GID 10000 bridges privileged and unprivileged containers. + +## Onboarding a new LXC + +1. `pct set <id> -mp0 /mnt/library,mp=/mnt/library` (if not already mounted). +2. Inside the container: + ``` + groupadd -g 10000 media + usermod -aG media <service-user> # for every user that needs library access + ``` +3. If the container is **unprivileged** (check `pct config <id> | grep unprivileged`), append this idmap block to `/etc/pve/lxc/<id>.conf` (back up first): + ``` + lxc.idmap: u 0 100000 65536 + lxc.idmap: g 0 100000 10000 + lxc.idmap: g 10000 10000 1 + lxc.idmap: g 10001 110001 55535 + ``` + Then `pct stop <id> && pct start <id>`. +4. For systemd services running with `User=root` (not typical — but [syncthing (109)](../containers/109-syncthing.md) is one), add a drop-in with `SupplementaryGroups=media`. Systemd skips `initgroups()` for `User=root`. +5. `pct exec` sessions don't get supplementary groups (no initgroups). Use `sudo -i` or `su - <user>` inside the container to verify membership interactively. Real services use `initgroups` and work correctly. + +## State snapshot + +### Host + +- Group `media` GID 10000 exists. +- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000). +- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid): + - `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries + - `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / booklore + - `downloads` — \*arr stack output + - `images` — photoprism / immich / mulita + - `roms` — emu frontends + - `syncthing` — cross-device sync +- Container-specific subtrees intentionally **not** migrated (keep their own owner:group): + - `documents` (paperless, `www-data:www-data 750`) + - `homecloud` (nextcloud — its own permission model, easy to break) + - `marimo` (marimo venv) — *LXC since destroyed; review whether subtree still serves a purpose* + - `notes`, `heaper`, `sophia` (single-container use) + - `repos` (owner UID 102 GID 105 from inside [gitea](../containers/104-gitea.md) — don't touch) + +### LXCs with media-group membership + +| ID | Name | Priv | Media-group members | +| --- | --------------------------------------------- | ---- | --------------------------------------------- | +| 101 | [jellyfin](../containers/101-jellyfin.md) | **unpriv + idmap** | jellyfin | +| 103 | [paperless](../containers/103-paperless.md) | priv | www-data | +| 104 | [gitea](../containers/104-gitea.md) | priv | www-data, gitea | +| 105 | [apps](../containers/105-apps.md) | priv | www-data | +| 109 | [syncthing](../containers/109-syncthing.md) | **unpriv + idmap** | root (via `SupplementaryGroups=media` drop-in on `syncthing@.service`) | +| 114 | [nextcloud](../containers/114-nextcloud.md) | priv | www-data | +| 116 | [heaper](../containers/116-heaper.md) | priv | www-data | +| 119 | [sophia](../containers/119-sophia.md) | priv | www-data | +| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data | +| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr | + +> Some entries from the older snapshot — 100 (arr-yunohost), 107 (marimo), 110 (photoprism), 112 (immich) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology). + +Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*`. + +## Gotchas + +- **[apps (105)](../containers/105-apps.md) is a Docker host.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000` or `user: "<uid>:10000"` in compose. Booklore, audiobookshelf-in-docker, etc. need this per-container. +- **`pct exec` does NOT run initgroups.** So `pct exec 109 -- id` shows only the primary group. For interactive verification, use `pct exec 109 -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine. +- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed. +- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page. +- **`/etc/subgid`** must retain both `root:100000:65536` AND `root:10000:1`. Dropping the second breaks startup of any unprivileged LXC with the idmap block. +- **\*arr "Set Permissions" options** can override the setgid inheritance by explicitly chown'ing files. Leave those off, or set the group to `media`. Relevant to Sonarr/Radarr/qBittorrent on [arriman (122)](../containers/122-arriman.md). +- **Nextcloud** files under `/mnt/library/homecloud` are deliberately NOT in the media group. NC manages its own permission model. See [nextcloud (114)](../containers/114-nextcloud.md). +- **\*arr stack on arriman** required `MEDIACENTER_GID=10000` (not 13000) in `.env` because s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. See [arriman (122)](../containers/122-arriman.md#changelog). + +## Related +- [Hubris host](../hosts/hubris.md) +- All container pages list whether they're in the standard + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-26 — `MEDIACENTER_GID` fix on [arriman (122)](../containers/122-arriman.md) +qBit was erroring every torrent with "Permission denied" because `MEDIACENTER_GID=13000` was set as a supplementary GID via `group_add:`. Changed to 10000 (primary GID); fix described above is now standard. + +### 2026-04-20 — standard rolled out +GID 10000 hostgroup, idmap blocks for unprivileged LXCs, setgid 2775 on shared subtrees, `media` membership for service users in every participating LXC. diff --git a/infrastructure/mesh.md b/infrastructure/mesh.md new file mode 100644 index 0000000..5d1dbb0 --- /dev/null +++ b/infrastructure/mesh.md @@ -0,0 +1,94 @@ +# Mesh — Tailscale → Netbird migration + +The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target end-state. In-progress as of 2026-04-21. + +## Current state + +- **PVE host** uses Netbird (`wt0`, `100.122.38.109/16`). Its resolver is the local netbird daemon, which forwards to LAN/upstream — so the PVE host gets `*.hubris.network → 192.168.8.175` via the system resolver chain. +- **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration. +- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service. + +## Consequence — every LXC wired to Authentik needs an internal override + +Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`. + +Two techniques. Pick by HTTP-client behavior. + +### A) `/etc/hosts` override + +Works for libc `getaddrinfo` clients: curl, wget, most Go/Python/Ruby apps, gitea. + +- Place the line **outside** the `# --- BEGIN PVE ---` / `# --- END PVE ---` markers. Proxmox rewrites everything inside that block on every container start. +- Belt-and-suspenders: `/etc/systemd/system/hubris-hosts-override.service` (oneshot, enabled, idempotent). + +### B) Local dnsmasq + +Required for clients that bypass `/etc/hosts`. **Nextcloud (PHP Guzzle + `OC\Http\Client\DnsPinMiddleware`) is one** — uses `dns_get_record()`, not `getaddrinfo`. Likely candidates for the same treatment: Python/Ruby/Java apps that do explicit DNS resolution before connecting. + +Recipe: +``` +apt install dnsmasq + +cat > /etc/dnsmasq.d/hubris-internal.conf <<EOF +address=/auth.hubris.network/192.168.8.175 +server=192.168.8.1 +server=1.1.1.1 +interface=lo +bind-interfaces +no-hosts +no-resolv +EOF + +# Set LXC default nameservers and live resolv.conf +pct set <id> --nameserver "127.0.0.1 192.168.8.1 1.1.1.1" +# Then update /etc/resolv.conf inside the LXC too. +``` + +### Known overrides applied + +| LXC | Technique | Notes | +| ------------------------------------------ | ---------------------------------------- | ----- | +| [104 (gitea)](../containers/104-gitea.md) | `/etc/hosts` + `hubris-hosts-override.service` | Standard | +| [114 (nextcloud)](../containers/114-nextcloud.md) | local dnsmasq @ 127.0.0.1 + resolv.conf reorder | Guzzle bypasses /etc/hosts | +| [105 (apps)](../containers/105-apps.md), inner containers | `extra_hosts:` in compose | Booklore, mulita, WriteFreely each ship with this | + +## Adding new LXCs + +- Don't add new LXCs to Tailscale; add them to Netbird. Tailscale is being decommissioned on hubris. +- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. If nameserver is `192.168.8.1` or `100.100.100.100`, add the hosts override. If it's the netbird daemon IP, skip. + +## Long-term fix + +Either: +- Split-horizon DNS at LAN/router level so `*.hubris.network → 192.168.8.175` for every LAN client. Eliminates all per-LXC overrides. +- Once Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain on the PVE host. + +## CRITICAL — never `docker compose up` Portainer-managed stacks + +[apps (105)](../containers/105-apps.md) runs multiple stacks deployed via the Portainer UI (`/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`). Running `docker compose up -d <svc>` from the host shell on these triggers a recreate of OTHER services in the stack — labels drift, compose treats the project as reconciling, containers get destroyed and remade. **This wiped Booklore's mariadb data on 2026-04-22** (bind mount `./mariadb/config` re-initialized fresh). + +Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-managed stacks: +1. Edit the compose in Portainer UI → **Stacks → <stack> → Editor → Update stack**. Portainer handles the recreate cleanly with its own state tracking. +2. Do NOT edit `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/docker-compose.yml` directly. +3. For data-preserving DNS overrides on stateful services, prefer Caddy-side handling (forward-auth) or the app's built-in OIDC over container-level `extra_hosts` when possible. + +## Related +- [DNS split-horizon](dns.md) +- [Authentik (124)](../containers/124-authentik.md) — the IdP that triggers most of these overrides +- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B +- [Gitea (104)](../containers/104-gitea.md) — example of Technique A +- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-22 — Booklore mariadb data wiped (lesson recorded) +The "never docker compose up Portainer-managed stacks" rule comes from this. See [apps (105)](../containers/105-apps.md#changelog). + +### 2026-04-22 — netbird mgmt host joined its own mesh +`82.165.190.79` is now a peer (`100.122.165.149`). LAN access + split-horizon DNS via PVE peer. See [VPS hardening](vps-hardening.md). + +### 2026-04-21 — overrides applied to gitea (104) and nextcloud (114) +Two techniques documented; Nextcloud forced the dnsmasq route because of Guzzle. diff --git a/infrastructure/monitoring.md b/infrastructure/monitoring.md new file mode 100644 index 0000000..79c5796 --- /dev/null +++ b/infrastructure/monitoring.md @@ -0,0 +1,75 @@ +# Monitoring — `claudio-monitor` + +Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only. + +## Why + +After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly. + +## Components + +- **Repo:** `dtoro/claudio-monitor` +- **Checkout:** `/opt/claudio-monitor` on the PVE host +- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually. +- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`. +- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`. +- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`). +- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped. +- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks). + +## Per-LXC data + +A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned. + +## Matrix-side plugin + +`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands: +- `/monitor status` +- `/monitor history [N]` +- `/monitor clear <key>` + +Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`. + +Events posted to claudio-bot `/notify`: +```json +{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}} +``` +`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`. + +## Current thresholds & active alerts + +Active warnings at the time monitor was deployed (2026-04-21): +- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9% +- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8% + +## Testing + +- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override). +- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`. +- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`. + +## Not yet deployed (tracked in plan file) + +- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md). +- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`. + +## Related repos + +- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`. +- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`. +- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon. + +## Related pages +- [Hubris host](../hosts/hubris.md) +- [claudio-bot (123)](../containers/123-claudio-bot.md) +- [HAOS VM (108)](../vms/108-haos.md) +- [Backups (disabled)](backups.md) +- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-21 — claudio-monitor stood up; thermal-watch removed +General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`. diff --git a/infrastructure/ssh-access.md b/infrastructure/ssh-access.md new file mode 100644 index 0000000..0cb907b --- /dev/null +++ b/infrastructure/ssh-access.md @@ -0,0 +1,45 @@ +# SSH access + +How to reach hubris and the VPS over SSH, and the dual-server gotcha. + +## Hubris + +Two SSH endpoints — easy to hit the wrong one. + +| Server | Listen | Auth | Notes | +| -------------- | ---------------------------- | --------------------------------- | ----- | +| OpenSSH | `0.0.0.0:22` | `authorized_keys` at `/etc/pve/priv/authorized_keys` (Proxmox cluster-synced; symlinked from `/root/.ssh/authorized_keys`) | Standard. | +| Netbird SSH | `100.122.38.109:22022` | OIDC / browser auth — bypasses `authorized_keys` | If a client lands here it'll open a browser tab to authenticate, then sometimes hang. Force port 22 or use the LAN IP. | + +### Authorized root keys +- `root@hubris` (self, RSA) — original. +- `d.toro.v@pm.me` (ed25519) — user's iMac (`mac-mini.netbird.selfhosted`, LAN `192.168.8.174`), added 2026-04-22. + +### Notes +- Password auth is enabled on hubris but the root password is **not** the one the user expects. Prefer key flows; don't try `ssh-copy-id` blind. +- Off-LAN access from the iMac uses the LAN path. As of 2026-04-22 the iMac's Netbird tunnel to hubris was P2P healthy but no packets were captured on `wt0`; needs revisit if remote access becomes critical. + +## VPS (`82.165.190.79` / `100.122.165.149`) + +- **Mesh-only.** Public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. +- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`. +- `ListenAddress` is still 0.0.0.0; gating is firewall-layer. +- Authorized root keys: PVE (`root@hubris`), iMac (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer before disabling its access paths. + +See [VPS hardening](vps-hardening.md) for the firewall + fail2ban rules and recovery paths. + +## Related +- [Hubris host](../hosts/hubris.md) +- [Mesh migration](mesh.md) +- [VPS hardening](vps-hardening.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-23 — VPS SSH hardened to mesh-only +Public `:22` blocked at nftables. Key-only sshd. See [VPS hardening](vps-hardening.md). + +### 2026-04-22 — iMac key authorized on hubris +`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`. diff --git a/infrastructure/vps-hardening.md b/infrastructure/vps-hardening.md new file mode 100644 index 0000000..138fca4 --- /dev/null +++ b/infrastructure/vps-hardening.md @@ -0,0 +1,84 @@ +# VPS hardening — `82.165.190.79` / `100.122.165.149` + +IONOS VPS that runs the Netbird control plane and the [public ingress traefik](ingress.md). Hardened 2026-04-23 from its stock-Plesk state. + +## At a glance +- **Hostname:** `inspiring-ramanujan.82-165-190-79.plesk.page` +- **OS:** Debian 13 +- **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)). +- **Public:** `82.165.190.79` (`ens6`). +- **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`. + +## SSH + +- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`. +- **Mesh-only**: public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. `ListenAddress` itself is still `0.0.0.0` — gating is firewall-layer. +- Authorized root keys: PVE (`root@hubris`), Mac Mini (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer **before** disabling its access paths. + +## Firewall — nftables (`inet hubris-fw`) + +Config at `/etc/nftables.conf`, service enabled. + +- Public iface `ens6`. Wireguard iface `wt0`. +- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6. Everything else drops. +- `wt0` fully accepted in INPUT. `lo` accepted. +- **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below. +- Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops. +- Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too. + +## fail2ban + +- **Jail `traefik-4xx`** tails `/var/log/traefik/access.log` (bind-mounted from container). Filter at `/etc/fail2ban/filter.d/traefik-4xx.conf` matches 401/403/404/429 from `blog-public@file` or `artifacto-public@file` routers only — netbird-grpc traffic isn't considered. +- Tunables: `findtime=600, maxretry=30, bantime=3600`. +- **Action** at `/etc/fail2ban/action.d/nft-hubris.conf` adds/removes elements from `inet hubris-fw banned4` with per-element timeout. + +### CRITICAL invariant — wireguard / fail2ban + +**Bans must never affect `wt0` or wireguard UDP.** The FORWARD chain explicitly `accept`s the following *before* the ban check: +- `udp 51820` (wireguard) +- `udp 3478` (STUN) +- `ct state established,related` + +The INPUT ban rule is scoped to `iifname "ens6"`. + +Violating this takes the mesh down for every home device (they share one public IP) and the only recovery is IONOS console → `nft flush set inet hubris-fw banned4`. + +## Traefik access log + +- Written to `/var/log/traefik/access.log` on the host via a bind mount added to `/opt/docker-compose.yml` (traefik volumes include `/var/log/traefik:/logs`) plus `--accesslog.filepath=/logs/access.log`. +- CLF format. Real client IP arrives correctly because docker userland-proxy is off — see [public ingress](ingress.md). + +## Plesk / mail / FTP / Dr.Web + +Stopped and disabled (not uninstalled). All of: +`dovecot`, `dovecot.socket`, `postfix`, `postfix@-`, `pc-remote`, `xinetd`, `plesk-task-manager`, `plesk-web-socket`, `sw-cp-server`, `sw-engine`, `plesk-repaird`, `plesk-repaird.socket`, `drwebd`. + +`psa.service` is masked (was a one-shot boot bootstrap). `/etc/cron.d/plesk-backup-manager-task` renamed to `.disabled`. + +Reverse: `systemctl unmask psa; systemctl enable --now <svc>`. + +## Auto-patching + +- `unattended-upgrades` enabled (stock). +- Drop-in at `/etc/apt/apt.conf.d/52hubris-reboot.conf` sets auto-reboot at **04:00 UTC** when `/var/run/reboot-required` is set. +- Runs inside the stock `apt-daily-upgrade.timer`. + +## Recovery paths + +Ordered by preference: +1. **SSH via mesh** — primary. Any mesh peer with an authorized key. +2. **IONOS web console** (my.ionos.com → VPS → Console) — uses the system password, not SSH keys. Bypasses any firewall misconfig. +3. **IONOS rescue mode** — boot rescue, mount rootfs, edit `/etc/nftables.conf` or `/etc/ssh/sshd_config.d/10-hubris-hardening.conf` to a known-good state, reboot. + +## Related +- [Public ingress (VPS traefik)](ingress.md) +- [Mesh migration](mesh.md) — VPS as a mesh peer +- [SSH access](ssh-access.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-23 — hardened +nftables firewall, mesh-only SSH, fail2ban traefik jail, Plesk disabled, auto-reboot 04:00 UTC, wireguard/fail2ban invariant established. diff --git a/investigations/2026-04-21-hubris-crash-loop.md b/investigations/2026-04-21-hubris-crash-loop.md new file mode 100644 index 0000000..7e164b8 --- /dev/null +++ b/investigations/2026-04-21-hubris-crash-loop.md @@ -0,0 +1,86 @@ +# 2026-04-21 — Hubris crash loop (thermal + USB drive) + +## Summary + +[`hubris`](../hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders. + +## Timeline + +### 2026-04-19 — drive attached +External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../infrastructure/backups.md). Pre-attach uptime had been 33 days stable. + +### 2026-04-19 → 2026-04-21 — first crashes +Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. + +### 2026-04-21 — crash cadence escalates +23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once `cpu-epp.service` started applying `EPP=balance_power`. + +### 2026-04-21 — first set of mitigations applied + +- **`cpu-epp.service`** deployed. Sets `scaling_governor=powersave` + EPP=`balance_power` at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. +- **Crash capture**: `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). +- **`rasdaemon`** installed (Debian pkg; mcelog is retired) — logs MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query `ras-mc-ctl --summary`. +- **UAS blacklist** for the drive: `/etc/modprobe.d/usb-storage-quirks.conf` → `options usb-storage quirks=090c:2320:u`. Forces BOT instead of UAS for the SMI bridge. +- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running. + +### 2026-04-22 — recurrence after 30 h 37 m +Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty. + +### 2026-04-22 — `cpu-epp.service` design bug fixed +Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`. + +### 2026-04-22 — drive removed (A/B test) +User physically removed the external USB drive. [Backup timers disabled](../infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause. + +### 2026-04-23 — SSD cooling + thermal pads installed +Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically. + +#### 2026-04-23 thermal-pad verdict +Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. **The load-bearing signal is the lifetime warning-time counter, not absolute sensor1** — watch that going forward, not sensor1 threshold. + +### 2026-04-28 — A/B test passing so far +3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established. + +## Root cause + +Two-contributor: +1. **Thermal**: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under `performance` governor. Mitigated by `cpu-epp.service`. In `amd-pstate=active` mode the governor MUST be `powersave` for EPP values to take effect — `performance` silently ignores them. +2. **Storage**: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far). + +Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so `fwupdmgr` can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+. + +## Diagnostic notes + +- **Pure hardware hangs leave no trace.** `pstore` stays empty across reboots despite `panic_on_oops/hardlockup/softlockup/rcu` + softdog `soft_panic=1` being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic. +- **First check** if hubris crash-loops again with the same silent-cutoff signature: `cpu-epp.service` is active and `/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` is `balance_power`. +- **Non-ECC RAM means memory faults are silent.** If `cpu-epp` is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out `pcie_aspm=off pci=nomsi` workarounds in `/etc/default/grub`. +- **Grub escalation** (`pcie_aspm=off pcie_port_pm=off pci=nomsi`) **deliberately NOT applied** yet to keep the drive-removal test clean. Enable only if crashes continue without the drive. +- `mcelog` is not in Debian 13 repos; `rasdaemon` is the replacement. + +## BIOS update path + +Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a `.exe` flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email `service@gmktec.com` with the S/N to confirm a newer BIOS / AGESA level actually exists. `fwupd` is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically. + +## Mitigations applied + +| Mitigation | Status | Notes | +| ----------------------------------------- | -------- | ---------------------------------------------------- | +| `cpu-epp.service` (powersave + balance_power) | Active | Unit ordering fixed 2026-04-22 to run before guests | +| Crash capture (sysctl + softdog + watchdog) | Active | Catches everything but pure silicon hangs | +| `rasdaemon` | Active | Persistent telemetry for MCE / memory / PCIe AER | +| UAS blacklist for `090c:2320` | Active (drive currently absent) | `/etc/modprobe.d/usb-storage-quirks.conf` | +| Mount-on-demand (`backup-usb.sh`) | Active (drive currently absent) | Backup unit `ExecStartPre`/`ExecStopPost` hooks | +| Drive physically removed | Active | A/B test in progress | +| Thermal pads on NVMes | Active 2026-04-23 | Steady-state composite 47 °C / sensor1 60 °C | +| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive | + +## Affected nodes +- [Hubris host](../hosts/hubris.md) +- [Backups (disabled)](../infrastructure/backups.md) +- [Monitoring](../infrastructure/monitoring.md) + +## Open questions +- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.) +- If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?) +- Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation. +- Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed. diff --git a/investigations/index.md b/investigations/index.md new file mode 100644 index 0000000..c018325 --- /dev/null +++ b/investigations/index.md @@ -0,0 +1,16 @@ +# Investigations + +Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here. + +## Index + +| Date | Title | Status | +| ------------ | ------------------------------------------------------------------ | ------------- | +| 2026-04-21 | [Hubris crash loop — thermal + USB drive](2026-04-21-hubris-crash-loop.md) | Drive removal A/B test passing as of 2026-04-28 (3+ days uptime) | + +## Conventions + +- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days. +- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions. +- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom. +- Link back from every node's changelog that's affected. diff --git a/operations/commands.md b/operations/commands.md new file mode 100644 index 0000000..33c298a --- /dev/null +++ b/operations/commands.md @@ -0,0 +1,56 @@ +# Operations cheatsheet + +Run from the [hubris host](../hosts/hubris.md) as root. When working from `/root` on Linux you're already on hubris — don't `ssh hubris` / `ping hubris`. + +## Proxmox CLI + +| Command | Use | +| --- | --- | +| `pct list` / `qm list` | List LXC containers / VMs | +| `pct config <id>` / `qm config <id>` | Container / VM config | +| `pct exec <id> -- <cmd>` | Run command inside an LXC without entering it (no initgroups — see [media permissions](../infrastructure/media-permissions.md)) | +| `pct enter <id>` | Shell into a container | +| `pct start <id>` / `pct stop <id>` | Boot / halt a container | +| `pvesm status` | Storage pools status | +| `pvesh get /nodes --output-format json` | Node summary as JSON | +| `pvesh get /nodes/hubris/lxc/<id>/status/current` | Live container status | +| `pvesh get /cluster/resources --type vm --output-format json` | Bulk per-LXC CPU/mem/disk (used by [claudio-monitor](../infrastructure/monitoring.md)) | +| `pveversion` | PVE version | +| `journalctl -u pve-cluster -n 100` | PVE service logs | + +## Storage + +- Shared mount: `/mnt/library` (ext4 on lvmthin `library`). +- Bind into a container: `pct set <id> -mp<N> /mnt/library/<sub>,mp=/data` +- For the standard whole-tree mount: `pct set <id> -mp0 /mnt/library,mp=/mnt/library`. See [media permissions](../infrastructure/media-permissions.md) for the GID-10000 onboarding recipe. + +## Reverse proxy + +- Caddyfile: `/etc/caddy/Caddyfile` on [LXC 121](../containers/121-caddy.md). Tracked in `dtoro/caddy-conf`. Edits should be committed + pushed — see [auto-deploy](../infrastructure/auto-deploy.md). +- Hot reload: `pct exec 121 -- systemctl reload caddy`. +- Validate: `pct exec 121 -- caddy validate --config /etc/caddy/Caddyfile`. + +## DNS + +- Split-horizon entries: `/etc/dnsmasq.d/hubris-split.conf` on [LXC 124](../containers/124-authentik.md). Hard restart on edit: `pct exec 124 -- systemctl restart dnsmasq`. SIGHUP isn't reliable. +- Verify: `dig @192.168.8.180 +short <host>.hubris.network`. +- See [DNS](../infrastructure/dns.md). + +## Web access + +- `https://proxmox.hubris.network` or `https://192.168.8.77:8006` — Proxmox UI + +## Telemetry quick checks + +- `ras-mc-ctl --summary` — summary of any RAS events (memory / PCIe AER / thermal) since boot +- `ras-mc-ctl --errors` — full event log +- `cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` — should be `balance_power` +- `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` — should be `powersave` +- `ls /sys/fs/pstore/ /var/lib/systemd/pstore/` — panic traces from a previous crash (empty for pure hardware hangs — see [investigation](../investigations/2026-04-21-hubris-crash-loop.md)) + +## Related +- [Hubris host](../hosts/hubris.md) +- [Containers index](../containers/index.md) +- [DNS](../infrastructure/dns.md) +- [Monitoring](../infrastructure/monitoring.md) +- [Auto-deploy](../infrastructure/auto-deploy.md) diff --git a/vms/108-haos.md b/vms/108-haos.md new file mode 100644 index 0000000..6ac7ca6 --- /dev/null +++ b/vms/108-haos.md @@ -0,0 +1,44 @@ +# 108 — `haos-16.3` + +Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC, hence the qm tenant). + +## At a glance +- **Type:** QEMU VM +- **HAOS version:** 16.3 (last verified) +- **IP:** `192.168.8.101` +- **Resources:** 4 GiB RAM, 32 GiB boot disk +- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → `192.168.8.101:8123` + +## Auth + +Native OIDC via the HACS integration `christiaangoossens/hass-oidc-auth` (repo `https://github.com/christiaangoossens/hass-oidc-auth`). + +Key gotchas: +- HAOS containers don't honor the Network-panel DNS. Set Supervisor DNS via: + ``` + ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1" + ``` + so OIDC discovery resolves internally to [authentik (124)](124-authentik.md). +- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen). +- YAML config: + - `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created). + - `features.default_redirect: true` — skip the welcome-splash so users land on the normal HA login page. +- Run `ha core restart` after each config change — HA caches DNS and OIDC discovery across frontend reloads. + +## Telemetry + +HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting. + +## Related +- [Authentik (124)](124-authentik.md) +- [Caddy (121)](121-caddy.md) +- [DNS](../infrastructure/dns.md) +- [Monitoring](../infrastructure/monitoring.md) + +## Changelog + +### 2026-04-28 — wiki entry created +Initial documentation. + +### 2026-04-21 — wired into Authentik via HACS hass-oidc-auth +Supervisor DNS pointed at LXC 124 dnsmasq; YAML features for auto-linking + default redirect.