dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

docs: move narrative wiki under knowledge/wiki/ (phase 3)

Browse Source 
Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-07-06 14:35:23 +02:00
parent bb5c0717a1
commit 8a6422bd7d
129 changed files with 249 additions and 264 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
knowledge/wiki/containers/101-jellyfin.md Normal file
View File

@@ -0,0 +1,160 @@
# 101 — `jellyfin`
Media server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.
## At a glance
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **IP:** `192.168.8.246`
- **Privilege:** privileged (recreated on strong as priv)
- **Resources:** 4 cores / 8 GiB RAM / 1 GiB swap / 16 GiB rootfs
- **GPU:** `/dev/dri/renderD128` + `/dev/dri/card0` (AMD Radeon 680M iGPU + RX 7600 dGPU) passed via `dev0` / `dev1` in LXC config
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096`
- **Version:** Jellyfin 10.11.11 (apt package, Ubuntu 24.04 noble repo)
- **FFmpeg:** jellyfin-ffmpeg7 7.1.4
## Service / port map
| Service | Listen | Notes |
| -------- | ------ | ----- |
| jellyfin | `:8096` | HTTP (caddy terminates TLS) |
## Hardware acceleration (VAAPI)
GPU is passed through to the LXC via `dev0: /dev/dri/renderD128,gid=993` and
`dev1: /dev/dri/card0,gid=44` in
`/etc/pve/lxc/101.conf` (strong). The `jellyfin` user is in the `render` (GID 993) and
`video` groups inside the container.
| GPU | Model | Role |
|-----|-------|------|
| Radeon 680M | iGPU (AMD Ryzen 7 PRO 6850U) | Primary VAAPI encoder/decoder |
| RX 7600 | dGPU (add-in) | Secondary transcode, HEVC/AV1 encoding |
Encoding settings (`/etc/jellyfin/encoding.xml`):
- `HardwareAccelerationType`: `vaapi`
- `VaapiDevice`: `/dev/dri/renderD128`
- `EnableHardwareEncoding`: `true`
- `AllowHevcEncoding`: `true`
- `AllowAv1Encoding`: `true`
- `EnableTonemapping`: `true`
- `HardwareDecodingCodecs`: h264, hevc, vc1, vp9, av1
- `EnableThrottling`: `true`
- `EnableSegmentDeletion`: `true`
Trickplay (`/etc/jellyfin/system.xml`):
- `EnableHwAcceleration`: `true`
- `EnableHwEncoding`: `true`
## Authentik SSO (OIDC)
Jellyfin uses the [SSO-Auth plugin](https://github.com/9p4/jellyfin-plugin-sso)
v4.0.0.4 for Authentik OIDC login. No Caddy forward-auth gate — the SSO plugin
handles auth directly via OIDC redirect flow.
### Architecture
```
User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :8096
↓
Login page with "Sign in with Authentik" button
↓ (click)
/sso/OID/start/Authentik
↓ (302 redirect)
auth.hubris.network OIDC
↓ (login)
/sso/OID/redirect/Authentik?code=...&state=...
↓
Jellyfin SSO plugin validates token → logged in
```
### Components
1. **SSO-Auth plugin** — installed at `/var/lib/jellyfin/plugins/SSO-Auth_4.0.0.4/`
- Config: `/var/lib/jellyfin/plugins/configurations/SSO-Auth.xml`
- Provider name: `Authentik`
- OIDC endpoint: `https://auth.hubris.network/application/o/jellyfin/`
- `SchemeOverride`: `https` (required — without it, plugin generates
`http://` redirect URIs that Authentik rejects)
- `EnableAuthorization`: `false` (prevents plugin from overwriting admin
permissions on each SSO login — see
[jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md))
- `OidScopes`: `["email"]` (openid+profile added by default by the plugin;
must be non-null or `OidChallenge()` throws `ArgumentNullException`)
2. **Authentik OIDC provider** — `Provider for Jellyfin` (PK 6)
- Client ID: `vt61t5Y2ZVtN6l3QjitkBvwUJjFKvSyl4TDBXcJx`
- Redirect URI: `https://media.hubris.network/sso/OID/redirect/Authentik`
- Application slug: `jellyfin`
3. **SSO button injection** — `/usr/share/jellyfin/web/sso-inject.js`
- Injected via `<script defer src="sso-inject.js?v=3">` in `index.html`
- Polls for `.readOnlyContent` div on the login page, adds "Sign in with
Authentik" button linking to `/sso/OID/start/Authentik`
- Cache-busted with `?v=N` parameter (bump on changes)
- **Lost on apt upgrade** — re-inject the script tag and copy the JS file
after `apt-get upgrade jellyfin*`
4. **Caddy config** — no forward-auth gate for media.hubris.network:
```caddy
media.hubris.network {
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
reverse_proxy 192.168.8.206:8096
}
```
### Known issues / pitfalls
- **`OidScopes` must be non-null** — if the field is missing from the plugin
config XML, `OidChallenge()` throws `System.ArgumentNullException`. Always
include `OidScopes` in the provider config (even if empty array).
- **`SchemeOverride: "https"` is required** — without it, the plugin generates
`http://` redirect URIs (from the internal HTTP listener). Authentik rejects
them with "Redirect URI Error".
- **SSO button JS is not served by the plugin** — the `__plugin/SSO-Auth.js`
endpoint returns 404 on Jellyfin 10.11.x when the plugin is installed
manually (not via Jellyfin's plugin manager). The `sso-inject.js` workaround
in `index.html` is the fallback.
- **No Caddy forward-auth gate** — the SSO plugin's OIDC redirect flow is
incompatible with Caddy's `import authentik` forward-auth. If both are
enabled, the forward-auth intercepts the OIDC callback and breaks the flow.
Use one or the other, not both. SSO plugin (OIDC redirect) is preferred.
- **API key for setup** — a temp API key can be inserted directly into the
`ApiKeys` SQLite table for automated configuration:
```sql
INSERT INTO ApiKeys VALUES (1, '2026-07-04', '2026-07-04', 'setup', 'jf-setup-key-...');
```
## Permissions
Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; `/mnt/media_local` on strong's ludo-lvm is owned `root:media` with mode `2775`.
## Related
- [Caddy reverse proxy](121-caddy.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads
- [DNS split-horizon](../infrastructure/dns.md)
- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config
## Changelog
### 2026-07-06 — wiki: IP, host, GPU, mount path updated for strong migration
Updated At-a-glance: IP 206→246, host hubris→strong, mount /mnt/library→/mnt/media_local, GPU Radeon 760M→680M+RX7600, privilege unpriv→priv. Permissions section updated. Changelog entry in 122-arriman.md updated similarly.
### 2026-07-04 — VAAPI hardware acceleration + Authentik SSO + resource bump
- Upgraded Jellyfin 10.11.8 → 10.11.11 (purge + reinstall to fix DB migration bug)
- Enabled VAAPI hardware acceleration (Radeon 760M): h264/hevc/vc1/vp9/av1 decode + encode
- Bumped resources: 2→4 cores, 4→8 GiB RAM, 512→1024 MiB swap
- Enabled trickplay HW acceleration + throttling + segment deletion
- Installed SSO-Auth plugin v4.0.0.4 with Authentik OIDC provider
- Configured `SchemeOverride: https`, `OidScopes: ["email"]`, `EnableAuthorization: false`
- Injected SSO button via `sso-inject.js` in web `index.html`
- Removed Caddy forward-auth gate (incompatible with SSO plugin OIDC flow)
- **Database was wiped** during cache relocation attempt — no LVM snapshot
existed. All watch states, user accounts, and library configs lost.
Libraries re-added via setup wizard.
### 2026-04-28 — wiki entry created
Initial documentation. No config changes.
### 2026-04-20 — joined the `media` GID 10000 standard
Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.

89
knowledge/wiki/containers/102-nfs-export.md Normal file
View File

@@ -0,0 +1,89 @@
# 102 — `nfs-export`
Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clients that can't use the host's PVE virtiofs path — currently only [100-zimaos](../vms/100-zimaos.md), which ships a kernel without virtiofs support.
## At a glance
- **Hostname:** `nfs-export`
- **IP:** `192.168.8.200` (static; LAN-only, no Caddy in front because NFS is L4)
- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy)
- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, apps)
## What it does
```
/mnt/library (host ext4 on nvme1n1)
│
├── bind-mounted into 7 other LXCs (jellyfin, paperless, …)
└── bind-mounted into LXC 102
│
└── nfs-kernel-server exports /mnt/library
│
└── consumed by VM 100 (ZimaOS)
```
Same inodes, same page cache. The NFS server is just one more access path on top of a tree that 8 other consumers already share — see [media permissions](../infrastructure/media-permissions.md).
## Export config
`/etc/exports`:
```
/mnt/library 192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)
```
Initially started as `ro`; promoted to `rw` on 2026-05-14 after the Files-UI evaluation confirmed (a) the library renders correctly as a folder under `/DATA`, (b) thumbnails are generated, (c) the squash works — a write from ZimaOS appears on `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention used by Nextcloud and mule-images.
### Guardrails (in order of importance)
1. **`all_squash,anonuid=33,anongid=10000`.** Every write from ZimaOS records on disk as `www-data:media` (uid 33, gid 10000), the same identity Nextcloud and mule-images use. Keeps the existing tree convention from drifting. See [`project_media_perms`](../infrastructure/media-permissions.md).
3. **Subnet restriction `192.168.8.0/24`.** No public/mesh access; LAN only.
4. **`no_subtree_check`** + **`sec=sys`** — standard performance/auth pair for a homelab.
5. **No `crossmnt`** because `/mnt/library` is a single ext4 filesystem on the host (no nested mounts to traverse).
### What we're *not* doing yet
- No per-subdir export. ZimaOS sees the whole tree; access is controlled by filesystem permissions (`drwxr-x---` private dirs like `documents/`, `notes/`, `heaper/` aren't readable by `media` group, so ZimaOS-as-squashed-uid won't see them either).
- No write-back. Until we promote to `rw`, ZimaOS can't write — eliminates the lock-domain split concern between NFS clients (NLM/v4) and local LXCs (POSIX flock).
- No Authentik / forward-auth. NFS doesn't sit behind HTTP, so the standard caddy+authentik path doesn't apply. Subnet ACL is the only auth.
## Why this LXC exists (vs serving from host)
We considered three options before building this:
| Option | Outcome |
|---|---|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../../investigations/index.md)). Rejected. |
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
Rationale lives in [the install plan](https://git.hubris.network/dtoro/Homelab-Docs) `/root/.claude/plans/i-wannt-you-to-nifty-muffin.md` on hubris.
## Operations
- **Reload exports** after editing `/etc/exports`: `pct exec 102 -- exportfs -ra`
- **List active exports:** `pct exec 102 -- exportfs -v`
- **Watch from outside:** `showmount -e 192.168.8.200`
- **Service health:** `pct exec 102 -- systemctl is-active nfs-server rpcbind`
- **Restart cleanly:** `pct restart 102` (ZimaOS will retry the mount via `nofail`)
- **Destroy + rebuild:** `pct stop 102 && pct destroy 102 --purge` — reversible in seconds; only ZimaOS notices
## Open items
- Consider tightening the export to subdirs (e.g. `movies`, `tv`, `music`, `audiobooks`, `books`, `images`, `podcasts`, `roms`) if you don't want ZimaOS reachable into `documents/`, `notes/`, `heaper/`, etc. — though those private subdirs are already invisible to `media`-group perms.
- ZimaOS architecture finding: the Drives panel only enumerates physical/block devices via `GET /v2/local_storage/storages` (read-only API, no POST). Network shares **cannot** appear as Drives — they show up as folders in Files. This is intentional in CasaOS's design; don't try to work around it. Library-as-folder is the supported model.
- Consider adding Samba to this LXC if a future Mac/iOS client needs SMB on the same tree — same LXC, no host changes.
- No PBS backup (no PBS configured on hubris); the container is fully described in this page + `pct config 102`, rebuild from scratch in <2 min if lost.
## Related
- [100-zimaos](../vms/100-zimaos.md) — the only consumer today
- [media permissions](../infrastructure/media-permissions.md) — uid 33 / gid 10000 standard
- [DNS](../infrastructure/dns.md) — `nfs-export.hubris.network` entry (direct, no Caddy)
## Changelog
### 2026-05-14 — Promoted to rw; squash behaviour verified
After ZimaOS Files UI evaluation passed (library renders as folder under `/DATA`, thumbnails work, ZimaOS Drives panel ignores NFS by design), flipped export to `rw`. Tested: writing `/DATA/library/.zimaos-rw-test` from ZimaOS appears on hubris's `/mnt/library` owned `www-data:media` (uid 33, gid 10000), confirming `all_squash,anonuid=33,anongid=10000` works as designed. Also discovered the dead end: ZimaOS's `GET /v2/local_storage/storages` is the source of the Drives panel; it returns only physical storage and rejects POST/PUT — network shares cannot be promoted to Drives.
### 2026-05-14 — LXC built; NFS export live (read-only)
Privileged Debian 13 container created with bind-mount `/mnt/library`. `nfs-kernel-server` installed and enabled; export `/mnt/library` to `192.168.8.0/24` with `ro,all_squash,anonuid=33,anongid=10000`. Smoke-tested from hubris host (mounted, listed library tree, confirmed RO). `nfs-export.hubris.network` added to LXC 124 dnsmasq.

41
knowledge/wiki/containers/103-paperless.md Normal file
View File

@@ -0,0 +1,41 @@
# 103 — `paperless`
Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/documents` and serves OCR'd indexed copies.
## At a glance
- **Hostname:** `paperless`
- **IP:** `192.168.8.130`
- **Privilege:** privileged
- **Resources:** 2 cores / 3 GiB RAM / 8 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (consumes `documents/`)
- **Public hostname:** [`paperless.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8000`
## Service / port map
| Service | Listen | Notes |
| ------------------- | ------- | -------------------------------- |
| paperless-webserver | `:8000` | HTTP |
| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
## Auth
Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
## Storage
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
## Known issues
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [Authentik](124-authentik.md)
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — wired into Authentik forward-auth
Cookie-domain-level proxy provider, embedded outpost binding. `/api/*` bypass added in Caddy site block to keep mobile clients working.

61
knowledge/wiki/containers/104-gitea.md Normal file
View File

@@ -0,0 +1,61 @@
# 104 — `gitea`
Self-hosted git server. Source of truth for all `dtoro/*` repositories that auto-deploy across the lab.
## At a glance
- **Hostname:** `gitea`
- **IP:** `192.168.8.121`
- **Privilege:** privileged
- **Resources:** 1 core / 1 GiB RAM / 8 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (under `/mnt/library/repos`, owner UID 102 / GID 105 — don't touch from other containers)
- **Public hostname:** [`git.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000`
- **Version:** 1.25.4 (last verified)
## Service / port map
| Service | Listen | Notes |
| ------- | -------- | ------------------------------ |
| gitea | `:3000` | HTTP web + git over HTTP |
| gitea SSH | `:22` | git over SSH |
| `gitea-deploy-webhook` | `127.0.0.1:9797` | Receives push events for the customizations repo |
## Customizations
`/var/lib/gitea/custom` is a checkout of [`dtoro/gitea-customizations`](https://git.hubris.network/dtoro/gitea-customizations).
- `templates/custom/footer.tmpl` injects:
- **Online3DViewer** v0.18.0 (vendored under `public/assets/o3dv/`) for inline 3D preview of CAD raw-file links: 3dm, 3ds, 3mf, amf, bim, brep, dae, fbx, fcstd, glb, gltf, ifc, igs, iges, stp, step, stl, obj, off, ply, wrl.
- **PlantUML client JS** (vanilla, vendored from `gitea.com/davidsvantesson/plantuml-code-highlight`). Browser calls `https://git.hubris.network/_plantuml/img/<encoded>` — caddy `handle_path /_plantuml/*` rewrites and proxies to the PlantUML server on [apps (105)](105-apps.md) `:8079`.
- `home.tmpl` is the anonymous landing page (Homesauce hero + sign-in/explore buttons).
- O3DV binary assets are *not* tracked — `deploy.sh` fetches them on first run.
## Auto-deploy
Push to `dtoro/gitea-customizations` master → gitea webhook → `http://127.0.0.1:9797/deploy` (loopback) → `gitea-deploy-webhook.service` validates HMAC against `/etc/gitea-deploy/secret` → runs `/opt/gitea-customizations/deploy.sh`. Then `systemctl restart gitea` if templates changed.
`app.ini` `[webhook] ALLOWED_HOST_LIST` includes `127.0.0.1`, `192.168.8.175`, `192.168.8.205`, `192.168.8.230`, `192.168.8.136`, `192.168.8.77` — every webhook target across the lab. **Don't strip these when editing app.ini.** See [auto-deploy](../infrastructure/auto-deploy.md) for the full list.
Git creds for the deploy service: `/etc/gitea-deploy/git-credentials` (mode 600), referenced via `credential.helper = store --file=/etc/gitea-deploy/git-credentials` in the repo's `.git/config` (because `ProtectHome=true` blocks `/root`).
## DNS workaround
LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (router DNS resolves the public IONOS A record, which is a dead end for LAN clients — see [DNS](../infrastructure/dns.md)). Belt-and-suspenders systemd unit `hubris-hosts-override.service` re-applies it idempotently. Override line is **outside** the `# --- BEGIN PVE ---` markers (Proxmox rewrites inside).
## Related
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
- [PlantUML server lives on apps (105)](105-apps.md)
## Changelog
### 2026-06-24 — terminalito deploy webhook (id 12)
Push webhook on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` ([trmnl (128)](128-trmnl.md)); `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — `app.ini` ALLOWED_HOST_LIST extended for Artifacto webhook
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
For OIDC integration with [authentik (124)](124-authentik.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.

142
knowledge/wiki/containers/105-apps.md Normal file
View File

@@ -0,0 +1,142 @@
# 105 — `apps`
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. Booklore migrated to [grimmory (130)](130-grimmory.md) on 2026-06-29.
## At a glance
- **Hostname:** `apps`
- **IP:** `192.168.8.205`
- **Privilege:** privileged (so container UID = host UID for bind-mount sanity)
- **Resources:** 2 cores / 4 GiB RAM / 30 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library`
- **Public hostnames:** see table below
## Public hostname → service map
| Hostname | Container | Backend port | Notes |
| --------------------------------- | ---------------- | ------------ | ----- |
| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. |
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
| `mcp.hubris.network` | homelab-mcp | `:9810` | MCP server over SSE. No SSO — agent-facing, gated at source-IP layer. See [homelab-context](../infrastructure/homelab-context.md). |
| `secrets.hubris.network` | secrets-issuance | `:9820` | Per-client age-key issuance. Mesh+LAN source-IP gated (`MESH_SUBNETS`). |
## Stacks
### Artifacto (`/opt/artifacto/`)
Self-hosted HTML-artifact share (Go + SQLite). Repo `dtoro/Artifacto` is the working tree itself — `git push origin main` triggers rebuild.
- Compose: tracked `docker-compose.yml` binds `./data:/data` and `127.0.0.1:3100:3000`. `docker-compose.override.yml` uses the `!override` YAML tag on `volumes:` and `ports:` to redirect `data` to `/mnt/library/documents/artifacts`.
- Data dir: `/mnt/library/documents/artifacts` — owner **UID 10001 : GID 10001** (matches the `app` user baked into the Dockerfile), mode 750. Parent `/mnt/library/documents` stays `www-data:www-data 750`; bind targets the child directly so no traversal needed.
- `.env` (untracked) holds `ADMIN_PASSWORD`, `SESSION_SECRET` (`openssl rand -hex 32`), `BASE_URL=https://artifacto.hubris.network`, `SSO_GATEWAY_SECRET` (must match `ARTIFACTO_GATEWAY_SECRET` in `/etc/caddy/caddy.env` on [caddy](121-caddy.md)). Don't `git checkout .env`.
- Auth: Authentik forward-auth on the admin surface (`/`, `/login`, `/a/*`, `/api/*`). `X-Authentik-Username` is honored only when the `X-Artifacto-Gateway` header matches `SSO_GATEWAY_SECRET` — prevents direct LAN connections from spoofing identity.
- Public exposure: VPS traefik gates path scope to share URLs; admin surface returns 404 at the edge. See [ingress](../infrastructure/ingress.md).
#### Auto-deploy pipeline (Artifacto)
Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webhook.py`, secret in `/etc/artifacto-deploy/secret`, systemd unit `artifacto-deploy-webhook.service`. Listens on `0.0.0.0:9798`, validates `X-Gitea-Signature` HMAC-SHA256, filters to `refs/heads/main`, runs deploy in a daemon thread (returns 202 immediately). Gitea webhook id 7 on `dtoro/Artifacto`.
- Logs: `pct exec 105 -- journalctl -u artifacto-deploy-webhook -f`
- Manual deploy: `pct exec 105 -- /opt/artifacto-deploy/deploy.sh`
- Health: `pct exec 105 -- curl -s http://127.0.0.1:9798/health` → `ok`
- Slug length bumped 5 → 8 chars (in `internal/store/artifacts.go:41`) once the endpoint went public.
### PlantUML (`/opt/plantuml/`)
`plantuml/plantuml-server:jetty` listening on `192.168.8.205:8079`, context path `/plantuml` (`BASE_URL=plantuml`). Reached only via [gitea](104-gitea.md)'s caddy `handle_path /_plantuml/*` rewrite — no DNS record of its own. Same-origin from gitea so the embedded markdown previewer can call it without CORS.
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
### homelab-mcp (`/opt/homelab-mcp/`)
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
`/opt/homelab-context/` (this LXC is itself an enrolled
[homelab-context](../infrastructure/homelab-context.md) client). Source
is `dtoro/Homelab-Docs/mcp/server.py`; service unit
`homelab-mcp.service` binds `0.0.0.0:9810`. DNS-rebinding protection
disabled at the FastMCP layer because mesh+LAN gating is the actual
trust boundary.
- Endpoint: `https://mcp.hubris.network/mcp` (Caddy → `:9810`). StreamableHTTP transport (POST `/mcp`).
- 14 tools registered: `get_host`, `list_services`, `find_service`,
`get_topology`, `search_docs`, `get_page`, `get_changelog`, `whoami`,
`list_my_secrets` (context); `get_service_status`, `tail_log`,
`list_lxcs`, `get_lxc_state`, `ping_service` (read-only management).
- Management tools shell out via SSH using a dedicated `mcp-reader` key
(planned; not provisioned yet — management tools fail until then).
#### Auto-deploy pipeline (homelab-mcp)
Receiver at `/opt/homelab-mcp/mcp/deploy/webhook/` (in-tree alongside
the server code). Listens on `0.0.0.0:9811`, secret in
`/etc/homelab-mcp-deploy/secret`, systemd unit
`homelab-mcp-deploy.service`. Gitea webhook id 10 on
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
restarts on push.
### secrets-issuance (`/opt/secrets-issuance/`)
Tiny HTTP service that issues per-client age keypairs the first time
each client calls `/issue`. Idempotent: subsequent calls return the
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
`secrets-issuance.service`. State (per-client keys + denylist) lives at
`/var/lib/secrets-issuance/` (NOT in the repo).
- Endpoint: `https://secrets.hubris.network/issue` (Caddy → `:9820`).
- `/revoke` is admin-token-gated by `/etc/secrets-issuance/admin-token`;
shreds the local key file and adds the hostname to the denylist.
Called by `homelab client remove`.
- Trust subnets today: `100.122.0.0/16` (Netbird), `100.64.0.0/10`
(Tailscale), `192.168.8.0/24` (LAN). Tighten if the LAN gets
untrusted devices.
#### Auto-deploy pipeline (secrets-issuance)
Receiver at `/opt/secrets-issuance/secrets-issuance/deploy/webhook/`,
port `9821`, secret in `/etc/secrets-issuance-deploy/secret`, unit
`secrets-issuance-deploy.service`. Gitea webhook id 11.
### WriteFreely (`/opt/writefreely-docker/`)
Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts.
## Permissions
- LXC is privileged → in-container UID == host UID.
- Member of [media GID 10000](../infrastructure/media-permissions.md). Adding `media` to the LXC alone is *not* enough for inner Docker containers — each container needs `--group-add 10000` or `user: "<uid>:10000"` to write into `/mnt/library`.
## Related
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](124-authentik.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
## Changelog
### 2026-06-29 — Booklore migrated to Grimmory on LXC 130
Booklore stack removed from Portainer. MariaDB dump taken first, then restored into [grimmory (130)](130-grimmory.md)'s fresh MariaDB. `books.hubris.network` Caddy backend updated to `192.168.8.213:6060`. Authentik OIDC provider updated to Public client type (PKCE) for Grimmory compatibility.
### 2026-05-20 — homelab-mcp + secrets-issuance live
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
`homelab-mcp.service` on `:9810` (MCP read+management surface) and
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
Caddy fronts both with Let's Encrypt; new vhosts on
[caddy](121-caddy.md), split-horizon DNS entries on
[authentik (124)](124-authentik.md). Gitea webhook ids 10 + 11 wire
auto-deploy. LXC is itself an enrolled context client
(`/opt/homelab-context/`).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — Artifacto deployed + Authentik gateway-secret auto-login wired
Repo `dtoro/Artifacto`. Compose, data dir, `.env`, webhook (id 7), Caddyfile site block, gateway-secret pattern, public-internet share exposure on the VPS — all set up that day. Slug length bumped 5 → 8 for brute-force resistance.
### 2026-04-22 — Booklore mariadb data wiped (lesson, not a feature)
`docker compose up -d` from the host shell on a Portainer-managed stack recreated services and re-initialized the bind-mounted `./mariadb/config`. Recovered by reseeding from snapshot. Rule recorded: edit Portainer-managed stacks via the Portainer UI.
### 2026-04-22 — Portainer wired into Authentik (native OAuth2)
Custom-claim `portainer_uid` scope. CLI added `--trusted-origins docker.hubris.network`.
### 2026-04-22 — `app.ini` host whitelist on gitea extended
Added `192.168.8.205` so the Artifacto webhook delivery wouldn't be rejected. See [gitea (104)](104-gitea.md).
### 2026-04-21 — Booklore + WriteFreely wired into Authentik OIDC
Native OIDC for both. `extra_hosts` override added inside each container so DNS resolves `auth.hubris.network` to the Caddy IP.
### 2026-04-20 — PlantUML server deployed
Used by [gitea (104)](104-gitea.md)'s footer.tmpl injection. Same-origin via caddy `handle_path /_plantuml/*` rewrite — no separate hostname.

56
knowledge/wiki/containers/106-auth-outpost.md Normal file
View File

@@ -0,0 +1,56 @@
# 106 — `auth-outpost`
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../../investigations/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
## At a glance
- **Hostname:** `auth-outpost`
- **IP:** `192.168.8.6` (static)
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
- **Mounts:** none
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](124-authentik.md)
## Role
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../../investigations/2026-05-31-authentik-vps-migration.md).
## Service / port map
| Service | Listen | Notes |
| --- | --- | --- |
| authentik proxy outpost | `192.168.8.6:9000` | `/outpost.goauthentik.io/*` (ping, auth/caddy) |
## Config paths
- `/opt/authentik-outpost/docker-compose.yml` — `ghcr.io/goauthentik/proxy:2026.5.2`, `AUTHENTIK_HOST=https://auth.hubris.network`, `AUTHENTIK_INSECURE=false`, port bound `192.168.8.6:9000`.
- `/opt/authentik-outpost/.env` (mode 600, **untracked**) — `AUTHENTIK_TOKEN` for outpost `hubris-lan-outpost` (sops-encrypt into `secrets/` — TODO Phase 5).
## The outpost object (VPS Authentik)
- Outpost **`hubris-lan-outpost`** (type proxy), providers: `hubris-forward-auth` (domain, Paperless + domain-level apps), `Provider for Torrent` (qBittorrent), `Provider for Artifacto` (single).
- Health: VPS admin → Applications → Outposts → last-seen current; `curl http://192.168.8.6:9000/outpost.goauthentik.io/ping` → `204`; container log `Successfully connected websocket`.
## Caddy wiring
[Caddy (121)](121-caddy.md) `(authentik)` snippet `reverse_proxy`/`forward_auth` → `http://192.168.8.6:9000` (was `192.168.8.180:9000` on LXC 124). Tracked in `dtoro/caddy-conf`.
## `sso.hubris.network` — the callback domain (critical)
Domain-level (`forward_domain`) providers redirect the browser to `{external_host}/outpost.goauthentik.io/callback` after login. On LXC 124 that `external_host` was `auth.hubris.network`, which resolved (LAN) to Caddy → the 124 outpost — same box, so it worked. Post-migration `auth.hubris.network` points to the **VPS core**, so the callback hit the wrong outpost → **400 / redirect-uri mismatch**.
Fix: the LAN outpost gets its **own** domain.
- DNS: `sso.hubris.network → 192.168.8.175` (Caddy). *(in dnsmasq today — must be carried into Technitium in DNS Phase 2.)*
- Caddy: site `sso.hubris.network { tls dns ionos; reverse_proxy 192.168.8.6:9000 }` (in `caddy-conf`).
- Authentik: `hubris-forward-auth` and `Provider for Torrent` have `external_host=https://sso.hubris.network`; their `redirect_uris` must match (`set_oauth_defaults()` regenerates them from `external_host` — changing `external_host` alone does NOT update them).
- `Provider for Artifacto` is `forward_single` on its own domain, so it was unaffected.
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
## Related
- [124 — authentik](124-authentik.md) — old embedded-outpost host (now DNS-only)
- [Caddy (121)](121-caddy.md) — forward-auth consumer
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
- [Authentik VPS migration](../../../investigations/2026-05-31-authentik-vps-migration.md)
## Changelog
### 2026-06-06 — Authentik session lifetime extended to 30 days
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../../investigations/2026-06-06-authentik-session-lifetime.md).
### 2026-06-01 — created; forward-auth cut over from LXC 124
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../../investigations/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](124-authentik.md).

67
knowledge/wiki/containers/107-dns.md Normal file
View File

@@ -0,0 +1,67 @@
# 107 — `dns`
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](124-authentik.md); single-purpose, one job.
## At a glance
- **Hostname:** `dns`
- **IP:** `192.168.8.2` (static — stable, decoupled from any app)
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 1 GiB / 8 GiB rootfs
- **Created:** 2026-06-01, Debian 13. Its own resolver is `1.1.1.1` (no circular dependency on the DNS it serves).
## Role
Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recursive forwarding (`1.1.1.1`, `8.8.8.8`) for everything else. Technitium runs in Docker (`technitium/dns-server:latest`, `network_mode: host`), web console on `:5380`.
## The `hubris.network` zone
- Specific A overrides: app names → `192.168.8.175` (Caddy), `nfs-export → 192.168.8.200`, `auth/sso/...` as needed.
- `auth.hubris.network → 82.165.190.79` (VPS Authentik), `sso.hubris.network → 192.168.8.175` (LAN forward-auth outpost).
- **Wildcard** `*.hubris.network → 82.165.190.79` — mirrors the public IONOS wildcard so undefined names (e.g. `netbird`) resolve to the VPS, matching public behaviour.
- **MX / SPF-TXT / CAA replicated from public** so an authoritative zone doesn't shadow `hubris.network` email/cert records.
## Config / access
- `/opt/technitium/docker-compose.yml`; admin password in `/opt/technitium/admin_password.txt` (mode 600 — sops-encrypt in Phase 5).
- Console: `http://192.168.8.2:5380` (user `admin`).
- API: `http://192.168.8.2:5380/api/...` (token via `/api/user/login`). Zone was built via the API.
## Who points here
- **NetBird mesh peers:** resolve via the **NetBird managed DNS zone**, kept in sync *from* this Technitium (see dns-sync below). The `home-lab-dns` nameserver group (`→ 192.168.8.2`) is a thin fallback forwarder.
- **Homelab DHCP clients:** Technitium's own DHCP scope hands out `192.168.8.2` as the DNS server for `192.168.8.x` leases (see DHCP section below).
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
## dns-sync (Technitium = authoring source)
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
## DHCP
Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
- **Scope:** `homelab` — `192.168.8.241 – 192.168.8.254`
- **Gateway:** `192.168.8.1` (Proxmox `vmbr0` alias)
- **DNS:** `192.168.8.2` (self)
- **Lease time:** 24 h
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk.
## Related
- [124 — authentik](124-authentik.md) — retired host of the old dnsmasq
- [DNS split-horizon](../infrastructure/dns.md)
- [Mesh](../infrastructure/mesh.md)
## Changelog
### 2026-06-24 — A record `trmnl.hubris.network → 192.168.8.175`
Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)); propagated to the NetBird managed zone by `dns-sync`.
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
### 2026-06-03 — DHCP pool narrowed to `.241–.254`
Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../plans/2026-06-03-dhcp-pool-exclude-static-ips.md).
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
### 2026-06-02 — DHCP server enabled; replaces Slate AX DHCP
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
### 2026-06-01 — created; replaced dnsmasq on 124
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](124-authentik.md).

111
knowledge/wiki/containers/114-nextcloud.md Normal file
View File

@@ -0,0 +1,111 @@
# 114 — `nextcloud`
Personal cloud / file collaboration. Source-of-truth for the photo libraries surfaced by [mulita (120)](120-mule-images.md).
## At a glance
- **Hostname:** `nextcloud`
- **IP:** `192.168.8.224`
- **Privilege:** privileged
- **Resources:** 4 cores / 6 GiB RAM / 25 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (Nextcloud user files under `/mnt/library/homecloud`)
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
## Auth
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](124-authentik.md#per-app-username-override-pattern-authentik) for the full pattern.
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
## DNS workaround (load-bearing)
Nextcloud's PHP HTTP client (`OC\Http\Client\DnsPinMiddleware`) calls `dns_get_record()` directly — it bypasses `/etc/hosts`. So the Authentik OIDC discovery URL needs **real DNS** answering with `192.168.8.175` for `auth.hubris.network`. Recipe applied:
- Local `dnsmasq` listening on `127.0.0.1:53`, `/etc/dnsmasq.d/hubris-internal.conf` with `address=/auth.hubris.network/192.168.8.175`, `server=192.168.8.1`, `server=1.1.1.1`, `interface=lo`, `bind-interfaces`, `no-hosts`, `no-resolv`.
- `pct set 114 --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"` so the LXC starts with the right resolver order.
- `/etc/hosts` override kept as belt-and-suspenders.
Also needs `allow_local_remote_servers=true` in `config.php`.
> Once internal DNS at the router level handles `*.hubris.network`, this LXC-local dnsmasq can go away. See [DNS](../infrastructure/dns.md).
## Storage
Files at `/mnt/library/homecloud`. Owned by Nextcloud's own permission model — **deliberately not on the `media` group**, NC manages it itself and would complain. See [media permissions](../infrastructure/media-permissions.md).
## Web stack (Apache event MPM + php-fpm)
Apache 2.4 with **`mpm_event`** as the MPM and PHP served via **php-fpm 8.4** over the Unix socket `/run/php/php8.4-fpm.sock` (mod_php disabled). Routing to FPM is via the distro's `/etc/apache2/conf-available/php8.4-fpm.conf` (gated by `<IfModule !mod_php8.c>`, so just `a2dismod php8.4` flips Apache onto FPM).
Why this matters: with mod_php every Apache worker is forced onto `mpm_prefork` (one heavy process per connection, full PHP interpreter loaded in each). On FPM + event, Apache workers are ~5–8 MB and async; PHP work is done by a separate FPM pool that keeps opcache hot. Concurrent upload throughput jumps significantly because Apache no longer blocks per-connection on a heavy PHP process.
### PHP-FPM pool
- Socket: `/run/php/php8.4-fpm.sock` (owner/group `www-data`)
- `pm = dynamic`, `pm.max_children = 30`, `pm.start_servers = 6`, `pm.min/max_spare_servers = 4/12`, `pm.max_requests = 500`
- Steady-state ~6 idle workers at ~80 MB each — bump `max_children` only if peak concurrency exceeds 30.
### PHP limits (FPM + CLI both)
Set in `/etc/php/8.4/fpm/php.ini` **and** `/etc/php/8.4/cli/php.ini` (the latter so `occ`/cron see the same ceilings):
| key | value |
|---|---|
| `upload_max_filesize` | `16G` |
| `post_max_size` | `16G` |
| `memory_limit` | `512M` |
| `max_execution_time` | `3600` |
| `max_input_time` | `3600` |
| `output_buffering` | `Off` |
After edits: `systemctl reload php8.4-fpm`. Caddy in front (`cloud.hubris.network`) imposes no body-size limit, so these are the effective ceiling.
### Opcache + JIT
`/etc/php/8.4/mods-available/opcache.ini` — full config maintained, not commented stubs:
- `opcache.enable=1`, `opcache.memory_consumption=256`, `opcache.interned_strings_buffer=32`, `opcache.max_accelerated_files=20000`
- `opcache.jit=tracing`, `opcache.jit_buffer_size=64M`
### Apache mod_reqtimeout
`/etc/apache2/mods-available/reqtimeout.conf` — relaxed body trickle so slow-WAN uploads don't get killed:
- `RequestReadTimeout header=20-40,MinRate=500`
- `RequestReadTimeout body=20,MinRate=100` (was `body=10,MinRate=500`)
## MariaDB tuning
Overrides in `/etc/mysql/mariadb.conf.d/99-nextcloud-tuning.cnf`:
- `innodb_buffer_pool_size = 1G` (default was 128M)
- `innodb_log_file_size = 256M`
- `innodb_flush_log_at_trx_commit = 2` (group-commit; trades a few ms of durability for write throughput — acceptable for NC)
- `innodb_flush_method = O_DIRECT`, `innodb_io_capacity = 2000` / `..._max = 4000`
- `max_allowed_packet = 256M`, `tmp_table_size = 64M`, `max_heap_table_size = 64M`
Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` needs a clean restart).
## Related
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
- [Authentik (124)](124-authentik.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-05-13 — Apache keepalive bumped (and TCP BBR on host)
Follow-up to the FPM/MPM/MariaDB round earlier the same day. After a WiFi-bound Linux client kept seeing ~2 MB/s downloads despite a 152 Mbps speed test (so the WiFi medium wasn't the cap), bumped Apache `KeepAliveTimeout 5 → 60` and `MaxKeepAliveRequests 100 → 1000` so the mirall desktop client reuses one TLS+TCP connection across many sequential file fetches instead of paying handshake + slow-start each time. Paired with TCP BBR + `fq` qdisc enabled on the [hubris host](../hosts/hubris.md#2026-05-13--tcp-bbr--fq-qdisc-enabled), which directly addresses single-stream throughput on jittery links. Server-side loopback baseline post-BBR is ~404 MB/s for a single 23 MB CR2 download via Caddy. Client-side diagnosis pending (curl single-stream + parallel tests) to determine if the remaining gap is mirall config (bandwidth limit / `parallelNetworkRequests`) or path shaping.
### 2026-05-13 — throughput tuning: FPM + event MPM + opcache JIT + MariaDB
Investigated upload/download speed. Bottlenecks were inside this LXC — LAN (33.9 Gbit/s caddy→NC iperf3) and `/mnt/library` NVMe (714 MB/s sustained) were not the limit. Changes:
- **Apache mod_php → php-fpm 8.4** (`a2dismod php8.4`). Flips PHP from in-process to FPM-over-Unix-socket. Apache workers now ~5–8 MB instead of ~80 MB each.
- **MPM prefork → event** (`a2dismod mpm_prefork && a2enmod mpm_event`). Async I/O, far better concurrency for parallel uploads.
- **opcache JIT enabled** (`tracing`, 64M buffer), `memory_consumption=256`, `interned_strings_buffer=32`, `max_accelerated_files=20000`.
- **MariaDB**: `innodb_buffer_pool_size 128M → 1G`, `innodb_log_file_size 96M → 256M`, `flush_log_at_trx_commit 1 → 2`, `O_DIRECT`, higher I/O capacity. New file `99-nextcloud-tuning.cnf`.
- **Apache mod_reqtimeout**: body trickle relaxed from `body=10,MinRate=500` → `body=20,MinRate=100` so slow-WAN uploads aren't killed.
- **PHP-FPM ini** synced to match what the apache2/cli inis already had (FPM was stuck at 2M/8M because FPM wasn't serving traffic before, but now it does).
Backups: `/etc/php/8.4/fpm/php.ini.bak.20260513`, `/etc/apache2/mods-available/reqtimeout.conf.bak.20260513`, `/etc/php/8.4/mods-available/opcache.ini.bak.20260513`. MariaDB tuning is a new file (rollback = `rm`).
Verified: `apache2ctl configtest` Syntax OK, `occ status` clean, `occ setupchecks` shows ✓ on memory_limit / output_buffering / DB indices, `status.php` returns 200 in <40 ms via Caddy with HTTP/2.
### 2026-05-10 — PHP upload limits raised
Apache + CLI php.ini bumped from distro defaults (`upload_max_filesize=2M`, `post_max_size=8M`, `memory_limit=128M`, `max_execution_time=30`) to NC-recommended (`16G/16G/512M/3600`). Backups at `/etc/php/8.4/{apache2,cli}/php.ini.bak-20260510-*`. Was causing slow/failing uploads via web UI.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — Nextcloud-rooted libraries shipped (mulita)
[mulita (120)](120-mule-images.md) now reads photos from `/nextcloud-users/<nc_user>/files/...` and writes mutations back via WebDAV. Pre-migration DB dump kept at `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump`.
### 2026-04-21 — wired into Authentik (native OIDC + nc_uid override)
Local `dnsmasq` added on this LXC because Guzzle bypasses `/etc/hosts`. Resolver order set to `127.0.0.1 192.168.8.1 1.1.1.1`. Username override `dtoro → admin`.

83
knowledge/wiki/containers/118-elementsynapse.md Normal file
View File

@@ -0,0 +1,83 @@
# 118 — `elementsynapse`
Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
## At a glance
- **Hostname:** `elementsynapse`
- **IP:** `192.168.8.242`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
- **Mounts:** none from `/mnt/library`
- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008`
- **`server_name`:** `avispero` (intentionally different from the public URL).
- **Database:** Synapse runs on **SQLite** (`/var/lib/matrix-synapse/homeserver.db`, ~90 MiB). PostgreSQL on `127.0.0.1:5432` is dedicated to the mautrix bridge databases (one per bridge), not Synapse itself.
## Service / port map
| Service | Listen | Notes |
| -------------- | ------- | -------------------------------- |
| synapse | `:8008` | Matrix client-server / federation HTTP. Caddy terminates TLS for `matrix.hubris.network`. |
| synapse-admin | `:5173` | Web UI (vite dev server) for Synapse admin API. |
| mautrix-whatsapp | `:29318` | Docker, `network_mode: host`, compose in `/root/mautrix-whatsapp/`. |
| mautrix-meta | `127.0.0.1:29319` | Facebook/Instagram bridge, `/root/mautrix-meta/`. |
| mautrix-signal | `127.0.0.1:29328` | `/root/mautrix-signal/`. |
| mautrix-slack | `127.0.0.1:29335` | `/root/mautrix-slack/`. |
| mautrix-linkedin | `127.0.0.1:29341` | `/root/mautrix-linkedin/`. |
## Bridges (mautrix)
All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name>/` — no systemd unit, `restart: unless-stopped`. Their appservice registration YAMLs live in `/etc/matrix-synapse/` and are listed in `homeserver.yaml` under `app_service_config_files`. If a bridge daemon is down, Synapse logs `push_bulk … ConnectionRefusedError` and retries every 512 s; restart the bridge with `cd /root/mautrix-<name> && docker compose up -d`.
## Operational notes
- Rootfs is on `pve` LVM-thin, grow with `pct resize 118 rootfs +<N>G` from the host — resize2fs runs automatically.
- `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
## Known issues
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-06-06 — DHCP drift fixed: internal `/etc/network/interfaces` was `dhcp` despite Proxmox static config
**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
**Fix:**
- Replaced `iface eth0 inet dhcp` with `iface eth0 inet static` + `address 192.168.8.239/24` + `gateway 192.168.8.1`
- `ifdown eth0 && ifup eth0` applied the static IP
- Killed lingering dhclient process
- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
### 2026-05-15 — phantom-notification cleanup for `@admin`
After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.
2. **The noisiest room** (a Signal-bridged DM) was flooded with `m.room.redaction` events from `@signalbot:avispero` (disappearing messages being deleted). Synapse counts redactions as notifications by default, but Element X doesn't render them — so they were invisible "ghosts".
Fix (after stopping Synapse and backing up `homeserver.db`):
- `DELETE FROM event_push_actions WHERE stream_ordering <= max(receipt_stream)` per (user, room) — clears receipt-superseded rows.
- `DELETE FROM event_push_actions WHERE event_id IN (SELECT event_id FROM events WHERE type='m.room.redaction')` — removes redaction-driven notifications.
- Rebuild `event_push_summary` from the remaining rows (preserve `thread_id='main'` to satisfy the NOT-NULL check constraint).
Result: 125 → 4 (legitimate WhatsApp unreads).
Then, to stop this recurring, an **override push-rule** was installed for `@admin` via the client API:
```
PUT /_matrix/client/v3/pushrules/global/override/suppress_redactions
{ "actions": [], "conditions": [{ "kind": "event_match", "key": "type", "pattern": "m.room.redaction" }] }
```
With empty `actions`, all `m.room.redaction` events are silently ignored by the push system. Inspect/remove via the same endpoint. (Note: rule ids beginning with `.` are reserved for server defaults — custom ids cannot start with a dot.)
### 2026-05-15 — disk-full incident, rootfs grown to 16 GiB, bridges restarted
Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed `OSError: [Errno 28] No space left on device` during flush. Fix: `pct resize 118 rootfs +8G` (host `pve` VG had plenty of headroom), `journalctl --vacuum-size=50M` to recover ~200 M, `systemctl restart matrix-synapse`, then `docker compose up -d` in each `/root/mautrix-*` directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).
### 2026-04-28 — wiki entry created
Initial documentation.

23
knowledge/wiki/containers/119-sophia.md Normal file
View File

@@ -0,0 +1,23 @@
# 119 — `sophia`
Single-container workload "sophia". Reads/writes under `/mnt/library/sophia`.
## At a glance
- **Hostname:** `sophia`
- **IP:** `192.168.8.157`
- **Privilege:** privileged
- **Resources:** 2 cores / 1 GiB RAM / 10 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library`
- **Public hostname:** none
## Permissions
LXC has `www-data` in the [media GID 10000 group](../infrastructure/media-permissions.md). The `/mnt/library/sophia` subtree itself is single-container and not on the setgid shared group.
## Related
- [Hubris host](../hosts/hubris.md)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-04-28 — wiki entry created
Stub. Document what runs inside next time we work on this LXC.

330
knowledge/wiki/containers/120-mule-images.md Normal file
View File

@@ -0,0 +1,330 @@
# 120 — `mule-images`
Hosts `mule-image` — the photos app at `photos.hubris.network`. PhotoPrism + Go sidecar + SvelteKit, replacing the legacy FastAPI/Celery stack as of 2026-05-22 (see Changelog). Auto-deploys from `dtoro/mule-image` on `git push origin main`.
## At a glance
- **Hostname:** `mule-images`
- **IP:** `192.168.8.136`
- **Privilege:** privileged
- **Resources:** 6 cores / 12 GiB RAM / 60 GiB rootfs / 2 GiB swap
- **Mounts:** `/mnt/library` ↔ `/mnt/library`; `/dev/dri/{card0,renderD128}` passed through for VA-API video accel on the AMD Phoenix1 iGPU.
- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → path-routed to PhotoPrism `:2342` / sidecar `:8000` / nginx-static `:3000`.
## Stack (`/opt/mule-image`)
`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose stack: base `docker-compose.yml` + LAN-port-rebind `docker-compose.override.yml` (untracked) + VA-API `docker-compose.gpu.yml`. The SvelteKit frontend is built on the host and served as a static bundle by nginx — no `vite dev` in prod.
| Service / process | Port (LAN) | Notes |
| ------------------------ | --------------------- | ----- |
| pp-mariadb | (internal 3306) | MariaDB 11; holds PhotoPrism's `photoprism` DB + sidecar's `mule_sidecar.*` |
| pp-app | `0.0.0.0:2342` | PhotoPrism `:latest`; `PHOTOPRISM_FFMPEG_ENCODER=h264_vaapi` |
| pp-sidecar | `0.0.0.0:8000` | Go service (file rename / folder ops / heap convert / dup scan / per-photo marks); also reconciles `USER_BASEPATHS` into PhotoPrism's `auth_users` every 60 s |
| nginx (host process) | `0.0.0.0:3000` | Serves the SvelteKit static bundle from `web/build/` |
**Per-user scoping** is driven by `USER_BASEPATHS=admin:admin/files, muli:muli/files` in `.env`. Sidecar applies it to PhotoPrism's `auth_users` table on boot + every 60s, `mkdir -p`s each target so PhotoPrism's ACL filter has somewhere to point.
`.env` is **untracked** — `git reset --hard` won't touch it, but `git clean -fdx` would. Holds PP_*, MariaDB passwords, SIDECAR_DB_PASSWORD, OIDC_* (existing `mule-image` Authentik app, redirect URI now `/api/v1/oidc/redirect`), `USER_BASEPATHS`, and `PP_FFMPEG_ENCODER=vaapi`.
`docker-compose.override.yml` is also untracked — it `!override`s the upstream `127.0.0.1:port` mappings to `0.0.0.0:port` so cross-host Caddy on [LXC 121](121-caddy.md) can reach pp-app + sidecar.
## Library access
PhotoPrism reads the library **directly off the bind-mounted filesystem** — no Nextcloud webhook integration in the new stack. The base path is `/mnt/library/homecloud` and per-user scoping comes from `USER_BASEPATHS` (see Stack section above).
- `dtoro` is mapped to NC user `admin` → `/mnt/library/homecloud/admin/files/`
- `muli` is mapped to NC user `muli` → `/mnt/library/homecloud/muli/files/`
PhotoPrism's container user is `33:10000` (`www-data:media`), matching the host ownership of the NC tree. The legacy `oc_webhook_listeners` rows + `NEXTCLOUD_WEBHOOK_SECRET` are gone — when NC writes via WebDAV (still its own primary surface), PhotoPrism picks up the new file on its next index pass.
## Authentication
OIDC via Authentik. App slug `mule-image`, redirect URI `https://photos.hubris.network/api/v1/oidc/redirect` (PhotoPrism's auto-derived path; PhotoPrism builds it from `PHOTOPRISM_SITE_URL`). `OIDC_REGISTER=true` auto-creates a PhotoPrism user at role `user` on first SSO login. The sidecar's basepath reconciler then assigns their scoped folder.
The `mule-image` Authentik app's redirect URI was migrated from the legacy FastAPI `/api/v1/auth/oidc/callback` path on 2026-05-22 — same client ID/secret were reused. The separate `mule-photos-new` Authentik app was deleted in the same operation.
## Auto-deploy
Push to `dtoro/mule-image` `main` → gitea webhook → `http://192.168.8.136:9797/deploy` → `mule-deploy-webhook.service`:
- Validates HMAC against `/etc/mule-deploy/secret`
- Filters to `refs/heads/main`
- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
- `git fetch && git reset --hard origin/main`, force-recreates **only the sidecar** image (so PhotoPrism's `Session` HMAC key in `pp/storage/config/hub.yml` doesn't rotate and invalidate every in-flight OIDC state cookie), reconciles `pp-app` + `mariadb` in place, `cd web && npm ci && npm run build`, `systemctl reload nginx`.
- Conditionally layers `docker-compose.gpu.yml` when `/dev/dri/renderD128` exists, and always layers `docker-compose.override.yml` (the LAN-port rebind) when present.
Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`. Gitea webhook id 6. `app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`.
Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`.
For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials` (mode 600) — same token as `/etc/caddy-deploy/git-credentials` on [caddy](121-caddy.md).
## Related
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
- [Authentik (124)](124-authentik.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Gitea (104)](104-gitea.md)
## Changelog
### 2026-05-22 — Cutover to PhotoPrism stack (Mulimage 2.0)
The `new` branch of `dtoro/mule-image` was merged into `main` as commit
`70dc1b6`. The merge replaces the legacy FastAPI + PostgreSQL + Celery
+ React stack with PhotoPrism + Go sidecar + MariaDB + SvelteKit, lifted
in from the M0 evaluation on (now-destroyed) [LXC 127](#).
Cutover on 120 (this session):
- Bumped to 6 cores / 2 GB swap. `/dev/dri/{card0,renderD128}` already
passed through; added an opt-in `docker-compose.gpu.yml` overlay that
the deploy script layers in when the device is present. PhotoPrism now
uses `h264_vaapi` instead of `libx264`.
- Installed host nginx + a small `photos.conf` vhost serving the SvelteKit
static bundle from `/opt/mule-image/web/build/` on `:3000` — no Vite
dev server in prod. SvelteKit's `adapter-static` produces a real SPA
bundle.
- Replaced `/opt/mule-deploy/deploy.sh` with the 127-style multi-compose
deploy (force-recreates only sidecar to preserve PhotoPrism's Session
HMAC key; reconciles pp-app + mariadb in place; rebuilds web/ and
reloads nginx).
- `/opt/mule-image/.env` was rewritten to the PhotoPrism shape. Reused
passwords from `/root/mule-photos-new-secrets.txt`. `USER_BASEPATHS=
admin:admin/files, muli:muli/files` exposes both libraries.
- `docker-compose.override.yml` (untracked) `!override`s pp-app +
sidecar ports to `0.0.0.0` so cross-LXC caddy on [121](121-caddy.md)
can reach them.
- Authentik: `mule-image` app's redirect URI updated to `/api/v1/oidc/redirect`;
the separate `mule-photos-new` app deleted.
- Caddyfile (`dtoro/caddy-conf`): `photos.hubris.network` switched from
plain passthrough to path-matchers (PhotoPrism API + sidecar API +
nginx static catch-all), and `photos-new.hubris.network` was removed
entirely.
- Cleanups: 4 Nextcloud `oc_webhook_listeners` rows for the legacy
`/api/v1/internal/nc-webhook` endpoint deleted; gitea webhook id 9
(the `refs/heads/new` deploy hook for 127) deleted; `192.168.8.181`
removed from gitea's `ALLOWED_HOST_LIST`; `photos-new.hubris.network`
removed from dnsmasq.
- LXC 127 destroyed via `pct destroy 127` after the new stack passed
curl verification end-to-end (PP `/api/v1/status` returns
`{"status":"operational"}` through caddy; sidecar `/api/sidecar/healthz`
returns `{"ok":true,...}`).
- Rollback safety net: legacy `mulita` postgres dump at
`/root/backups/mulita-legacy-final-20260521-212036.sql.gz` on hubris
(10 MB, 16,155 photo rows). The `mule-image_mule_db_data` docker
volume on 120 retains the on-disk postgres data for at least 24 h
before housekeeping removes it.
### 2026-05-14 — Drop AI/vision pipeline, plain Postgres, DB↔FS refresh
**AI removal (commits `a27267f` + `423a73a`).** The OpenCLIP-on-ONNX
binary classifier (photography vs other) and all its scaffolding gone:
- Backend: `app/services/vision/`, `app/tasks/vision.py`,
`app/services/feature_flags.py`, `app/routers/features.py` deleted.
Admin `/admin/feature-flags`, `/admin/ai/{backfill,rescan}` endpoints
removed. `Photo.needs_review` column + `ix_photos_needs_review`
index dropped (alembic `0019_drop_ai_remnants`).
- Frontend: AI Settings tab, `useFeaturesQuery`, "Needs Review" sidebar
entry + filter, `needs_review` URL param plumbing, FeatureFlag types
all gone.
- Infra: `worker-vision` compose service + `models_data` volume
deleted. `worker-light` no longer runs `python -m
app.services.vision.bootstrap_models` before celery. Backend
Dockerfile drops the dedicated torch RUN layer. `requirements.txt`
drops `torch`, `torchvision`, `open-clip-torch`, `onnxruntime`.
**Postgres image swap.** `pgvector/pgvector:pg16` → `postgres:16`. The
0019 migration drops the (now-unused) `vector` extension first; the
SQL was pre-applied via `psql` against the still-pgvector container
and `alembic_version` stamped to `0019` so the new backend's
`bootstrap.py` upgrade-head was a no-op. After the swap surfaced a
collation-version mismatch (Bookworm glibc 2.36 → Trixie glibc 2.41),
the fix was `REINDEX DATABASE mulita` + `ALTER DATABASE … REFRESH
COLLATION VERSION` on `mulita`, `postgres`, `template1`. The data
volume was preserved across the image change.
**One-shot DB ↔ filesystem refresh.** New script
`backend/scripts/full_refresh.py` (run as
`docker exec -w /app mulita-backend python -m scripts.full_refresh
[--dry-run]`). Phases: `cleanup_data_integrity` → inline scan of every
active SourceRoot → `prune_missing_photos(dry_run=False)` → new
`prune_orphan_thumbnails` helper that removes
`/data/thumbs/{user}/{photo}/` dirs for photo_ids that left the DB.
First run: 0 missing photos, 1 stale folder row, **7982 orphan
thumbnail dirs** reaped.
**Manual cleanup post-deploy.** `docker compose up -d --build
--force-recreate` doesn't reap services removed from the YAML, so
`mulita-worker-vision` had to be `docker rm -f`'d by hand. The
`mule-image_models_data` volume was likewise removed manually.
### 2026-05-11 — Stop duplicating Nextcloud's indexing (Phase 1 + 2)
Two big architectural shifts in one session, both aimed at killing work mule was doing that NC already does for the same source files.
**Phase 1 — thumbnail proxy (commits `576b0c2`, `28738ac`).** Photos table gains `nextcloud_fileid` (migration `0018`). `GET /api/v1/photos/{id}/thumb/{size}` proxies NC's `/index.php/core/preview` keyed by that fileid, authenticated with the owner's encrypted app password (small=240, medium=640, large=1280). Worker now only writes the `medium` size to `/data/thumbs` (the vision worker still loads it from disk); `small` + `large` no longer touch disk. Disk fallback stays wired for legacy rows and the rare 404 from NC (iPhone JPEGs mis-extensioned as `.DNG` — verified). Existing 17,591 photos backfilled via `backend/scripts/backfill_nextcloud_fileid.py`. Tested with real CR2, real DNG, fake DNG, JPEG, HEIC — all green.
**Range support for `/original` (commit `18dce33`).** Pre-existing bug surfaced by Phase 1 testing: `<video>` couldn't play `.mov` because `FileResponse` was returning 200 with the whole body and no `Accept-Ranges` header, so browsers reported "format not supported." Now parses `Range: bytes=START-END`, returns 206 with `Content-Range`, streams in 1 MB chunks.
**Fix: backfill_gps was the actual CPU drain, not the watcher (commit `d24c64e`).** `_scan_all_source_roots_async` auto-queued `backfill_gps` 30 s after every container boot, which then re-queued `extract_metadata` for every photo with `latitude IS NULL` — ~60k pointless tasks per deploy, pinning worker-light at 180+% CPU for ~30 min. Killed the auto-trigger; manual `POST /api/v1/library/backfill-gps` still works.
**Phase 2 — webhook receiver replaces watchfiles (commits `362fbc6`, `f657e2c`).** New endpoint `POST /api/v1/internal/nc-webhook` (auth via `Authorization: Bearer $NEXTCLOUD_WEBHOOK_SECRET`, `hmac.compare_digest`) handles `NodeCreated|NodeWritten|NodeDeleted|NodeRenamed`. Maps NC's `/admin/files/...` path to the bind-mounted `/nextcloud-users/admin/files/...` and dispatches the same `scan_folder` / `handle_file_deletion` machinery the watcher used. Registered against NC via `backend/scripts/register_nc_webhooks.py` (idempotent: deletes existing webhooks targeting the same URL first). `mulita-worker-watcher` container retired; celery `--beat` folded into `worker-light` so `discard_missing_photos_beat` still fires. End-to-end verified by uploading a test JPEG through WebDAV — both `NodeCreated` and `NodeWritten` fire, mule receives + dispatches, 200 OK.
**NC cron tightened to `*/1` so webhook delivery latency drops from up to 5 min to ~60 s** (`crontab -u www-data` on LXC 114). NC dispatches webhook calls through its background-job queue; the cron interval = the worst-case latency. `*/5` was the default; `*/1` is the upstream recommendation anyway.
**Post-Phase-2 delete-roundtrip patches (commit `9408825`).** End-to-end testing of the NC↔mule deletion paths surfaced two real gaps:
1. *Folder delete* — NC fires one `NodeDeletedEvent` for the folder, not one per child. The webhook handler bailed with "unsupported extension" and photos under the deleted folder kept `is_discarded=false` until the 30-min reconcile sweep. Fix: `handle_directory_deletion()` does a single `UPDATE photos SET is_discarded=true WHERE filepath LIKE 'dir/%'` when the deleted path has no supported image extension.
2. *Resurrect on rewrite* — PUT-overwrite of a previously-discarded file fired `NodeWrittenEvent` → `scan_folder`, but scan_folder's "photo exists, skip" branch left `is_discarded=true`. Fix: when the existing row is discarded, flip `is_discarded=false`, clear `discarded_at`, re-queue `extract_metadata`.
Verified end-to-end on `Photos/MuleTestFolder/{test1,test2,nested/test3}.jpg`: `DELETE Photos/MuleTestFolder/` discarded all 3 in one shot; `PUT test1.jpg` back resurrected only that one.
**Known remaining gap — trashbin restore.** Moving a file out of `/dav/trashbin/.../trash/foo.jpg.dXXXX` back to `/files/...` via WebDAV MOVE fires **no event mule subscribes to**. NC's trashbin app emits its own internal event class that isn't in the `OCP\Files\Events\Node\*` set we registered with `webhook_listeners`. Workaround: re-upload via PUT (covered by the resurrect-on-rewrite fix above) or wait for the 30-min reconcile sweep.
**Folder rename round-trip (commits `f4a03b6` + `f27f3cb`).** Both directions now work:
- *NC-side rename → mule:* webhook's "renamed" branch detects directory rename (neither path has a supported extension) and calls new `handle_directory_rename(old, new)` in `scan.py`. The helper iterates matching rows in Python and prefix-rewrites `Photo.filepath`, `Folder.path`, `SourceRoot.path` in one transaction. Cross-source-root case (rare) discards the old subtree and lets scan_folder add fresh rows under the new root.
- *Mule-side rename → NC:* the existing `PATCH /api/v1/folders/{id}` endpoint (`folders.py:202`) already does WebDAV MOVE via `nextcloud_dav.move_for_user` and rewrites mule's DB inline. The NodeRenamedEvent that bounces back through the webhook hits `handle_directory_rename`, which finds 0 rows under the old prefix and is a no-op — feedback loop is idempotent.
**asyncpg gotcha**: the original implementation of `handle_directory_rename` used raw SQL with `SUBSTRING(filepath FROM LENGTH(:old_prefix) + 1)` so the offset would be computed server-side. asyncpg's type inference miscategorises the `LENGTH()` result and rejects the parameter as "$2: int (expected str)". The fix iterates in Python (same pattern as the existing PATCH endpoint). Lesson: avoid passing `LENGTH(:x)+1` as an argument to `SUBSTRING(...)` via asyncpg + sqlalchemy `text()`.
**Phase 3 — Memories-backed `extract_metadata` (commit `2a5759c`).** Memories app re-enabled on NC. `extract_metadata` now tries `GET /index.php/apps/memories/api/image/info/{fileid}` (auth: Basic + `OCS-APIRequest: true` header to bypass CSRF) before falling back to ExifTool. Replaces ~80 ms of subprocess with ~1-2 ms HTTP for ongoing imports. New helpers: `nextcloud_dav.fetch_memories_info_async()` + `metadata._apply_memories_metadata()`. We kept mule's full date-fallback chain (SubSec → DateTimeOriginal → CreateDate → MediaCreateDate → TrackCreateDate → filename heuristic → mtime) because 35% of the library (`taken_at_source='path'`) depends on the filename heuristic, and Memories alone would silently mis-date those photos to mtime. PhotoInfoPanel reads `exif.Make/Model/ISO/FNumber` from `photos.exif_json` — Memories' `exif` blob uses those exact plain key names, so no frontend adapter was needed. ExifTool subprocess is still in place as the fallback for brand-new photos racing the NC scan, non-NC photos, and any NC HTTP failure.
**Phase 4 — retire `/data/thumbs` (commits `5a67ed7` + `7a1c6b6`).** Vision worker now fetches NC's 640px preview via a new sync helper `nextcloud_dav.get_preview_bytes()` instead of reading `/data/thumbs/{id}/medium.webp`. `thumbs.WORKER_THUMB_SIZES = set()` — `generate_thumbnails` still computes pHash on the original-res pixels (perceptual dedup is mule-only) but stops writing files. All ~22k `medium.webp` purged after verification; `/data/thumbs` shrank from 4.1 GB → 94 MB residual.
**SECRET_KEY bug found mid-deploy (commit `7a1c6b6`).** `docker-compose.yml` only set `SECRET_KEY` on the `backend` service, not on the workers. Workers' Fernet-based `decrypt(nextcloud_app_password_enc)` silently returned empty and `_credentials_for()` raised `NextcloudCredentialsMissing`. This meant Phase 3's `extract_metadata` in worker-light had been silently falling back to ExifTool *the entire time*, and Phase 4's vision worker couldn't fetch NC previews at all. Fix replicates `SECRET_KEY=${SECRET_KEY:-...}` to all worker services. After any compose edit, run `docker exec mulita-<svc> env | grep SECRET_KEY` to confirm propagation.
**Caveats worth knowing for Phase 4:**
- Vision is `disabled` in production (`redis GET mulita:flags:vision.enabled == "false"`). The Phase 4 vision-from-NC path is correct but unexercised by live traffic; it'll matter whenever vision is re-enabled.
- NC's preview generator hasn't covered the whole library at the 640px+ tier. A sample of 6 photos: 4 with proper sizes, 1 stuck at 160px (`IMG_4954.DNG`), 1 unfetchable (`IMG_0193.DNG`). If vision re-enables and quality matters, run `pct exec 114 -- sudo -u www-data php occ preview:generate-all` to backfill.
- The HTTP `/thumb/{size}` endpoint's inline-regen disk fallback still writes one WebP per NC-404 event (mis-extensioned RAW etc). `/data/thumbs` will grow back very slowly from that path.
### 2026-05-11 — Duplicates view shows parent folder + full-path tooltip
`GET /api/v1/library/duplicates/groups` now includes `filepath` in each member payload. `DuplicatesView` renders a dark bottom-strip on every thumbnail showing the last two folder segments — `…/<grandparent>/<parent>` (e.g. `…/files/Phone Photos starting 01-26`) via a `duplicatePathLabel` helper. Going up two levels avoids the failure mode where two copies sit under matching parent names (e.g. duplicate `2023/` subfolders under different archives) and the label would collapse. Full filepath surfaces through the native `title` tooltip on hover. Dimensions chip moves from bottom-left to top-left so the path strip can run edge-to-edge. `memberToPhoto` finally stops faking `filepath = filename` — the synthetic Photo it hands to `PhotoThumbnail` now carries the real path.
**Infra side-effect during the deploy:** two consecutive `--force-recreate` cycles (deploy.sh and a manual `down/up`) raced and left **orphan duplicate worker containers** (`<id>_mulita-worker-light` + the named `mulita-worker-light`, same for vision) both pulling from the same Redis queue. Worker concurrency was effectively doubled, host load climbed past 120, and `pct exec` into LXC 120 hung for several minutes. Recovery: `docker rm -f` on both orphan IDs. The deploy-gotcha entry below covers the pattern; the new contribution from this session is "look for renamed `<id>_<service>` containers and remove them if you ever see load spike + pct hang after a deploy".
### 2026-05-11 — Drop Title field, add bulk Notes editor
- Removed the `Title` (user_title) row from `PhotoInfoPanel.tsx`: the form field, its `titleDraft`/`setTitleDraft` state, the `commitTitle` handler, and the `photo?.user_title` watcher in the draft-sync `useEffect`. Column stays on `photos` and on the backend model — only the UI affordance went away.
- New bulk action `set_notes` in `backend/app/routers/photos.py` `bulk_action`: validates the value is a string (or null/empty to clear), then sets `user_notes` on every photo in the selection in one transaction.
- Frontend wiring: `bulkSetNotes` in `services/api.ts`, `bulkNotes` mutation in `hooks/useBulkPhotoMutations.ts` (optimistic patch with empty-string → null collapse, full rollback on error), surfaced in `RightSidebar.tsx`'s multi-photo bulk panel as a 2-row Textarea + Clear + Apply pair below the Tags section. Doesn't auto-fire on every keystroke — clicking Apply is the explicit commit (otherwise every keystroke would PATCH N rows).
- Shared `Photo` interface in `types/photo.ts` now includes `user_notes?: string | null` so the optimistic-patch typing accepts the field; previously it only existed on `PhotoInfoPanel`'s local `PhotoDetails` shape.
### 2026-05-10 — Right sidebar compact rebuild
Right sidebar previously had `<ActiveHeapCard />` + `<Header />` + a scroll region holding two parallel collapsibles ("Edit" + "Metadata"), with three nested `<Section>` sub-collapsibles (Basic Info / Camera / Location) inside Metadata. A lot of chrome for a per-photo form.
- **Removed** `frontend/src/components/heaps/ActiveHeapCard.tsx` entirely (no remaining importers; active-heap state stays in the store, Select/Discard buttons still consult it).
- `RightSidebar.tsx` single-photo branch also drops its `<Header />` strip — the new "METADATA" collapsible trigger inside `PhotoInfoPanel` is the visible title. Multi-photo branch keeps the Header for "N Photos Selected".
- `PhotoInfoPanel.tsx` is now two stacked collapsibles:
- **Metadata** (default expanded): readonly fields (Size / Dimensions grid, Path, GPS inlined when present), a thin `<hr>`, then editable form (Filename, Title, Date Taken, Notes, Tags, Rating + Color on one row, Flag).
- **Camera** (default expanded): isolated EXIF block (make+model, lens, 2×2 ISO/Aperture/Shutter/Focal grid).
- Compact density: Notes `rows=3 → 2`, stars/swatches `h-5 → h-4`, `space-y-2.5 → 2`, Flag buttons `text-sm → text-xs`, grid `gap-2 → gap-x-2 gap-y-1`, empty "No GPS data" chip dropped (now hidden when there are no coordinates).
- Local `Section` helper deleted from `PhotoInfoPanel.tsx` (no longer used).
`PreviewView` reuses `RightSidebar` inside its overlay aside, so the change applies in both grid and preview.
### 2026-05-10 — Timeline scroll-anchor on section switch
Clicking a folder in the left sidebar (or any other navigation that changed `currentSection` — All Photos, Rated, Discarded, etc.) didn't reset the timeline's scroll position. If the user was deep in All Photos and clicked a folder with fewer rows, the new section loaded with the previous scroll offset preserved, often landing on empty space below the last row.
`Timeline.tsx` already had a section-change effect that cleared selection and reset the auto-focus guard, and a separate auto-focus effect that selects `photos[0]` once the new query resolves. The latter has an "ensure visible" scrollTo, but it only fires if the cell is out of view AND only after the next render — by then the user has already seen the wrong scroll position. Added `parentRef.current.scrollTop = 0` synchronously inside the section-change effect so the first paint of the new section anchors at the top; the auto-focus selectPhoto then highlights photo[0] as before.
### 2026-05-10 — Filter bar 422-on-date-from
User reported "all filters broken, no photos shown" right after the metadata-extraction overhaul shipped. Tracing actual API traffic on a fresh session showed `GET /api/v1/photos?per_page=100&date_from=2026-04-10&sort=taken_at&order=desc` returning **422 Unprocessable Entity**: pydantic v2's `datetime` parser rejects bare-date strings (`"2026-04-10"`) for `Optional[datetime]` query params.
The frontend's `filtersToParams` in `store/filterStore.ts` had been padding `date_to` with `T23:59:59` for inclusive end-of-day, but `date_from` went out as a bare `YYYY-MM-DD` — so every date-range filter request 422'd, and TanStack Query's failure-state shows an empty grid. From the user's perspective it looked like "filters return nothing across the board"; from the backend it was a single endpoint signature problem.
Fix on both sides:
- **Frontend**: pad `date_from` with `T00:00:00` symmetric to `date_to`'s `T23:59:59`.
- **Backend**: switch `date_from` / `date_to` to `Optional[str]` and parse with `datetime.fromisoformat` inside the handler. `fromisoformat` accepts both bare dates (→ midnight) and full ISO strings, so any older client / curl that sends a date-only value still works. Tz-aware values get coerced to naive UTC, matching the `taken_at` column shape and the same fix applied to `PATCH /photos/{id}` earlier in the session. Bad input now returns 400 with a clear message instead of pydantic's 422.
### 2026-05-10 — Date extraction overhaul
User reported wrong "Date Taken" on stills (JPEG / HEIC / DNG). Tracing the pipeline showed the trusted-EXIF list at `services/metadata.py:230-244` ended in `EXIF:ModifyDate`, which is set every time a file is re-saved (Lightroom export, batch resize, EXIF strip), so any photo that lost its original capture metadata during editing was being labeled `taken_at_source='exif'` with the *edit* timestamp.
Pipeline rewrite:
- New trusted-EXIF priority: `SubSecDateTimeOriginal` → `DateTimeOriginal` → `CreateDate` → `QuickTime:MediaCreateDate` → `QuickTime:CreateDate`. **`ModifyDate` removed entirely.**
- Fall back to `guess_date_from_path()` (the same heuristic that already powers `has_date_warning`) when no trusted EXIF date is found. New `taken_at_source='path'` value with a "PATH" badge in the info panel; tooltip explains the date came from filename / folder rather than real EXIF.
- Skip the date-write block entirely when `photo.taken_at_source == 'manual'` so a rescan can't clobber a user correction. (Previous behavior overwrote manual edits.)
- `parse_exif_datetime` accepts tz-aware variants (`%z`), normalizes to naive UTC, and rejects the `0000:00:00` placeholder.
Backfill: new `backfill_taken_at` celery task + `POST /api/v1/library/maintenance/backfill-taken-at` endpoint. Re-enqueues `extract_metadata` for every non-manual, non-trashed photo so the new rules apply across the existing library. Snapshot before the sweep started: 11,223 `exif` + 6,039 `filesystem` + 10 `path`. Will reshape over the next ~45 min.
Side note: the default celery queue had ~209k pending tasks at the time we fired the backfill — the watcher's 5-minute restart loop (since fixed) had been re-enqueuing scans, and tasks for the 4,158 photos we hard-deleted earlier today were still sitting around. Most fail fast (`Photo not found`, ~24ms each); real work runs at ~100ms. Decided to let it drain instead of flushing — safer.
### 2026-05-10 — Two cascade-delete + datetime fixes after the hard-remove shipped
The first cut of `delete_nextcloud_source_root` blew up with `folders_parent_id_fkey` violations on the first real test (`Taco and Muli - 2024 onward`, 35 folders / 4,158 photos). Two iterations to get it right:
1. **NULL `parent_id` before deleting Folders.** Folders have a self-referential `parent_id` FK with no `ON DELETE` rule; postgres checks the constraint per row regardless of insertion order, so a flat `DELETE FROM folders WHERE id IN (...)` of the whole subtree fails on the parents whose children appear later in the same statement. Fixed with an `UPDATE folders SET parent_id = NULL WHERE id IN (folder_ids)` first.
2. **Widen the NULL UPDATE to cross-source-root children.** A "Leóns 1st Year" SourceRoot at `.../Taco and Muli - 2024 onward/Leóns 1st Year` had its own folder rows whose `parent_id` pointed into the Taco SourceRoot's hierarchy. The first patch only NULLed `parent_id` for folders whose `id` was in the delete set; the Leóns folders weren't in that set so they kept their references and the FK still tripped. Fix: `UPDATE folders SET parent_id = NULL WHERE parent_id IN (folder_ids)` — kills any incoming reference into the delete set, internal or external.
After both fixes: `DELETE /api/v1/nextcloud/source-roots/{id}` for `Taco and Muli - 2024 onward` cleared 4,158 photos and 35 folders in a single request and returned 200. UI swaps the trash icon for a `Loader2` spinner while the request is in flight (`removeRoot.isPending && removeRoot.variables === r.id`) so the multi-second cascade is visible.
Separate fix in the same session: `PATCH /api/v1/photos/{id}` returned 500 with `can't subtract offset-naive and offset-aware datetimes` when the frontend sent a tz-aware ISO string for `taken_at` (the datetime-local input is supposed to be naive but real-world locales / paste flows occasionally include `+02:00`). The DB column is `timestamp without time zone`, so asyncpg refused to bind. Normalize on the server with `astimezone(timezone.utc).replace(tzinfo=None)` if `tzinfo is not None`.
Deploy infra learning: the new `--force-recreate` in `deploy.sh` does NOT reliably recreate containers on every push — saw two consecutive deploys leave the backend at the previous `StartedAt` despite a fresh image. Manual `docker compose up -d --no-deps --force-recreate <svc>` after deploy still races with whatever compose state the auto-deploy left mid-flight, twice landing the stack in a half-broken state (orphaned `<id>_mulita-backend` rename containers, db / redis stopped). Recovery: `docker compose down && docker compose up -d`. Open question — `--force-recreate` may need to be replaced with something more explicit.
### 2026-05-10 — Hard-remove Nextcloud SourceRoot + reliable delete sync
`DELETE /api/v1/nextcloud/source-roots/{id}` was a soft-deactivate (`is_active=false`) — the trash icon in Settings only hid the SourceRoot from active queries while leaving every Folder + Photo row in the DB forever. Re-adding the same path resurrected ghosts; `prune-missing` reported zero deletes for the soft-removed entry because the cleanup code skipped inactive source roots.
Endpoint now hard-deletes the SourceRoot, every Folder under it (chunked at 500), and every Photo in those folders. `photo_tags` and `heap_photos` cascade automatically via `ON DELETE CASCADE` on the join tables. `FolderShare` uses a stringly-typed `folder_id` (no FK) so cleaned manually for both `folder_type='folder'` and `folder_type='source_root'`. Returns `{deleted_photos, deleted_folders}` so the UI can toast a count. Files in Nextcloud are untouched.
Sync side: the `watch_folders` celery task in `backend/app/tasks/scan.py` already detected filesystem deletions and soft-marked photos `is_discarded=true`, but the global `task_soft_time_limit=300` in `app/tasks/celery.py` was killing the watcher every five minutes and dropping every FS event during the restart window. The `soft_time_limit=None` on the decorator was being interpreted as "use worker default" rather than "unlimited". Override to `soft_time_limit=0, time_limit=0` (Celery convention for unbounded). Backstop: a new `discard_missing_photos()` in `app/services/cleanup.py` — soft variant of `prune_missing_photos`, walks every `_sr_state == 'present'` source root and flips `is_discarded=true` on Photo rows whose file is gone. Wired as `discard_missing_photos_beat` celery task scheduled every 30 minutes via `beat_schedule` on the celery app. Beat runs in-process on `worker-watcher` (`--beat` flag in `docker-compose.yml`) — the watcher is already a Redis-locked singleton so no need for a separate beat container.
Manual `POST /api/v1/library/maintenance/prune-missing` remains the hard-delete path for when the user wants to permanently drop orphan rows; the new beat job only soft-discards (file gone → mule-image trash, restorable).
### 2026-05-10 — OIDC auto-redirect on LoginPage
`OIDC_ENABLED=true` was already set in `.env`, so the LoginPage rendered a "Sign in with Authentik" button next to the password form. With a single trusted IdP and a logged-in Authentik session, that extra click was friction without upside.
`LoginPage.tsx` now reads `/auth/config` on mount and, if OIDC is on, immediately navigates to the OIDC login URL. Authentik recognizes the existing session and bounces back through the callback with no user interaction. Two escape hatches: `?password=1` in the URL forces the password form, and a `skipAutoSso` sessionStorage flag (set by `AuthContext.logout` and by the OIDC callback's error branch) suppresses the next auto-redirect so logouts actually log out and OIDC failures surface their error instead of looping. While the redirect is in flight the page shows "Signing in with Authentik..." plus a "Use password instead" link.
### 2026-05-10 — right sidebar restructure (heap pinned, single scroll, collapsible Metadata)
The right sidepanel had three stacked flex regions: `<ActiveHeapCard />`, `<Header />`, and `<PhotoInfoPanel />` — with `PhotoInfoPanel` carrying its own internal scroll. That left the editable fields (filename, title, notes, rating, color, flag) stuck above the readonly metadata scroll, effectively two scroll boundaries on one sidebar.
- Moved the scroll boundary up to `RightSidebar.tsx`: only `ActiveHeapCard` and `Header` stay pinned now. Editable + readonly content scroll together in one `flex-1 overflow-y-auto` region beneath them.
- `PhotoInfoPanel.tsx` dropped its `h-full` / inner `flex-1 overflow-y-auto`.
- The four readonly sections (Tags / Basic Info / Camera / Location) are now wrapped in a single outer "Metadata" `Collapsible`. Default expanded, one click hides the whole block. Sub-sections stay individually collapsible.
- Second pass: the editable form (filename / title / notes / rating / color / flag) got the same treatment under an outer "Edit" `Collapsible` so the panel is now two equal collapsible groups below the title strip. Dropped the X (clear-selection) button from the Header; Esc and grid-empty-area-click still clear.
- Third pass: split editable vs read-only between the two groups consistently. `TagsEditor` and `TakenAtEditor` had been buried inside the readonly Metadata sub-sections — `Tags` as its own Section, taken-at wedged into `Basic Info` between size/dims and the filepath. Moved both into the Edit collapsible, ordered identification → description → categorization: Filename · Title · Date Taken · Notes · Tags · Rating · Color · Flag. Metadata now holds only readonly: `Basic Info` (size, dims, path), `Camera`, `Location`.
`PreviewView` reuses `RightSidebar` under an `<aside class="overflow-hidden">`, so the change applies in both the grid sidebar and the preview overlay.
### 2026-05-10 — `photos.hubris.network` perf sweep
User reported slow load. Five fixes shipped, in order:
1. **Backend out of dev mode.** `docker-compose.yml` `command:` was running `uvicorn … --reload` (single worker, file-watcher). Page loads fan out ~15 parallel API calls; they all serialized on one event loop. → `--workers 2 --proxy-headers`. Two uvicorn worker procs now.
2. **iPhone Apple ProRAW / Linear DNG decode fixed.** rawpy 0.26.1 + LibRaw 0.22 rejects Apple Linear DNG (`Photometric Interpretation: Linear Raw`, 8-bit) as "Unsupported file format or not RAW file". Those files are TIFF containers with developed RGB inside and have no embedded preview to fall back to. Added PIL `Image.open(src_path)` as the last fallback in both `_generate_proxy_webp` (routers/photos.py) and `process_raw_image` (tasks/thumbs.py). ~1,300 iPhone DNGs were 415-ing on every detail view; now decode in <1s via PIL.
3. **Reconcile DB with disk (renamed-folder case).** `prune_missing_photos` was skipping all 4,154 orphaned photo rows under SourceRoot `Taco and Muli - 2024 onward` (renamed in Nextcloud to `Photo Archive 2004-2024`) because the leaf path didn't resolve and the code assumed "drive unmounted → must skip". Added `_sr_state()` to classify as `present` / `renamed` (parent mount fine, leaf gone) / `unmounted` (parent inaccessible). Only `unmounted` still skips. Two stale source roots logged with a clearer hint pointing at `POST /api/v1/library/maintenance/prune-missing`. User has not yet been asked to run that — endpoint is ready when they are.
4. **Frontend page size + idle polling.** `usePhotosQuery.ts` was first-fetching `per_page=500` (slow paint + 500 thumb requests at once). Split into `PER_PAGE_INITIAL=100` for first paint, `PER_PAGE_BACKGROUND=500` for the cursor-chain prefetch. Idle polling for `scan-status` and `worker-status` (`useScanActivity.ts`, `ScanProgress.tsx`) bumped from 10s/15s to 30s/30s while idle; active cadence (2s/3s) unchanged.
5. **Partial index on `photos`.** Default list query `WHERE NOT is_trashed AND NOT is_hidden ORDER BY taken_at DESC NULLS LAST, id DESC LIMIT N` was doing a seq-scan + top-N heapsort (~25ms standalone, worse under concurrency). Added migration `0017_photos_list_index` creating `ix_photos_list_visible` (partial index on the sort key, restricted to visible rows). EXPLAIN now shows an Index Only Scan → **24.7ms → 0.097ms**, ~250× speedup.
**Deploy gotcha — fixed (with caveat).** The original `docker compose up -d --build` in `deploy.sh` did not reliably recreate containers when only runtime config (`command:`, env-only) or migration files changed; image hash would change but compose treated the existing container as "current enough". Bit three times this session before `/opt/mule-deploy/deploy.sh` was updated to `docker compose up -d --build --force-recreate`. Trade-off accepted: an extra restart cycle on deploys where nothing user-visible changed.
**Caveat:** the first auto-deploy after the flag flip raced with my own earlier manual `docker compose up -d --no-deps --force-recreate frontend` and landed the stack in a half-broken state — `mulita-frontend` got stuck under a renamed temp container, several services dropped off `mule-image_mulita-network`, frontend nginx restarted in a loop with `host not found in upstream "backend"`. Fixed by `docker compose down && docker compose up -d`. Don't issue a manual `--force-recreate` on a single service while the auto-deploy webhook is also expected to fire — let the deploy own the lifecycle.
**Data drift still outstanding.** 4,154 photo rows + 1 unregistered folder (`Photo Archive 2004-2024`) on disk that's not a SourceRoot. The reconcile endpoints now work — user decides when to call them. The new folder needs to be added as a SourceRoot via the Settings UI before its files will be indexed.
**Proxy cache still empty** (`mule-image_proxies_data` volume is 4 KB). Pre-generating ~500–1,500 WebP proxies for non-web-safe formats would make first-open of every RAW/HEIC photo instant. Deferred — needs a one-shot script and the disk-space tradeoff isn't worth it until the data-drift reconcile happens first.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — Nextcloud-rooted libraries shipped
Bind `/mnt/library/homecloud` into the workers, reads via filesystem, writes via WebDAV. `users.nextcloud_username` override field added; `dtoro → admin` mapping. Surviving SourceRoots cleaned up to NC paths.
### 2026-04-22 — native OIDC via Authentik
Authlib-based code in `backend/app/auth_oidc.py`. `extra_hosts` override for `auth.hubris.network` in compose override (gitignored).
### 2026-04-21 — auto-deploy pipeline shipped
Webhook receiver at `:9797`, async deploy returning 202. Mirrors caddy-conf / gitea-customizations.

106
knowledge/wiki/containers/121-caddy.md Normal file
View File

@@ -0,0 +1,106 @@
# 121 — `caddy`
The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the LAN/mesh and forwards to the right backend.
## At a glance
- **Hostname:** `caddy`
- **IP:** `192.168.8.175`
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 512 MiB RAM / 6 GiB rootfs
- **Mounts:** none from `/mnt/library`
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
## Sites currently served (live as of 2026-07-06)
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
- `books.hubris.network` → [grimmory (130)](130-grimmory.md) `:6060`
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
- `house.hubris.network` → [house (129)](129-house.md) `:3000`
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
- `paperless.hubris.network` → [paperless (103)](103-paperless.md) `:8000`
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
- `roms.hubris.network` → [romm (134)](134-romm.md) `:80`
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
- `teddy.hubris.network` → LXC 131 `192.168.8.150:8443`
- `trmnl.hubris.network` → [trmnl (128)](128-trmnl.md) `:9851`
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
## Snippet: `(authentik)` forward-auth
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](124-authentik.md#forward-auth-domain-level-setup).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](124-authentik.md).
## Caddy environment
Untracked secrets in `/etc/caddy/caddy.env`:
- `IONOS_AUTH_API_TOKEN` — DNS-01 ACME
- `ARTIFACTO_GATEWAY_SECRET` — must match `SSO_GATEWAY_SECRET` on [apps (105)](105-apps.md). Rotate together.
## Auto-deploy
Push to `dtoro/caddy-conf` master → gitea webhook → `http://192.168.8.175:9797/deploy` → `caddy-deploy-webhook.service` validates HMAC against `/etc/caddy-deploy/secret` → runs `/etc/caddy/scripts/deploy.sh`:
1. `git pull`
2. `caddy validate`
3. `systemctl reload caddy`
Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scripts, systemd unit all live in the repo under `scripts/`. The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). Git credentials for the root-run service are at `/etc/caddy-deploy/git-credentials` (mode 600).
> **Systemd unit changes inside the repo do NOT auto-reinstall** — you have to re-run `scripts/webhook/install.sh` manually after editing the unit.
## Related
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
- [Authentik (124)](124-authentik.md) — forward-auth + IdP
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
- [Gitea (104)](104-gitea.md) — webhook source
- All container pages list their Caddy backend.
## Changelog
### 2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082
`sab.hubris.network` now uses `import authentik` inside a `handle` block. SABnzbd own auth disabled, `local_ranges` set for transparent proxy. Port bumped from `:8081` to `:8082` (fix from 2026-06-04) now documented.
### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
**Fix:**
- Restored Caddyfile from `origin/master` (`git checkout --force origin/master -- Caddyfile`)
- `systemctl reload caddy`
**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
### 2026-06-02 — caddy.service unit missing; recreated
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
### 2026-04-28 — wiki entry created
Initial documentation. 16 active sites at this date.
### 2026-04-22 — Artifacto + Authentik gateway-secret pattern wired
Caddy site `artifacto.hubris.network`: forward-auth on admin paths, `header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}`, public matchers `/p/*`, `/static/*`, `/healthz`. `caddy.env` extended with the gateway secret.
### 2026-04-21 — `(authentik)` snippet added
Top-of-file snippet used by every gated site. Targets `http://192.168.8.180:9000` directly to dodge hairpin TLS round-trip.
### 2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)
For the [arriman (122)](122-arriman.md) docker stack. Caddy + certs work; needed corresponding [DNS split-horizon](../infrastructure/dns.md) entries to be added to land cleanly.
### 2026-04-20 — auto-deploy pipeline shipped
Repo `dtoro/caddy-conf`. Webhook receiver at `:9797`, validate + reload on push to master.

158
knowledge/wiki/containers/122-arriman.md Normal file
View File

@@ -0,0 +1,158 @@
# 122 — `arriman`
Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr) compose. Replaced the old yunohost-based [LXC 100](index.md#recently-destroyed-kept-for-archaeology) on 2026-04-21.
## At a glance
- **Hostname:** `arriman`
- **IP:** `192.168.8.245`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged
- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
## Compose
`/home/arr/ezarr/docker-compose.yml`. Network `ezarr_default` `172.18.0.0/16`. Service users all run with primary GID 10000 (host `media` group) inside their containers — see [media permissions](../infrastructure/media-permissions.md). Per-app `/config/<app>-config/` dirs bind to `/config` inside.
`.env` contents:
- `DOWNLOADS_DIR=/mnt/library/downloads/`
- `MEDIA_DIR=/mnt/library/`
- `MEDIACENTER_GID=10000` ← **must be 10000, not 13000.** s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. Without this, qBit errors every torrent with "Permission denied".
- `SONARR_UID=13001`, `RADARR_UID=13002`, `LIDARR_UID=13003`, `PROWLARR_UID=13006`, `JELLYSEERR_UID=13012`, `BAZARR_UID=13013`, `QBITTORRENT_UID`, `SABNZBD_UID`
## Service / port map
All services route through gluetun's network namespace. Ports are exposed via
the gluetun container:
| Service | Host:Container | Public hostname |
| ------------- | -------------- | ------------------------------------ |
| gluetun (VPN) | — | — |
| sonarr | `8989:8989` | direct only (via gluetun) |
| radarr | `7878:7878` | direct only (via gluetun) |
| lidarr | `8686:8686` | direct only (via gluetun) |
| prowlarr | `9696:9696` | direct only (via gluetun) |
| bazarr | `6767:6767` | direct only (via gluetun) |
| jellyseerr | `5056:5055` | [`jellyseerr.hubris.network`](../infrastructure/dns.md) |
| qbittorrent | `8080:8080` | [`qbit.hubris.network`](../infrastructure/dns.md) |
| sabnzbd | `8082:8082` HTTP, `9090:9090` HTTPS | [`sab.hubris.network`](../infrastructure/dns.md) |
| flaresolverr | `8191:8191` | internal only |
| homarr | `7575:7575` | internal only |
Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `localhost:<port>`
(services share gluetun's shared network namespace). External services reach them
via `gluetun:<port>` (e.g. Sonarr → qBittorrent at `localhost:8080` or
`gluetun:8080`).
## Categories (qBit + SAB + *arr)
| Category | qBit savePath | SAB dir | Consumer |
| -------- | --------------------------- | ------- | --------------------------------- |
| tv | `/data/torrents/tv` | tv | Sonarr |
| movies | `/data/torrents/movies` | movies | Radarr |
| music | `/data/torrents/music` | music | Lidarr |
| anime | `/data/torrents/anime` | anime | Sonarr (2nd download client, root `/data/media/anime`) |
| books / audiobooks / comics / podcasts | `/data/torrents/<cat>` | `<cat>` | (reserved, none in use yet) |
Path mapping: host `/mnt/library/<cat>` ↔ container `/data/media/<cat>`. Downloads: host `/mnt/library/downloads/<torrents|usenet>/<cat>` ↔ container `/data/torrents/<cat>` and `/data/usenet/<cat>`.
## Auth (reverse-proxy + Authentik forward-auth)
### qBit
Auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
- `WebUI\\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
- `WebUI\\ReverseProxySupportEnabled=true`
- `WebUI\\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
> **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`.
Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy.
### SABnzbd
Gated with Authentik forward-auth (applied 2026-06-13). Caddy `sab.hubris.network` block uses `import authentik` inside a `handle` block. SABnzbd's own web auth is disabled:
- `html_login = 0` → no HTML login form
- `username` / `password` cleared → CherryPy Basic Auth not activated
- `local_ranges = 172.18.0.0/16, 192.168.8.0/24, 127.0.0.0/8` → proxied requests from Caddy (192.168.8.x) and Docker-proxy (172.18.x) pass without auth
**API key** (`67ef5a45e4e04157994e977005a33878`) still works for internal service-to-service calls (Sonarr/Radarr/Lidarr via Docker internal networking — they talk to SAB at `localhost:8082`, not through Caddy).
`host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
## Credentials
Backups of all migration-time secrets at `/mnt/library/backup/arr-migration-2026-04-21/state/`:
- `lxc100-apikey-{sonarr,radarr,prowlarr,lidarr}.txt`
- `qbit-creds.txt`, `sab-creds.txt`
- `lxc100-jellyseerr-settings.json`
\*arr API keys were preserved from the yunohost DB — don't rotate unless needed.
qBittorrent: `admin` / `CDyjxMKvjfGupCKzXFkjsKw4nRrXMRL`.
## Migration gotchas (apply if ever re-importing)
- Imported \*arr `config.xml`: must have `<BindAddress>*</BindAddress>`, `<Port>{ezarr-port}</Port>`, `<UrlBase></UrlBase>`. Otherwise app listens only on 127.0.0.1, wrong port, wrong path.
- Imported DB paths must be rewritten `/mnt/library/` → `/data/media/` (RootFolders / Series / Movies / Artists tables). Without this, scans and imports fail silently.
- Bazarr `config/config.yaml` (yaml, not ini) — `ip: 0.0.0.0`, `port: 6767`, empty top-level `base_url`; sonarr/radarr connection sections normalized to docker service names.
- Staging dir for migrations: `/mnt/library/backup/stage/<app>/` (visible to both old LXC 100 and arriman because both bind-mount the same host path).
## Permissions
Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has `www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr` in the `media` group. Each docker container also receives GID 10000 as its primary group via `MEDIACENTER_GID=10000`.
## Rollback material (kept until 2026-04-28+)
`/mnt/library/backup/arr-migration-2026-04-21/`:
- `tars/` — per-subtree tars of yunohost app data (granular restore)
- `snapshots/` — vzdumps of LXC 100 (pre-migration) and LXC 122 (pre-import), plus post-stop vzdumps of LXC 100 + 106
- `stage/<app>-preimport.tar.gz` — original empty ezarr `/config/<app>-config/` dirs
- `state/` — service dumps, listening ports, API keys, jellyseerr settings at migration time
- `DESTROY_AFTER_2026-04-28.md` — destroy checklist + post-destroy restore commands
## Related
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](124-authentik.md) — forward-auth wiring + per-app `/api/*` bypass
- [DNS](../infrastructure/dns.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Hubris host](../hosts/hubris.md)
## Changelog
### 2026-06-13 — SABnzbd gated with Authentik forward-auth
SABnzbd now uses Authentik forward-auth (same `import authentik` Caddy pattern as qBit). SABnzbd's own web auth disabled: `html_login=0`, credentials cleared, `local_ranges` extended to cover Docker bridge + homelab LAN. API key still works for internal *arr service calls. See [Auth section](#auth-reverse-proxy--authentik-forward-auth) above.
### 2026-06-04 — all arr services moved behind gluetun VPN; SAB port conflict fixed
- All services (sonarr, radarr, lidarr, bazarr, prowlarr, jellyseerr, homarr,
flaresolverr) now use `network_mode: service:gluetun` — whole stack routes
through the VPN
- Port mappings moved from individual services to gluetun container
- **Fixed SABnzbd port conflict**: was crashing in a restart loop because
qBittorrent held port 8080 inside the shared gluetun namespace. Changed
SAB internal port to 8082 (config at `/config/sabnzbd-config/sabnzbd.ini`)
- Caddy `sab.hubris.network` updated to point to `:8082`
- Jellyseerr's `extra_hosts` (auth.hubris.network) moved to gluetun since
`extra_hosts` conflicts with `network_mode`
### 2026-06-02 — ProtonVPN added (gluetun); LXC IP set static
- Added `gluetun` container to compose as a WireGuard VPN sidecar (ProtonVPN, server AL#57, located in Tirana, Albania)
- **qbittorrent** and **sabnzbd** now use `network_mode: service:gluetun` — all traffic routes through the VPN
- Ports 8080 (qBit WebUI), 6881 tcp/udp (qBit BT), 8081 (SAB WebUI) exposed through gluetun
- gluetun config at `gluetun-config/wireguard/wg0.conf` (read-only mount)
- Healthcheck on gluetun; qBit/SAB wait for `service_healthy` before starting
- LXC IP changed from DHCP to static (`192.168.8.132`) via `pct set` + `/etc/network/interfaces`
- **After first start:** Sonarr/Radarr/Lidarr download client host needs updating from `qbittorrent` → `gluetun` (SAB similarly `sabnzbd` → `gluetun`)
- **Also fixed:** 7 other DHCP LXCs (101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy) set to static IPs to prevent floating on reboot. See infrastructure/dns.md.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — `MEDIACENTER_GID` flipped 13000 → 10000
qBit was erroring every torrent with "Permission denied" because s6-setuidgid only honors the primary PGID and `group_add:` doesn't propagate. Setting the primary GID to 10000 (the host `media` group) fixed it.
### 2026-04-21 — \*arr stack migrated yunohost → docker-compose
Old LXC 100 stopped; ezarr stack stood up here; data and DBs imported with path rewrites. flaresolverr folded in from old LXC 106. Three Caddy sites added (jellyseerr, qbit, sab) → see [caddy](121-caddy.md). Rollback artifacts under `/mnt/library/backup/arr-migration-2026-04-21/`.

48
knowledge/wiki/containers/128-trmnl.md Normal file
View File

@@ -0,0 +1,48 @@
# 128 — `trmnl`
Self-hosted middleware for TRMNL e-ink plugins. TRMNL cloud polls it; it fetches/shapes live data into JSON the plugin's Liquid template renders.
## At a glance
- **Hostname:** `trmnl`
- **IP:** `192.168.8.211`
- **Privilege:** unprivileged
- **Resources:** 1 core / 768 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** `trmnl.hubris.network` (via [VPS ingress](../infrastructure/ingress.md))
## Role
Runs one FastAPI aggregator (`server.app:app`, port 9851) that mounts a router per plugin from the `dtoro/terminalito` repo. First consumer: `munich-home` (`/munich-home/dashboard`) — weather (Open-Meteo), MVG transit, Google Calendar, plus server-side Kita/quote logic. Talks out to the public internet for those APIs; TRMNL cloud polls it inbound every 15 min. Bearer-token gated (`TRMNL_POLL_TOKEN`); `/health` is open.
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `trmnl-plugins` | `0.0.0.0:9851` | uvicorn aggregator; `EnvironmentFile=/etc/trmnl-plugins/env` |
## Storage / config paths
- `/opt/terminalito` — git checkout (origin = internal gitea `http://192.168.8.121:3000/dtoro/terminalito.git`)
- `/opt/terminalito/server/.venv` — venv
- `/etc/trmnl-plugins/env` — `TRMNL_POLL_TOKEN` (+ Google/MVG creds once enrolled)
- `/etc/systemd/system/trmnl-plugins.service`
## Auto-deploy
Wired — [auto-deploy](../infrastructure/auto-deploy.md) Shape B, webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` (`terminalito-deploy.service`). Push to `main` → `server/deploy/deploy.sh` (`git pull` + pip + reinstall units + restart `trmnl-plugins`). Secret `/etc/terminalito-deploy/secret`; git creds `/etc/terminalito-deploy/git-credentials` wired as a repo-local `credential.helper`. Manual: `pct exec 128 -- /opt/terminalito/server/deploy/deploy.sh`.
## Secrets
Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env`. Google Calendar + MVG creds are pending: enroll via `homelab client add trmnl` + bootstrap, add `secrets/trmnl-oauth.yaml`, then `server/deploy/render-env.sh` builds the env from `homelab secret trmnl-oauth`. Until then calendar/transit cards degrade to empty; weather works.
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`trmnl.hubris.network → 192.168.8.211:9851`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy)
- [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito`
- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md)
## Changelog
### 2026-06-24 — auto-deploy + LAN DNS wired
Gitea Shape-B deploy pipeline (webhook id 12, `:9797`) — push to `dtoro/terminalito` redeploys; verified end-to-end. Technitium A record `trmnl.hubris.network → 192.168.8.175` added on [dns (107)](107-dns.md) (propagated to the NetBird managed zone via dns-sync), so LAN clients take the short path through [Caddy (121)](121-caddy.md). See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-06-24 — public path live
Verified end-to-end from the internet: `https://trmnl.hubris.network/munich-home/dashboard` → 200 with token, 401 without; `/health` 200. The provision-time outage was the netbird `home-lab-network` (192.168.8.0/24) route having no active routing peer — the **mac-mini routing peer's netbird daemon was down** (artifacto/blog were 504 too). Bringing netbird up on mac-mini restored the route; the edge recovered with no config change. See [ingress](../infrastructure/ingress.md) / [mesh](../infrastructure/mesh.md).
### 2026-06-24 — provisioned
LXC 128 created (Debian 13, unprivileged, `192.168.8.211`). Deployed `trmnl-plugins.service` on :9851 from `dtoro/terminalito`. Caddy block added (`dtoro/caddy-conf`) + LE cert via IONOS DNS-01; verified `/health` 200 and `/munich-home/dashboard` (live weather) through Caddy. Cert mirrored to VPS (`trmnl.fullchain.crt`/`trmnl.privkey.key`) + traefik router `trmnl-public` → `192.168.8.211:9851` added to `/opt/traefik-dynamic.yaml`. **Public path pending**: VPS↔home netbird route was down at provision time (`No networks available`, 3/6 peers — artifacto/blog also 504); resolves when the mesh route recovers. **LAN pending**: Technitium A record not yet added. Not SOPS-enrolled; Google/MVG creds pending.

52
knowledge/wiki/containers/129-house.md Normal file
View File

@@ -0,0 +1,52 @@
# 129 — `house`
Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modules: calendar, tasks, meals, groceries, budget, documents, notes, contacts, birthdays, housekeeping, recipes, reminders.
## At a glance
- **Hostname:** `house`
- **IP:** `192.168.8.244`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** unprivileged
- **Resources:** 1 core / 1344 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** [`house.hubris.network`](../infrastructure/ingress.md) → VPS traefik → Caddy
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `oikos` (Yuvomi) | `0.0.0.0:3000` | Docker Compose at `/opt/yuvomi/`, image `ghcr.io/ulsklyc/yuvomi` |
## Integrations
- **Authentik SSO (OIDC):** Provider `Provider for Yuvomi` (PK 31) in Authentik on VPS. Env vars in `/opt/yuvomi/.env`: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`. Redirect URI: `https://house.hubris.network/auth/oidc/callback`.
- **Paperless DMS connector (native):** Yuvomi connects directly to Paperless-ngx API at `http://192.168.8.130:8000/`. API token stored in SQLite `dms_accounts` table. Search, link, and upload documents from Yuvomi to Paperless via Settings → Documents → DMS.
- **Weather widget:** Open-Meteo (free, no API key). Munich coordinates set.
- **Google Calendar:** OAuth configured via env vars (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI`). Redirect URI: `https://house.hubris.network/api/v1/calendar/google/callback`. Authorize in Settings → Calendar → Connect Google Calendar.
## Config paths
- `/opt/yuvomi/docker-compose.yml` — downloaded from upstream
- `/opt/yuvomi/.env` — config including secrets (untracked)
- `/opt/yuvomi/data/` — SQLCipher SQLite DB (`oikos.db`)
- `/opt/yuvomi/backups/` — auto backups
- `/opt/yuvomi/modules/` — Yuvomi modules (empty for now)
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`house.hubris.network → 192.168.8.212:3000`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy)
- [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`)
- [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source
- [Deployment plan](../../../plans/2026-06-25-yuvomi-deployment.md)
## Changelog
### 2026-06-27 — Google Calendar OAuth env vars configured
`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` set in `.env`. New OAuth client ID (`-bho4iq..`).
### 2026-06-26 — provisioned
LXC 129 created (Debian 13, unprivileged, `192.168.8.212`). Docker installed. Yuvomi container running on `:3000` from `ghcr.io/ulsklyc/yuvomi:latest`. Caddy block + DNS A record + VPS traefik router `house-public` for public access. Authentik OIDC provider created (PK 31). WebDAV document bridge on paperless LXC (103) at `:8088` for Paperless auto-import.

66
knowledge/wiki/containers/130-grimmory.md Normal file
View File

@@ -0,0 +1,66 @@
# 130 — `grimmory`
Self-hosted digital library (eBooks, comics, audiobooks). Community fork/successor of Booklore, with smart shelves, metadata enrichment, Kobo/KOReader sync, OPDS, and a built-in EPUB/PDF reader. Migrated from [apps (105)](105-apps.md) on 2026-06-29.
## At a glance
- **Hostname:** `grimmory`
- **IP:** `192.168.8.247`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13)
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
- **Public hostname:** `books.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| Grimmory | `192.168.8.247:6060` | Docker Compose at `/opt/grimmory/` |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/grimmory/docker-compose.yml`. Key points:
- Image: `ghcr.io/grimmory-tools/grimmory:latest`
- Books library: `/mnt/library/books` → `/books` (read-write; media GID 10000 via `GROUP_ID=10000`)
- Bookdrop (watched folder for auto-import): `/opt/grimmory/bookdrop` → `/bookdrop`
- App data (covers, DB config): `/opt/grimmory/data` → `/app/data`
- MariaDB config: `/opt/grimmory/mariadb/config` → `/config` (linuxserver/mariadb image)
- `extra_hosts: auth.hubris.network:192.168.8.175` — routes Authentik OIDC discovery to Caddy from inside the container
- `FORCE_DISABLE_OIDC=false` — OIDC stays enabled; provider configured via Grimmory admin UI
Credentials live in `/opt/grimmory/.env` (untracked):
- `DATABASE_PASSWORD` / `MYSQL_PASSWORD` — MariaDB Grimmory user password
- `MYSQL_ROOT_PASSWORD` — MariaDB root password
## Authentik OIDC
Uses Confidential client (client secret stored in Grimmory's DB — migrated from Booklore). The OIDC config carried over in the database dump; no manual re-entry needed.
- **Authentik provider:** `Provider for Grimmory` (renamed from `Provider for Booklore` on migration)
- **Client ID:** `L1u0eFsNhbKgiIvvFeIr2mvZdbtFyzidCq2h6thL`
- **Client type:** Confidential (client secret in `oidc_provider_details` in MariaDB `app_settings`)
- **Redirect URI:** `https://books.hubris.network/oauth2-callback`
- **Scopes:** openid, profile, email, offline_access
- **Back-channel logout:** `http://192.168.8.247:6060/api/v1/auth/oidc/backchannel-logout`
- **Application slug:** `booklore` → Issuer URI: `https://auth.hubris.network/application/o/booklore/`
## Media permissions
LXC is privileged → in-container UID = host UID. Docker container gets media GID via `GROUP_ID=10000` env var (Grimmory/linuxserver pattern). The `/mnt/library/books` subtree is owned `:media` mode `2775` (setgid). See [media-permissions](../infrastructure/media-permissions.md).
## Related
- [apps (105)](105-apps.md) — previous host (Booklore)
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060`
- [Authentik (124)](124-authentik.md) — OIDC provider `Grimmory`
- [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-06-29 — provisioned; Booklore migrated
LXC 130 created (Debian 13, privileged, `192.168.8.213`). Docker installed. Grimmory compose deployed at `/opt/grimmory/`. MariaDB dump from Booklore (LXC 105) restored — schema-compatible since Grimmory is a direct fork. Caddy `books.hubris.network` backend updated from `192.168.8.205:6060` to `192.168.8.213:6060`. Authentik provider updated: Booklore → Grimmory, Confidential → Public (PKCE). Booklore stack removed from Portainer on LXC 105.

62
knowledge/wiki/containers/131-teddycloud.md Normal file
View File

@@ -0,0 +1,62 @@
# 131 — `teddycloud`
Self-hosted [TeddyCloud](https://github.com/toniebox-reverse-engineering/teddycloud), a
reimplementation of the Toniebox cloud backend — lets Tonie figurines play custom/ripped
audio content against a local server instead of the official cloud.
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
[OIKOS.md](../../../OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
for it — but it never got its own inventory entry or doc page until now.
## At a glance
- **Hostname:** `teddycloud`
- **Host:** hubris (confirmed via `pct config 131` run directly on hubris — the original
drift finding's `pct list` source)
- **IP:** `192.168.8.150` (static; was briefly `192.168.8.243` via DHCP until the
2026-07-05 strong-migration work assigned it a fixed address — see hosts/strong.md)
- **Privilege:** unconfirmed (not checked — read-only investigation didn't need it)
- **Resources:** 1 core / 1 GiB RAM / 512 MiB swap / 16 GiB rootfs (`local-lvm`), Debian 12
(bookworm)
- **Mounts:** `/mnt/library` (`mp0`)
- **Public hostname:** `teddy.hubris.network` → `192.168.8.150:8443` (see
[caddy (121)](121-caddy.md))
- **Enrollment:** none — no `age_pubkey`, not a `homelab` CLI client. It's a plain
docker-compose app container, not a fleet-managed host. No action needed unless it starts
needing secrets.
## Service
Runs via `docker compose` at `/opt/teddycloud` — container `teddycloud-teddycloud-1`,
image `ghcr.io/toniebox-reverse-engineering/teddycloud:latest`, publishing `80`, `443`,
`8080`, and `8443`. Caddy routes `teddy.hubris.network` to the `:8443` port.
## Risk notes
**No Caddy forward-auth gate** — unlike `sab.hubris.network` on the same Caddyfile (which
is explicitly annotated `(Authentik forward-auth)`), `teddy.hubris.network` has no auth
annotation. It's reachable to anyone on the LAN/mesh who can resolve the hostname. Not
addressed as part of this doc pass — flagging it here since it's now visible in one place
for the first time.
## Related
- [Hubris host](../hosts/hubris.md)
- [Caddy (121)](121-caddy.md) — terminates `teddy.hubris.network`
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
taken by this container
- [Containers index](index.md)
- [OIKOS.md](../../../OIKOS.md) — drift detector that caught this
## Changelog
### 2026-07-06 — documented for the first time (drift-caught)
Added to `inventory.yaml` and given this page. Verified live via read-only `pct config 131`
+ `pct exec 131 -- ...` on hubris: hostname, IP, resources, and that it runs via
`docker compose` (not a raw binary or systemd unit). No changes made to the running
container — this is pure documentation catch-up.

199
knowledge/wiki/containers/132-rclone.md Normal file
View File

@@ -0,0 +1,199 @@
# 132 — `rclone`
Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive**
with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing
and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md).
Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container,
so this landed on **132**.)
## At a glance
- **Hostname:** `rclone`
- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik)
- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree,
incl. `homecloud/` and `documents/`, regardless of owner)
- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03
after `rclone-rcd.service` was OOM-killed under real load (see "Known issue" below)
- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup
job must never be able to write into the library
- **Public hostname:** none — the UI is **LAN-only, no auth** (by design)
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status |
| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) |
## Backup design
- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**).
- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay
browsable in Proton's web UI).
- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line
(`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other
disks once bind-mounted into this LXC.
- **Path mapping:** source `S` → `proton:library-backup/<S without leading slash>`
(e.g. `/mnt/library/notes` → `proton:library-backup/mnt/library/notes`).
- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept
the path stable) — no arg = every enabled line. Submits each folder as an **async job through the
rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`),
so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on
Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The
rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with
query params 404s.
- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/<safe>-<ts>.log`; one-line
JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`.
- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today;
**TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md).
## rclone + Proton Drive
- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present
(`rclone v1.74.3`).
- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600.
**This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never
commit it.** Escrow the Proton account creds in the password manager.
- **Config gotchas** (from rclone docs/forum):
- Log into Proton via a **browser at least once** first, or key generation fails.
- For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates
codes; obscure with `rclone obscure`.
- Passwords with **extended-ASCII** characters are known to break auth.
- Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner).
- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted
to this LXC's age key, so the remote can be rebuilt after a re-provision.
## The UI (rclone Web GUI)
`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first
start). Reach it at **http://192.168.8.214:5572** on the LAN.
> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data —
> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is
> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik.
## Tracked config (deferred)
**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files
directly on the LXC — fully functional, just not version-controlled or auto-deployed. A
`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and
staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like
[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation.
See [auto-deploy](../infrastructure/auto-deploy.md).
**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G),
`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as
a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free
after this set.
## Enrollment gotcha: `pct exec` PATH
`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so
`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own
`command -v sops` post-install check failed under `pct exec` even though the binary installed fine.
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
[agent-enrollment.md troubleshooting](../../../operations/agent-enrollment.md#troubleshooting) if it recurs
on future LXC bootstraps.
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
What looked like repeated "protondrive silently stalls" was actually **`rclone-rcd.service` (the rc
API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting
OOM-killed** under the original 1 GiB RAM allocation — `journalctl` confirms
`A process of this unit has been killed by the OOM killer` at the exact moment a transfer had
"frozen." systemd's own `Restart=on-failure` (5s) auto-respawns it, but every in-flight job's state
is lost on the kill, which looked identical to a silent backend hang from the outside (frozen
`core/stats`, no new log lines). **Fix: bumped the LXC's memory to 2 GiB** (`pct set 132 -memory
2048` — applies live via the host cgroup, confirmed via `cat /sys/fs/cgroup/lxc/132/memory.max` on
hubris, no container reboot needed). After the bump, the full folder set (`cloud` 287G, `documents`
249M, `repos` 83M) completed cleanly with no further kills.
**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
haven't moved for 15 minutes, it restarts both `rclone-rcd.service` (clears any stuck/orphaned job —
this is the actual daemon holding the work, not the thin wrapper) and then `rclone-backup.service`
(`--no-block`, load-bearing — see below). Kept as a safety net even after the RAM fix, in case
memory pressure returns under a larger folder set later. State kept in
`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running.
**Two watchdog design bugs found and fixed while chasing this (2026-07-03):**
1. **Wrong stats-group key.** Per-job progress polling queried `core/stats` under `job/<jobid>`,
but rclone tracks stats under whatever `_group` name the job was submitted with. Made a perfectly
healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same
`group` variable consistently. **Lesson: distrust the per-run log's "progress bytes=" line during
an incident; cross-check with unfiltered `core/stats` first.**
2. **Watchdog restarted only the thin wrapper, and blocked doing it.** The actual `rclone sync` work
runs inside `rclone-rcd.service`, not `rclone-backup.service` — restarting the wrapper alone left
any stuck job orphaned inside `rcd` while a new wrapper submitted a duplicate job on top. Worse,
`systemctl restart rclone-backup.service` (no `--no-block`) blocks until the *new* invocation's
long-running `ExecStart` exits — which could be hours — so the watchdog's own oneshot service
never logged "Finished," and `OnUnitActiveSec` (which schedules relative to the previous run
*finishing*) never fired again. The watchdog silently disabled itself after exactly one use.
Fixed: restart `rclone-rcd.service` first, then `rclone-backup.service` with `--no-block`.
## Related
- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
- [Hubris host](../hosts/hubris.md) — owns `/mnt/library`
- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library`
- [Containers index](index.md)
## Changelog
### 2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed
What looked like repeated silent "protondrive stalls" turned out to be **`rclone-rcd.service`
getting OOM-killed** under the original 1 GiB RAM allocation — confirmed via
`journalctl -u rclone-rcd.service` showing `killed by the OOM killer` at the exact freeze point.
Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: `cloud` (287G) completed cleanly
(exit 0), `documents` (249M) completed with 1 minor error (259.7 MB transferred), `repos` (83M)
completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong
stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a
blocking `systemctl restart`, causing it to silently disable itself after one use) — see "Known
issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM
bump addresses the actual root cause.
### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web
GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote
`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with
`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling
`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this
rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line
as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running,
completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather
than truncating in place, specifically so it wouldn't risk corrupting the still-running original
`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was
unaffected, kept running to completion under the old in-memory script content). The already-running
scheduled sync from before this change is a standalone process and won't retroactively appear in the
GUI; every run after this point will.
### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync
Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was
entered instead of the TOTP secret — reconfigured with the secret, auth now works
(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes`
(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not
an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB
plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection:
`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the
actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early
rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by
choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays
staged at `/root/rclone-repo` for later.
### 2026-07-01 — provisioned; enrolled
LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3
installed from the official binary (`protondrive` backend present). Runner + monthly timer +
`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into
homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared
secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST`
updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive
remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain
operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB
backup deprecated in the same change.

61
knowledge/wiki/containers/133-seanime.md Normal file
View File

@@ -0,0 +1,61 @@
# seanime (LXC 133) — Seanime anime media server
## Summary
| Field | Value |
|-------|-------|
| VMID | 133 |
| Host | strong |
| Role | Anime media server (online streaming + torrent client) |
| LAN IP | 192.168.8.248/28 (vmbr1) |
| Public URL | https://seanime.hubris.network |
| Backend | Seanime v3.9.0, systemd service |
| Port | 43211 |
## Mounts
- `/anime` → `/mnt/media_local/anime` (ludo-lvm, bind mount) — existing anime collection
## Services
### Seanime (native binary)
- Binary: `/opt/seanime/bin/seanime`
- Data: `/opt/seanime/data/`
- Config: `/opt/seanime/data/config.toml`
- Service: `systemctl status seanime`
- Flags: `--host 0.0.0.0 --port 43211 --datadir /opt/seanime/data --disable-password`
- Config highlights:
- `secureMode = 'lax'` (allows non-local Caddy proxy)
- `trustedProxies = ['192.168.8.0/24']`
### Torrent client (qBittorrent on arriman)
- Host: `192.168.8.245:8080`
- Auth: subnet whitelist (no password needed from homelab LAN)
- qBittorrent config updated: `AuthSubnetWhitelist=192.168.8.0/24`
### Extensions installed
**Online streaming (8):**
HiAnime, AniWatch, KickAssAnime, Anicrush, Animo, AniNeko, Senshi, Sudatchi
## Caddy
- Config: `seanime.hubris.network` block in `/etc/caddy/Caddyfile` (dtoro/caddy-conf repo)
- TLS: Let's Encrypt DNS-01 via IONOS API
- Backend: `192.168.8.248:43211`
## DNS
- `seanime.hubris.network` A record → `192.168.8.175` (Caddy)
## Changelog
### 2026-07-05 — initial creation
- LXC 133 created on strong (Debian 13, 2 cores / 2 GiB)
- Seanime v3.9.0 installed as native binary + systemd service
- /anime bind mount from ludo-lvm
- qBittorrent on arriman configured as torrent client
- Caddy reverse proxy + DNS records set up
- Online streaming extensions installed

82
knowledge/wiki/containers/134-romm.md Normal file
View File

@@ -0,0 +1,82 @@
# 134 — `romm`
Self-hosted ROM manager ([RomM](https://romm.app)). Browse, search, and play
your retro game library from the browser. Runs on Docker Compose with a
MariaDB sidecar.
## At a glance
- **Hostname:** `romm`
- **IP:** `192.168.8.249/28` (static, vmbr1 on strong)
- **Privilege:** privileged
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13, ludo-lvm)
- **Mounts:** `/mnt/media_local` → `/mnt/library`
- **Public hostname:** `roms.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| RomM | `192.168.8.249:80` | HTTP (Caddy terminates TLS) |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/romm/docker-compose.yml`. Key points:
- Image: `rommapp/romm:latest`
- DB sidecar: `mariadb:latest` with healthcheck
- ROM library: `/mnt/library/roms` → `/romm/library` (writable)
- Resources (covers, etc.): Docker named volume `romm_resources` → `/romm/resources`
- Saves/states: `/opt/romm/assets` → `/romm/assets`
- Config: `/opt/romm/config` → `/romm/config`
- Auth key: auto-generated, stored in `/opt/romm/.env`
Environment (`/opt/romm/.env`):
- `DB_ROOT_PASSWD` / `DB_PASSWD` — MariaDB credentials
- `DB_USER=romm-user` / `DB_NAME=romm`
- `ROMM_AUTH_SECRET_KEY` — auto-generated
## ROM library structure
RomM expects `/mnt/library/roms/<platform>/<game>/<rom>`. Create platform
directories as needed:
```
/mnt/media_local/roms/
├── gba/
│ └── Pokemon - Emerald/
│ └── Pokemon Emerald.gba
├── snes/
│ └── Super Mario World/
│ └── Super Mario World.sfc
└── psx/
└── Final Fantasy VII/
└── Final Fantasy VII.bin
```
## Media permissions
The `/mnt/media_local/roms` directory is owned `root:media` with mode `2775`
(setgid). New files inherit the `media` GID (10000). The LXC is privileged so
no idmap block is needed — in-container UID/GID matches the host. Docker
containers within the LXC run as-is (read-only mount).
## Related
- [Strong host](../hosts/strong.md)
- [Caddy (121)](121-caddy.md) — `roms.hubris.network → 192.168.8.249:80`
- [DNS (107)](107-dns.md) — `roms.hubris.network A 192.168.8.175`
- [Media permissions](../infrastructure/media-permissions.md)
- [RomM docs](https://docs.romm.app)
## Changelog
### 2026-07-05 — provisioned
LXC 134 created on strong (Debian 13, privileged, `192.168.8.249/28`).
Docker + Compose installed. RomM stack deployed at `/opt/romm/`.
Created `/mnt/media_local/roms` with setgid `media:GID=10000` mode `2775`.
Caddy `roms.hubris.network` → `192.168.8.249:80`.
DNS `roms A 192.168.8.175` added to Technitium.
Hubris /32 route for `.249` added to `50-strong-route`.

90
knowledge/wiki/containers/archive/123-claudio-bot.md Normal file
View File

@@ -0,0 +1,90 @@
# 123 — `claudio-bot` (DEPRECATED — destroyed 2026-06-04)
> **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini.
> Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob.
> Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea.
> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details.
Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications.
## At a glance
- **Hostname:** `claudio-bot`
- **IP:** `192.168.8.230`
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 512 MiB RAM / 8 GiB rootfs
- **Mounts:** none from `/mnt/library`
- **Public hostname:** none
## Stack
Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`).
Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed).
## Configuration
Secrets at `/etc/claudio-bot/` (mode 600):
- `config.yaml`
- `matrix.token`
- `anthropic.key`
- `lmstudio.key`
- `ipc.token`
Deploy git creds at `/etc/claudio-deploy/git-credentials`.
### LLM backend (pluggable)
`llm.backend` in `config.yaml`:
- `anthropic` — Claude API via `anthropic.key`
- `lmstudio` — OpenAI-compat HTTP via `lmstudio.key` (bearer)
Currently set to `lmstudio` → `google/gemma-4-e4b` on the Mac mini at `192.168.8.174:1234` (since 2026-04-25). Switch back with `backend:` + `systemctl restart claudio-bot`. Original config saved at `/etc/claudio-bot/config.yaml.bak`. The LM Studio agent in `bot_core/lmstudio.py` translates Anthropic → OpenAI tool definitions.
## IPC
`http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by:
- [claudio-monitor on hubris](../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`)
- The (currently disabled) [restic backup wrapper](../infrastructure/backups.md) (token in `/etc/restic/bot.token`)
> Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together.
## Plugins
Module under `plugins/<name>.py` exposing a `Plugin` class; add `<name>` to `plugins:` in `config.yaml`. Plugins can register slash commands, Claude tools, and `on_notify` / `on_proposal_response` hooks.
Active plugins:
- `system` — `ping`, `help`, `status`, `list_plugins`
- `backup` — ingests `/notify` from the backup wrapper (currently silent — backups disabled)
- `monitor` — `/monitor status`, `/monitor history [N]`, `/monitor clear <key>`. Tools: `get_alerts`, `get_monitor_history` for NL queries via Claude.
## Auto-deploy
Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf.
`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.230`.
## Related
- [elementsynapse (118)](118-elementsynapse.md)
- [Monitoring (claudio-monitor)](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
## Changelog
### 2026-06-04 — LXC destroyed; replaced by Hermes Agent
LXC 123 destroyed via `pct destroy 123 --purge`. Bot service stopped, systemd
units disabled. `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived on
Gitea. Monitoring replaced by Hermes `homelab-health-watchdog` cron job.
`@claudio:avispero` Matrix account decommissioned.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-25 — LLM backend switched to LM Studio
`backend: lmstudio` → `google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field.
### 2026-04-21 — `monitor` plugin added
Receives events from [claudio-monitor](../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`.
### 2026-04-20 — claudio-bot deployed
LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server.

313
knowledge/wiki/containers/archive/127-mule-photos-new.md Normal file
View File

@@ -0,0 +1,313 @@
# 127 — `mule-photos-new`
Side-by-side **PhotoPrism M0 test** of the `dtoro/mule-image` `new` branch
at `photos-new.hubris.network`. Production [LXC 120](120-mule-images.md) keeps
running on the legacy stack at `photos.hubris.network` until M5 cutover.
## At a glance
- **Hostname:** `mule-photos-new`
- **IP:** `192.168.8.181`
- **Privilege:** unpriv
- **Resources:** 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap
- **Features:** `nesting=1,fuse=1,keyctl=1`
- **Mounts:** *(none — see scratch copy below)*
- **Public hostname:** [`photos-new.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`)
## Stack (`/opt/mule-image`)
`/opt/mule-image` is the working tree of `dtoro/mule-image` on branch
`new`. Compose lives at `docker-compose.photoprism.yml`; LXC-127-only
overrides at `docker-compose.photoprism.override.yml` (untracked — see
[Why an override exists](#why-an-override-exists)). Invoked with
`--env-file .env.photoprism`.
| Service | Container | Port | Notes |
| ----------- | ------------- | ----------------- | -------------------------------------------------- |
| mariadb | `pp-mariadb` | `127.0.0.1:3306` | MariaDB 11; named volume `pp_mariadb_data`; init SQL provisions `mule_sidecar.marks` table |
| photoprism | `pp-app` | `:2342` | `docker.io/photoprism/photoprism:latest`, runs as uid 33 (PP_UID/GID env) |
| sidecar | `pp-sidecar` | `:8000` (overridden) | Go + Gin service for rename / folder mutations / heap convert / dup detect |
| vite (host) | systemd unit | `:5173` | SvelteKit dev server (`mule-vite.service`), `npm run dev` in `/opt/mule-image/web` |
PhotoPrism is enterprise-tier (`tier: 1`) per the build tag (`-Plus`). TF
vision pipeline and EXIF backwrite are **disabled** in M0 — `PP_READONLY=true`
keeps the originals view read-only as the M0 safety net.
## Library — writable rsync scratch copy (NOT the real admin Photos)
Unlike LXC 120 (which mounts `/mnt/library` directly), this LXC has **no
bind-mount of the production library**. Instead a one-shot rsync of the
admin's Photos lives on the LXC's own rootfs:
- **Host source (read-only reference):** `/mnt/library/homecloud/admin/files/Photos` (~4.9 GB)
- **LXC scratch (writable):** `/srv/photos-scratch` (owner `www-data:media`, mode 0775)
This means **sidecar rename / folder mutation operations land in the
scratch copy, not the real admin library**. The scratch is not
auto-synced — it's a snapshot from `2026-05-17`. To refresh from
production:
```bash
# on hubris (LXC 127 must be stopped to mount its rootfs)
pct stop 127
pct mount 127
rsync -aHAX --info=stats2 --chown=100033:110000 --no-perms \
--chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r \
/mnt/library/homecloud/admin/files/Photos/ \
/var/lib/lxc/127/rootfs/srv/photos-scratch/
pct unmount 127
pct start 127
```
`--chown=100033:110000` accounts for the unprivileged-LXC ID shift
(host 100033 = LXC `www-data`, host 110000 = LXC `media`). Don't try a
bind-mount of `/mnt/library/...` — the admin Photos tree is 0750 and
unprivileged LXCs can't see through.
## Auth — Authentik OIDC
PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](124-authentik.md).
- **Provider/Application slug:** `mule-photos-new`
- **Issuer:** `https://auth.hubris.network/application/o/mule-photos-new/`
- **Redirect URI:** `https://photos-new.hubris.network/api/v1/oidc/redirect`
- **Scopes:** `openid profile email`
- **Initiated by clicking the OIDC button** at `/library/login` → `GET /api/v1/oidc/login` → 302 to Authentik authorize.
- `OIDC_REGISTER=true` + `OIDC_ROLE=admin` so the first SSO login auto-creates a PhotoPrism admin account.
Local PhotoPrism admin (username `admin`, password in
`/root/mule-photos-new-secrets.txt` on hubris) stays available as a
fallback.
## Why an override exists
`docker-compose.photoprism.override.yml` is **only on LXC 127** (not in
the git repo). After the 2026-05-17/18 fixes landed upstream
(commits `cce1d87` for OIDC env names and `3d8e050` for worker caps),
the override has shrunk to just one stanza — the cross-host sidecar
bind:
```yaml
services:
sidecar:
ports: !override
- "0.0.0.0:8000:8000"
```
Upstream binds the sidecar to `127.0.0.1:8000` because the M4 design
colocates Caddy with the sidecar. On this test LXC Caddy lives on a
different host (LXC 121), so the port has to be reachable from the
LAN. Delete this file at M4 cutover.
## Auto-deploy
Mirrors the LXC 120 pattern.
- **Webhook listener:** `mule-deploy-webhook.service` → `python3 /opt/mule-deploy/webhook.py` on `0.0.0.0:9797`.
- **Branch filter:** `refs/heads/new` (LXC 120 still owns `main`).
- **HMAC secret:** `/etc/mule-deploy/secret` (mode 0600).
- **Deploy script:** `/opt/mule-deploy/deploy.sh` — `git fetch && git reset --hard origin/new`, fix `pp/{storage,import}` ownership to `33:10000` (PP container uid), `docker compose ... up -d --build --force-recreate` with both compose files, `systemctl restart mule-vite`.
- **Gitea webhook id 9** on `dtoro/mule-image` pointed at `http://192.168.8.181:9797/deploy`.
Push to the `new` branch on [git.hubris.network/dtoro/mule-image](http://git.hubris.network/dtoro/mule-image) → webhook fires → rebuild. The legacy LXC 120 watches `main` and is unaffected.
**Gitea gotcha:** the receiver IP must be in `[webhook] ALLOWED_HOST_LIST`
in `/etc/gitea/app.ini` on [LXC 104](104-gitea.md). LXC 127's
`192.168.8.181` was missing on first bring-up; every push delivered
status 0 with the message `webhook can only call allowed HTTP servers`.
Adding the IP and `systemctl restart gitea` is enough — same list is
also the gating mechanism for the LXC 120 webhook. Verify any future
test instance is added before relying on auto-deploy.
## Bootstrap secrets
Saved on hubris at `/root/mule-photos-new-secrets.txt` (mode 0600):
- `PP_ADMIN_PASSWORD` — initial PhotoPrism `admin` login
- `PP_DB_PASSWORD` — MariaDB `photoprism` user
- `PP_DB_ROOT_PASSWORD` — MariaDB root
- `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` — generated by `ak shell` against Authentik
`SIDECAR_DB_PASSWORD` is still the literal placeholder
`replace-at-m4-bringup` because `mariadb/init/01-sidecar.sql` hardcodes
it; rotate before this stack ever goes public.
## Health checks
```bash
# from hubris
pct exec 127 -- curl -sf http://127.0.0.1:2342/api/v1/status # PP
pct exec 127 -- curl -sf http://127.0.0.1:8000/api/sidecar/healthz # sidecar
pct exec 127 -- curl -sf http://127.0.0.1:5173/ # Vite
# through Caddy
curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \
https://photos-new.hubris.network/api/v1/oidc/login -i | head -2 # 302 → auth.hubris.network
```
> **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack
> validated here was promoted into production on [LXC 120](120-mule-images.md)
> via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This
> page is retained for archaeology; everything below is historic. See the
> 2026-05-22 entry in [120-mule-images.md](120-mule-images.md#changelog) for
> the cutover detail.
## Changelog
### 2026-05-22 — Destroyed
`pct destroy 127` after curl verification of the new 120 stack passed
end-to-end. dnsmasq `photos-new.hubris.network` line removed; gitea
webhook id 9 + `192.168.8.181` ALLOWED_HOST_LIST entry removed; caddy
`photos-new.hubris.network` site block dropped from `dtoro/caddy-conf`.
Authentik `mule-photos-new` app + provider deleted.
### 2026-05-18 (pm) — OIDC state-cookie fix: deploy.sh no longer recreates pp-app
User reported `failed to get state: securecookie: the value is not valid` after
authenticating at Authentik. Root cause: PhotoPrism rotates the `Session:` HMAC
key in `pp/storage/config/hub.yml` on **every container start** (it's the hub
auto-refresh; no env flag disables it, and with `Status: ""` it regenerates
each boot). The key signs OIDC state cookies, so every restart invalidated
every in-flight login.
`/opt/mule-deploy/deploy.sh` was doing `docker compose up -d --build
--force-recreate` after a `pull --ignore-buildable`. Both moves recreate
pp-app: `--force-recreate` unconditionally, and the pull bumps the
`photoprism:latest` digest which makes plain `up -d` recreate too. Every
auto-deploy on a code push therefore broke every OIDC login.
Fixed by:
1. Dropping the `pull` step from the auto-deploy. Image refreshes for pp-app
/ mariadb are now an admin operation (run `docker compose pull` manually
when you want a new PhotoPrism build).
2. Force-recreating only the **sidecar** (its image rebuilds on every push
anyway). `up -d photoprism mariadb` reconciles in place — only restarts
if their compose declaration actually changes.
Verified by running `deploy.sh` twice and watching `hub.yml`'s `Session:`
field and pp-app's container PID; both stayed stable across the deploy.
The sidecar's PID changed as expected.
The fix lives in `/opt/mule-deploy/deploy.sh` on LXC 127. `/opt/mule-deploy`
is not a git checkout — keep this Changelog entry as the source of truth.
### 2026-05-18 — OIDC bridge + indexer + folder fixes
The `new` branch's PhotoPrism stack required several iterations to be
actually usable. Fixes pushed upstream so they apply to anyone running
the M0 compose; the LXC override file shrank to just the cross-host
sidecar port (see [Why an override exists](#why-an-override-exists)).
**OIDC end-to-end** (upstream commits `4abe6d7`, `9a3ad3e`, `cce1d87`,
plus Caddy `/library/* → /` bounce on LXC 121).
- The SvelteKit `/login` had a `// OIDC SSO ships in M4` placeholder
but no button. Added a "Sign in with {provider}" button conditional
on `/api/v1/config.ext.oidc.enabled`; click sends the browser to
`/api/v1/oidc/login`.
- The compose file passed OIDC values through `PHOTOPRISM_OIDC_ISSUER_URL`
/ `_CLIENT_ID` / `_CLIENT_SECRET` / `_PROVIDER_NAME` — names PhotoPrism
silently ignores. The actual env-var names are `PHOTOPRISM_OIDC_URI`
/ `_CLIENT` / `_SECRET` / `_PROVIDER` (see `photoprism show config`).
Renamed upstream; user-facing keys in `.env.photoprism` (OIDC_ISSUER_URL,
OIDC_CLIENT_ID, …) are unchanged.
- PhotoPrism's OIDC callback does **not** set `auth_token` / `auth_session`
cookies. It returns an HTML page that writes the session into
`localStorage` under `pp:<storageNamespace>:session.{id,token,user,provider}`
and then runs `window.location.href = "/library/login"`. Caddy on this
test instance bounces `/library/*` back to `/`, and the SvelteKit root
layout (`bootstrapSessionFromPhotoPrism()`) reads those localStorage
entries on mount, fetches `/api/v1/session/<id>` with the cookied
token, and adopts the session into the SPA store.
**Indexer caps** (upstream commit `3d8e050`). A fresh index of ~1.2k
photos pushed the LXC load average above 50 with the default
`PHOTOPRISM_INDEX_WORKERS` (NumCPU/2 = 3 here, each forking TF +
ffmpeg + libvips). Compose now reads `PP_WORKERS` / `PP_INDEX_WORKERS`
from `.env.photoprism`, defaulting to 2. Both set explicitly on this
LXC to keep sibling containers happy.
**Library went RW.** Flipped `PP_READONLY=false` and `PP_ORIGINALS_MODE=rw`
in `.env.photoprism` so the indexer can actually run — `READONLY=true`
disables it entirely. Safe because `/srv/photos-scratch` is the
rsync scratch copy on the LXC rootfs, not the real admin Photos tree.
**Admin role.** OIDC creates users with `OIDC_ROLE` ONLY on first
registration. The `dtoro` user was created in an earlier flow before
the env-var-name fix, so it landed as `guest` and saw no photos in
the UI. Promoted manually:
```sql
UPDATE auth_users SET user_role='admin', super_admin=1, can_invite=1
WHERE user_name='dtoro';
DELETE FROM auth_sessions WHERE user_name='dtoro';
```
Stale sessions are dropped so a fresh OIDC login mints an admin token.
**Video pre-transcode pass** (no upstream change — operational fix on
this LXC). Only 11/45 `.mov` originals had a `.avc` sidecar; the rest
forced 12–21 s inline libx264 transcodes on first playback, serialised
one ffmpeg at a time. Measured cold vs warm:
| Path | TTFB |
| --------------------------------- | ------- |
| Thumbnail `fit_1280` (warm) | ~2 ms |
| Video playback with `.avc` sidecar | ~2 ms |
| Video playback without sidecar | 12–21 s |
| Thumbnail GET *during* a transcode | ~2 ms (no sibling slowdown — 6-core LXC, ffmpeg ~6%/core, nvme util 0.01%) |
Mitigation:
```bash
pct exec 127 -- docker exec -d pp-app /opt/photoprism/bin/photoprism convert
```
Walks the library, builds every missing `.avc` next to its original
(`/photoprism/storage/sidecar/<year>/<month>/<name>.mov.avc`),
idempotent on re-run, two ffmpeg processes in parallel. Took ~8 min
to bring coverage to 45/45. Previously-cold videos verified to serve
at ~2 ms TTFB after the pass. Future imports get AVC sidecars
automatically as part of indexing; a re-run is only needed if videos
ever land outside the indexer's path.
**Folder tree** (upstream commits `8083328`, `505fef5`, `cfd85a1`).
PhotoPrism's `path:` operator is exact-match by default but supports
a `*` wildcard. Without it, every internal tree node (year folders,
since photos always nest under YYYY/MM) returned zero hits — both in
the timeline and in the sidecar's folder-count fan-out. Fixed both
to emit `path:"<folder>*"`. Also relaxed the root-folder client-side
filter (was clipping to `Path === ''`, which is always empty) so `/`
shows the whole library. Root badge in the sidebar now reads
`config.count.all` directly instead of subtracting Σ(folderCounts) —
the subtraction double-counted after the recursive switch.
### 2026-05-17 — Bring-up
LXC 127 created from `debian-13-standard_13.1-2`, joined to vmbr0 with
static IP `192.168.8.181`. Docker engine + Node 20 installed.
`dtoro/mule-image` cloned at branch `new`, compose stack
(`docker-compose.photoprism.yml`) brought up: MariaDB 11 + PhotoPrism
`:latest` (`-Plus` build) + Go sidecar (built locally). Vite dev server
running as `mule-deploy` via `mule-vite.service` on port 5173.
Authentik OIDC application `mule-photos-new` provisioned via `ak shell`
(`OAuth2Provider` + `Application` + STRICT `RedirectURI`). PhotoPrism's
OIDC button delegates to Authentik; `OIDC_REGISTER=true` /
`OIDC_ROLE=admin` so the first SSO login becomes admin.
Admin's Photos library rsynced (~4.9 GB, 1206 files) into
`/srv/photos-scratch` on the LXC rootfs (no bind-mount). Sidecar
mutations land in the scratch copy, not the real library.
Caddy site `photos-new.hubris.network` added in `dtoro/caddy-conf`;
dnsmasq entry on LXC 124 → `192.168.8.175`. dnsmasq required a `restart`
(not `reload`) for the new `address=` line to take effect.
`docker-compose.photoprism.override.yml` (LXC-only, untracked) pins two
upstream issues: sidecar bound to `127.0.0.1` (cross-host Caddy can't
reach), and OIDC env-var name mismatch
(`PHOTOPRISM_OIDC_ISSUER_URL` vs `PHOTOPRISM_OIDC_URI` and friends).
Should land upstream on `new` next iteration.

57
knowledge/wiki/containers/index.md Normal file
View File

@@ -0,0 +1,57 @@
# LXC containers — index
Most containers live on [`hubris`](../hosts/hubris.md). Some have been
[migrated to `strong`](../hosts/strong.md) (Phase 1+2, 2026-07-05).
| ID | Name | Host | IP | Priv | Cores | RAM | Disk | Mounts | Public hostname | Status |
| --- | ---------------- | ------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- |
| 101 | [jellyfin](101-jellyfin.md) | **strong** | 192.168.8.246 | priv | 4 | 8 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `media.hubris.network` | running |
| 103 | [paperless](103-paperless.md) | hubris | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running |
| 104 | [gitea](104-gitea.md) | hubris | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running |
| 105 | [apps](105-apps.md) | hubris | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `artifacto` / `blog` | running |
| 114 | [nextcloud](114-nextcloud.md) | hubris | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running |
| 118 | [elementsynapse](118-elementsynapse.md) | **strong** | 192.168.8.242 | unpriv | 2 | 4 GiB | 32 GiB | — | `matrix.hubris.network` | running |
| 119 | [sophia](119-sophia.md) | hubris | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running |
| 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running |
| 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
| 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running |
| 124 | [authentik](124-authentik.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
| 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running |
| 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running |
| 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running |
| 131 | [teddycloud](131-teddycloud.md) | hubris | 192.168.8.150 | — | 1 | 1 GiB | 16 GiB | `/mnt/library` | `teddy.hubris.network` (no auth gate) | running |
|| 132 | [rclone](132-rclone.md) | hubris | 192.168.8.214 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` (**ro**) | — (LAN-only UI `:5572`) | running |
|| 134 | [romm](134-romm.md) | **strong** | 192.168.8.249 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `roms.hubris.network` | running |
## Recently destroyed (kept for archaeology)
| ID | Name | Destroyed | Reason |
| --- | ---------------- | --------------- | --------------------------------------------- |
| 127 | mule-photos-new | 2026-05-22 | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/2026-06-04_130000-deprecate-claudio-bot.md) |
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned |
> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
## Conventions
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range.
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong.
- Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
## Related
- [Hubris host](../hosts/hubris.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Caddy](121-caddy.md) — terminates every public hostname
- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain