dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion

Browse Source 
- Migrations 010 (content_hash) + 011 (search tsvector column)
- new: internal/knowledge/seed.go — knowledge seed ingest engine
- new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge
- wire knowledge ingest into oikos seed pipeline
- convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml
- archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/
- delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py
- remove empty knowledge/ directory tree
This commit is contained in:
authentik Default Admin
2026-07-07 20:22:30 +02:00
parent b2bfa26f64
commit 6b75f7302d
125 changed files with 7557 additions and 1938 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
archive/knowledge/GLOSSARY.md Normal file
View File

@@ -0,0 +1,33 @@
# Glossary
Terms and abbreviations used throughout the homelab wiki.
| Term | Meaning |
|------|---------|
| **Authentik** | SSO/identity provider. Core runs on the VPS; forward-auth outpost at LXC 106 on hubris |
| **Caddy** | Reverse proxy (LXC 121). Terminates TLS for every `*.hubris.network` hostname |
| **Caveman** | Terse communication standard for agent responses — no filler, keep substance |
| **Forward-auth** | Caddy snippet that delegates authentication to an Authentik outpost. Protects web UIs like qBit, SABnzbd |
| **Gitea** | Git server at `git.hubris.network`. Hosts all tracked config repos |
| **Gluetun** | WireGuard VPN sidecar on arriman. All \*arr traffic routes through it |
| **HAOS** | Home Assistant Operating System. VM 108 on hubris |
| **Hubris** | Primary Proxmox VE node (GMKtec NucBox M6 Ultra). PVE hostname, cluster member 1 |
| **LXC** | Linux Container (Proxmox). VM-like isolation without a full OS kernel |
| **LVM-thin** | Thin-provisioned logical volume manager. Used for all container/VM storage |
| **MCP** | Model Context Protocol (MCP server at `mcp.hubris.network`). Structured tools for agents to query homelab state |
| **Mesh** | Overlay VPN for off-LAN connectivity. Netbird is current; Tailscale is legacy |
| **Netbird** | Preferred mesh VPN. VPS hosts the management plane; all homelab nodes are members |
| **OIDC** | OpenID Connect. Protocol used by Authentik for SSO login flows |
| **Oikos** | Agent operating model ([.agents/OIKOS.md](../.agents/OIKOS.md)). OODA loop, risk classes, policy, ontology |
| **PVE** | Proxmox Virtual Environment — the hypervisor on both hubris and strong |
| **SOPS** | `sops` — Mozilla SOPS. Encrypts secrets with age keys so they live in the git repo |
| **Strong** | Secondary Proxmox VE node. Cluster member 2 (hostname `strong`, nickname ludo/ludo-mini) |
| **Traefik** | Reverse proxy on IONOS VPS. Serves `*.hubris.network` to the public internet |
| **VAAPI** | Video Acceleration API. Intel/AMD GPU-based hardware transcode for Jellyfin |
| **VPS** | Virtual Private Server at IONOS (`82.165.190.79`). Runs Authentik core + Netbird management |
| **\\*arr** | Media automation suite: Sonarr (TV), Radarr (movies), Lidarr (music), Prowlarr (indexer), Bazarr (subtitles), Readarr (books — not in use) |
## See also
- [Infrastructure index](wiki/infrastructure/index.md) — cross-cutting systems each with their own doc page
- [OIKOS operating model](../.agents/OIKOS.md) — agent policy, risk classes, lifecycle

160
archive/knowledge/containers/101-jellyfin.md Normal file
View File

@@ -0,0 +1,160 @@
# 101 — `jellyfin`
Media server: serves the movies / TV / anime / music libraries from `/mnt/media_local` to LAN clients. Hardware transcoding via AMD Radeon 680M + RX 7600 VAAPI. Authentik SSO via OIDC.
## At a glance
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **IP:** `192.168.8.246`
- **Privilege:** privileged (recreated on strong as priv)
- **Resources:** 4 cores / 8 GiB RAM / 1 GiB swap / 16 GiB rootfs
- **GPU:** `/dev/dri/renderD128` + `/dev/dri/card0` (AMD Radeon 680M iGPU + RX 7600 dGPU) passed via `dev0` / `dev1` in LXC config
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
- **Public hostname:** [`media.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8096`
- **Version:** Jellyfin 10.11.11 (apt package, Ubuntu 24.04 noble repo)
- **FFmpeg:** jellyfin-ffmpeg7 7.1.4
## Service / port map
| Service | Listen | Notes |
| -------- | ------ | ----- |
| jellyfin | `:8096` | HTTP (caddy terminates TLS) |
## Hardware acceleration (VAAPI)
GPU is passed through to the LXC via `dev0: /dev/dri/renderD128,gid=993` and
`dev1: /dev/dri/card0,gid=44` in
`/etc/pve/lxc/101.conf` (strong). The `jellyfin` user is in the `render` (GID 993) and
`video` groups inside the container.
| GPU | Model | Role |
|-----|-------|------|
| Radeon 680M | iGPU (AMD Ryzen 7 PRO 6850U) | Primary VAAPI encoder/decoder |
| RX 7600 | dGPU (add-in) | Secondary transcode, HEVC/AV1 encoding |
Encoding settings (`/etc/jellyfin/encoding.xml`):
- `HardwareAccelerationType`: `vaapi`
- `VaapiDevice`: `/dev/dri/renderD128`
- `EnableHardwareEncoding`: `true`
- `AllowHevcEncoding`: `true`
- `AllowAv1Encoding`: `true`
- `EnableTonemapping`: `true`
- `HardwareDecodingCodecs`: h264, hevc, vc1, vp9, av1
- `EnableThrottling`: `true`
- `EnableSegmentDeletion`: `true`
Trickplay (`/etc/jellyfin/system.xml`):
- `EnableHwAcceleration`: `true`
- `EnableHwEncoding`: `true`
## Authentik SSO (OIDC)
Jellyfin uses the [SSO-Auth plugin](https://github.com/9p4/jellyfin-plugin-sso)
v4.0.0.4 for Authentik OIDC login. No Caddy forward-auth gate — the SSO plugin
handles auth directly via OIDC redirect flow.
### Architecture
```
User → media.hubris.network → Caddy (TLS, no forward-auth) → Jellyfin :8096
↓
Login page with "Sign in with Authentik" button
↓ (click)
/sso/OID/start/Authentik
↓ (302 redirect)
auth.hubris.network OIDC
↓ (login)
/sso/OID/redirect/Authentik?code=...&state=...
↓
Jellyfin SSO plugin validates token → logged in
```
### Components
1. **SSO-Auth plugin** — installed at `/var/lib/jellyfin/plugins/SSO-Auth_4.0.0.4/`
- Config: `/var/lib/jellyfin/plugins/configurations/SSO-Auth.xml`
- Provider name: `Authentik`
- OIDC endpoint: `https://auth.hubris.network/application/o/jellyfin/`
- `SchemeOverride`: `https` (required — without it, plugin generates
`http://` redirect URIs that Authentik rejects)
- `EnableAuthorization`: `false` (prevents plugin from overwriting admin
permissions on each SSO login — see
[jellyfin-sso-plugin](../../../devops/homelab-authentik-admin/references/jellyfin-sso-plugin.md))
- `OidScopes`: `["email"]` (openid+profile added by default by the plugin;
must be non-null or `OidChallenge()` throws `ArgumentNullException`)
2. **Authentik OIDC provider** — `Provider for Jellyfin` (PK 6)
- Client ID: `vt61t5Y2ZVtN6l3QjitkBvwUJjFKvSyl4TDBXcJx`
- Redirect URI: `https://media.hubris.network/sso/OID/redirect/Authentik`
- Application slug: `jellyfin`
3. **SSO button injection** — `/usr/share/jellyfin/web/sso-inject.js`
- Injected via `<script defer src="sso-inject.js?v=3">` in `index.html`
- Polls for `.readOnlyContent` div on the login page, adds "Sign in with
Authentik" button linking to `/sso/OID/start/Authentik`
- Cache-busted with `?v=N` parameter (bump on changes)
- **Lost on apt upgrade** — re-inject the script tag and copy the JS file
after `apt-get upgrade jellyfin*`
4. **Caddy config** — no forward-auth gate for media.hubris.network:
```caddy
media.hubris.network {
tls { dns ionos {env.IONOS_AUTH_API_TOKEN} }
reverse_proxy 192.168.8.206:8096
}
```
### Known issues / pitfalls
- **`OidScopes` must be non-null** — if the field is missing from the plugin
config XML, `OidChallenge()` throws `System.ArgumentNullException`. Always
include `OidScopes` in the provider config (even if empty array).
- **`SchemeOverride: "https"` is required** — without it, the plugin generates
`http://` redirect URIs (from the internal HTTP listener). Authentik rejects
them with "Redirect URI Error".
- **SSO button JS is not served by the plugin** — the `__plugin/SSO-Auth.js`
endpoint returns 404 on Jellyfin 10.11.x when the plugin is installed
manually (not via Jellyfin's plugin manager). The `sso-inject.js` workaround
in `index.html` is the fallback.
- **No Caddy forward-auth gate** — the SSO plugin's OIDC redirect flow is
incompatible with Caddy's `import authentik` forward-auth. If both are
enabled, the forward-auth intercepts the OIDC callback and breaks the flow.
Use one or the other, not both. SSO plugin (OIDC redirect) is preferred.
- **API key for setup** — a temp API key can be inserted directly into the
`ApiKeys` SQLite table for automated configuration:
```sql
INSERT INTO ApiKeys VALUES (1, '2026-07-04', '2026-07-04', 'setup', 'jf-setup-key-...');
```
## Permissions
Member of the [media GID 10000](../infrastructure/media-permissions.md) standard. Service user `jellyfin` is in the `media` group inside the container; `/mnt/media_local` on strong's ludo-lvm is owned `root:media` with mode `2775`.
## Related
- [Caddy reverse proxy](121-caddy.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [arriman](122-arriman.md) — \*arr stack writes the libraries jellyfin reads
- [DNS split-horizon](../infrastructure/dns.md)
- [Authentik admin](../../../devops/homelab-authentik-admin/SKILL.md) — OIDC provider creation, SSO plugin config
## Changelog
### 2026-07-06 — wiki: IP, host, GPU, mount path updated for strong migration
Updated At-a-glance: IP 206→246, host hubris→strong, mount /mnt/library→/mnt/media_local, GPU Radeon 760M→680M+RX7600, privilege unpriv→priv. Permissions section updated. Changelog entry in 122-arriman.md updated similarly.
### 2026-07-04 — VAAPI hardware acceleration + Authentik SSO + resource bump
- Upgraded Jellyfin 10.11.8 → 10.11.11 (purge + reinstall to fix DB migration bug)
- Enabled VAAPI hardware acceleration (Radeon 760M): h264/hevc/vc1/vp9/av1 decode + encode
- Bumped resources: 2→4 cores, 4→8 GiB RAM, 512→1024 MiB swap
- Enabled trickplay HW acceleration + throttling + segment deletion
- Installed SSO-Auth plugin v4.0.0.4 with Authentik OIDC provider
- Configured `SchemeOverride: https`, `OidScopes: ["email"]`, `EnableAuthorization: false`
- Injected SSO button via `sso-inject.js` in web `index.html`
- Removed Caddy forward-auth gate (incompatible with SSO plugin OIDC flow)
- **Database was wiped** during cache relocation attempt — no LVM snapshot
existed. All watch states, user accounts, and library configs lost.
Libraries re-added via setup wizard.
### 2026-04-28 — wiki entry created
Initial documentation. No config changes.
### 2026-04-20 — joined the `media` GID 10000 standard
Idmap block applied; in-container `media` group at GID 10000 mapped to host GID 10000. See [media permissions](../infrastructure/media-permissions.md). Config backup: `/root/101.conf.bak.*`.

89
archive/knowledge/containers/102-nfs-export.md Normal file
View File

@@ -0,0 +1,89 @@
# 102 — `nfs-export`
Dedicated, single-purpose LXC that re-exports `/mnt/library` over NFSv4 to clients that can't use the host's PVE virtiofs path — currently only [100-zimaos](../vms/100-zimaos.md), which ships a kernel without virtiofs support.
## At a glance
- **Hostname:** `nfs-export`
- **IP:** `192.168.8.200` (static; LAN-only, no Caddy in front because NFS is L4)
- **LAN DNS:** `nfs-export.hubris.network` → `192.168.8.200` (direct, no Caddy)
- **Privilege:** privileged (`unprivileged: 0`) + `lxc.apparmor.profile: unconfined` — required for `nfs-kernel-server`
- **Resources:** 1 core / 512 MiB RAM / 2 GiB rootfs / 256 MiB swap
- **Mounts:** host `/mnt/library` ↔ container `/mnt/library` (same path on both sides — matches the bind-mount convention used by jellyfin, paperless, arriman, nextcloud, mule-images, apps)
## What it does
```
/mnt/library (host ext4 on nvme1n1)
│
├── bind-mounted into 7 other LXCs (jellyfin, paperless, …)
└── bind-mounted into LXC 102
│
└── nfs-kernel-server exports /mnt/library
│
└── consumed by VM 100 (ZimaOS)
```
Same inodes, same page cache. The NFS server is just one more access path on top of a tree that 8 other consumers already share — see [media permissions](../infrastructure/media-permissions.md).
## Export config
`/etc/exports`:
```
/mnt/library 192.168.8.0/24(rw,all_squash,anonuid=33,anongid=10000,no_subtree_check,sec=sys)
```
Initially started as `ro`; promoted to `rw` on 2026-05-14 after the Files-UI evaluation confirmed (a) the library renders correctly as a folder under `/DATA`, (b) thumbnails are generated, (c) the squash works — a write from ZimaOS appears on `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention used by Nextcloud and mule-images.
### Guardrails (in order of importance)
1. **`all_squash,anonuid=33,anongid=10000`.** Every write from ZimaOS records on disk as `www-data:media` (uid 33, gid 10000), the same identity Nextcloud and mule-images use. Keeps the existing tree convention from drifting. See [`project_media_perms`](../infrastructure/media-permissions.md).
3. **Subnet restriction `192.168.8.0/24`.** No public/mesh access; LAN only.
4. **`no_subtree_check`** + **`sec=sys`** — standard performance/auth pair for a homelab.
5. **No `crossmnt`** because `/mnt/library` is a single ext4 filesystem on the host (no nested mounts to traverse).
### What we're *not* doing yet
- No per-subdir export. ZimaOS sees the whole tree; access is controlled by filesystem permissions (`drwxr-x---` private dirs like `documents/`, `notes/`, `heaper/` aren't readable by `media` group, so ZimaOS-as-squashed-uid won't see them either).
- No write-back. Until we promote to `rw`, ZimaOS can't write — eliminates the lock-domain split concern between NFS clients (NLM/v4) and local LXCs (POSIX flock).
- No Authentik / forward-auth. NFS doesn't sit behind HTTP, so the standard caddy+authentik path doesn't apply. Subnet ACL is the only auth.
## Why this LXC exists (vs serving from host)
We considered three options before building this:
| Option | Outcome |
|---|---|
| **NFS on hubris bare-metal host** | Best performance, but adds long-lived NFS/RPC daemons to a host with a recent crash episode ([hubris crash 2026-04-21/22](../../sources/investigations/index.md)). Rejected. |
| **SMB on host** | Same host-blast-radius problem, plus 30–50% lower throughput than NFS on Linux↔Linux. Rejected. |
| **NFS in a dedicated LXC** ← this | Within ~2% of host performance (LXC is namespace isolation; IO path is unchanged), zero new daemons on hubris, matches the existing fleet pattern. Selected. |
Rationale lives in [the install plan](https://git.hubris.network/dtoro/Homelab-Docs) `/root/.claude/plans/i-wannt-you-to-nifty-muffin.md` on hubris.
## Operations
- **Reload exports** after editing `/etc/exports`: `pct exec 102 -- exportfs -ra`
- **List active exports:** `pct exec 102 -- exportfs -v`
- **Watch from outside:** `showmount -e 192.168.8.200`
- **Service health:** `pct exec 102 -- systemctl is-active nfs-server rpcbind`
- **Restart cleanly:** `pct restart 102` (ZimaOS will retry the mount via `nofail`)
- **Destroy + rebuild:** `pct stop 102 && pct destroy 102 --purge` — reversible in seconds; only ZimaOS notices
## Open items
- Consider tightening the export to subdirs (e.g. `movies`, `tv`, `music`, `audiobooks`, `books`, `images`, `podcasts`, `roms`) if you don't want ZimaOS reachable into `documents/`, `notes/`, `heaper/`, etc. — though those private subdirs are already invisible to `media`-group perms.
- ZimaOS architecture finding: the Drives panel only enumerates physical/block devices via `GET /v2/local_storage/storages` (read-only API, no POST). Network shares **cannot** appear as Drives — they show up as folders in Files. This is intentional in CasaOS's design; don't try to work around it. Library-as-folder is the supported model.
- Consider adding Samba to this LXC if a future Mac/iOS client needs SMB on the same tree — same LXC, no host changes.
- No PBS backup (no PBS configured on hubris); the container is fully described in this page + `pct config 102`, rebuild from scratch in <2 min if lost.
## Related
- [100-zimaos](../vms/100-zimaos.md) — the only consumer today
- [media permissions](../infrastructure/media-permissions.md) — uid 33 / gid 10000 standard
- [DNS](../infrastructure/dns.md) — `nfs-export.hubris.network` entry (direct, no Caddy)
## Changelog
### 2026-05-14 — Promoted to rw; squash behaviour verified
After ZimaOS Files UI evaluation passed (library renders as folder under `/DATA`, thumbnails work, ZimaOS Drives panel ignores NFS by design), flipped export to `rw`. Tested: writing `/DATA/library/.zimaos-rw-test` from ZimaOS appears on hubris's `/mnt/library` owned `www-data:media` (uid 33, gid 10000), confirming `all_squash,anonuid=33,anongid=10000` works as designed. Also discovered the dead end: ZimaOS's `GET /v2/local_storage/storages` is the source of the Drives panel; it returns only physical storage and rejects POST/PUT — network shares cannot be promoted to Drives.
### 2026-05-14 — LXC built; NFS export live (read-only)
Privileged Debian 13 container created with bind-mount `/mnt/library`. `nfs-kernel-server` installed and enabled; export `/mnt/library` to `192.168.8.0/24` with `ro,all_squash,anonuid=33,anongid=10000`. Smoke-tested from hubris host (mounted, listed library tree, confirmed RO). `nfs-export.hubris.network` added to LXC 124 dnsmasq.

41
archive/knowledge/containers/103-paperless.md Normal file
View File

@@ -0,0 +1,41 @@
# 103 — `paperless`
Paperless-ngx for document management. Ingests scans / PDFs from `/mnt/library/documents` and serves OCR'd indexed copies.
## At a glance
- **Hostname:** `paperless`
- **IP:** `192.168.8.130`
- **Privilege:** privileged
- **Resources:** 2 cores / 3 GiB RAM / 8 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (consumes `documents/`)
- **Public hostname:** [`paperless.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8000`
## Service / port map
| Service | Listen | Notes |
| ------------------- | ------- | -------------------------------- |
| paperless-webserver | `:8000` | HTTP |
| paperless-task-queue, paperless-scheduler, paperless-consumer | — | systemd workers |
## Auth
Behind [Authentik forward-auth](106-auth-outpost.md). API path `/api/*` bypasses forward-auth (mobile clients can't follow the browser login redirect; bearer token still enforces auth on `/api`). Header propagation: `PAPERLESS_ENABLE_HTTP_REMOTE_USER=true` and `PAPERLESS_HTTP_REMOTE_USER_HEADER_NAME=HTTP_X_AUTHENTIK_USERNAME` in `/opt/paperless/paperless.conf`. Django auto-creates matching users on first SSO login.
## Storage
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
## Known issues
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- [Authentik](106-auth-outpost.md)
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — wired into Authentik forward-auth
Cookie-domain-level proxy provider, embedded outpost binding. `/api/*` bypass added in Caddy site block to keep mobile clients working.

61
archive/knowledge/containers/104-gitea.md Normal file
View File

@@ -0,0 +1,61 @@
# 104 — `gitea`
Self-hosted git server. Source of truth for all `dtoro/*` repositories that auto-deploy across the lab.
## At a glance
- **Hostname:** `gitea`
- **IP:** `192.168.8.121`
- **Privilege:** privileged
- **Resources:** 1 core / 1 GiB RAM / 8 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (under `/mnt/library/repos`, owner UID 102 / GID 105 — don't touch from other containers)
- **Public hostname:** [`git.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000`
- **Version:** 1.25.4 (last verified)
## Service / port map
| Service | Listen | Notes |
| ------- | -------- | ------------------------------ |
| gitea | `:3000` | HTTP web + git over HTTP |
| gitea SSH | `:22` | git over SSH |
| `gitea-deploy-webhook` | `127.0.0.1:9797` | Receives push events for the customizations repo |
## Customizations
`/var/lib/gitea/custom` is a checkout of [`dtoro/gitea-customizations`](https://git.hubris.network/dtoro/gitea-customizations).
- `templates/custom/footer.tmpl` injects:
- **Online3DViewer** v0.18.0 (vendored under `public/assets/o3dv/`) for inline 3D preview of CAD raw-file links: 3dm, 3ds, 3mf, amf, bim, brep, dae, fbx, fcstd, glb, gltf, ifc, igs, iges, stp, step, stl, obj, off, ply, wrl.
- **PlantUML client JS** (vanilla, vendored from `gitea.com/davidsvantesson/plantuml-code-highlight`). Browser calls `https://git.hubris.network/_plantuml/img/<encoded>` — caddy `handle_path /_plantuml/*` rewrites and proxies to the PlantUML server on [apps (105)](105-apps.md) `:8079`.
- `home.tmpl` is the anonymous landing page (Homesauce hero + sign-in/explore buttons).
- O3DV binary assets are *not* tracked — `deploy.sh` fetches them on first run.
## Auto-deploy
Push to `dtoro/gitea-customizations` master → gitea webhook → `http://127.0.0.1:9797/deploy` (loopback) → `gitea-deploy-webhook.service` validates HMAC against `/etc/gitea-deploy/secret` → runs `/opt/gitea-customizations/deploy.sh`. Then `systemctl restart gitea` if templates changed.
`app.ini` `[webhook] ALLOWED_HOST_LIST` includes `127.0.0.1`, `192.168.8.175`, `192.168.8.205`, `192.168.8.230`, `192.168.8.136`, `192.168.8.77` — every webhook target across the lab. **Don't strip these when editing app.ini.** See [auto-deploy](../infrastructure/auto-deploy.md) for the full list.
Git creds for the deploy service: `/etc/gitea-deploy/git-credentials` (mode 600), referenced via `credential.helper = store --file=/etc/gitea-deploy/git-credentials` in the repo's `.git/config` (because `ProtectHome=true` blocks `/root`).
## DNS workaround
LXC has `/etc/hosts` override mapping `auth.hubris.network → 192.168.8.175` (router DNS resolves the public IONOS A record, which is a dead end for LAN clients — see [DNS](../infrastructure/dns.md)). Belt-and-suspenders systemd unit `hubris-hosts-override.service` re-applies it idempotently. Override line is **outside** the `# --- BEGIN PVE ---` markers (Proxmox rewrites inside).
## Related
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy pipelines](../infrastructure/auto-deploy.md)
- [PlantUML server lives on apps (105)](105-apps.md)
## Changelog
### 2026-06-24 — terminalito deploy webhook (id 12)
Push webhook on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` ([trmnl (128)](128-trmnl.md)); `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — `app.ini` ALLOWED_HOST_LIST extended for Artifacto webhook
Added `192.168.8.205`. See [Artifacto auto-deploy on apps (105)](105-apps.md).
### 2026-04-21 — `/etc/hosts` override for `auth.hubris.network` added
For OIDC integration with [authentik (124)](106-auth-outpost.md). Outside the PVE markers, with a hubris-hosts-override.service for idempotency.
### 2026-04-20 — gitea customizations + auto-deploy pipeline shipped
`dtoro/gitea-customizations` repo created; webhook receiver at loopback `:9797` validates HMAC and runs `deploy.sh`. CAD and PlantUML loaders live in `footer.tmpl`.

154
archive/knowledge/containers/105-apps.md Normal file
View File

@@ -0,0 +1,154 @@
# 105 — `apps`
Docker host for everything that doesn't justify its own LXC. Currently runs Artifacto, PlantUML server, Portainer (and historically WriteFreely / blog), plus the [homelab-context distribution services](../infrastructure/homelab-context.md) (MCP + secrets-issuance) since 2026-05-20. Booklore migrated to [grimmory (130)](130-grimmory.md) on 2026-06-29.
## At a glance
- **Hostname:** `apps`
- **IP:** `192.168.8.205`
- **Privilege:** privileged (so container UID = host UID for bind-mount sanity)
- **Resources:** 2 cores / 4 GiB RAM / 30 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library`
- **Public hostnames:** see table below
## Public hostname → service map
| Hostname | Container | Backend port | Notes |
| --------------------------------- | ---------------- | ------------ | ----- |
| `docker.hubris.network` | Portainer | `:9443` | Native OAuth2 via Authentik. Trusted-origins requires hostname only (no scheme/port). |
| `artifacto.hubris.network` | Artifacto | `:3100` | Public `/p/*`, `/static/*`, `/healthz` exposed via [VPS traefik](../infrastructure/ingress.md). |
| `blog.hubris.network` | WriteFreely | `:8080` | Native OIDC via `[oauth.generic]`. |
| `git.hubris.network/_plantuml/*` | PlantUML server | `:8079` | Same-origin route from [gitea (104)](104-gitea.md). |
| `mcp.hubris.network` | homelab-mcp | `:9810` | MCP server over SSE. No SSO — agent-facing, gated at source-IP layer. See [homelab-context](../infrastructure/homelab-context.md). |
| `secrets.hubris.network` | secrets-issuance | `:9820` | Per-client age-key issuance. Mesh+LAN source-IP gated (`MESH_SUBNETS`). |
## Stacks
### Artifacto (`/opt/artifacto/`)
Self-hosted HTML-artifact share (Go + SQLite). Repo `dtoro/Artifacto` is the working tree itself — `git push origin main` triggers rebuild.
- Compose: tracked `docker-compose.yml` binds `./data:/data` and `127.0.0.1:3100:3000`. `docker-compose.override.yml` uses the `!override` YAML tag on `volumes:` and `ports:` to redirect `data` to `/mnt/library/documents/artifacts`.
- Data dir: `/mnt/library/documents/artifacts` — owner **UID 10001 : GID 10001** (matches the `app` user baked into the Dockerfile), mode 750. Parent `/mnt/library/documents` stays `www-data:www-data 750`; bind targets the child directly so no traversal needed.
- `.env` (untracked) holds `ADMIN_PASSWORD`, `SESSION_SECRET` (`openssl rand -hex 32`), `BASE_URL=https://artifacto.hubris.network`, `SSO_GATEWAY_SECRET` (must match `ARTIFACTO_GATEWAY_SECRET` in `/etc/caddy/caddy.env` on [caddy](121-caddy.md)). Don't `git checkout .env`.
- Auth: Authentik forward-auth on the admin surface (`/`, `/login`, `/a/*`, `/api/*`). `X-Authentik-Username` is honored only when the `X-Artifacto-Gateway` header matches `SSO_GATEWAY_SECRET` — prevents direct LAN connections from spoofing identity.
- Public exposure: VPS traefik gates path scope to share URLs; admin surface returns 404 at the edge. See [ingress](../infrastructure/ingress.md).
#### Auto-deploy pipeline (Artifacto)
Receiver at `/opt/artifacto-deploy/` (outside the app repo): `deploy.sh` + `webhook.py`, secret in `/etc/artifacto-deploy/secret`, systemd unit `artifacto-deploy-webhook.service`. Listens on `0.0.0.0:9798`, validates `X-Gitea-Signature` HMAC-SHA256, filters to `refs/heads/main`, runs deploy in a daemon thread (returns 202 immediately). Gitea webhook id 7 on `dtoro/Artifacto`.
- Logs: `pct exec 105 -- journalctl -u artifacto-deploy-webhook -f`
- Manual deploy: `pct exec 105 -- /opt/artifacto-deploy/deploy.sh`
- Health: `pct exec 105 -- curl -s http://127.0.0.1:9798/health` → `ok`
- Slug length bumped 5 → 8 chars (in `internal/store/artifacts.go:41`) once the endpoint went public.
### PlantUML (`/opt/plantuml/`)
`plantuml/plantuml-server:jetty` listening on `192.168.8.205:8079`, context path `/plantuml` (`BASE_URL=plantuml`). Reached only via [gitea](104-gitea.md)'s caddy `handle_path /_plantuml/*` rewrite — no DNS record of its own. Same-origin from gitea so the embedded markdown previewer can call it without CORS.
### Portainer
Native OAuth2 (Settings → Authentication → OAuth → Custom). Manual endpoints (no OIDC discovery). Uses `portainer-uid` custom-claim scope from Authentik. Container is **not** compose-managed — safe to `docker run` recreate; data lives in named volume `portainer_data`. CLI flag: `--trusted-origins docker.hubris.network` (hostname only — `IsTrustedOrigin` rejects strings containing `://`).
### homelab-mcp (`/opt/homelab-mcp/`) — DEPRECATED (Go rewrite, Phase 6)
> **Status:** This Python MCP server is being replaced by the Go `oikos api` binary
> running in Docker on mac-mini. Cutover pending — see
> [scripts/cutover-checklist.md](../../scripts/cutover-checklist.md) for the
> execution plan. The Go MCP uses the official MCP Go SDK (Streamable HTTP, not
> FastMCP) with 15 tools including `get_blast_radius`, `request_execution`, and
> `get_agent_activity`. Source: `internal/mcp/server.go`.
**Current (Python) implementation — DO NOT MODIFY, awaiting cutover:**
FastMCP server (Python venv at `/opt/homelab-mcp/.venv`). Reads from
`/opt/homelab-context/` (this LXC is itself an enrolled
[homelab-context](../infrastructure/homelab-context.md) client). Source
is `dtoro/Homelab-Docs/mcp/server.py`; service unit
`homelab-mcp.service` binds `0.0.0.0:9810`. DNS-rebinding protection
disabled at the FastMCP layer because mesh+LAN gating is the actual
trust boundary.
- Endpoint: `https://mcp.hubris.network/mcp` (Caddy → `:9810`). StreamableHTTP transport (POST `/mcp`).
- 14 tools registered: `get_host`, `list_services`, `find_service`,
`get_topology`, `search_docs`, `get_page`, `get_changelog`, `whoami`,
`list_my_secrets` (context); `get_service_status`, `tail_log`,
`list_lxcs`, `get_lxc_state`, `ping_service` (read-only management).
- Management tools shell out via SSH using a dedicated `mcp-reader` key
(planned; not provisioned yet — management tools fail until then).
#### Auto-deploy pipeline (homelab-mcp)
Receiver at `/opt/homelab-mcp/mcp/deploy/webhook/` (in-tree alongside
the server code). Listens on `0.0.0.0:9811`, secret in
`/etc/homelab-mcp-deploy/secret`, systemd unit
`homelab-mcp-deploy.service`. Gitea webhook id 10 on
`dtoro/Homelab-Docs`. Deploy script reinstalls the service unit and
restarts on push.
### secrets-issuance (`/opt/secrets-issuance/`) — DEPRECATED (Go Phase 5)
> **Status:** Replaced by `internal/secrets/` in the Go rewrite. Machine identities
> are now managed via Infisical (`docker compose --profile infisical up`).
Tiny HTTP service that issues per-client age keypairs the first time
each client calls `/issue`. Idempotent: subsequent calls return the
same key. Mesh+LAN source-IP gated via the `MESH_SUBNETS` env in
`secrets-issuance.service`. State (per-client keys + denylist) lives at
`/var/lib/secrets-issuance/` (NOT in the repo).
- Endpoint: `https://secrets.hubris.network/issue` (Caddy → `:9820`).
- `/revoke` is admin-token-gated by `/etc/secrets-issuance/admin-token`;
shreds the local key file and adds the hostname to the denylist.
Called by `homelab client remove`.
- Trust subnets today: `100.122.0.0/16` (Netbird), `100.64.0.0/10`
(Tailscale), `192.168.8.0/24` (LAN). Tighten if the LAN gets
untrusted devices.
#### Auto-deploy pipeline (secrets-issuance)
Receiver at `/opt/secrets-issuance/secrets-issuance/deploy/webhook/`,
port `9821`, secret in `/etc/secrets-issuance-deploy/secret`, unit
`secrets-issuance-deploy.service`. Gitea webhook id 11.
### WriteFreely (`/opt/writefreely-docker/`)
Native OIDC via `[oauth.generic]` in `config/config.ini`. `host = https://auth.hubris.network`, scope `openid profile email`, `map_user_id = sub`. Redirect URI in Authentik: `https://blog.hubris.network/oauth/callback/generic`. Container has `extra_hosts: auth.hubris.network:192.168.8.175`. First-time users with a local WF account must connect SSO via Customize/Settings → Linked Accounts.
## Permissions
- LXC is privileged → in-container UID == host UID.
- Member of [media GID 10000](../infrastructure/media-permissions.md). Adding `media` to the LXC alone is *not* enough for inner Docker containers — each container needs `--group-add 10000` or `user: "<uid>:10000"` to write into `/mnt/library`.
## Related
- [Gitea (104)](104-gitea.md) — uses the PlantUML server
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](106-auth-outpost.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (Artifacto + blog)](../infrastructure/ingress.md)
## Changelog
### 2026-06-29 — Booklore migrated to Grimmory on LXC 130
Booklore stack removed from Portainer. MariaDB dump taken first, then restored into [grimmory (130)](130-grimmory.md)'s fresh MariaDB. `books.hubris.network` Caddy backend updated to `192.168.8.213:6060`. Authentik OIDC provider updated to Public client type (PKCE) for Grimmory compatibility.
### 2026-05-20 — homelab-mcp + secrets-issuance live
Two new services from the [homelab-context distribution plan](../infrastructure/homelab-context.md):
`homelab-mcp.service` on `:9810` (MCP read+management surface) and
`secrets-issuance.service` on `:9820` (per-client age-key provisioning).
Caddy fronts both with Let's Encrypt; new vhosts on
[caddy](121-caddy.md), split-horizon DNS entries on
[authentik (124)](106-auth-outpost.md). Gitea webhook ids 10 + 11 wire
auto-deploy. LXC is itself an enrolled context client
(`/opt/homelab-context/`).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — Artifacto deployed + Authentik gateway-secret auto-login wired
Repo `dtoro/Artifacto`. Compose, data dir, `.env`, webhook (id 7), Caddyfile site block, gateway-secret pattern, public-internet share exposure on the VPS — all set up that day. Slug length bumped 5 → 8 for brute-force resistance.
### 2026-04-22 — Booklore mariadb data wiped (lesson, not a feature)
`docker compose up -d` from the host shell on a Portainer-managed stack recreated services and re-initialized the bind-mounted `./mariadb/config`. Recovered by reseeding from snapshot. Rule recorded: edit Portainer-managed stacks via the Portainer UI.
### 2026-04-22 — Portainer wired into Authentik (native OAuth2)
Custom-claim `portainer_uid` scope. CLI added `--trusted-origins docker.hubris.network`.
### 2026-04-22 — `app.ini` host whitelist on gitea extended
Added `192.168.8.205` so the Artifacto webhook delivery wouldn't be rejected. See [gitea (104)](104-gitea.md).
### 2026-04-21 — Booklore + WriteFreely wired into Authentik OIDC
Native OIDC for both. `extra_hosts` override added inside each container so DNS resolves `auth.hubris.network` to the Caddy IP.
### 2026-04-20 — PlantUML server deployed
Used by [gitea (104)](104-gitea.md)'s footer.tmpl injection. Same-origin via caddy `handle_path /_plantuml/*` rewrite — no separate hostname.

56
archive/knowledge/containers/106-auth-outpost.md Normal file
View File

@@ -0,0 +1,56 @@
# 106 — `auth-outpost`
Authentik **forward-auth outpost** for LAN-gated apps. A stateless proxy that connects outbound to the [VPS Authentik core](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md) and serves forward-auth locally, so [Caddy (121)](121-caddy.md) never hairpins auth through VPS Traefik.
## At a glance
- **Hostname:** `auth-outpost`
- **IP:** `192.168.8.6` (static)
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 512 MiB / 4 GiB rootfs
- **Mounts:** none
- **Created:** 2026-06-01, Debian 13, replacing the embedded outpost on [124](106-auth-outpost.md)
## Role
Runs one container — `ghcr.io/goauthentik/proxy` — that opens an outbound websocket to `https://auth.hubris.network` (the VPS core), pulls its proxy-provider config, and answers Caddy's `forward_auth` subrequests on `192.168.8.6:9000` (LAN-only bind). Because the call path is **Caddy → outpost (LAN)**, with no Traefik in between, `X-Forwarded-Host` is preserved — the failure that 404s when Caddy is pointed at `https://auth.hubris.network` directly (Traefik rewrites the header). See the [migration investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
## Service / port map
| Service | Listen | Notes |
| --- | --- | --- |
| authentik proxy outpost | `192.168.8.6:9000` | `/outpost.goauthentik.io/*` (ping, auth/caddy) |
## Config paths
- `/opt/authentik-outpost/docker-compose.yml` — `ghcr.io/goauthentik/proxy:2026.5.2`, `AUTHENTIK_HOST=https://auth.hubris.network`, `AUTHENTIK_INSECURE=false`, port bound `192.168.8.6:9000`.
- `/opt/authentik-outpost/.env` (mode 600, **untracked**) — `AUTHENTIK_TOKEN` for outpost `hubris-lan-outpost` (sops-encrypt into `secrets/` — TODO Phase 5).
## The outpost object (VPS Authentik)
- Outpost **`hubris-lan-outpost`** (type proxy), providers: `hubris-forward-auth` (domain, Paperless + domain-level apps), `Provider for Torrent` (qBittorrent), `Provider for Artifacto` (single).
- Health: VPS admin → Applications → Outposts → last-seen current; `curl http://192.168.8.6:9000/outpost.goauthentik.io/ping` → `204`; container log `Successfully connected websocket`.
## Caddy wiring
[Caddy (121)](121-caddy.md) `(authentik)` snippet `reverse_proxy`/`forward_auth` → `http://192.168.8.6:9000` (was `192.168.8.180:9000` on LXC 124). Tracked in `dtoro/caddy-conf`.
## `sso.hubris.network` — the callback domain (critical)
Domain-level (`forward_domain`) providers redirect the browser to `{external_host}/outpost.goauthentik.io/callback` after login. On LXC 124 that `external_host` was `auth.hubris.network`, which resolved (LAN) to Caddy → the 124 outpost — same box, so it worked. Post-migration `auth.hubris.network` points to the **VPS core**, so the callback hit the wrong outpost → **400 / redirect-uri mismatch**.
Fix: the LAN outpost gets its **own** domain.
- DNS: `sso.hubris.network → 192.168.8.175` (Caddy). *(in dnsmasq today — must be carried into Technitium in DNS Phase 2.)*
- Caddy: site `sso.hubris.network { tls dns ionos; reverse_proxy 192.168.8.6:9000 }` (in `caddy-conf`).
- Authentik: `hubris-forward-auth` and `Provider for Torrent` have `external_host=https://sso.hubris.network`; their `redirect_uris` must match (`set_oauth_defaults()` regenerates them from `external_host` — changing `external_host` alone does NOT update them).
- `Provider for Artifacto` is `forward_single` on its own domain, so it was unaffected.
**Lesson:** when the IdP core and the forward-auth outpost live on different hosts, the outpost needs a dedicated domain distinct from the core's — and proxy-provider `redirect_uris` must be regenerated, not just `external_host`.
## Related
- [124 — authentik](106-auth-outpost.md) — old embedded-outpost host (now DNS-only)
- [Caddy (121)](121-caddy.md) — forward-auth consumer
- [Ingress (VPS traefik)](../infrastructure/ingress.md)
- [Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)
## Changelog
### 2026-06-06 — Authentik session lifetime extended to 30 days
VPS Authentik core `user_login` stage updated: `session_duration` changed from `seconds=0` (session cookie, cleared on browser close) to `days=30` (persistent 30-day cookie). Also set `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` in `/opt/authentik.env` on the VPS. See [investigation](../../sources/investigations/2026-06-06-authentik-session-lifetime.md).
### 2026-06-01 — created; forward-auth cut over from LXC 124
New dedicated LXC for the LAN forward-auth outpost (Phase 1 of the [architecture migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Deployed `goauthentik/proxy:2026.5.2` pointed at the VPS core; repointed Caddy `(authentik)` from `192.168.8.180:9000` → `192.168.8.6:9000`. Verified Paperless/qBittorrent/Artifacto return the SSO redirect with **124-Authentik stopped**, confirming the frozen instance is out of the path. dnsmasq stays on 124 until [DNS is relocated](106-auth-outpost.md).

67
archive/knowledge/containers/107-dns.md Normal file
View File

@@ -0,0 +1,67 @@
# 107 — `dns`
Homelab DNS server (Technitium). Replaces the dnsmasq that lived on [124 — authentik](106-auth-outpost.md); single-purpose, one job.
## At a glance
- **Hostname:** `dns`
- **IP:** `192.168.8.2` (static — stable, decoupled from any app)
- **Privilege:** privileged (Docker-in-LXC, `features: nesting=1`)
- **Resources:** 1 core / 1 GiB / 8 GiB rootfs
- **Created:** 2026-06-01, Debian 13. Its own resolver is `1.1.1.1` (no circular dependency on the DNS it serves).
## Role
Authoritative split-horizon DNS for `hubris.network` on the LAN/mesh, plus recursive forwarding (`1.1.1.1`, `8.8.8.8`) for everything else. Technitium runs in Docker (`technitium/dns-server:latest`, `network_mode: host`), web console on `:5380`.
## The `hubris.network` zone
- Specific A overrides: app names → `192.168.8.175` (Caddy), `nfs-export → 192.168.8.200`, `auth/sso/...` as needed.
- `auth.hubris.network → 82.165.190.79` (VPS Authentik), `sso.hubris.network → 192.168.8.175` (LAN forward-auth outpost).
- **Wildcard** `*.hubris.network → 82.165.190.79` — mirrors the public IONOS wildcard so undefined names (e.g. `netbird`) resolve to the VPS, matching public behaviour.
- **MX / SPF-TXT / CAA replicated from public** so an authoritative zone doesn't shadow `hubris.network` email/cert records.
## Config / access
- `/opt/technitium/docker-compose.yml`; admin password in `/opt/technitium/admin_password.txt` (mode 600 — sops-encrypt in Phase 5).
- Console: `http://192.168.8.2:5380` (user `admin`).
- API: `http://192.168.8.2:5380/api/...` (token via `/api/user/login`). Zone was built via the API.
## Who points here
- **NetBird mesh peers:** resolve via the **NetBird managed DNS zone**, kept in sync *from* this Technitium (see dns-sync below). The `home-lab-dns` nameserver group (`→ 192.168.8.2`) is a thin fallback forwarder.
- **Homelab DHCP clients:** Technitium's own DHCP scope hands out `192.168.8.2` as the DNS server for `192.168.8.x` leases (see DHCP section below).
- **Plain LAN clients (`192.168.178.x`):** Fritz!Box DHCP still hands out Fritz!Box itself (`192.168.178.1`) as DNS — no split-horizon for non-mesh clients. Changing this requires a secondary DNS fallback, which Fritz!OS 8.x doesn't expose in a single DHCP field.
## dns-sync (Technitium = authoring source)
`/opt/dns-sync/sync.py` (cron `*/10`, logs `/var/log/dns-sync.log`) reconciles this zone's named A-records → the NetBird managed DNS zone via the NetBird API (`/api/dns/zones/{id}/records`). Token at `/opt/dns-sync/netbird-token` (mode 600; source of truth in sops `secrets/netbird-pat.yaml`). **Edit DNS only here**; the sync propagates to the mesh. It deletes NetBird records absent from Technitium. Tracked: [scripts/dns-sync.py](../../../scripts/dns-sync.py). *Why this exists:* NetBird won't forward to Technitium for mesh peers (self-IP / nameserver-group quirks), so we sync into the managed zone instead — see [dns.md](../infrastructure/dns.md).
## DHCP
Technitium also runs a DHCP server for the homelab subnet (enabled 2026-06-02):
- **Scope:** `homelab` — `192.168.8.241 – 192.168.8.254`
- **Gateway:** `192.168.8.1` (Proxmox `vmbr0` alias)
- **DNS:** `192.168.8.2` (self)
- **Lease time:** 24 h
Replaces the DHCP that was previously served by the Slate AX router. Static-IP LXCs (`.101–.239`) are excluded from the pool. Pool narrowed from `.100–.240` to `.241–.254` on 2026-06-03 to eliminate IP conflict risk.
## Related
- [124 — authentik](106-auth-outpost.md) — retired host of the old dnsmasq
- [DNS split-horizon](../infrastructure/dns.md)
- [Mesh](../infrastructure/mesh.md)
## Changelog
### 2026-06-24 — A record `trmnl.hubris.network → 192.168.8.175`
Added for [trmnl (128)](128-trmnl.md) (LAN path via [Caddy (121)](121-caddy.md)); propagated to the NetBird managed zone by `dns-sync`.
### 2026-06-06 — dns-sync cron installed (had been missing since deployment)
Although the 2026-06-03 changelog claimed "cron */10", **no crontab was actually configured** on the LXC. The sync was running only via ad-hoc manual invocations during incident debugging. Fixed by adding `/etc/cron.d/dns-sync`.
### 2026-06-03 — DHCP pool narrowed to `.241–.254`
Previous pool `.100–.240` overlapped with all static LXCs/VMs (`.101–.239`). Shrunk via API (`/api/dhcp/scopes/set`). 11 stale DHCP leases in `.101–.110` remain until natural expiry (2026-06-04). See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-06-03 — dns-sync added (Technitium → NetBird managed zone)
This Technitium became the single DNS authoring source; `/opt/dns-sync/sync.py` (cron */10) reconciles named A-records into the NetBird managed zone via the API. Fixed previously-broken mesh names (`sso`, `nfs-export`, `mcp`, `secrets`) by adding them to the managed zone; reaped obsolete `files`/`photos-new`. See [dns.md](../infrastructure/dns.md).
### 2026-06-02 — DHCP server enabled; replaces Slate AX DHCP
Enabled Technitium's built-in DHCP server for `192.168.8.0/24` (scope `homelab`, range `.100–.240`, gateway `192.168.8.1`, DNS self). Previously the Slate AX sub-router served DHCP for the homelab subnet. With the Slate AX retired and Proxmox now the subnet router, Technitium takes over DHCP. Configured via the Technitium API (`/api/dhcp/scopes/set`). DHCP LXCs kept their Slate AX leases until expiry, then renewed from Technitium.
### 2026-06-01 — created; replaced dnsmasq on 124
Stood up Technitium at `192.168.8.2`, imported the split-horizon zone (specific A + wildcard + MX/SPF/CAA), made it the primary nameserver in the NetBird `home-lab-dns` group. Verified all names resolve with dnsmasq/124 stopped; [LXC 124 retired](106-auth-outpost.md).

111
archive/knowledge/containers/114-nextcloud.md Normal file
View File

@@ -0,0 +1,111 @@
# 114 — `nextcloud`
Personal cloud / file collaboration. Source-of-truth for the photo libraries surfaced by [mulita (120)](120-mule-images.md).
## At a glance
- **Hostname:** `nextcloud`
- **IP:** `192.168.8.224`
- **Privilege:** privileged
- **Resources:** 4 cores / 6 GiB RAM / 25 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library` (Nextcloud user files under `/mnt/library/homecloud`)
- **Public hostname:** [`cloud.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md)
## Auth
Native OIDC via `user_oidc` app. **Username override pattern**: Authentik user `dtoro` maps to local Nextcloud user `admin` via the `nc_uid` custom-claim scope. Configured via `occ user_oidc:provider <name> --mapping-uid=nc_uid` and `--scope="openid profile email <app>-uid"`. See [Authentik](106-auth-outpost.md#per-app-username-override-pattern-authentik) for the full pattern.
Redirect URI: `/index.php/apps/user_oidc/code` (NOT `/apps/...` — pretty URLs aren't on).
## DNS workaround (load-bearing)
Nextcloud's PHP HTTP client (`OC\Http\Client\DnsPinMiddleware`) calls `dns_get_record()` directly — it bypasses `/etc/hosts`. So the Authentik OIDC discovery URL needs **real DNS** answering with `192.168.8.175` for `auth.hubris.network`. Recipe applied:
- Local `dnsmasq` listening on `127.0.0.1:53`, `/etc/dnsmasq.d/hubris-internal.conf` with `address=/auth.hubris.network/192.168.8.175`, `server=192.168.8.1`, `server=1.1.1.1`, `interface=lo`, `bind-interfaces`, `no-hosts`, `no-resolv`.
- `pct set 114 --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"` so the LXC starts with the right resolver order.
- `/etc/hosts` override kept as belt-and-suspenders.
Also needs `allow_local_remote_servers=true` in `config.php`.
> Once internal DNS at the router level handles `*.hubris.network`, this LXC-local dnsmasq can go away. See [DNS](../infrastructure/dns.md).
## Storage
Files at `/mnt/library/homecloud`. Owned by Nextcloud's own permission model — **deliberately not on the `media` group**, NC manages it itself and would complain. See [media permissions](../infrastructure/media-permissions.md).
## Web stack (Apache event MPM + php-fpm)
Apache 2.4 with **`mpm_event`** as the MPM and PHP served via **php-fpm 8.4** over the Unix socket `/run/php/php8.4-fpm.sock` (mod_php disabled). Routing to FPM is via the distro's `/etc/apache2/conf-available/php8.4-fpm.conf` (gated by `<IfModule !mod_php8.c>`, so just `a2dismod php8.4` flips Apache onto FPM).
Why this matters: with mod_php every Apache worker is forced onto `mpm_prefork` (one heavy process per connection, full PHP interpreter loaded in each). On FPM + event, Apache workers are ~5–8 MB and async; PHP work is done by a separate FPM pool that keeps opcache hot. Concurrent upload throughput jumps significantly because Apache no longer blocks per-connection on a heavy PHP process.
### PHP-FPM pool
- Socket: `/run/php/php8.4-fpm.sock` (owner/group `www-data`)
- `pm = dynamic`, `pm.max_children = 30`, `pm.start_servers = 6`, `pm.min/max_spare_servers = 4/12`, `pm.max_requests = 500`
- Steady-state ~6 idle workers at ~80 MB each — bump `max_children` only if peak concurrency exceeds 30.
### PHP limits (FPM + CLI both)
Set in `/etc/php/8.4/fpm/php.ini` **and** `/etc/php/8.4/cli/php.ini` (the latter so `occ`/cron see the same ceilings):
| key | value |
|---|---|
| `upload_max_filesize` | `16G` |
| `post_max_size` | `16G` |
| `memory_limit` | `512M` |
| `max_execution_time` | `3600` |
| `max_input_time` | `3600` |
| `output_buffering` | `Off` |
After edits: `systemctl reload php8.4-fpm`. Caddy in front (`cloud.hubris.network`) imposes no body-size limit, so these are the effective ceiling.
### Opcache + JIT
`/etc/php/8.4/mods-available/opcache.ini` — full config maintained, not commented stubs:
- `opcache.enable=1`, `opcache.memory_consumption=256`, `opcache.interned_strings_buffer=32`, `opcache.max_accelerated_files=20000`
- `opcache.jit=tracing`, `opcache.jit_buffer_size=64M`
### Apache mod_reqtimeout
`/etc/apache2/mods-available/reqtimeout.conf` — relaxed body trickle so slow-WAN uploads don't get killed:
- `RequestReadTimeout header=20-40,MinRate=500`
- `RequestReadTimeout body=20,MinRate=100` (was `body=10,MinRate=500`)
## MariaDB tuning
Overrides in `/etc/mysql/mariadb.conf.d/99-nextcloud-tuning.cnf`:
- `innodb_buffer_pool_size = 1G` (default was 128M)
- `innodb_log_file_size = 256M`
- `innodb_flush_log_at_trx_commit = 2` (group-commit; trades a few ms of durability for write throughput — acceptable for NC)
- `innodb_flush_method = O_DIRECT`, `innodb_io_capacity = 2000` / `..._max = 4000`
- `max_allowed_packet = 256M`, `tmp_table_size = 64M`, `max_heap_table_size = 64M`
Apply with `systemctl restart mariadb` (not reload — `innodb_log_file_size` needs a clean restart).
## Related
- [mulita (120)](120-mule-images.md) — reads NC user trees + writes back via WebDAV
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Mesh migration (DNS overrides explained)](../infrastructure/mesh.md)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-05-13 — Apache keepalive bumped (and TCP BBR on host)
Follow-up to the FPM/MPM/MariaDB round earlier the same day. After a WiFi-bound Linux client kept seeing ~2 MB/s downloads despite a 152 Mbps speed test (so the WiFi medium wasn't the cap), bumped Apache `KeepAliveTimeout 5 → 60` and `MaxKeepAliveRequests 100 → 1000` so the mirall desktop client reuses one TLS+TCP connection across many sequential file fetches instead of paying handshake + slow-start each time. Paired with TCP BBR + `fq` qdisc enabled on the [hubris host](../hosts/hubris.md#2026-05-13--tcp-bbr--fq-qdisc-enabled), which directly addresses single-stream throughput on jittery links. Server-side loopback baseline post-BBR is ~404 MB/s for a single 23 MB CR2 download via Caddy. Client-side diagnosis pending (curl single-stream + parallel tests) to determine if the remaining gap is mirall config (bandwidth limit / `parallelNetworkRequests`) or path shaping.
### 2026-05-13 — throughput tuning: FPM + event MPM + opcache JIT + MariaDB
Investigated upload/download speed. Bottlenecks were inside this LXC — LAN (33.9 Gbit/s caddy→NC iperf3) and `/mnt/library` NVMe (714 MB/s sustained) were not the limit. Changes:
- **Apache mod_php → php-fpm 8.4** (`a2dismod php8.4`). Flips PHP from in-process to FPM-over-Unix-socket. Apache workers now ~5–8 MB instead of ~80 MB each.
- **MPM prefork → event** (`a2dismod mpm_prefork && a2enmod mpm_event`). Async I/O, far better concurrency for parallel uploads.
- **opcache JIT enabled** (`tracing`, 64M buffer), `memory_consumption=256`, `interned_strings_buffer=32`, `max_accelerated_files=20000`.
- **MariaDB**: `innodb_buffer_pool_size 128M → 1G`, `innodb_log_file_size 96M → 256M`, `flush_log_at_trx_commit 1 → 2`, `O_DIRECT`, higher I/O capacity. New file `99-nextcloud-tuning.cnf`.
- **Apache mod_reqtimeout**: body trickle relaxed from `body=10,MinRate=500` → `body=20,MinRate=100` so slow-WAN uploads aren't killed.
- **PHP-FPM ini** synced to match what the apache2/cli inis already had (FPM was stuck at 2M/8M because FPM wasn't serving traffic before, but now it does).
Backups: `/etc/php/8.4/fpm/php.ini.bak.20260513`, `/etc/apache2/mods-available/reqtimeout.conf.bak.20260513`, `/etc/php/8.4/mods-available/opcache.ini.bak.20260513`. MariaDB tuning is a new file (rollback = `rm`).
Verified: `apache2ctl configtest` Syntax OK, `occ status` clean, `occ setupchecks` shows ✓ on memory_limit / output_buffering / DB indices, `status.php` returns 200 in <40 ms via Caddy with HTTP/2.
### 2026-05-10 — PHP upload limits raised
Apache + CLI php.ini bumped from distro defaults (`upload_max_filesize=2M`, `post_max_size=8M`, `memory_limit=128M`, `max_execution_time=30`) to NC-recommended (`16G/16G/512M/3600`). Backups at `/etc/php/8.4/{apache2,cli}/php.ini.bak-20260510-*`. Was causing slow/failing uploads via web UI.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — Nextcloud-rooted libraries shipped (mulita)
[mulita (120)](120-mule-images.md) now reads photos from `/nextcloud-users/<nc_user>/files/...` and writes mutations back via WebDAV. Pre-migration DB dump kept at `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump`.
### 2026-04-21 — wired into Authentik (native OIDC + nc_uid override)
Local `dnsmasq` added on this LXC because Guzzle bypasses `/etc/hosts`. Resolver order set to `127.0.0.1 192.168.8.1 1.1.1.1`. Username override `dtoro → admin`.

83
archive/knowledge/containers/118-elementsynapse.md Normal file
View File

@@ -0,0 +1,83 @@
# 118 — `elementsynapse`
Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
## At a glance
- **Hostname:** `elementsynapse`
- **IP:** `192.168.8.242`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
- **Mounts:** none from `/mnt/library`
- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008`
- **`server_name`:** `avispero` (intentionally different from the public URL).
- **Database:** Synapse runs on **SQLite** (`/var/lib/matrix-synapse/homeserver.db`, ~90 MiB). PostgreSQL on `127.0.0.1:5432` is dedicated to the mautrix bridge databases (one per bridge), not Synapse itself.
## Service / port map
| Service | Listen | Notes |
| -------------- | ------- | -------------------------------- |
| synapse | `:8008` | Matrix client-server / federation HTTP. Caddy terminates TLS for `matrix.hubris.network`. |
| synapse-admin | `:5173` | Web UI (vite dev server) for Synapse admin API. |
| mautrix-whatsapp | `:29318` | Docker, `network_mode: host`, compose in `/root/mautrix-whatsapp/`. |
| mautrix-meta | `127.0.0.1:29319` | Facebook/Instagram bridge, `/root/mautrix-meta/`. |
| mautrix-signal | `127.0.0.1:29328` | `/root/mautrix-signal/`. |
| mautrix-slack | `127.0.0.1:29335` | `/root/mautrix-slack/`. |
| mautrix-linkedin | `127.0.0.1:29341` | `/root/mautrix-linkedin/`. |
## Bridges (mautrix)
All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name>/` — no systemd unit, `restart: unless-stopped`. Their appservice registration YAMLs live in `/etc/matrix-synapse/` and are listed in `homeserver.yaml` under `app_service_config_files`. If a bridge daemon is down, Synapse logs `push_bulk … ConnectionRefusedError` and retries every 512 s; restart the bridge with `cd /root/mautrix-<name> && docker compose up -d`.
## Operational notes
- Rootfs is on `pve` LVM-thin, grow with `pct resize 118 rootfs +<N>G` from the host — resize2fs runs automatically.
- `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
## Known issues
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
## Related
- ~~[claudio-bot (123)](archive/123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
- [Caddy](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-06-06 — DHCP drift fixed: internal `/etc/network/interfaces` was `dhcp` despite Proxmox static config
**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
**Fix:**
- Replaced `iface eth0 inet dhcp` with `iface eth0 inet static` + `address 192.168.8.239/24` + `gateway 192.168.8.1`
- `ifdown eth0 && ifup eth0` applied the static IP
- Killed lingering dhclient process
- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
### 2026-05-15 — phantom-notification cleanup for `@admin`
After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.
2. **The noisiest room** (a Signal-bridged DM) was flooded with `m.room.redaction` events from `@signalbot:avispero` (disappearing messages being deleted). Synapse counts redactions as notifications by default, but Element X doesn't render them — so they were invisible "ghosts".
Fix (after stopping Synapse and backing up `homeserver.db`):
- `DELETE FROM event_push_actions WHERE stream_ordering <= max(receipt_stream)` per (user, room) — clears receipt-superseded rows.
- `DELETE FROM event_push_actions WHERE event_id IN (SELECT event_id FROM events WHERE type='m.room.redaction')` — removes redaction-driven notifications.
- Rebuild `event_push_summary` from the remaining rows (preserve `thread_id='main'` to satisfy the NOT-NULL check constraint).
Result: 125 → 4 (legitimate WhatsApp unreads).
Then, to stop this recurring, an **override push-rule** was installed for `@admin` via the client API:
```
PUT /_matrix/client/v3/pushrules/global/override/suppress_redactions
{ "actions": [], "conditions": [{ "kind": "event_match", "key": "type", "pattern": "m.room.redaction" }] }
```
With empty `actions`, all `m.room.redaction` events are silently ignored by the push system. Inspect/remove via the same endpoint. (Note: rule ids beginning with `.` are reserved for server defaults — custom ids cannot start with a dot.)
### 2026-05-15 — disk-full incident, rootfs grown to 16 GiB, bridges restarted
Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed `OSError: [Errno 28] No space left on device` during flush. Fix: `pct resize 118 rootfs +8G` (host `pve` VG had plenty of headroom), `journalctl --vacuum-size=50M` to recover ~200 M, `systemctl restart matrix-synapse`, then `docker compose up -d` in each `/root/mautrix-*` directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).
### 2026-04-28 — wiki entry created
Initial documentation.

23
archive/knowledge/containers/119-sophia.md Normal file
View File

@@ -0,0 +1,23 @@
# 119 — `sophia`
Single-container workload "sophia". Reads/writes under `/mnt/library/sophia`.
## At a glance
- **Hostname:** `sophia`
- **IP:** `192.168.8.157`
- **Privilege:** privileged
- **Resources:** 2 cores / 1 GiB RAM / 10 GiB rootfs
- **Mounts:** `/mnt/library` ↔ `/mnt/library`
- **Public hostname:** none
## Permissions
LXC has `www-data` in the [media GID 10000 group](../infrastructure/media-permissions.md). The `/mnt/library/sophia` subtree itself is single-container and not on the setgid shared group.
## Related
- [Hubris host](../hosts/hubris.md)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-04-28 — wiki entry created
Stub. Document what runs inside next time we work on this LXC.

330
archive/knowledge/containers/120-mule-images.md Normal file
View File

@@ -0,0 +1,330 @@
# 120 — `mule-images`
Hosts `mule-image` — the photos app at `photos.hubris.network`. PhotoPrism + Go sidecar + SvelteKit, replacing the legacy FastAPI/Celery stack as of 2026-05-22 (see Changelog). Auto-deploys from `dtoro/mule-image` on `git push origin main`.
## At a glance
- **Hostname:** `mule-images`
- **IP:** `192.168.8.136`
- **Privilege:** privileged
- **Resources:** 6 cores / 12 GiB RAM / 60 GiB rootfs / 2 GiB swap
- **Mounts:** `/mnt/library` ↔ `/mnt/library`; `/dev/dri/{card0,renderD128}` passed through for VA-API video accel on the AMD Phoenix1 iGPU.
- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → path-routed to PhotoPrism `:2342` / sidecar `:8000` / nginx-static `:3000`.
## Stack (`/opt/mule-image`)
`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose stack: base `docker-compose.yml` + LAN-port-rebind `docker-compose.override.yml` (untracked) + VA-API `docker-compose.gpu.yml`. The SvelteKit frontend is built on the host and served as a static bundle by nginx — no `vite dev` in prod.
| Service / process | Port (LAN) | Notes |
| ------------------------ | --------------------- | ----- |
| pp-mariadb | (internal 3306) | MariaDB 11; holds PhotoPrism's `photoprism` DB + sidecar's `mule_sidecar.*` |
| pp-app | `0.0.0.0:2342` | PhotoPrism `:latest`; `PHOTOPRISM_FFMPEG_ENCODER=h264_vaapi` |
| pp-sidecar | `0.0.0.0:8000` | Go service (file rename / folder ops / heap convert / dup scan / per-photo marks); also reconciles `USER_BASEPATHS` into PhotoPrism's `auth_users` every 60 s |
| nginx (host process) | `0.0.0.0:3000` | Serves the SvelteKit static bundle from `web/build/` |
**Per-user scoping** is driven by `USER_BASEPATHS=admin:admin/files, muli:muli/files` in `.env`. Sidecar applies it to PhotoPrism's `auth_users` table on boot + every 60s, `mkdir -p`s each target so PhotoPrism's ACL filter has somewhere to point.
`.env` is **untracked** — `git reset --hard` won't touch it, but `git clean -fdx` would. Holds PP_*, MariaDB passwords, SIDECAR_DB_PASSWORD, OIDC_* (existing `mule-image` Authentik app, redirect URI now `/api/v1/oidc/redirect`), `USER_BASEPATHS`, and `PP_FFMPEG_ENCODER=vaapi`.
`docker-compose.override.yml` is also untracked — it `!override`s the upstream `127.0.0.1:port` mappings to `0.0.0.0:port` so cross-host Caddy on [LXC 121](121-caddy.md) can reach pp-app + sidecar.
## Library access
PhotoPrism reads the library **directly off the bind-mounted filesystem** — no Nextcloud webhook integration in the new stack. The base path is `/mnt/library/homecloud` and per-user scoping comes from `USER_BASEPATHS` (see Stack section above).
- `dtoro` is mapped to NC user `admin` → `/mnt/library/homecloud/admin/files/`
- `muli` is mapped to NC user `muli` → `/mnt/library/homecloud/muli/files/`
PhotoPrism's container user is `33:10000` (`www-data:media`), matching the host ownership of the NC tree. The legacy `oc_webhook_listeners` rows + `NEXTCLOUD_WEBHOOK_SECRET` are gone — when NC writes via WebDAV (still its own primary surface), PhotoPrism picks up the new file on its next index pass.
## Authentication
OIDC via Authentik. App slug `mule-image`, redirect URI `https://photos.hubris.network/api/v1/oidc/redirect` (PhotoPrism's auto-derived path; PhotoPrism builds it from `PHOTOPRISM_SITE_URL`). `OIDC_REGISTER=true` auto-creates a PhotoPrism user at role `user` on first SSO login. The sidecar's basepath reconciler then assigns their scoped folder.
The `mule-image` Authentik app's redirect URI was migrated from the legacy FastAPI `/api/v1/auth/oidc/callback` path on 2026-05-22 — same client ID/secret were reused. The separate `mule-photos-new` Authentik app was deleted in the same operation.
## Auto-deploy
Push to `dtoro/mule-image` `main` → gitea webhook → `http://192.168.8.136:9797/deploy` → `mule-deploy-webhook.service`:
- Validates HMAC against `/etc/mule-deploy/secret`
- Filters to `refs/heads/main`
- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
- `git fetch && git reset --hard origin/main`, force-recreates **only the sidecar** image (so PhotoPrism's `Session` HMAC key in `pp/storage/config/hub.yml` doesn't rotate and invalidate every in-flight OIDC state cookie), reconciles `pp-app` + `mariadb` in place, `cd web && npm ci && npm run build`, `systemctl reload nginx`.
- Conditionally layers `docker-compose.gpu.yml` when `/dev/dri/renderD128` exists, and always layers `docker-compose.override.yml` (the LAN-port rebind) when present.
Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`. Gitea webhook id 6. `app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`.
Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`.
For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials` (mode 600) — same token as `/etc/caddy-deploy/git-credentials` on [caddy](121-caddy.md).
## Related
- [Nextcloud (114)](114-nextcloud.md) — source of truth for photo libraries
- [Authentik (124)](106-auth-outpost.md)
- [Caddy (121)](121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Gitea (104)](104-gitea.md)
## Changelog
### 2026-05-22 — Cutover to PhotoPrism stack (Mulimage 2.0)
The `new` branch of `dtoro/mule-image` was merged into `main` as commit
`70dc1b6`. The merge replaces the legacy FastAPI + PostgreSQL + Celery
+ React stack with PhotoPrism + Go sidecar + MariaDB + SvelteKit, lifted
in from the M0 evaluation on (now-destroyed) [LXC 127](#).
Cutover on 120 (this session):
- Bumped to 6 cores / 2 GB swap. `/dev/dri/{card0,renderD128}` already
passed through; added an opt-in `docker-compose.gpu.yml` overlay that
the deploy script layers in when the device is present. PhotoPrism now
uses `h264_vaapi` instead of `libx264`.
- Installed host nginx + a small `photos.conf` vhost serving the SvelteKit
static bundle from `/opt/mule-image/web/build/` on `:3000` — no Vite
dev server in prod. SvelteKit's `adapter-static` produces a real SPA
bundle.
- Replaced `/opt/mule-deploy/deploy.sh` with the 127-style multi-compose
deploy (force-recreates only sidecar to preserve PhotoPrism's Session
HMAC key; reconciles pp-app + mariadb in place; rebuilds web/ and
reloads nginx).
- `/opt/mule-image/.env` was rewritten to the PhotoPrism shape. Reused
passwords from `/root/mule-photos-new-secrets.txt`. `USER_BASEPATHS=
admin:admin/files, muli:muli/files` exposes both libraries.
- `docker-compose.override.yml` (untracked) `!override`s pp-app +
sidecar ports to `0.0.0.0` so cross-LXC caddy on [121](121-caddy.md)
can reach them.
- Authentik: `mule-image` app's redirect URI updated to `/api/v1/oidc/redirect`;
the separate `mule-photos-new` app deleted.
- Caddyfile (`dtoro/caddy-conf`): `photos.hubris.network` switched from
plain passthrough to path-matchers (PhotoPrism API + sidecar API +
nginx static catch-all), and `photos-new.hubris.network` was removed
entirely.
- Cleanups: 4 Nextcloud `oc_webhook_listeners` rows for the legacy
`/api/v1/internal/nc-webhook` endpoint deleted; gitea webhook id 9
(the `refs/heads/new` deploy hook for 127) deleted; `192.168.8.181`
removed from gitea's `ALLOWED_HOST_LIST`; `photos-new.hubris.network`
removed from dnsmasq.
- LXC 127 destroyed via `pct destroy 127` after the new stack passed
curl verification end-to-end (PP `/api/v1/status` returns
`{"status":"operational"}` through caddy; sidecar `/api/sidecar/healthz`
returns `{"ok":true,...}`).
- Rollback safety net: legacy `mulita` postgres dump at
`/root/backups/mulita-legacy-final-20260521-212036.sql.gz` on hubris
(10 MB, 16,155 photo rows). The `mule-image_mule_db_data` docker
volume on 120 retains the on-disk postgres data for at least 24 h
before housekeeping removes it.
### 2026-05-14 — Drop AI/vision pipeline, plain Postgres, DB↔FS refresh
**AI removal (commits `a27267f` + `423a73a`).** The OpenCLIP-on-ONNX
binary classifier (photography vs other) and all its scaffolding gone:
- Backend: `app/services/vision/`, `app/tasks/vision.py`,
`app/services/feature_flags.py`, `app/routers/features.py` deleted.
Admin `/admin/feature-flags`, `/admin/ai/{backfill,rescan}` endpoints
removed. `Photo.needs_review` column + `ix_photos_needs_review`
index dropped (alembic `0019_drop_ai_remnants`).
- Frontend: AI Settings tab, `useFeaturesQuery`, "Needs Review" sidebar
entry + filter, `needs_review` URL param plumbing, FeatureFlag types
all gone.
- Infra: `worker-vision` compose service + `models_data` volume
deleted. `worker-light` no longer runs `python -m
app.services.vision.bootstrap_models` before celery. Backend
Dockerfile drops the dedicated torch RUN layer. `requirements.txt`
drops `torch`, `torchvision`, `open-clip-torch`, `onnxruntime`.
**Postgres image swap.** `pgvector/pgvector:pg16` → `postgres:16`. The
0019 migration drops the (now-unused) `vector` extension first; the
SQL was pre-applied via `psql` against the still-pgvector container
and `alembic_version` stamped to `0019` so the new backend's
`bootstrap.py` upgrade-head was a no-op. After the swap surfaced a
collation-version mismatch (Bookworm glibc 2.36 → Trixie glibc 2.41),
the fix was `REINDEX DATABASE mulita` + `ALTER DATABASE … REFRESH
COLLATION VERSION` on `mulita`, `postgres`, `template1`. The data
volume was preserved across the image change.
**One-shot DB ↔ filesystem refresh.** New script
`backend/scripts/full_refresh.py` (run as
`docker exec -w /app mulita-backend python -m scripts.full_refresh
[--dry-run]`). Phases: `cleanup_data_integrity` → inline scan of every
active SourceRoot → `prune_missing_photos(dry_run=False)` → new
`prune_orphan_thumbnails` helper that removes
`/data/thumbs/{user}/{photo}/` dirs for photo_ids that left the DB.
First run: 0 missing photos, 1 stale folder row, **7982 orphan
thumbnail dirs** reaped.
**Manual cleanup post-deploy.** `docker compose up -d --build
--force-recreate` doesn't reap services removed from the YAML, so
`mulita-worker-vision` had to be `docker rm -f`'d by hand. The
`mule-image_models_data` volume was likewise removed manually.
### 2026-05-11 — Stop duplicating Nextcloud's indexing (Phase 1 + 2)
Two big architectural shifts in one session, both aimed at killing work mule was doing that NC already does for the same source files.
**Phase 1 — thumbnail proxy (commits `576b0c2`, `28738ac`).** Photos table gains `nextcloud_fileid` (migration `0018`). `GET /api/v1/photos/{id}/thumb/{size}` proxies NC's `/index.php/core/preview` keyed by that fileid, authenticated with the owner's encrypted app password (small=240, medium=640, large=1280). Worker now only writes the `medium` size to `/data/thumbs` (the vision worker still loads it from disk); `small` + `large` no longer touch disk. Disk fallback stays wired for legacy rows and the rare 404 from NC (iPhone JPEGs mis-extensioned as `.DNG` — verified). Existing 17,591 photos backfilled via `backend/scripts/backfill_nextcloud_fileid.py`. Tested with real CR2, real DNG, fake DNG, JPEG, HEIC — all green.
**Range support for `/original` (commit `18dce33`).** Pre-existing bug surfaced by Phase 1 testing: `<video>` couldn't play `.mov` because `FileResponse` was returning 200 with the whole body and no `Accept-Ranges` header, so browsers reported "format not supported." Now parses `Range: bytes=START-END`, returns 206 with `Content-Range`, streams in 1 MB chunks.
**Fix: backfill_gps was the actual CPU drain, not the watcher (commit `d24c64e`).** `_scan_all_source_roots_async` auto-queued `backfill_gps` 30 s after every container boot, which then re-queued `extract_metadata` for every photo with `latitude IS NULL` — ~60k pointless tasks per deploy, pinning worker-light at 180+% CPU for ~30 min. Killed the auto-trigger; manual `POST /api/v1/library/backfill-gps` still works.
**Phase 2 — webhook receiver replaces watchfiles (commits `362fbc6`, `f657e2c`).** New endpoint `POST /api/v1/internal/nc-webhook` (auth via `Authorization: Bearer $NEXTCLOUD_WEBHOOK_SECRET`, `hmac.compare_digest`) handles `NodeCreated|NodeWritten|NodeDeleted|NodeRenamed`. Maps NC's `/admin/files/...` path to the bind-mounted `/nextcloud-users/admin/files/...` and dispatches the same `scan_folder` / `handle_file_deletion` machinery the watcher used. Registered against NC via `backend/scripts/register_nc_webhooks.py` (idempotent: deletes existing webhooks targeting the same URL first). `mulita-worker-watcher` container retired; celery `--beat` folded into `worker-light` so `discard_missing_photos_beat` still fires. End-to-end verified by uploading a test JPEG through WebDAV — both `NodeCreated` and `NodeWritten` fire, mule receives + dispatches, 200 OK.
**NC cron tightened to `*/1` so webhook delivery latency drops from up to 5 min to ~60 s** (`crontab -u www-data` on LXC 114). NC dispatches webhook calls through its background-job queue; the cron interval = the worst-case latency. `*/5` was the default; `*/1` is the upstream recommendation anyway.
**Post-Phase-2 delete-roundtrip patches (commit `9408825`).** End-to-end testing of the NC↔mule deletion paths surfaced two real gaps:
1. *Folder delete* — NC fires one `NodeDeletedEvent` for the folder, not one per child. The webhook handler bailed with "unsupported extension" and photos under the deleted folder kept `is_discarded=false` until the 30-min reconcile sweep. Fix: `handle_directory_deletion()` does a single `UPDATE photos SET is_discarded=true WHERE filepath LIKE 'dir/%'` when the deleted path has no supported image extension.
2. *Resurrect on rewrite* — PUT-overwrite of a previously-discarded file fired `NodeWrittenEvent` → `scan_folder`, but scan_folder's "photo exists, skip" branch left `is_discarded=true`. Fix: when the existing row is discarded, flip `is_discarded=false`, clear `discarded_at`, re-queue `extract_metadata`.
Verified end-to-end on `Photos/MuleTestFolder/{test1,test2,nested/test3}.jpg`: `DELETE Photos/MuleTestFolder/` discarded all 3 in one shot; `PUT test1.jpg` back resurrected only that one.
**Known remaining gap — trashbin restore.** Moving a file out of `/dav/trashbin/.../trash/foo.jpg.dXXXX` back to `/files/...` via WebDAV MOVE fires **no event mule subscribes to**. NC's trashbin app emits its own internal event class that isn't in the `OCP\Files\Events\Node\*` set we registered with `webhook_listeners`. Workaround: re-upload via PUT (covered by the resurrect-on-rewrite fix above) or wait for the 30-min reconcile sweep.
**Folder rename round-trip (commits `f4a03b6` + `f27f3cb`).** Both directions now work:
- *NC-side rename → mule:* webhook's "renamed" branch detects directory rename (neither path has a supported extension) and calls new `handle_directory_rename(old, new)` in `scan.py`. The helper iterates matching rows in Python and prefix-rewrites `Photo.filepath`, `Folder.path`, `SourceRoot.path` in one transaction. Cross-source-root case (rare) discards the old subtree and lets scan_folder add fresh rows under the new root.
- *Mule-side rename → NC:* the existing `PATCH /api/v1/folders/{id}` endpoint (`folders.py:202`) already does WebDAV MOVE via `nextcloud_dav.move_for_user` and rewrites mule's DB inline. The NodeRenamedEvent that bounces back through the webhook hits `handle_directory_rename`, which finds 0 rows under the old prefix and is a no-op — feedback loop is idempotent.
**asyncpg gotcha**: the original implementation of `handle_directory_rename` used raw SQL with `SUBSTRING(filepath FROM LENGTH(:old_prefix) + 1)` so the offset would be computed server-side. asyncpg's type inference miscategorises the `LENGTH()` result and rejects the parameter as "$2: int (expected str)". The fix iterates in Python (same pattern as the existing PATCH endpoint). Lesson: avoid passing `LENGTH(:x)+1` as an argument to `SUBSTRING(...)` via asyncpg + sqlalchemy `text()`.
**Phase 3 — Memories-backed `extract_metadata` (commit `2a5759c`).** Memories app re-enabled on NC. `extract_metadata` now tries `GET /index.php/apps/memories/api/image/info/{fileid}` (auth: Basic + `OCS-APIRequest: true` header to bypass CSRF) before falling back to ExifTool. Replaces ~80 ms of subprocess with ~1-2 ms HTTP for ongoing imports. New helpers: `nextcloud_dav.fetch_memories_info_async()` + `metadata._apply_memories_metadata()`. We kept mule's full date-fallback chain (SubSec → DateTimeOriginal → CreateDate → MediaCreateDate → TrackCreateDate → filename heuristic → mtime) because 35% of the library (`taken_at_source='path'`) depends on the filename heuristic, and Memories alone would silently mis-date those photos to mtime. PhotoInfoPanel reads `exif.Make/Model/ISO/FNumber` from `photos.exif_json` — Memories' `exif` blob uses those exact plain key names, so no frontend adapter was needed. ExifTool subprocess is still in place as the fallback for brand-new photos racing the NC scan, non-NC photos, and any NC HTTP failure.
**Phase 4 — retire `/data/thumbs` (commits `5a67ed7` + `7a1c6b6`).** Vision worker now fetches NC's 640px preview via a new sync helper `nextcloud_dav.get_preview_bytes()` instead of reading `/data/thumbs/{id}/medium.webp`. `thumbs.WORKER_THUMB_SIZES = set()` — `generate_thumbnails` still computes pHash on the original-res pixels (perceptual dedup is mule-only) but stops writing files. All ~22k `medium.webp` purged after verification; `/data/thumbs` shrank from 4.1 GB → 94 MB residual.
**SECRET_KEY bug found mid-deploy (commit `7a1c6b6`).** `docker-compose.yml` only set `SECRET_KEY` on the `backend` service, not on the workers. Workers' Fernet-based `decrypt(nextcloud_app_password_enc)` silently returned empty and `_credentials_for()` raised `NextcloudCredentialsMissing`. This meant Phase 3's `extract_metadata` in worker-light had been silently falling back to ExifTool *the entire time*, and Phase 4's vision worker couldn't fetch NC previews at all. Fix replicates `SECRET_KEY=${SECRET_KEY:-...}` to all worker services. After any compose edit, run `docker exec mulita-<svc> env | grep SECRET_KEY` to confirm propagation.
**Caveats worth knowing for Phase 4:**
- Vision is `disabled` in production (`redis GET mulita:flags:vision.enabled == "false"`). The Phase 4 vision-from-NC path is correct but unexercised by live traffic; it'll matter whenever vision is re-enabled.
- NC's preview generator hasn't covered the whole library at the 640px+ tier. A sample of 6 photos: 4 with proper sizes, 1 stuck at 160px (`IMG_4954.DNG`), 1 unfetchable (`IMG_0193.DNG`). If vision re-enables and quality matters, run `pct exec 114 -- sudo -u www-data php occ preview:generate-all` to backfill.
- The HTTP `/thumb/{size}` endpoint's inline-regen disk fallback still writes one WebP per NC-404 event (mis-extensioned RAW etc). `/data/thumbs` will grow back very slowly from that path.
### 2026-05-11 — Duplicates view shows parent folder + full-path tooltip
`GET /api/v1/library/duplicates/groups` now includes `filepath` in each member payload. `DuplicatesView` renders a dark bottom-strip on every thumbnail showing the last two folder segments — `…/<grandparent>/<parent>` (e.g. `…/files/Phone Photos starting 01-26`) via a `duplicatePathLabel` helper. Going up two levels avoids the failure mode where two copies sit under matching parent names (e.g. duplicate `2023/` subfolders under different archives) and the label would collapse. Full filepath surfaces through the native `title` tooltip on hover. Dimensions chip moves from bottom-left to top-left so the path strip can run edge-to-edge. `memberToPhoto` finally stops faking `filepath = filename` — the synthetic Photo it hands to `PhotoThumbnail` now carries the real path.
**Infra side-effect during the deploy:** two consecutive `--force-recreate` cycles (deploy.sh and a manual `down/up`) raced and left **orphan duplicate worker containers** (`<id>_mulita-worker-light` + the named `mulita-worker-light`, same for vision) both pulling from the same Redis queue. Worker concurrency was effectively doubled, host load climbed past 120, and `pct exec` into LXC 120 hung for several minutes. Recovery: `docker rm -f` on both orphan IDs. The deploy-gotcha entry below covers the pattern; the new contribution from this session is "look for renamed `<id>_<service>` containers and remove them if you ever see load spike + pct hang after a deploy".
### 2026-05-11 — Drop Title field, add bulk Notes editor
- Removed the `Title` (user_title) row from `PhotoInfoPanel.tsx`: the form field, its `titleDraft`/`setTitleDraft` state, the `commitTitle` handler, and the `photo?.user_title` watcher in the draft-sync `useEffect`. Column stays on `photos` and on the backend model — only the UI affordance went away.
- New bulk action `set_notes` in `backend/app/routers/photos.py` `bulk_action`: validates the value is a string (or null/empty to clear), then sets `user_notes` on every photo in the selection in one transaction.
- Frontend wiring: `bulkSetNotes` in `services/api.ts`, `bulkNotes` mutation in `hooks/useBulkPhotoMutations.ts` (optimistic patch with empty-string → null collapse, full rollback on error), surfaced in `RightSidebar.tsx`'s multi-photo bulk panel as a 2-row Textarea + Clear + Apply pair below the Tags section. Doesn't auto-fire on every keystroke — clicking Apply is the explicit commit (otherwise every keystroke would PATCH N rows).
- Shared `Photo` interface in `types/photo.ts` now includes `user_notes?: string | null` so the optimistic-patch typing accepts the field; previously it only existed on `PhotoInfoPanel`'s local `PhotoDetails` shape.
### 2026-05-10 — Right sidebar compact rebuild
Right sidebar previously had `<ActiveHeapCard />` + `<Header />` + a scroll region holding two parallel collapsibles ("Edit" + "Metadata"), with three nested `<Section>` sub-collapsibles (Basic Info / Camera / Location) inside Metadata. A lot of chrome for a per-photo form.
- **Removed** `frontend/src/components/heaps/ActiveHeapCard.tsx` entirely (no remaining importers; active-heap state stays in the store, Select/Discard buttons still consult it).
- `RightSidebar.tsx` single-photo branch also drops its `<Header />` strip — the new "METADATA" collapsible trigger inside `PhotoInfoPanel` is the visible title. Multi-photo branch keeps the Header for "N Photos Selected".
- `PhotoInfoPanel.tsx` is now two stacked collapsibles:
- **Metadata** (default expanded): readonly fields (Size / Dimensions grid, Path, GPS inlined when present), a thin `<hr>`, then editable form (Filename, Title, Date Taken, Notes, Tags, Rating + Color on one row, Flag).
- **Camera** (default expanded): isolated EXIF block (make+model, lens, 2×2 ISO/Aperture/Shutter/Focal grid).
- Compact density: Notes `rows=3 → 2`, stars/swatches `h-5 → h-4`, `space-y-2.5 → 2`, Flag buttons `text-sm → text-xs`, grid `gap-2 → gap-x-2 gap-y-1`, empty "No GPS data" chip dropped (now hidden when there are no coordinates).
- Local `Section` helper deleted from `PhotoInfoPanel.tsx` (no longer used).
`PreviewView` reuses `RightSidebar` inside its overlay aside, so the change applies in both grid and preview.
### 2026-05-10 — Timeline scroll-anchor on section switch
Clicking a folder in the left sidebar (or any other navigation that changed `currentSection` — All Photos, Rated, Discarded, etc.) didn't reset the timeline's scroll position. If the user was deep in All Photos and clicked a folder with fewer rows, the new section loaded with the previous scroll offset preserved, often landing on empty space below the last row.
`Timeline.tsx` already had a section-change effect that cleared selection and reset the auto-focus guard, and a separate auto-focus effect that selects `photos[0]` once the new query resolves. The latter has an "ensure visible" scrollTo, but it only fires if the cell is out of view AND only after the next render — by then the user has already seen the wrong scroll position. Added `parentRef.current.scrollTop = 0` synchronously inside the section-change effect so the first paint of the new section anchors at the top; the auto-focus selectPhoto then highlights photo[0] as before.
### 2026-05-10 — Filter bar 422-on-date-from
User reported "all filters broken, no photos shown" right after the metadata-extraction overhaul shipped. Tracing actual API traffic on a fresh session showed `GET /api/v1/photos?per_page=100&date_from=2026-04-10&sort=taken_at&order=desc` returning **422 Unprocessable Entity**: pydantic v2's `datetime` parser rejects bare-date strings (`"2026-04-10"`) for `Optional[datetime]` query params.
The frontend's `filtersToParams` in `store/filterStore.ts` had been padding `date_to` with `T23:59:59` for inclusive end-of-day, but `date_from` went out as a bare `YYYY-MM-DD` — so every date-range filter request 422'd, and TanStack Query's failure-state shows an empty grid. From the user's perspective it looked like "filters return nothing across the board"; from the backend it was a single endpoint signature problem.
Fix on both sides:
- **Frontend**: pad `date_from` with `T00:00:00` symmetric to `date_to`'s `T23:59:59`.
- **Backend**: switch `date_from` / `date_to` to `Optional[str]` and parse with `datetime.fromisoformat` inside the handler. `fromisoformat` accepts both bare dates (→ midnight) and full ISO strings, so any older client / curl that sends a date-only value still works. Tz-aware values get coerced to naive UTC, matching the `taken_at` column shape and the same fix applied to `PATCH /photos/{id}` earlier in the session. Bad input now returns 400 with a clear message instead of pydantic's 422.
### 2026-05-10 — Date extraction overhaul
User reported wrong "Date Taken" on stills (JPEG / HEIC / DNG). Tracing the pipeline showed the trusted-EXIF list at `services/metadata.py:230-244` ended in `EXIF:ModifyDate`, which is set every time a file is re-saved (Lightroom export, batch resize, EXIF strip), so any photo that lost its original capture metadata during editing was being labeled `taken_at_source='exif'` with the *edit* timestamp.
Pipeline rewrite:
- New trusted-EXIF priority: `SubSecDateTimeOriginal` → `DateTimeOriginal` → `CreateDate` → `QuickTime:MediaCreateDate` → `QuickTime:CreateDate`. **`ModifyDate` removed entirely.**
- Fall back to `guess_date_from_path()` (the same heuristic that already powers `has_date_warning`) when no trusted EXIF date is found. New `taken_at_source='path'` value with a "PATH" badge in the info panel; tooltip explains the date came from filename / folder rather than real EXIF.
- Skip the date-write block entirely when `photo.taken_at_source == 'manual'` so a rescan can't clobber a user correction. (Previous behavior overwrote manual edits.)
- `parse_exif_datetime` accepts tz-aware variants (`%z`), normalizes to naive UTC, and rejects the `0000:00:00` placeholder.
Backfill: new `backfill_taken_at` celery task + `POST /api/v1/library/maintenance/backfill-taken-at` endpoint. Re-enqueues `extract_metadata` for every non-manual, non-trashed photo so the new rules apply across the existing library. Snapshot before the sweep started: 11,223 `exif` + 6,039 `filesystem` + 10 `path`. Will reshape over the next ~45 min.
Side note: the default celery queue had ~209k pending tasks at the time we fired the backfill — the watcher's 5-minute restart loop (since fixed) had been re-enqueuing scans, and tasks for the 4,158 photos we hard-deleted earlier today were still sitting around. Most fail fast (`Photo not found`, ~24ms each); real work runs at ~100ms. Decided to let it drain instead of flushing — safer.
### 2026-05-10 — Two cascade-delete + datetime fixes after the hard-remove shipped
The first cut of `delete_nextcloud_source_root` blew up with `folders_parent_id_fkey` violations on the first real test (`Taco and Muli - 2024 onward`, 35 folders / 4,158 photos). Two iterations to get it right:
1. **NULL `parent_id` before deleting Folders.** Folders have a self-referential `parent_id` FK with no `ON DELETE` rule; postgres checks the constraint per row regardless of insertion order, so a flat `DELETE FROM folders WHERE id IN (...)` of the whole subtree fails on the parents whose children appear later in the same statement. Fixed with an `UPDATE folders SET parent_id = NULL WHERE id IN (folder_ids)` first.
2. **Widen the NULL UPDATE to cross-source-root children.** A "Leóns 1st Year" SourceRoot at `.../Taco and Muli - 2024 onward/Leóns 1st Year` had its own folder rows whose `parent_id` pointed into the Taco SourceRoot's hierarchy. The first patch only NULLed `parent_id` for folders whose `id` was in the delete set; the Leóns folders weren't in that set so they kept their references and the FK still tripped. Fix: `UPDATE folders SET parent_id = NULL WHERE parent_id IN (folder_ids)` — kills any incoming reference into the delete set, internal or external.
After both fixes: `DELETE /api/v1/nextcloud/source-roots/{id}` for `Taco and Muli - 2024 onward` cleared 4,158 photos and 35 folders in a single request and returned 200. UI swaps the trash icon for a `Loader2` spinner while the request is in flight (`removeRoot.isPending && removeRoot.variables === r.id`) so the multi-second cascade is visible.
Separate fix in the same session: `PATCH /api/v1/photos/{id}` returned 500 with `can't subtract offset-naive and offset-aware datetimes` when the frontend sent a tz-aware ISO string for `taken_at` (the datetime-local input is supposed to be naive but real-world locales / paste flows occasionally include `+02:00`). The DB column is `timestamp without time zone`, so asyncpg refused to bind. Normalize on the server with `astimezone(timezone.utc).replace(tzinfo=None)` if `tzinfo is not None`.
Deploy infra learning: the new `--force-recreate` in `deploy.sh` does NOT reliably recreate containers on every push — saw two consecutive deploys leave the backend at the previous `StartedAt` despite a fresh image. Manual `docker compose up -d --no-deps --force-recreate <svc>` after deploy still races with whatever compose state the auto-deploy left mid-flight, twice landing the stack in a half-broken state (orphaned `<id>_mulita-backend` rename containers, db / redis stopped). Recovery: `docker compose down && docker compose up -d`. Open question — `--force-recreate` may need to be replaced with something more explicit.
### 2026-05-10 — Hard-remove Nextcloud SourceRoot + reliable delete sync
`DELETE /api/v1/nextcloud/source-roots/{id}` was a soft-deactivate (`is_active=false`) — the trash icon in Settings only hid the SourceRoot from active queries while leaving every Folder + Photo row in the DB forever. Re-adding the same path resurrected ghosts; `prune-missing` reported zero deletes for the soft-removed entry because the cleanup code skipped inactive source roots.
Endpoint now hard-deletes the SourceRoot, every Folder under it (chunked at 500), and every Photo in those folders. `photo_tags` and `heap_photos` cascade automatically via `ON DELETE CASCADE` on the join tables. `FolderShare` uses a stringly-typed `folder_id` (no FK) so cleaned manually for both `folder_type='folder'` and `folder_type='source_root'`. Returns `{deleted_photos, deleted_folders}` so the UI can toast a count. Files in Nextcloud are untouched.
Sync side: the `watch_folders` celery task in `backend/app/tasks/scan.py` already detected filesystem deletions and soft-marked photos `is_discarded=true`, but the global `task_soft_time_limit=300` in `app/tasks/celery.py` was killing the watcher every five minutes and dropping every FS event during the restart window. The `soft_time_limit=None` on the decorator was being interpreted as "use worker default" rather than "unlimited". Override to `soft_time_limit=0, time_limit=0` (Celery convention for unbounded). Backstop: a new `discard_missing_photos()` in `app/services/cleanup.py` — soft variant of `prune_missing_photos`, walks every `_sr_state == 'present'` source root and flips `is_discarded=true` on Photo rows whose file is gone. Wired as `discard_missing_photos_beat` celery task scheduled every 30 minutes via `beat_schedule` on the celery app. Beat runs in-process on `worker-watcher` (`--beat` flag in `docker-compose.yml`) — the watcher is already a Redis-locked singleton so no need for a separate beat container.
Manual `POST /api/v1/library/maintenance/prune-missing` remains the hard-delete path for when the user wants to permanently drop orphan rows; the new beat job only soft-discards (file gone → mule-image trash, restorable).
### 2026-05-10 — OIDC auto-redirect on LoginPage
`OIDC_ENABLED=true` was already set in `.env`, so the LoginPage rendered a "Sign in with Authentik" button next to the password form. With a single trusted IdP and a logged-in Authentik session, that extra click was friction without upside.
`LoginPage.tsx` now reads `/auth/config` on mount and, if OIDC is on, immediately navigates to the OIDC login URL. Authentik recognizes the existing session and bounces back through the callback with no user interaction. Two escape hatches: `?password=1` in the URL forces the password form, and a `skipAutoSso` sessionStorage flag (set by `AuthContext.logout` and by the OIDC callback's error branch) suppresses the next auto-redirect so logouts actually log out and OIDC failures surface their error instead of looping. While the redirect is in flight the page shows "Signing in with Authentik..." plus a "Use password instead" link.
### 2026-05-10 — right sidebar restructure (heap pinned, single scroll, collapsible Metadata)
The right sidepanel had three stacked flex regions: `<ActiveHeapCard />`, `<Header />`, and `<PhotoInfoPanel />` — with `PhotoInfoPanel` carrying its own internal scroll. That left the editable fields (filename, title, notes, rating, color, flag) stuck above the readonly metadata scroll, effectively two scroll boundaries on one sidebar.
- Moved the scroll boundary up to `RightSidebar.tsx`: only `ActiveHeapCard` and `Header` stay pinned now. Editable + readonly content scroll together in one `flex-1 overflow-y-auto` region beneath them.
- `PhotoInfoPanel.tsx` dropped its `h-full` / inner `flex-1 overflow-y-auto`.
- The four readonly sections (Tags / Basic Info / Camera / Location) are now wrapped in a single outer "Metadata" `Collapsible`. Default expanded, one click hides the whole block. Sub-sections stay individually collapsible.
- Second pass: the editable form (filename / title / notes / rating / color / flag) got the same treatment under an outer "Edit" `Collapsible` so the panel is now two equal collapsible groups below the title strip. Dropped the X (clear-selection) button from the Header; Esc and grid-empty-area-click still clear.
- Third pass: split editable vs read-only between the two groups consistently. `TagsEditor` and `TakenAtEditor` had been buried inside the readonly Metadata sub-sections — `Tags` as its own Section, taken-at wedged into `Basic Info` between size/dims and the filepath. Moved both into the Edit collapsible, ordered identification → description → categorization: Filename · Title · Date Taken · Notes · Tags · Rating · Color · Flag. Metadata now holds only readonly: `Basic Info` (size, dims, path), `Camera`, `Location`.
`PreviewView` reuses `RightSidebar` under an `<aside class="overflow-hidden">`, so the change applies in both the grid sidebar and the preview overlay.
### 2026-05-10 — `photos.hubris.network` perf sweep
User reported slow load. Five fixes shipped, in order:
1. **Backend out of dev mode.** `docker-compose.yml` `command:` was running `uvicorn … --reload` (single worker, file-watcher). Page loads fan out ~15 parallel API calls; they all serialized on one event loop. → `--workers 2 --proxy-headers`. Two uvicorn worker procs now.
2. **iPhone Apple ProRAW / Linear DNG decode fixed.** rawpy 0.26.1 + LibRaw 0.22 rejects Apple Linear DNG (`Photometric Interpretation: Linear Raw`, 8-bit) as "Unsupported file format or not RAW file". Those files are TIFF containers with developed RGB inside and have no embedded preview to fall back to. Added PIL `Image.open(src_path)` as the last fallback in both `_generate_proxy_webp` (routers/photos.py) and `process_raw_image` (tasks/thumbs.py). ~1,300 iPhone DNGs were 415-ing on every detail view; now decode in <1s via PIL.
3. **Reconcile DB with disk (renamed-folder case).** `prune_missing_photos` was skipping all 4,154 orphaned photo rows under SourceRoot `Taco and Muli - 2024 onward` (renamed in Nextcloud to `Photo Archive 2004-2024`) because the leaf path didn't resolve and the code assumed "drive unmounted → must skip". Added `_sr_state()` to classify as `present` / `renamed` (parent mount fine, leaf gone) / `unmounted` (parent inaccessible). Only `unmounted` still skips. Two stale source roots logged with a clearer hint pointing at `POST /api/v1/library/maintenance/prune-missing`. User has not yet been asked to run that — endpoint is ready when they are.
4. **Frontend page size + idle polling.** `usePhotosQuery.ts` was first-fetching `per_page=500` (slow paint + 500 thumb requests at once). Split into `PER_PAGE_INITIAL=100` for first paint, `PER_PAGE_BACKGROUND=500` for the cursor-chain prefetch. Idle polling for `scan-status` and `worker-status` (`useScanActivity.ts`, `ScanProgress.tsx`) bumped from 10s/15s to 30s/30s while idle; active cadence (2s/3s) unchanged.
5. **Partial index on `photos`.** Default list query `WHERE NOT is_trashed AND NOT is_hidden ORDER BY taken_at DESC NULLS LAST, id DESC LIMIT N` was doing a seq-scan + top-N heapsort (~25ms standalone, worse under concurrency). Added migration `0017_photos_list_index` creating `ix_photos_list_visible` (partial index on the sort key, restricted to visible rows). EXPLAIN now shows an Index Only Scan → **24.7ms → 0.097ms**, ~250× speedup.
**Deploy gotcha — fixed (with caveat).** The original `docker compose up -d --build` in `deploy.sh` did not reliably recreate containers when only runtime config (`command:`, env-only) or migration files changed; image hash would change but compose treated the existing container as "current enough". Bit three times this session before `/opt/mule-deploy/deploy.sh` was updated to `docker compose up -d --build --force-recreate`. Trade-off accepted: an extra restart cycle on deploys where nothing user-visible changed.
**Caveat:** the first auto-deploy after the flag flip raced with my own earlier manual `docker compose up -d --no-deps --force-recreate frontend` and landed the stack in a half-broken state — `mulita-frontend` got stuck under a renamed temp container, several services dropped off `mule-image_mulita-network`, frontend nginx restarted in a loop with `host not found in upstream "backend"`. Fixed by `docker compose down && docker compose up -d`. Don't issue a manual `--force-recreate` on a single service while the auto-deploy webhook is also expected to fire — let the deploy own the lifecycle.
**Data drift still outstanding.** 4,154 photo rows + 1 unregistered folder (`Photo Archive 2004-2024`) on disk that's not a SourceRoot. The reconcile endpoints now work — user decides when to call them. The new folder needs to be added as a SourceRoot via the Settings UI before its files will be indexed.
**Proxy cache still empty** (`mule-image_proxies_data` volume is 4 KB). Pre-generating ~500–1,500 WebP proxies for non-web-safe formats would make first-open of every RAW/HEIC photo instant. Deferred — needs a one-shot script and the disk-space tradeoff isn't worth it until the data-drift reconcile happens first.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — Nextcloud-rooted libraries shipped
Bind `/mnt/library/homecloud` into the workers, reads via filesystem, writes via WebDAV. `users.nextcloud_username` override field added; `dtoro → admin` mapping. Surviving SourceRoots cleaned up to NC paths.
### 2026-04-22 — native OIDC via Authentik
Authlib-based code in `backend/app/auth_oidc.py`. `extra_hosts` override for `auth.hubris.network` in compose override (gitignored).
### 2026-04-21 — auto-deploy pipeline shipped
Webhook receiver at `:9797`, async deploy returning 202. Mirrors caddy-conf / gitea-customizations.

106
archive/knowledge/containers/121-caddy.md Normal file
View File

@@ -0,0 +1,106 @@
# 121 — `caddy`
The reverse proxy. Terminates TLS for every `*.hubris.network` hostname on the LAN/mesh and forwards to the right backend.
## At a glance
- **Hostname:** `caddy`
- **IP:** `192.168.8.175`
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 512 MiB RAM / 6 GiB rootfs
- **Mounts:** none from `/mnt/library`
- **Config:** `/etc/caddy/Caddyfile` is a [git checkout of `dtoro/caddy-conf`](#auto-deploy)
- **Cert source:** Let's Encrypt **DNS-01** via IONOS API (`IONOS_AUTH_API_TOKEN`).
## Sites currently served (live as of 2026-07-06)
- `artifacto.hubris.network` → [apps (105)](105-apps.md) `:3100`
- `blog.hubris.network` → [apps (105)](105-apps.md) `:8080`
- `books.hubris.network` → [grimmory (130)](130-grimmory.md) `:6060`
- `cloud.hubris.network` → [nextcloud (114)](114-nextcloud.md) `:443`
- `docker.hubris.network` → [apps (105)](105-apps.md) `:9443`
- `git.hubris.network` → [gitea (104)](104-gitea.md) `:3000` (+ `handle_path /_plantuml/*` → apps `:8079`)
- `home.hubris.network` → [haos VM (108)](../vms/108-haos.md) `192.168.8.101:8123`
- `house.hubris.network` → [house (129)](129-house.md) `:3000`
- `jellyseerr.hubris.network` → [arriman (122)](122-arriman.md) `:5056`
- `matrix.hubris.network` → [elementsynapse (118)](118-elementsynapse.md) `:8008`
- `media.hubris.network` → [jellyfin (101)](101-jellyfin.md) `:8096`
- `paperless.hubris.network` → [paperless (103)](103-paperless.md) `:8000`
- `photos.hubris.network` → [mule-images (120)](120-mule-images.md) `:3000`
- `proxmox.hubris.network` → [hubris host](../hosts/hubris.md) `:8006`
- `qbit.hubris.network` → [arriman (122)](122-arriman.md) `:8080`
- `roms.hubris.network` → [romm (134)](134-romm.md) `:80`
- `sab.hubris.network` → [arriman (122)](122-arriman.md) `:8082` (Authentik forward-auth)
- `teddy.hubris.network` → LXC 131 `192.168.8.150:8443`
- `trmnl.hubris.network` → [trmnl (128)](128-trmnl.md) `:9851`
> **Reminder:** Caddy alone isn't enough to make a new subdomain reachable on the LAN. Each one needs an entry in [DNS split-horizon](../infrastructure/dns.md) too.
## Snippet: `(authentik)` forward-auth
A snippet at the top of the Caddyfile (used as `import authentik` in any site block) wires forward-auth to the embedded Authentik outpost. It points at `http://192.168.8.180:9000` directly (NOT `https://auth.hubris.network`) to avoid Caddy-to-self round-tripping that strips `X-Forwarded-Host`. The forward-auth block must explicitly set `header_up X-Forwarded-Host {host}`. See [Authentik](106-auth-outpost.md#forward-auth-domain-level-setup).
For apps with mobile clients, `/api/*` (or equivalent) bypasses forward-auth — see the per-app gotchas in [Authentik](106-auth-outpost.md).
## Caddy environment
Untracked secrets in `/etc/caddy/caddy.env`:
- `IONOS_AUTH_API_TOKEN` — DNS-01 ACME
- `ARTIFACTO_GATEWAY_SECRET` — must match `SSO_GATEWAY_SECRET` on [apps (105)](105-apps.md). Rotate together.
## Auto-deploy
Push to `dtoro/caddy-conf` master → gitea webhook → `http://192.168.8.175:9797/deploy` → `caddy-deploy-webhook.service` validates HMAC against `/etc/caddy-deploy/secret` → runs `/etc/caddy/scripts/deploy.sh`:
1. `git pull`
2. `caddy validate`
3. `systemctl reload caddy`
Gitea webhook id 2 on `dtoro/caddy-conf`. Receiver, deploy script, install scripts, systemd unit all live in the repo under `scripts/`. The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`). Git credentials for the root-run service are at `/etc/caddy-deploy/git-credentials` (mode 600).
> **Systemd unit changes inside the repo do NOT auto-reinstall** — you have to re-run `scripts/webhook/install.sh` manually after editing the unit.
## Related
- [DNS split-horizon](../infrastructure/dns.md) — must add entry for every new subdomain
- [Authentik (124)](106-auth-outpost.md) — forward-auth + IdP
- [Auto-deploy](../infrastructure/auto-deploy.md)
- [Public ingress (VPS traefik)](../infrastructure/ingress.md) — mirrors Caddy's certs to the VPS for public exposure
- [Gitea (104)](104-gitea.md) — webhook source
- All container pages list their Caddy backend.
## Changelog
### 2026-06-13 — sab.hubris.network gated with Authentik forward-auth; port fixed :8081→:8082
`sab.hubris.network` now uses `import authentik` inside a `handle` block. SABnzbd own auth disabled, `local_ranges` set for transparent proxy. Port bumped from `:8081` to `:8082` (fix from 2026-06-04) now documented.
### 2026-06-06 — Caddyfile truncated to 43 lines; restored from origin/master + safeguards added
**Symptom:** All `*.hubris.network` hosts except `photos` and `auth` (VPS-hosted) returned `tlsv1 alert internal error` or timeout. Only 3 site blocks (`photos`, `prism`, `photos2`) remained in the Caddyfile.
**Root cause:** The Caddyfile was manually edited directly on LXC 121 (not via the `dtoro/caddy-conf` git repo), overwriting 260 lines / 30+ site blocks with 43 lines of photo-only config.
**Fix:**
- Restored Caddyfile from `origin/master` (`git checkout --force origin/master -- Caddyfile`)
- `systemctl reload caddy`
**Permanent safeguards added to `/etc/caddy/scripts/deploy.sh`:**
- **Site-count guard:** refuses to reload if fewer than 20 `*.hubris.network` blocks detected
- **Dirty-tree auto-stash:** stashes local changes before `git pull --ff-only` so the webhook doesn't fail on local edits
- **Auto-backup:** saves `Caddyfile.bak.<timestamp>` before any modifications, keeps last 5
Also: [elementsynapse LXC 118](118-elementsynapse.md) found to have DHCP-overridden static IP (actual `.244` vs config `.239`) during incident investigation — fixed.
### 2026-06-02 — caddy.service unit missing; recreated
After the Slate AX → SODOLA network migration, Caddy was not listening (ports 80/443 dead). Root cause: the custom hubris1 Debian package (`caddy_1:2.11.3-hubris1_amd64`) does not ship a systemd service unit file. The unit had previously existed but was lost (likely on a package reinstall). Recreated at `/lib/systemd/system/caddy.service` with standard Caddy service config + `EnvironmentFile=/etc/caddy/caddy.env` (already present in `caddy.service.d/override.conf`). **Risk:** the unit will be lost again if the package is reinstalled without the file being tracked. Fix: add the service unit to the `caddy-conf` repo or rebuild the hubris1 package to include it.
### 2026-04-28 — wiki entry created
Initial documentation. 16 active sites at this date.
### 2026-04-22 — Artifacto + Authentik gateway-secret pattern wired
Caddy site `artifacto.hubris.network`: forward-auth on admin paths, `header_up X-Artifacto-Gateway {env.ARTIFACTO_GATEWAY_SECRET}`, public matchers `/p/*`, `/static/*`, `/healthz`. `caddy.env` extended with the gateway secret.
### 2026-04-21 — `(authentik)` snippet added
Top-of-file snippet used by every gated site. Targets `http://192.168.8.180:9000` directly to dodge hairpin TLS round-trip.
### 2026-04-21 — three new sites added (jellyseerr, qbit, sab → 192.168.8.132)
For the [arriman (122)](122-arriman.md) docker stack. Caddy + certs work; needed corresponding [DNS split-horizon](../infrastructure/dns.md) entries to be added to land cleanly.
### 2026-04-20 — auto-deploy pipeline shipped
Repo `dtoro/caddy-conf`. Webhook receiver at `:9797`, validate + reload on push to master.

158
archive/knowledge/containers/122-arriman.md Normal file
View File

@@ -0,0 +1,158 @@
# 122 — `arriman`
Docker host running the \*arr stack via [`ezarr`](https://github.com/ezarr/ezarr) compose. Replaced the old yunohost-based [LXC 100](index.md#recently-destroyed-kept-for-archaeology) on 2026-04-21.
## At a glance
- **Hostname:** `arriman`
- **IP:** `192.168.8.245`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged
- **Resources:** 4 cores / 8 GiB RAM / 24 GiB rootfs
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
- **Public hostnames:** `jellyseerr` / `qbit` / `sab` (see below)
## Compose
`/home/arr/ezarr/docker-compose.yml`. Network `ezarr_default` `172.18.0.0/16`. Service users all run with primary GID 10000 (host `media` group) inside their containers — see [media permissions](../infrastructure/media-permissions.md). Per-app `/config/<app>-config/` dirs bind to `/config` inside.
`.env` contents:
- `DOWNLOADS_DIR=/mnt/library/downloads/`
- `MEDIA_DIR=/mnt/library/`
- `MEDIACENTER_GID=10000` ← **must be 10000, not 13000.** s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. Without this, qBit errors every torrent with "Permission denied".
- `SONARR_UID=13001`, `RADARR_UID=13002`, `LIDARR_UID=13003`, `PROWLARR_UID=13006`, `JELLYSEERR_UID=13012`, `BAZARR_UID=13013`, `QBITTORRENT_UID`, `SABNZBD_UID`
## Service / port map
All services route through gluetun's network namespace. Ports are exposed via
the gluetun container:
| Service | Host:Container | Public hostname |
| ------------- | -------------- | ------------------------------------ |
| gluetun (VPN) | — | — |
| sonarr | `8989:8989` | direct only (via gluetun) |
| radarr | `7878:7878` | direct only (via gluetun) |
| lidarr | `8686:8686` | direct only (via gluetun) |
| prowlarr | `9696:9696` | direct only (via gluetun) |
| bazarr | `6767:6767` | direct only (via gluetun) |
| jellyseerr | `5056:5055` | [`jellyseerr.hubris.network`](../infrastructure/dns.md) |
| qbittorrent | `8080:8080` | [`qbit.hubris.network`](../infrastructure/dns.md) |
| sabnzbd | `8082:8082` HTTP, `9090:9090` HTTPS | [`sab.hubris.network`](../infrastructure/dns.md) |
| flaresolverr | `8191:8191` | internal only |
| homarr | `7575:7575` | internal only |
Internal *arr ↔ *arr / *arr ↔ qBit/SAB/flaresolverr comms run on `localhost:<port>`
(services share gluetun's shared network namespace). External services reach them
via `gluetun:<port>` (e.g. Sonarr → qBittorrent at `localhost:8080` or
`gluetun:8080`).
## Categories (qBit + SAB + *arr)
| Category | qBit savePath | SAB dir | Consumer |
| -------- | --------------------------- | ------- | --------------------------------- |
| tv | `/data/torrents/tv` | tv | Sonarr |
| movies | `/data/torrents/movies` | movies | Radarr |
| music | `/data/torrents/music` | music | Lidarr |
| anime | `/data/torrents/anime` | anime | Sonarr (2nd download client, root `/data/media/anime`) |
| books / audiobooks / comics / podcasts | `/data/torrents/<cat>` | `<cat>` | (reserved, none in use yet) |
Path mapping: host `/mnt/library/<cat>` ↔ container `/data/media/<cat>`. Downloads: host `/mnt/library/downloads/<torrents|usenet>/<cat>` ↔ container `/data/torrents/<cat>` and `/data/usenet/<cat>`.
## Auth (reverse-proxy + Authentik forward-auth)
### qBit
Auto-login behind forward-auth via IP whitelist. `qBittorrent.conf` lines:
- `WebUI\\AuthSubnetWhitelist=172.18.0.0/16, 172.17.0.0/16, 192.168.8.175/32`
- `WebUI\\ReverseProxySupportEnabled=true`
- `WebUI\\TrustedReverseProxiesList=192.168.8.175, 172.18.0.0/16`
> **Stop the container before editing `qBittorrent.conf`.** qBit writes its in-memory config on graceful shutdown and clobbers any live edits. Recipe: `docker stop qbittorrent && sed -i ... && docker start qbittorrent`.
Mobile/desktop clients keep working via `/api/v2/*` path bypass on Caddy.
### SABnzbd
Gated with Authentik forward-auth (applied 2026-06-13). Caddy `sab.hubris.network` block uses `import authentik` inside a `handle` block. SABnzbd's own web auth is disabled:
- `html_login = 0` → no HTML login form
- `username` / `password` cleared → CherryPy Basic Auth not activated
- `local_ranges = 172.18.0.0/16, 192.168.8.0/24, 127.0.0.0/8` → proxied requests from Caddy (192.168.8.x) and Docker-proxy (172.18.x) pass without auth
**API key** (`67ef5a45e4e04157994e977005a33878`) still works for internal service-to-service calls (Sonarr/Radarr/Lidarr via Docker internal networking — they talk to SAB at `localhost:8082`, not through Caddy).
`host_whitelist`: `sabnzbd, localhost, 127.0.0.1, 192.168.8.132, sab.hubris.network` — extend before accessing SAB from a new host.
## Credentials
Backups of all migration-time secrets at `/mnt/library/backup/arr-migration-2026-04-21/state/`:
- `lxc100-apikey-{sonarr,radarr,prowlarr,lidarr}.txt`
- `qbit-creds.txt`, `sab-creds.txt`
- `lxc100-jellyseerr-settings.json`
\*arr API keys were preserved from the yunohost DB — don't rotate unless needed.
qBittorrent: `admin` / `CDyjxMKvjfGupCKzXFkjsKw4nRrXMRL`.
## Migration gotchas (apply if ever re-importing)
- Imported \*arr `config.xml`: must have `<BindAddress>*</BindAddress>`, `<Port>{ezarr-port}</Port>`, `<UrlBase></UrlBase>`. Otherwise app listens only on 127.0.0.1, wrong port, wrong path.
- Imported DB paths must be rewritten `/mnt/library/` → `/data/media/` (RootFolders / Series / Movies / Artists tables). Without this, scans and imports fail silently.
- Bazarr `config/config.yaml` (yaml, not ini) — `ip: 0.0.0.0`, `port: 6767`, empty top-level `base_url`; sonarr/radarr connection sections normalized to docker service names.
- Staging dir for migrations: `/mnt/library/backup/stage/<app>/` (visible to both old LXC 100 and arriman because both bind-mount the same host path).
## Permissions
Member of [media GID 10000](../infrastructure/media-permissions.md). The LXC has `www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr` in the `media` group. Each docker container also receives GID 10000 as its primary group via `MEDIACENTER_GID=10000`.
## Rollback material (kept until 2026-04-28+)
`/mnt/library/backup/arr-migration-2026-04-21/`:
- `tars/` — per-subtree tars of yunohost app data (granular restore)
- `snapshots/` — vzdumps of LXC 100 (pre-migration) and LXC 122 (pre-import), plus post-stop vzdumps of LXC 100 + 106
- `stage/<app>-preimport.tar.gz` — original empty ezarr `/config/<app>-config/` dirs
- `state/` — service dumps, listening ports, API keys, jellyseerr settings at migration time
- `DESTROY_AFTER_2026-04-28.md` — destroy checklist + post-destroy restore commands
## Related
- [Caddy (121)](121-caddy.md)
- [Authentik (124)](106-auth-outpost.md) — forward-auth wiring + per-app `/api/*` bypass
- [DNS](../infrastructure/dns.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Hubris host](../hosts/hubris.md)
## Changelog
### 2026-06-13 — SABnzbd gated with Authentik forward-auth
SABnzbd now uses Authentik forward-auth (same `import authentik` Caddy pattern as qBit). SABnzbd's own web auth disabled: `html_login=0`, credentials cleared, `local_ranges` extended to cover Docker bridge + homelab LAN. API key still works for internal *arr service calls. See [Auth section](#auth-reverse-proxy--authentik-forward-auth) above.
### 2026-06-04 — all arr services moved behind gluetun VPN; SAB port conflict fixed
- All services (sonarr, radarr, lidarr, bazarr, prowlarr, jellyseerr, homarr,
flaresolverr) now use `network_mode: service:gluetun` — whole stack routes
through the VPN
- Port mappings moved from individual services to gluetun container
- **Fixed SABnzbd port conflict**: was crashing in a restart loop because
qBittorrent held port 8080 inside the shared gluetun namespace. Changed
SAB internal port to 8082 (config at `/config/sabnzbd-config/sabnzbd.ini`)
- Caddy `sab.hubris.network` updated to point to `:8082`
- Jellyseerr's `extra_hosts` (auth.hubris.network) moved to gluetun since
`extra_hosts` conflicts with `network_mode`
### 2026-06-02 — ProtonVPN added (gluetun); LXC IP set static
- Added `gluetun` container to compose as a WireGuard VPN sidecar (ProtonVPN, server AL#57, located in Tirana, Albania)
- **qbittorrent** and **sabnzbd** now use `network_mode: service:gluetun` — all traffic routes through the VPN
- Ports 8080 (qBit WebUI), 6881 tcp/udp (qBit BT), 8081 (SAB WebUI) exposed through gluetun
- gluetun config at `gluetun-config/wireguard/wg0.conf` (read-only mount)
- Healthcheck on gluetun; qBit/SAB wait for `service_healthy` before starting
- LXC IP changed from DHCP to static (`192.168.8.132`) via `pct set` + `/etc/network/interfaces`
- **After first start:** Sonarr/Radarr/Lidarr download client host needs updating from `qbittorrent` → `gluetun` (SAB similarly `sabnzbd` → `gluetun`)
- **Also fixed:** 7 other DHCP LXCs (101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy) set to static IPs to prevent floating on reboot. See infrastructure/dns.md.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — `MEDIACENTER_GID` flipped 13000 → 10000
qBit was erroring every torrent with "Permission denied" because s6-setuidgid only honors the primary PGID and `group_add:` doesn't propagate. Setting the primary GID to 10000 (the host `media` group) fixed it.
### 2026-04-21 — \*arr stack migrated yunohost → docker-compose
Old LXC 100 stopped; ezarr stack stood up here; data and DBs imported with path rewrites. flaresolverr folded in from old LXC 106. Three Caddy sites added (jellyseerr, qbit, sab) → see [caddy](121-caddy.md). Rollback artifacts under `/mnt/library/backup/arr-migration-2026-04-21/`.

48
archive/knowledge/containers/128-trmnl.md Normal file
View File

@@ -0,0 +1,48 @@
# 128 — `trmnl`
Self-hosted middleware for TRMNL e-ink plugins. TRMNL cloud polls it; it fetches/shapes live data into JSON the plugin's Liquid template renders.
## At a glance
- **Hostname:** `trmnl`
- **IP:** `192.168.8.211`
- **Privilege:** unprivileged
- **Resources:** 1 core / 768 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** `trmnl.hubris.network` (via [VPS ingress](../infrastructure/ingress.md))
## Role
Runs one FastAPI aggregator (`server.app:app`, port 9851) that mounts a router per plugin from the `dtoro/terminalito` repo. First consumer: `munich-home` (`/munich-home/dashboard`) — weather (Open-Meteo), MVG transit, Google Calendar, plus server-side Kita/quote logic. Talks out to the public internet for those APIs; TRMNL cloud polls it inbound every 15 min. Bearer-token gated (`TRMNL_POLL_TOKEN`); `/health` is open.
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `trmnl-plugins` | `0.0.0.0:9851` | uvicorn aggregator; `EnvironmentFile=/etc/trmnl-plugins/env` |
## Storage / config paths
- `/opt/terminalito` — git checkout (origin = internal gitea `http://192.168.8.121:3000/dtoro/terminalito.git`)
- `/opt/terminalito/server/.venv` — venv
- `/etc/trmnl-plugins/env` — `TRMNL_POLL_TOKEN` (+ Google/MVG creds once enrolled)
- `/etc/systemd/system/trmnl-plugins.service`
## Auto-deploy
Wired — [auto-deploy](../infrastructure/auto-deploy.md) Shape B, webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` (`terminalito-deploy.service`). Push to `main` → `server/deploy/deploy.sh` (`git pull` + pip + reinstall units + restart `trmnl-plugins`). Secret `/etc/terminalito-deploy/secret`; git creds `/etc/terminalito-deploy/git-credentials` wired as a repo-local `credential.helper`. Manual: `pct exec 128 -- /opt/terminalito/server/deploy/deploy.sh`.
## Secrets
Not yet SOPS-enrolled. The poll token is set directly in `/etc/trmnl-plugins/env`. Google Calendar + MVG creds are pending: enroll via `homelab client add trmnl` + bootstrap, add `secrets/trmnl-oauth.yaml`, then `server/deploy/render-env.sh` builds the env from `homelab secret trmnl-oauth`. Until then calendar/transit cards degrade to empty; weather works.
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`trmnl.hubris.network → 192.168.8.211:9851`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `trmnl → 192.168.8.175` (LAN path via Caddy)
- [Gitea (104)](104-gitea.md) — source repo `dtoro/terminalito`
- [Plan: 2026-06-24 TRMNL plugins LXC](../../../plans/2026-06-24-trmnl-plugins-lxc.md)
## Changelog
### 2026-06-24 — auto-deploy + LAN DNS wired
Gitea Shape-B deploy pipeline (webhook id 12, `:9797`) — push to `dtoro/terminalito` redeploys; verified end-to-end. Technitium A record `trmnl.hubris.network → 192.168.8.175` added on [dns (107)](107-dns.md) (propagated to the NetBird managed zone via dns-sync), so LAN clients take the short path through [Caddy (121)](121-caddy.md). See [auto-deploy](../infrastructure/auto-deploy.md).
### 2026-06-24 — public path live
Verified end-to-end from the internet: `https://trmnl.hubris.network/munich-home/dashboard` → 200 with token, 401 without; `/health` 200. The provision-time outage was the netbird `home-lab-network` (192.168.8.0/24) route having no active routing peer — the **mac-mini routing peer's netbird daemon was down** (artifacto/blog were 504 too). Bringing netbird up on mac-mini restored the route; the edge recovered with no config change. See [ingress](../infrastructure/ingress.md) / [mesh](../infrastructure/mesh.md).
### 2026-06-24 — provisioned
LXC 128 created (Debian 13, unprivileged, `192.168.8.211`). Deployed `trmnl-plugins.service` on :9851 from `dtoro/terminalito`. Caddy block added (`dtoro/caddy-conf`) + LE cert via IONOS DNS-01; verified `/health` 200 and `/munich-home/dashboard` (live weather) through Caddy. Cert mirrored to VPS (`trmnl.fullchain.crt`/`trmnl.privkey.key`) + traefik router `trmnl-public` → `192.168.8.211:9851` added to `/opt/traefik-dynamic.yaml`. **Public path pending**: VPS↔home netbird route was down at provision time (`No networks available`, 3/6 peers — artifacto/blog also 504); resolves when the mesh route recovers. **LAN pending**: Technitium A record not yet added. Not SOPS-enrolled; Google/MVG creds pending.

52
archive/knowledge/containers/129-house.md Normal file
View File

@@ -0,0 +1,52 @@
# 129 — `house`
Yuvomi family planner (formerly Oikos). Self-hosted family planner with 14 modules: calendar, tasks, meals, groceries, budget, documents, notes, contacts, birthdays, housekeeping, recipes, reminders.
## At a glance
- **Hostname:** `house`
- **IP:** `192.168.8.244`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** unprivileged
- **Resources:** 1 core / 1344 MiB RAM / 8 GiB rootfs (Debian 13)
- **Mounts:** none
- **Public hostname:** [`house.hubris.network`](../infrastructure/ingress.md) → VPS traefik → Caddy
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| `oikos` (Yuvomi) | `0.0.0.0:3000` | Docker Compose at `/opt/yuvomi/`, image `ghcr.io/ulsklyc/yuvomi` |
## Integrations
- **Authentik SSO (OIDC):** Provider `Provider for Yuvomi` (PK 31) in Authentik on VPS. Env vars in `/opt/yuvomi/.env`: `OIDC_ISSUER`, `OIDC_CLIENT_ID`, `OIDC_CLIENT_SECRET`. Redirect URI: `https://house.hubris.network/auth/oidc/callback`.
- **Paperless DMS connector (native):** Yuvomi connects directly to Paperless-ngx API at `http://192.168.8.130:8000/`. API token stored in SQLite `dms_accounts` table. Search, link, and upload documents from Yuvomi to Paperless via Settings → Documents → DMS.
- **Weather widget:** Open-Meteo (free, no API key). Munich coordinates set.
- **Google Calendar:** OAuth configured via env vars (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI`). Redirect URI: `https://house.hubris.network/api/v1/calendar/google/callback`. Authorize in Settings → Calendar → Connect Google Calendar.
## Config paths
- `/opt/yuvomi/docker-compose.yml` — downloaded from upstream
- `/opt/yuvomi/.env` — config including secrets (untracked)
- `/opt/yuvomi/data/` — SQLCipher SQLite DB (`oikos.db`)
- `/opt/yuvomi/backups/` — auto backups
- `/opt/yuvomi/modules/` — Yuvomi modules (empty for now)
## Related
- [Caddy (121)](121-caddy.md) — LAN reverse proxy (`house.hubris.network → 192.168.8.212:3000`)
- [VPS ingress](../infrastructure/ingress.md) — public edge (cert mirror + traefik router)
- [DNS (107)](107-dns.md) — Technitium A record `house → 192.168.8.175` (LAN path via Caddy)
- [Paperless (103)](103-paperless.md) — native DMS connector (API at `:8000`)
- [TRMNL (128)](128-trmnl.md) — Google Calendar tokens source
- [Deployment plan](../../../plans/done/2026-06-25-yuvomi-deployment.md)
## Changelog
### 2026-06-27 — Google Calendar OAuth env vars configured
`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, `GOOGLE_REDIRECT_URI` set in `.env`. New OAuth client ID (`-bho4iq..`).
### 2026-06-26 — provisioned
LXC 129 created (Debian 13, unprivileged, `192.168.8.212`). Docker installed. Yuvomi container running on `:3000` from `ghcr.io/ulsklyc/yuvomi:latest`. Caddy block + DNS A record + VPS traefik router `house-public` for public access. Authentik OIDC provider created (PK 31). WebDAV document bridge on paperless LXC (103) at `:8088` for Paperless auto-import.

66
archive/knowledge/containers/130-grimmory.md Normal file
View File

@@ -0,0 +1,66 @@
# 130 — `grimmory`
Self-hosted digital library (eBooks, comics, audiobooks). Community fork/successor of Booklore, with smart shelves, metadata enrichment, Kobo/KOReader sync, OPDS, and a built-in EPUB/PDF reader. Migrated from [apps (105)](105-apps.md) on 2026-06-29.
## At a glance
- **Hostname:** `grimmory`
- **IP:** `192.168.8.247`
- **Host:** **strong** (migrated from hubris 2026-07-05)
- **Privilege:** privileged (UID = host UID for `/mnt/library` media GID)
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13)
- **Mounts:** `/mnt/media_local` ↔ `/mnt/library`
- **Public hostname:** `books.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| Grimmory | `192.168.8.247:6060` | Docker Compose at `/opt/grimmory/` |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/grimmory/docker-compose.yml`. Key points:
- Image: `ghcr.io/grimmory-tools/grimmory:latest`
- Books library: `/mnt/library/books` → `/books` (read-write; media GID 10000 via `GROUP_ID=10000`)
- Bookdrop (watched folder for auto-import): `/opt/grimmory/bookdrop` → `/bookdrop`
- App data (covers, DB config): `/opt/grimmory/data` → `/app/data`
- MariaDB config: `/opt/grimmory/mariadb/config` → `/config` (linuxserver/mariadb image)
- `extra_hosts: auth.hubris.network:192.168.8.175` — routes Authentik OIDC discovery to Caddy from inside the container
- `FORCE_DISABLE_OIDC=false` — OIDC stays enabled; provider configured via Grimmory admin UI
Credentials live in `/opt/grimmory/.env` (untracked):
- `DATABASE_PASSWORD` / `MYSQL_PASSWORD` — MariaDB Grimmory user password
- `MYSQL_ROOT_PASSWORD` — MariaDB root password
## Authentik OIDC
Uses Confidential client (client secret stored in Grimmory's DB — migrated from Booklore). The OIDC config carried over in the database dump; no manual re-entry needed.
- **Authentik provider:** `Provider for Grimmory` (renamed from `Provider for Booklore` on migration)
- **Client ID:** `L1u0eFsNhbKgiIvvFeIr2mvZdbtFyzidCq2h6thL`
- **Client type:** Confidential (client secret in `oidc_provider_details` in MariaDB `app_settings`)
- **Redirect URI:** `https://books.hubris.network/oauth2-callback`
- **Scopes:** openid, profile, email, offline_access
- **Back-channel logout:** `http://192.168.8.247:6060/api/v1/auth/oidc/backchannel-logout`
- **Application slug:** `booklore` → Issuer URI: `https://auth.hubris.network/application/o/booklore/`
## Media permissions
LXC is privileged → in-container UID = host UID. Docker container gets media GID via `GROUP_ID=10000` env var (Grimmory/linuxserver pattern). The `/mnt/library/books` subtree is owned `:media` mode `2775` (setgid). See [media-permissions](../infrastructure/media-permissions.md).
## Related
- [apps (105)](105-apps.md) — previous host (Booklore)
- [Caddy (121)](121-caddy.md) — `books.hubris.network → 192.168.8.247:6060`
- [Authentik (124)](106-auth-outpost.md) — OIDC provider `Grimmory`
- [DNS (107)](107-dns.md) — `books.hubris.network → 192.168.8.175` (unchanged from Booklore)
- [Media permissions](../infrastructure/media-permissions.md)
## Changelog
### 2026-06-29 — provisioned; Booklore migrated
LXC 130 created (Debian 13, privileged, `192.168.8.213`). Docker installed. Grimmory compose deployed at `/opt/grimmory/`. MariaDB dump from Booklore (LXC 105) restored — schema-compatible since Grimmory is a direct fork. Caddy `books.hubris.network` backend updated from `192.168.8.205:6060` to `192.168.8.213:6060`. Authentik provider updated: Booklore → Grimmory, Confidential → Public (PKCE). Booklore stack removed from Portainer on LXC 105.

62
archive/knowledge/containers/131-teddycloud.md Normal file
View File

@@ -0,0 +1,62 @@
# 131 — `teddycloud`
Self-hosted [TeddyCloud](https://github.com/toniebox-reverse-engineering/teddycloud), a
reimplementation of the Toniebox cloud backend — lets Tonie figurines play custom/ripped
audio content against a local server instead of the official cloud.
Predates the client-enrollment convention entirely; nobody wrote it down. Found and
documented on 2026-07-06 after Oikos's drift detector (`oikos/drift.py`) flagged
`pve_id 131` as live on hubris (via `pct list`) with no `inventory.yaml` entry — see
[OIKOS.md](../../../.agents/OIKOS.md)'s Week 3 build-status note. `containers/132-rclone.md` had already
mentioned it in passing ("LXC 131 was already taken by an undocumented `teddycloud`
container"), and `hosts/strong.md`'s 2026-07-05 migration changelog fixed a DHCP conflict
for it — but it never got its own inventory entry or doc page until now.
## At a glance
- **Hostname:** `teddycloud`
- **Host:** hubris (confirmed via `pct config 131` run directly on hubris — the original
drift finding's `pct list` source)
- **IP:** `192.168.8.150` (static; was briefly `192.168.8.243` via DHCP until the
2026-07-05 strong-migration work assigned it a fixed address — see hosts/strong.md)
- **Privilege:** unconfirmed (not checked — read-only investigation didn't need it)
- **Resources:** 1 core / 1 GiB RAM / 512 MiB swap / 16 GiB rootfs (`local-lvm`), Debian 12
(bookworm)
- **Mounts:** `/mnt/library` (`mp0`)
- **Public hostname:** `teddy.hubris.network` → `192.168.8.150:8443` (see
[caddy (121)](121-caddy.md))
- **Enrollment:** none — no `age_pubkey`, not a `homelab` CLI client. It's a plain
docker-compose app container, not a fleet-managed host. No action needed unless it starts
needing secrets.
## Service
Runs via `docker compose` at `/opt/teddycloud` — container `teddycloud-teddycloud-1`,
image `ghcr.io/toniebox-reverse-engineering/teddycloud:latest`, publishing `80`, `443`,
`8080`, and `8443`. Caddy routes `teddy.hubris.network` to the `:8443` port.
## Risk notes
**No Caddy forward-auth gate** — unlike `sab.hubris.network` on the same Caddyfile (which
is explicitly annotated `(Authentik forward-auth)`), `teddy.hubris.network` has no auth
annotation. It's reachable to anyone on the LAN/mesh who can resolve the hostname. Not
addressed as part of this doc pass — flagging it here since it's now visible in one place
for the first time.
## Related
- [Hubris host](../hosts/hubris.md)
- [Caddy (121)](121-caddy.md) — terminates `teddy.hubris.network`
- [rclone (132)](132-rclone.md) — landed on pve_id 132 specifically because 131 was already
taken by this container
- [Containers index](index.md)
- [OIKOS.md](../../../.agents/OIKOS.md) — drift detector that caught this
## Changelog
### 2026-07-06 — documented for the first time (drift-caught)
Added to `inventory.yaml` and given this page. Verified live via read-only `pct config 131`
+ `pct exec 131 -- ...` on hubris: hostname, IP, resources, and that it runs via
`docker compose` (not a raw binary or systemd unit). No changes made to the running
container — this is pure documentation catch-up.

199
archive/knowledge/containers/132-rclone.md Normal file
View File

@@ -0,0 +1,199 @@
# 132 — `rclone`
Off-host backup appliance. Mirrors selected `/mnt/library` folders to **Proton Drive**
with a plain `rclone sync` (monthly), and serves rclone's Web GUI on the LAN for browsing
and ad-hoc runs. **Replaces** the disabled restic-on-USB job — see [backups](../infrastructure/backups.md).
Provisioned 2026-07-01. (LXC 131 was already taken by an undocumented `teddycloud` container,
so this landed on **132**.)
## At a glance
- **Hostname:** `rclone`
- **IP:** `192.168.8.214` (static, set in PVE `net0` config — same pattern as grimmory/authentik)
- **Privilege:** privileged (root in-container = host root → reads every `/mnt/library` subtree,
incl. `homecloud/` and `documents/`, regardless of owner)
- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs (Debian 13) — bumped from 1 GiB on 2026-07-03
after `rclone-rcd.service` was OOM-killed under real load (see "Known issue" below)
- **Mounts:** `/mnt/library` **read-only** (`mp0: /mnt/library,mp=/mnt/library,ro=1`) — a backup
job must never be able to write into the library
- **Public hostname:** none — the UI is **LAN-only, no auth** (by design)
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| rclone Web GUI (`rcd`) | `192.168.8.214:5572` | `rclone-rcd.service`, **`--rc-no-auth`**, LAN-only. Browse `/mnt/library` + `proton:`, run ad-hoc syncs, live job status |
| monthly mirror | — | `rclone-backup.service` + `.timer` (`OnCalendar=*-*-01 03:00`) |
## Backup design
- **Mode:** plain mirror — `rclone sync` (Proton mirrors local; deletions propagate; **no versioning**).
- **Encryption:** Proton Drive's built-in E2E only (no rclone `crypt` overlay → files stay
browsable in Proton's web UI).
- **Selected set:** `/etc/rclone-backup/folders.list` — one absolute source path per line
(`#`/blank ignored). This file *is* the picked set the monthly timer mirrors. Extensible to other
disks once bind-mounted into this LXC.
- **Path mapping:** source `S` → `proton:library-backup/<S without leading slash>`
(e.g. `/mnt/library/notes` → `proton:library-backup/mnt/library/notes`).
- **Runner:** `/usr/local/sbin/rclone-backup.sh [folder ...]` (Python, despite the `.sh` name — kept
the path stable) — no arg = every enabled line. Submits each folder as an **async job through the
rclone rc API** served by `rclone-rcd.service` (the same daemon backing the Web GUI on `:5572`),
so scheduled/ad-hoc runs show up live in the GUI's **Jobs panel**, not just in logs. Gentle on
Proton's rate limits (`Transfers=4, TPSLimit=8, FastList=true` via the rc `_config` payload). The
rc API here requires **POST for every call** including `job/status` and `core/stats` — GET with
query params 404s.
- **Logs / "past runs":** per-run logs in `/var/log/rclone-backup/<safe>-<ts>.log`; one-line
JSON summary per run appended to `/var/log/rclone-backup/runs.jsonl`.
- **Failure notify:** `OnFailure=rclone-backup-notify@%n.service` → logs to journal today;
**TODO** wire to Hermes `send_message` (Matrix) per [backups](../infrastructure/backups.md).
## rclone + Proton Drive
- **rclone** installed from the official binary (not apt) so the `protondrive` backend is present
(`rclone v1.74.3`).
- Remote **`proton:`** (type `protondrive`). Config at `/root/.config/rclone/rclone.conf`, mode 600.
**This file is a secret** (holds the obscured Proton password + TOTP secret + session) — **never
commit it.** Escrow the Proton account creds in the password manager.
- **Config gotchas** (from rclone docs/forum):
- Log into Proton via a **browser at least once** first, or key generation fails.
- For unattended runs, store the **TOTP _secret_** (not a 6-digit code) so rclone self-generates
codes; obscure with `rclone obscure`.
- Passwords with **extended-ASCII** characters are known to break auth.
- Proton's API is rate-limited → keep `--transfers`/`--tpslimit` conservative (baked into the runner).
- **DR escrow (pending):** store the Proton creds as sops secret `secrets/protondrive.yaml`, granted
to this LXC's age key, so the remote can be rebuilt after a re-provision.
## The UI (rclone Web GUI)
`rclone rcd --rc-web-gui --rc-no-auth --rc-addr 0.0.0.0:5572` (assets auto-downloaded on first
start). Reach it at **http://192.168.8.214:5572** on the LAN.
> **Security note:** `--rc-no-auth` exposes *full* rclone control — including deleting remote data —
> to anyone on the LAN (accepted per the design choice). The container has only a LAN NIC, so it is
> not publicly reachable. Harden later by adding `--rc-user/--rc-pass` or fronting it with Authentik.
## Tracked config (deferred)
**Not yet tracked.** The runner, systemd units, and `folders.list` currently live as plain files
directly on the LXC — fully functional, just not version-controlled or auto-deployed. A
`dtoro/rclone` gitea repo package (runner, units, `install.sh`, webhook receiver) is pre-built and
staged at `/root/rclone-repo` on the LXC for whenever this gets tracked (Shape A, like
[caddy](121-caddy.md)). Gitea `ALLOWED_HOST_LIST` already includes `192.168.8.214` in anticipation.
See [auto-deploy](../infrastructure/auto-deploy.md).
**Selected folders (live in `/etc/rclone-backup/folders.list`):** `/mnt/library/cloud` (287G),
`/mnt/library/documents` (249M), `/mnt/library/repos` (83M). `/mnt/library/notes` was synced once as
a connectivity test (not in the recurring set). Proton quota checked: 2 TiB plan, ~1.65 TiB free
after this set.
## Enrollment gotcha: `pct exec` PATH
`pct exec` (lxc-attach) does **not** source `/etc/environment` or run a login shell, so
`/usr/local/bin` (where bootstrap installs `sops`) isn't on `$PATH` by default — bootstrap's own
`command -v sops` post-install check failed under `pct exec` even though the binary installed fine.
Fixed by symlinking `/usr/local/bin/{sops,homelab}` into `/usr/bin` (always on the minimal PATH),
rather than relying on `/etc/environment`. Same category as the documented [`pct exec` no-initgroups
gotcha](../infrastructure/media-permissions.md#gotchas) — worth adding to
[agent-enrollment.md troubleshooting](../../../.agents/operations/agent-enrollment.md#troubleshooting) if it recurs
on future LXC bootstraps.
## Known issue: `rclone-rcd.service` OOM-killed under 1 GiB RAM (root cause, resolved)
What looked like repeated "protondrive silently stalls" was actually **`rclone-rcd.service` (the rc
API daemon backing the Web GUI and, since the rc-API redesign, all actual sync work) getting
OOM-killed** under the original 1 GiB RAM allocation — `journalctl` confirms
`A process of this unit has been killed by the OOM killer` at the exact moment a transfer had
"frozen." systemd's own `Restart=on-failure` (5s) auto-respawns it, but every in-flight job's state
is lost on the kill, which looked identical to a silent backend hang from the outside (frozen
`core/stats`, no new log lines). **Fix: bumped the LXC's memory to 2 GiB** (`pct set 132 -memory
2048` — applies live via the host cgroup, confirmed via `cat /sys/fs/cgroup/lxc/132/memory.max` on
hubris, no container reboot needed). After the bump, the full folder set (`cloud` 287G, `documents`
249M, `repos` 83M) completed cleanly with no further kills.
**`rclone-backup-watchdog.timer`** (every 5 min) → `rclone-backup-watchdog.sh`: if
`rclone-backup.service` is active but total transferred bytes (global `core/stats` on the rc API)
haven't moved for 15 minutes, it restarts both `rclone-rcd.service` (clears any stuck/orphaned job —
this is the actual daemon holding the work, not the thin wrapper) and then `rclone-backup.service`
(`--no-block`, load-bearing — see below). Kept as a safety net even after the RAM fix, in case
memory pressure returns under a larger folder set later. State kept in
`/var/lib/rclone-backup/watchdog-state.json`, cleared whenever the service isn't running.
**Two watchdog design bugs found and fixed while chasing this (2026-07-03):**
1. **Wrong stats-group key.** Per-job progress polling queried `core/stats` under `job/<jobid>`,
but rclone tracks stats under whatever `_group` name the job was submitted with. Made a perfectly
healthy sync look stalled at 0 bytes for 22+ hours in its own log. Fixed by using the same
`group` variable consistently. **Lesson: distrust the per-run log's "progress bytes=" line during
an incident; cross-check with unfiltered `core/stats` first.**
2. **Watchdog restarted only the thin wrapper, and blocked doing it.** The actual `rclone sync` work
runs inside `rclone-rcd.service`, not `rclone-backup.service` — restarting the wrapper alone left
any stuck job orphaned inside `rcd` while a new wrapper submitted a duplicate job on top. Worse,
`systemctl restart rclone-backup.service` (no `--no-block`) blocks until the *new* invocation's
long-running `ExecStart` exits — which could be hours — so the watchdog's own oneshot service
never logged "Finished," and `OnUnitActiveSec` (which schedules relative to the previous run
*finishing*) never fired again. The watchdog silently disabled itself after exactly one use.
Fixed: restart `rclone-rcd.service` first, then `rclone-backup.service` with `--no-block`.
## Related
- [Backups](../infrastructure/backups.md) — this job supersedes the disabled restic-on-USB backup
- [Hubris host](../hosts/hubris.md) — owns `/mnt/library`
- [Media permissions](../infrastructure/media-permissions.md) — read-only consumer of `/mnt/library`
- [Containers index](index.md)
## Changelog
### 2026-07-03 — root cause found (OOM, not Proton); RAM bumped to 2 GiB; full folder set completed
What looked like repeated silent "protondrive stalls" turned out to be **`rclone-rcd.service`
getting OOM-killed** under the original 1 GiB RAM allocation — confirmed via
`journalctl -u rclone-rcd.service` showing `killed by the OOM killer` at the exact freeze point.
Bumped the LXC to 2 GiB RAM (live, no reboot). After the bump: `cloud` (287G) completed cleanly
(exit 0), `documents` (249M) completed with 1 minor error (259.7 MB transferred), `repos` (83M)
completing as of this entry. Also fixed two real watchdog bugs found while chasing this (wrong
stats-group key making a healthy sync look frozen; watchdog restarting only the wrapper with a
blocking `systemctl restart`, causing it to silently disable itself after one use) — see "Known
issue" above for full detail. The watchdog is kept as a safety net going forward even though the RAM
bump addresses the actual root cause.
### 2026-07-02 — runner rewritten to submit jobs via the rc API (GUI job visibility)
The original runner (`rclone sync` invoked as a standalone CLI subprocess) was invisible to the Web
GUI's Jobs panel — the GUI only tracks work submitted through its own `rcd` process. Rewrote
`/usr/local/sbin/rclone-backup.sh` in Python, submitting each folder via `POST /sync/sync` with
`_async: true` against `http://127.0.0.1:5572` (the running `rclone-rcd.service`), then polling
`POST /job/status` + `POST /core/stats` (both **must be POST** — GET-with-querystring 404s on this
rc API) until finished, logging periodic progress snapshots and the same `runs.jsonl` summary line
as before. Verified live: submitted job visible in `POST /job/list`'s `runningIds` while running,
completed cleanly (`success: true`) once done. Deployed via atomic rename (write-then-`mv`) rather
than truncating in place, specifically so it wouldn't risk corrupting the still-running original
`cloud`+`documents`+`repos` sync mid-flight (verified after the fact: that sync's bash process was
unaffected, kept running to completion under the old in-memory script content). The already-running
scheduled sync from before this change is a standalone process and won't retroactively appear in the
GUI; every run after this point will.
### 2026-07-02 — Proton Drive auth fixed; real folder set enabled; first live sync
Initial `rclone config` failed 2FA (`422 ... auth/v4/2fa`) because a live 6-digit TOTP code was
entered instead of the TOTP secret — reconfigured with the secret, auth now works
(`rclone lsd proton:` lists the Drive). Verified end-to-end with a real sync of `/mnt/library/notes`
(219 objects, 5.964 MiB, exit 0) — confirmed files land as plain, browsable objects on Proton (not
an opaque archive), matching the plain-mirror + Proton-E2E design. Checked Proton quota (2 TiB
plan, 1.945 TiB free) before enabling a large folder. `folders.list` set to the real selection:
`cloud` (287G), `documents` (249M), `repos` (83M); a full sync of that set was kicked off via the
actual `rclone-backup.service` unit (not an ad-hoc call) to validate the real monthly path early
rather than waiting for the Aug 1 timer. Tracked-repo step (`dtoro/rclone` on gitea) deferred by
choice — runner/units/`folders.list` remain plain files on the LXC for now; the repo package stays
staged at `/root/rclone-repo` for later.
### 2026-07-01 — provisioned; enrolled
LXC 132 created (Debian 13, privileged, `192.168.8.214`, `/mnt/library` read-only). rclone v1.74.3
installed from the official binary (`protondrive` backend present). Runner + monthly timer +
`folders.list` deployed; rclone Web GUI (`rcd`, LAN-only no-auth) live on `:5572`. Enrolled into
homelab-context (`--no-mesh`, LAN-only issuance): age key issued, inventory finalized, shared
secrets granted, `homelab whoami` + `homelab secret hello` verified. Gitea `ALLOWED_HOST_LIST`
updated to include `192.168.8.214`. Hit and fixed a `pct exec` PATH gotcha (see below). Proton Drive
remote, `dtoro/rclone` tracked repo + webhook, and the `secrets/protondrive.yaml` escrow remain
operator-run follow-ups (credentialed steps — Proton password/2FA, repo creation). Restic-on-USB
backup deprecated in the same change.

61
archive/knowledge/containers/133-seanime.md Normal file
View File

@@ -0,0 +1,61 @@
# seanime (LXC 133) — Seanime anime media server
## Summary
| Field | Value |
|-------|-------|
| VMID | 133 |
| Host | strong |
| Role | Anime media server (online streaming + torrent client) |
| LAN IP | 192.168.8.248/28 (vmbr1) |
| Public URL | https://seanime.hubris.network |
| Backend | Seanime v3.9.0, systemd service |
| Port | 43211 |
## Mounts
- `/anime` → `/mnt/media_local/anime` (ludo-lvm, bind mount) — existing anime collection
## Services
### Seanime (native binary)
- Binary: `/opt/seanime/bin/seanime`
- Data: `/opt/seanime/data/`
- Config: `/opt/seanime/data/config.toml`
- Service: `systemctl status seanime`
- Flags: `--host 0.0.0.0 --port 43211 --datadir /opt/seanime/data --disable-password`
- Config highlights:
- `secureMode = 'lax'` (allows non-local Caddy proxy)
- `trustedProxies = ['192.168.8.0/24']`
### Torrent client (qBittorrent on arriman)
- Host: `192.168.8.245:8080`
- Auth: subnet whitelist (no password needed from homelab LAN)
- qBittorrent config updated: `AuthSubnetWhitelist=192.168.8.0/24`
### Extensions installed
**Online streaming (8):**
HiAnime, AniWatch, KickAssAnime, Anicrush, Animo, AniNeko, Senshi, Sudatchi
## Caddy
- Config: `seanime.hubris.network` block in `/etc/caddy/Caddyfile` (dtoro/caddy-conf repo)
- TLS: Let's Encrypt DNS-01 via IONOS API
- Backend: `192.168.8.248:43211`
## DNS
- `seanime.hubris.network` A record → `192.168.8.175` (Caddy)
## Changelog
### 2026-07-05 — initial creation
- LXC 133 created on strong (Debian 13, 2 cores / 2 GiB)
- Seanime v3.9.0 installed as native binary + systemd service
- /anime bind mount from ludo-lvm
- qBittorrent on arriman configured as torrent client
- Caddy reverse proxy + DNS records set up
- Online streaming extensions installed

82
archive/knowledge/containers/134-romm.md Normal file
View File

@@ -0,0 +1,82 @@
# 134 — `romm`
Self-hosted ROM manager ([RomM](https://romm.app)). Browse, search, and play
your retro game library from the browser. Runs on Docker Compose with a
MariaDB sidecar.
## At a glance
- **Hostname:** `romm`
- **IP:** `192.168.8.249/28` (static, vmbr1 on strong)
- **Privilege:** privileged
- **Resources:** 1 core / 2 GiB RAM / 16 GiB rootfs (Debian 13, ludo-lvm)
- **Mounts:** `/mnt/media_local` → `/mnt/library`
- **Public hostname:** `roms.hubris.network`
## Service / port map
| Service | Listen | Notes |
|---------|--------|-------|
| RomM | `192.168.8.249:80` | HTTP (Caddy terminates TLS) |
| MariaDB | internal only | Sidecar in the same compose stack |
## Compose
Located at `/opt/romm/docker-compose.yml`. Key points:
- Image: `rommapp/romm:latest`
- DB sidecar: `mariadb:latest` with healthcheck
- ROM library: `/mnt/library/roms` → `/romm/library` (writable)
- Resources (covers, etc.): Docker named volume `romm_resources` → `/romm/resources`
- Saves/states: `/opt/romm/assets` → `/romm/assets`
- Config: `/opt/romm/config` → `/romm/config`
- Auth key: auto-generated, stored in `/opt/romm/.env`
Environment (`/opt/romm/.env`):
- `DB_ROOT_PASSWD` / `DB_PASSWD` — MariaDB credentials
- `DB_USER=romm-user` / `DB_NAME=romm`
- `ROMM_AUTH_SECRET_KEY` — auto-generated
## ROM library structure
RomM expects `/mnt/library/roms/<platform>/<game>/<rom>`. Create platform
directories as needed:
```
/mnt/media_local/roms/
├── gba/
│ └── Pokemon - Emerald/
│ └── Pokemon Emerald.gba
├── snes/
│ └── Super Mario World/
│ └── Super Mario World.sfc
└── psx/
└── Final Fantasy VII/
└── Final Fantasy VII.bin
```
## Media permissions
The `/mnt/media_local/roms` directory is owned `root:media` with mode `2775`
(setgid). New files inherit the `media` GID (10000). The LXC is privileged so
no idmap block is needed — in-container UID/GID matches the host. Docker
containers within the LXC run as-is (read-only mount).
## Related
- [Strong host](../hosts/strong.md)
- [Caddy (121)](121-caddy.md) — `roms.hubris.network → 192.168.8.249:80`
- [DNS (107)](107-dns.md) — `roms.hubris.network A 192.168.8.175`
- [Media permissions](../infrastructure/media-permissions.md)
- [RomM docs](https://docs.romm.app)
## Changelog
### 2026-07-05 — provisioned
LXC 134 created on strong (Debian 13, privileged, `192.168.8.249/28`).
Docker + Compose installed. RomM stack deployed at `/opt/romm/`.
Created `/mnt/media_local/roms` with setgid `media:GID=10000` mode `2775`.
Caddy `roms.hubris.network` → `192.168.8.249:80`.
DNS `roms A 192.168.8.175` added to Technitium.
Hubris /32 route for `.249` added to `50-strong-route`.

57
archive/knowledge/containers/index.md Normal file
View File

@@ -0,0 +1,57 @@
# LXC containers — index
Most containers live on [`hubris`](../hosts/hubris.md). Some have been
[migrated to `strong`](../hosts/strong.md) (Phase 1+2, 2026-07-05).
| ID | Name | Host | IP | Priv | Cores | RAM | Disk | Mounts | Public hostname | Status |
| --- | ---------------- | ------- | --------------- | ---- | ----- | ----- | ----- | --------------------- | ------------------------------------- | -------- |
| 101 | [jellyfin](101-jellyfin.md) | **strong** | 192.168.8.246 | priv | 4 | 8 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `media.hubris.network` | running |
| 103 | [paperless](103-paperless.md) | hubris | 192.168.8.130 | priv | 2 | 3 GiB | 8 GiB | `/mnt/library` | `paperless.hubris.network` | running |
| 104 | [gitea](104-gitea.md) | hubris | 192.168.8.121 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` | `git.hubris.network` | running |
| 105 | [apps](105-apps.md) | hubris | 192.168.8.205 | priv | 2 | 4 GiB | 30 GiB | `/mnt/library` | `docker` / `artifacto` / `blog` | running |
| 114 | [nextcloud](114-nextcloud.md) | hubris | 192.168.8.224 | priv | 4 | 6 GiB | 25 GiB | `/mnt/library` | `cloud.hubris.network` | running |
| 118 | [elementsynapse](118-elementsynapse.md) | **strong** | 192.168.8.242 | unpriv | 2 | 4 GiB | 32 GiB | — | `matrix.hubris.network` | running |
| 119 | [sophia](119-sophia.md) | hubris | 192.168.8.157 | priv | 2 | 1 GiB | 10 GiB | `/mnt/library` | — | running |
| 120 | [mule-images](120-mule-images.md) | hubris | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU) | `photos.hubris.network` | running |
| 121 | [caddy](121-caddy.md) | hubris | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
| 122 | [arriman](122-arriman.md) | **strong** | 192.168.8.245 | priv | 4 | 8 GiB | 24 GiB | `/mnt/media_local` (via mp0) | `jellyseerr` / `qbit` / `sab` | running |
| 124 | [authentik](106-auth-outpost.md) | hubris | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
| 128 | [trmnl](128-trmnl.md) | hubris | 192.168.8.211 | unpriv | 1 | 768 MiB | 8 GiB | — | `trmnl.hubris.network` | running |
| 129 | [house](129-house.md) | **strong** | 192.168.8.244 | unpriv | 2 | 3 GiB | 8 GiB | — | `house.hubris.network` | running |
| 130 | [grimmory](130-grimmory.md) | **strong** | 192.168.8.247 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `books.hubris.network` | running |
| 131 | [teddycloud](131-teddycloud.md) | hubris | 192.168.8.150 | — | 1 | 1 GiB | 16 GiB | `/mnt/library` | `teddy.hubris.network` (no auth gate) | running |
|| 132 | [rclone](132-rclone.md) | hubris | 192.168.8.214 | priv | 1 | 1 GiB | 8 GiB | `/mnt/library` (**ro**) | — (LAN-only UI `:5572`) | running |
|| 134 | [romm](134-romm.md) | **strong** | 192.168.8.249 | priv | 1 | 2 GiB | 16 GiB | `/mnt/media_local` (via mp0) | `roms.hubris.network` | running |
## Recently destroyed (kept for archaeology)
| ID | Name | Destroyed | Reason |
| --- | ---------------- | --------------- | --------------------------------------------- |
| 127 | mule-photos-new | 2026-05-22 | PhotoPrism + sidecar + SvelteKit stack promoted to LXC 120 via Mulimage 2.0 merge (`70dc1b6`); M0 test LXC retired. Caddy + dnsmasq + gitea webhook + NC webhook listeners all cleaned up in the same cutover. |
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
| 126 | plato | 2026-06-28 | Notes/discovery workspace decommissioned; data at `/mnt/library/documents/plato` retained for archaeology |
| 123 | claudio-bot (destroyed — see [archive](archive/123-claudio-bot.md)) | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../../../plans/done/2026-06-04_130000-deprecate-claudio-bot.md) |
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 110 | photoprism | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 111 | karakeep | between 2026-04-21 and 2026-04-28 | Decommissioned |
| 112 | immich | between 2026-04-21 and 2026-04-28 | Replaced by [mulita](120-mule-images.md) |
| 115 | reticulum | between 2026-04-21 and 2026-04-28 | Decommissioned |
> Several `.conf.bak` files survive under `/etc/pve/lxc/` if you need to recover any of the configs.
## Conventions
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](106-auth-outpost.md) which is statically `192.168.8.180/24`. Containers on [strong](../hosts/strong.md) use `bridge=vmbr1` with static IPs in the `192.168.8.240/28` range.
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library` on hubris, or `mp0: /mnt/media_local,mp=/mnt/library` on strong.
- Most containers are privileged. Unprivileged ones require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
## Related
- [Hubris host](../hosts/hubris.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Caddy](121-caddy.md) — terminates every public hostname
- [DNS](../infrastructure/dns.md) — split-horizon entries for each subdomain

162
archive/knowledge/hosts/hubris.md Normal file
View File

@@ -0,0 +1,162 @@
# `hubris` — Proxmox host
Proxmox VE host running 1 VM and 13 LXC containers — the whole homelab's
workloads still live here. As of 2026-07-01, hubris is node 1 of the 2-node
`Homelab` cluster (see [Cluster](#cluster)); the second node is
[strong](strong.md), which hosts nothing yet.
## At a glance
- **Role:** Proxmox VE 9.1.2 hypervisor (kernel `6.14.11-4-pve`)
- **Hardware:** GMKtec NucBox M6 Ultra — AMD Ryzen 5 7640HS (Phoenix APU), 12 vCPU / ~28 GiB RAM, 2× Samsung 990 EVO Plus NVMe (one SSD primary, one for `library` LVM). 2× Realtek RTL8125 NICs (`r8169`).
- **BIOS:** 1.02 (2025-08-06) — vendor not on LVFS, no automated update path. See [investigations](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
- **Uplink:** `vmbr1` (slave: `eno1`) → SODOLA switch → Fritz!Box 7590. DHCP-reserved `192.168.178.10/24`, gateway `192.168.178.1`.
- **Homelab bridge:** `vmbr0` — portless internal bridge, `192.168.8.77/24` + `192.168.8.1/24` alias (LXC default gateway). All 16 LXCs and the HAOS VM are on `vmbr0`. Proxmox routes between `vmbr0` and `vmbr1`; Fritz!Box has a static route `192.168.8.0/24 → 192.168.178.10`.
- **WiFi:** disabled 2026-06-02 — `wlp3s0` removed from `/etc/network/interfaces`, wpa config deleted. Was used as a failover to the now-retired Slate AX AP.
- **Mesh:** Netbird `wt0` `100.122.38.109/16`. Resolver: `100.122.38.109` (the local netbird daemon, which forwards to LAN/upstream and learns `*.hubris.network` answers via that path). See [mesh](../infrastructure/mesh.md).
- **UI:** `https://proxmox.hubris.network` (via [caddy](../containers/121-caddy.md)) or `https://192.168.8.77:8006`.
## Storage
| Pool | Type | Size | Use |
| ------------ | --------- | ----- | -------------------------------------- |
| `local` | dir | ~95G | ISOs, templates, `/etc`, configs |
| `local-lvm` | lvmthin | 856G | LXC/VM rootfs |
| `library` | lvmthin | 3.7T | Backs `/mnt/library` ext4 (mounted as `/dev/mapper/library-library`) |
`/mnt/library` holds the shared media + data pool: `anime`, `audiobooks`, `books`, `comics`, `documents`, `downloads`, `heaper`, `homecloud`, `images`, `marimo`, `movies`, `music`, `notes`, `podcasts`, `repos`, `roms`, `sophia`. Bind-mounted into every container that needs it. Permissions standard: [media GID 10000](../infrastructure/media-permissions.md).
## Cluster
Member of `Homelab`, a 2-node Proxmox cluster with [strong](strong.md)
(cluster/OS hostname `strong`), formed 2026-07-01.
- **Corosync ring0:** hubris's internal `192.168.8.77` (the `vmbr0` address).
strong reaches it via the existing Fritz!Box static route
(`192.168.8.0/24 → 192.168.178.10`) — no dedicated corosync link, just the
household LAN. Fine for a home cluster; not latency-isolated.
- **Quorum:** 2 nodes, 1 vote each, no QDevice tiebreaker. Quorum needs both
votes — if either node is down (reboot, maintenance, network hiccup), the
survivor's running guests keep working but `/etc/pve` goes read-only:
no start/stop/create/edit until quorum returns. Decided to skip a QDevice
for now; revisit if hubris's periodic reboots (BIOS/thermal work, see
Quirks below) make this painful in practice.
- **Storage:** `local` / `local-lvm` are the standard per-node default IDs
(every node has its own, not actually shared). The `library` lvmthin pool
is explicitly restricted to `nodes hubris` in `/etc/pve/storage.cfg` since
it's a physical thinpool that only exists on this host's hardware.
- strong currently hosts no LXCs/VMs — it exists solely as a cluster
member so far. See [strong.md](strong.md) and the [library-SSD
migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)
for what comes next (physical drive move, service migration — not started).
## Tenants
### VMs
- [108 — `haos-16.3`](../vms/108-haos.md) — Home Assistant OS, 4 GiB / 32 GiB
### LXC containers
See [containers/index](../containers/index.md). 10 active on hubris (101, 118, 122, 129, 130 migrated to [strong](strong.md) 2026-07-05).
## Boot-time tuning (load-bearing)
- **TCP BBR + `fq` qdisc** in `/etc/sysctl.d/99-bbr.conf`: `net.ipv4.tcp_congestion_control=bbr`, `net.core.default_qdisc=fq`, socket buffers up to 64 MiB (`rmem_max`/`wmem_max`/`tcp_rmem`/`tcp_wmem`). `tcp_bbr` module pinned to load at boot via `/etc/modules-load.d/bbr.conf`. Improves single-TCP-stream throughput on jittery / lossy links (notably WiFi clients downloading from [114 — nextcloud](../containers/114-nextcloud.md)). Container traffic crosses this kernel, so every guest inherits the change.
- **`cpu-epp.service`** (enabled) sets `scaling_governor=powersave` + EPP=`balance_power` at boot — drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle. In `amd-pstate=active` mode the governor MUST be `powersave`, not `performance`, for EPP to apply. Unit ordering: `After=sysinit.target` + `Before=pve-guests.service` so it runs before the guest fleet starts (the original `After=multi-user.target` left the hottest boot window on `performance`). Fixed 2026-04-22.
- **Crash capture:** `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`). Pstore traces collected to `/var/lib/systemd/pstore/` by `systemd-pstore.service`. **Caveat:** silent CPU lockups leave pstore empty.
- **`rasdaemon`** (Debian pkg) collects MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query with `ras-mc-ctl --summary` / `--errors`. (mcelog is retired in Debian 13 — don't go looking for it.)
## Network performance baseline (2026-05-14)
| Path | Throughput | Notes |
| ---- | ---------- | ----- |
| WAN (Ookla, from host) | ↓ 113.5 Mbit/s · ↑ 19.9 Mbit/s · 24.6 ms | Asymmetric residential link. 20 Mbit upload is the cap for public ingress (artifacto, blog via VPS traefik). |
| `eno1` link | 1000 Mbit/s, Full duplex | Negotiated correctly to router. |
| Host → LXC over `vmbr0` | **34.7 Gbit/s** | Single TCP stream, iperf3 10 s, 0 retransmits. |
| LXC → LXC over `vmbr0` | **34.8 Gbit/s** | Single TCP stream, iperf3 5 s, 0 retransmits. |
Intra-host throughput is memory-bandwidth bound — not a bottleneck for any current workload.
**eno1 error counters** (uptime 14 d 6 h at measurement time): `tx_packets` 99.99 M / `rx_packets` 96.08 M; `rx_errors` 1,619,203 (~1.7 % of RX); `align_errors` 10,919; `tx_errors` 20. The 1.6 M rx_errors in 14 days is the noteworthy datapoint — likely cable / switch-port. Re-check this counter in ~1 week and confirm it is not still climbing at the same rate; if it is, swap the patch cable first.
## Host services owned by external repos
| What | Repo | Path on host |
| --------------------------------- | -------------------------- | ----------------------------- |
| `claudio-monitor` (5-min watchdog) | `dtoro/claudio-monitor` | `/opt/claudio-monitor` |
| `backup-library` (restic) | `dtoro/backup-library` | `/opt/backup-library` *(disabled)* |
| `hubris-public-cert-sync` | (script + systemd unit) | `/usr/local/bin/hubris-public-cert-sync.sh`, daily timer |
See [monitoring](../infrastructure/monitoring.md), [backups](../infrastructure/backups.md), [ingress](../infrastructure/ingress.md).
## Quirks
- `/etc/pve` is fuse — the Proxmox cluster filesystem, now genuinely cluster-synced (2-node) rather than the single-node-but-still-fuse case this note used to describe.
- ZFS is **not** in use; storage is LVM-thin + ext4.
- Two Realtek 8125 NICs use the in-tree `r8169` driver, not the OOT `r8125`.
- Hardware is thermally marginal. NVMe sensors live near warn temp under load. Thermal pads installed on the SSDs 2026-04-23; host relocated to a better-ventilated spot 2026-04-29.
- Non-ECC RAM: silent memory faults are possible; suspect DIMMs if `cpu-epp` is on and crashes still happen.
- BIOS update path requires a Windows-To-Go USB (no LVFS, no Linux flasher).
## Authorized SSH keys (root)
- `root@hubris` (self, RSA) — local
- `d.toro.v@pm.me` (ed25519) — user's iMac, added 2026-04-22
- `root@strong` (RSA) — strong's cluster-join key, added 2026-07-01 so
`pvecm add` could authenticate without a password prompt
OpenSSH on `0.0.0.0:22`. Netbird's built-in SSH server is on `100.122.38.109:22022` and bypasses `authorized_keys` (OIDC/browser). See [SSH access](../infrastructure/ssh-access.md) for the dual-server gotcha.
## Related
- [Containers index](../containers/index.md)
- [DNS split-horizon](../infrastructure/dns.md)
- [Caddy reverse proxy](../containers/121-caddy.md)
- [Mesh / Netbird](../infrastructure/mesh.md)
- [Media permissions](../infrastructure/media-permissions.md)
- [Monitoring](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
- [strong — Proxmox host](strong.md)
## Changelog
### 2026-07-01 — strong joined as a 2nd cluster node ("Homelab")
User reformatted `strong` (formerly a Linux dev workstation, `192.168.178.181`) to Proxmox VE 9.2.3. Cluster/OS hostname on that box is `strong` (left as-is from install). Bootstrapped root SSH on strong from a one-time console password (installed hubris's existing trusted key set: `root@hubris`, `d.toro.v@pm.me`), then generated a keypair on strong and pre-authorized it here (`root@strong`) so `pvecm add 192.168.8.77 --use_ssh 1` (run from strong) could join without an interactive password prompt. No cabling/routing changes needed — strong reaches hubris's corosync address (`192.168.8.77`) via the existing Fritz!Box static route. Cluster now 2 nodes, quorate, **no QDevice** (explicit choice — see [Cluster](#cluster) above for the quorum tradeoff this implies). strong hosts no guests yet; this is Phase 1 of the [library-SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md), nothing further from that plan has been executed.
### 2026-06-02 — Slate AX retired; SODOLA switch added; network restructured
Replaced GL.iNet Slate AX sub-router with SODOLA 5-Port 2.5Gbit managed switch. Fritz!OS 8.x lacks second-IP-network support on LAN ports, so Proxmox now acts as the subnet router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10/24`; `vmbr0` is a portless internal bridge holding all LXCs/VMs with `192.168.8.1` as an alias (unchanged LXC gateway). Fritz!Box static route `192.168.8.0/24 → 192.168.178.10` enables inbound routing. No LXC configs changed. Eliminated double-NAT. WiFi (`wlp3s0`) also removed — was pointing at the Slate AX SSID, no longer useful. See [network](../infrastructure/network.md) and [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-05-14 — LXC 109 (syncthing) decommissioned
User destroyed the syncthing LXC (had been stopped since 2026-04-21, never re-enabled). `pct destroy 109 --purge` cleaned `vm-109-disk-0` on `local-lvm` and the `/etc/pve/lxc/109.conf` entry. Data subtree `/mnt/library/syncthing` was already empty and retained as an empty dir. No DNS, Caddy, NFS-export, or claudio-monitor references to clean up. Entry moved to the "recently destroyed" table in [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology); references stripped from [README](../../../README.md), [media-permissions](../infrastructure/media-permissions.md), [vms/100-zimaos](../vms/100-zimaos.md), and [containers/102-nfs-export](../containers/102-nfs-export.md).
### 2026-05-14 — network performance baseline captured
First explicit speed snapshot: WAN ↓113.5 / ↑19.9 Mbit (24.6 ms), `eno1` 1 Gb full-duplex negotiated, intra-host `vmbr0` ~34.7 Gbit/s host↔LXC and ~34.8 Gbit/s LXC↔LXC (single TCP stream, zero retransmits). `iperf3` + `speedtest-cli` installed on host. Noted `eno1` `rx_errors` at 1.62 M (~1.7 % of 96 M RX packets in 14 d uptime) plus 10.9 k `align_errors` — flagged for follow-up; expect to recheck the trend in ~1 week, suspect patch cable / switch port first if still climbing. See new "Network performance baseline" section above.
### 2026-05-14 — LXC 116 (heaper) decommissioned
User destroyed the heaper LXC. No `116.conf.bak` left behind in `/etc/pve/lxc/`. Data subtree `/mnt/library/heaper` (224 MiB) retained for archaeology — see [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology) and [media-permissions](../infrastructure/media-permissions.md).
### 2026-05-13 — TCP BBR + fq qdisc enabled
`/etc/sysctl.d/99-bbr.conf` switches `net.ipv4.tcp_congestion_control` from `cubic` to `bbr` and `net.core.default_qdisc` from `fq_codel` to `fq`. Also bumps `rmem_max`/`wmem_max` to 64 MiB and widens `tcp_rmem`/`tcp_wmem`. `tcp_bbr` module pinned at boot via `/etc/modules-load.d/bbr.conf`. Triggered by Nextcloud client downloads from a WiFi laptop pulling ~2 MB/s despite a 152 Mbps link — server-side baseline through Caddy with BBR is ~400 MB/s single-stream loopback, so any client-perceived single-stream improvement is pure congestion-control win. Touches every LXC's outbound TCP since they all share this kernel.
### 2026-04-29 — relocated to better-ventilated spot
User physically moved the host to a new location with improved airflow. Post-move idle baseline (45 min uptime, light load): k10temp Tctl **47.2 °C**, amdgpu edge 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C, ACPI zone 47–49 °C. Compares well against the 2026-04-23 thermal-pad steady-state (nvme0 sensor1 60–61 °C). Watch the lifetime NVMe warning-time counter over the coming days for confirmation. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-29-physical-relocation).
### 2026-04-28 — Phase 1 WiFi failover
Host now dual-homed: LAN `192.168.8.77` (primary) + WiFi `192.168.8.141` (failover, metric 200) on the GL-AXT1800-714-5G AP. Installed `wpasupplicant`+`iw`; added `wlp3s0` stanza to `/etc/network/interfaces` with `wpa-conf`; ARP isolation sysctls in `post-up`. Built `wan-failover.service` to remove the vmbr0 default route on `eno1` carrier loss, since the bridge's carrier doesn't follow `eno1` (the LXC veths keep it `1`). LXC/VM guests are still LAN-only — Phase 2 will migrate them.
### 2026-04-28 — wiki started
This wiki created. Live state at this date: 14 LXCs running (109 syncthing stopped), 1 VM, kernel `6.14.11-4-pve`, uptime 3 d 0 h post drive-removal A/B test. Compared to memory snapshot from a week ago, **destroyed**: LXC 100 (yunohost arr), 106 (flaresolverr), 107 (marimo), 110 (photoprism), 111 (karakeep), 112 (immich), 115 (reticulum). 100 + 106 destroyed per the planned 2026-04-21 \*arr migration retention; the others removed since.
### 2026-04-23 — SSD cooling + thermal pads installed
Thermal pads on both NVMe drives. Steady-state nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes after install. Watch the lifetime warning-time counter going forward, not absolute sensor1. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md#2026-04-23-thermal-pad-verdict).
### 2026-04-22 — drive removal A/B test
Removed external USB backup drive (Silicon Motion `090c:2320`). Disabled the four `backup-library*.timer` units, commented the fstab entry. Goal: confirm whether the drive + UAS interaction on the AMD USB4 PCIe tunnel is the dominant root cause of the silent hard-locks. Pre-drive uptime was 33 days; with drive, repeated crashes despite UAS blacklist + mount-on-demand. **Result so far:** 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders thermal protection. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
### 2026-04-22 — `cpu-epp.service` ordering bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`, so the hottest boot window (20+ guests starting on `performance`) preceded EPP application. Now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-21 — crash-capture + RAS telemetry enabled
`60-crash-capture.conf`, softdog `soft_panic=1`, RuntimeWatchdog 15 s. `rasdaemon` installed and enabled. Pure silicon hangs still leave no trace; this catches everything else.
### 2026-04-21 — `cpu-epp.service` deployed
Pinned governor=`powersave`, EPP=`balance_power` at boot. Stopped the host idling at ~95 °C with everything pinned at 4.4 GHz. First fix in the [crash-loop incident](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).

9
archive/knowledge/hosts/index.md Normal file
View File

@@ -0,0 +1,9 @@
# Hosts
The two Proxmox VE nodes that run the fleet. Narrative pages; the machine-readable host records are
the generated `hosts/*.yaml` files at the repo root.
| Document | What it covers |
|----------|----------------|
| [hubris.md](hubris.md) | Primary PVE node (GMKtec NucBox M6 Ultra, `192.168.8.77`) — runs most LXCs plus the VMs. |
| [strong.md](strong.md) | Secondary PVE node / cluster member (`192.168.178.181`) — hosts the LXCs migrated from hubris. |

159
archive/knowledge/hosts/strong.md Normal file
View File

@@ -0,0 +1,159 @@
# `strong` — Proxmox host
Second node in the `Homelab` cluster, alongside [hubris](hubris.md). Formerly
a Linux dev workstation nicknamed "ludo" (or "ludo-mini") — reformatted to
Proxmox VE on 2026-07-01. No LXCs/VMs deployed on it yet.
## At a glance
- **Role:** Proxmox VE 9.2.3 cluster member (kernel `7.0.12-1-pve`) — hosts [7 LXCs](../containers/index.md) migrated from hubris (Phase 1+2, 2026-07-05)
- **Naming:** the OS/cluster hostname is `strong` (leftover from install,
kept as-is). This wiki page and `inventory.yaml` also use `strong` as of
2026-07-01 — earlier the same day the inventory entry was briefly named
`ludo-mini`, but that was renamed so the hostname bootstrap needs
(`hosts/$(hostname).yaml`) would just resolve. "Ludo"/"ludo-mini" remains
the machine's everyday nickname; some older docs (investigations, the
library-SSD migration plan) still refer to it that way — that's fine,
those are historical.
- **Hardware:** AMD Ryzen 7 PRO 6850U, 16 threads, 28 GiB RAM.
- `nvme0n1` — MasonSemi MC3100 1TB (boot/OS): `pve-root` 96G, `pve-data`
(thinpool) 815G, 8G swap.
- `nvme1n1` — WD_BLACK SN7100 2TB, added 2026-07-01. Arrived with an
existing APFS partition (previously used in a Mac) — wiped via
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1`, then turned
into its own LVM-thin pool via `pvesh create nodes/strong/disks/lvmthin`
(handles pvcreate/vgcreate/lvcreate + storage.cfg registration in one
step). Registered as Proxmox storage `ludo-lvm` (VG `ludo-lvm`, thinpool
`ludo-lvm`, ~1.8 TiB usable — kept the `ludo-lvm` name since that's what
was actually created live; not worth renaming), restricted `nodes strong`
in `/etc/pve/storage.cfg` — same pattern as hubris's `library` pool.
Empty so far; this is separate from the [library-SSD migration
plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md)'s
planned drive move from hubris (that hasn't happened) — this is
general-purpose VM/CT capacity.
- **Network:** `vmbr0` is bridged straight onto the household LAN —
`192.168.178.181/24`, gateway `192.168.178.1`. Unlike hubris, there is no
internal/uplink bridge split yet — `vmbr0` here plays the role hubris's
`vmbr1` plays there. Reachable from the homelab subnet (`192.168.8.0/24`)
via the existing Fritz!Box static route through hubris; no new cabling
or routing was needed to join the cluster.
- **Storage:** `local` + `local-lvm` (cluster-wide default names, shared
with hubris by convention, actually separate per-node volumes). Hubris's
`library` lvmthin pool is restricted to `nodes hubris` in
`/etc/pve/storage.cfg` and does not exist here.
- **Mesh:** Netbird not installed — fresh OS wiped whatever the old
workstation had. Reachable today only via LAN routing (confirmed DNS for
`*.hubris.network` already resolves correctly here via `192.168.8.2`).
Add to Netbird if off-LAN access to this host itself (distinct from any
future guests) is needed.
- **Homelab-context client enrollment:** done 2026-07-01 via
`bootstrap.sh --no-secrets` (reused the operator's existing Gitea PAT for
the initial clone). `/opt/homelab-context`, the `homelab` CLI, and the
5-min sync timer are live; `homelab whoami` resolves correctly. See
[agent-enrollment.md](../../../.agents/operations/agent-enrollment.md).
- **Age key / secrets:** issued the same day over plain LAN (no Netbird
needed — see the `--no-mesh` bootstrap.sh fix below). Key lives at
`/etc/age/key.txt`; pubkey `age1rtwvdct6avjkr3cyxv3vue3vqx4d524fjfr3vk7xrnvyrylnry5sm54sn4`
recorded in `inventory.yaml`. Not yet a recipient on any actual secret
(`hello.yaml`, `gitea-pat.yaml`, etc.) — that's a separate grant, see
["Granting a secret to a new client"](../../../.agents/operations/agent-enrollment.md#granting-a-secret-to-a-new-client).
## Cluster membership
Joined hubris's single-node cluster (`Homelab`) via `pvecm add` on
2026-07-01. See [hosts/hubris.md#cluster](hubris.md#cluster) for the full
cluster picture, node IDs, and the quorum tradeoff (2 nodes, no QDevice —
either node going down freezes management on the survivor).
## SSH
Root login via the same key set trusted on hubris (`root@hubris`,
`d.toro.v@pm.me`) — installed 2026-07-01 by appending to
`/root/.ssh/authorized_keys` (now symlinked to `/etc/pve/priv/authorized_keys`
post cluster-join, so it's cluster-synced same as hubris). No password auth
needed going forward.
## Related
- [hubris — Proxmox host](hubris.md)
- [Library SSD migration plan](../../../.hermes/plans/2026-06-03_110000-library-ssd-migration-to-ludo-mini.md) — the larger project this is Phase 1 of (filename kept as-is, historical)
- [Network](../infrastructure/network.md)
- [SSH access](../infrastructure/ssh-access.md)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
## Changelog
### 2026-07-05 — Phase 2d: grimmory (130) migrated to strong
Migrated book library from hubris (192.168.8.213) to strong (192.168.8.247).
Rsync'd /books (2.6G) from hubris library SSD to ludo-lvm/media. Container
mounts /mnt/media_local → /mnt/library locally. Caddy backend for
books.hubris.network updated.
### 2026-07-05 — Phase 2: arriman (122) + jellyfin (101) migrated; library on ludo-lvm
Migrated arriman (→ 192.168.8.245) and jellyfin (→ 192.168.8.246) to strong.
Created 1.5T thin volume on ludo-lvm for media library (/mnt/media_local).
Rsync'd 363G of media data (movies, tv, anime, downloads, music) from hubris
library SSD to ludo-lvm. Both containers now mount /mnt/media_local directly
as local ext4 — no NFS cross-node dependency. Jellyfin gets Radeon 680M iGPU
(for VAAPI) + RX 7600 dGPU passthrough via dev0/dev1. Caddy backends updated
for media, jellyseerr, qbit, sab. Hubris freed 16 GiB RAM (8+8).
Dhcp scope narrowed to 192.168.8.100-239.
### 2026-07-05 — house (LXC 129) migrated to strong; DHCP scope narrowed
Migrated Yuvomi family planner from hubris (192.168.8.212) to strong
(192.168.8.244). Followed same restore pattern as elementsynapse (Phase 1b).
Discovered IP conflict: teddycloud (LXC 131) had 192.168.8.243 via DHCP
(scope was 192.168.8.241-254). Narrowed DHCP scope to 192.168.8.100-239,
gave teddycloud static IP 192.168.8.150. Caddy updated: house.hubris.network
→ 192.168.8.244:3000, teddy.hubris.network → 192.168.8.150:8443.
### 2026-07-05 — elementsynapse (LXC 118) migrated to strong
Migrated Matrix/Synapse + Element client from hubris (192.168.8.239) to a new
LXC on strong (192.168.8.242). Added vmbr1 on strong at 192.168.8.241/28
(portless internal bridge) for strong-hosted guests. Enabled IP forwarding,
proxy ARP on hubris vmbr0, and specific /32 routes for strong's guest subnet.
Caddy updated: element.hubris.network → 192.168.8.242:8080.
VPS traefik updated: matrix.hubris.network backend → 192.168.8.242:8008.
This is Phase 1a of the strong migration plan — see .hermes/plans/2026-07-05_strong-migration-assessment.md.
### 2026-07-01 — age key issued over LAN; 3 bugs found/fixed in bootstrap.sh
Re-ran bootstrap without `--no-secrets` to get a real age key. Hit three real bugs live, fixed all three in `bootstrap.sh` and re-ran clean:
1. The `mcp`-CLI pipx-install step and the (unused, `--with-hermes`-only) Goose installer both called `sudo -u <user>` unconditionally — fails with "sudo: command not found" on a minimal root-only image with no `sudo` binary at all. Added a `run_as()` helper that only shells out to `sudo` when there's a real distinct invoking user.
2. `sops` isn't an apt/dnf package (matches what `agent-enrollment.md`'s manual-install recipe already does) — the auto-installer tried `apt-get install sops` and failed outright. Added `install_sops_binary()`, fetching the GitHub release binary directly on both dnf and apt paths.
3. Bigger one: running without `--no-secrets` unconditionally tries to install + interactively connect Netbird (device-code SSO), even though the very next check already knows how to accept plain LAN reachability instead. Over SSH with nobody watching, this hangs forever — had to manually kill a stuck `netbird up` process. Added `--no-mesh`, which skips the Netbird install/connect step but keeps the LAN-fallback path for secrets issuance. This run used `bootstrap.sh --no-mesh` and completed cleanly: `mesh: lan`, age key installed, `mcp` CLI installed via pipx (proving fix #1 too).
Result: age key at `/etc/age/key.txt`, pubkey recorded in `inventory.yaml`. Not yet granted access to any actual secret file — see the note above.
### 2026-07-01 — enrolled as a homelab-context client
Ran `bootstrap.sh --no-secrets` (reused the operator's existing personal Gitea PAT for the initial clone rather than minting a fresh read-only one). Installed git, cloned `/opt/homelab-context`, installed the 5-min systemd sync timer, symlinked `homelab` CLI and `AGENTS.md`. Skipped age-key/secrets issuance and Netbird per operator choice — but bootstrap's own connectivity check reported `mesh: lan`, i.e. the secrets-issuance endpoint is already reachable over plain LAN, so re-running without `--no-secrets` later wouldn't require a Netbird join. Known gap: the `mcp` pipx CLI install step silently failed (`sudo: command not found` — bootstrap.sh's pipx step assumes a `sudo` binary even when already root; harmless, only affects the `homelab mcp <tool>` shell subcommand).
### 2026-07-01 — inventory identity renamed ludo-mini → strong
Discovered while starting client enrollment: `bootstrap.sh` looks up
`hosts/$(hostname).yaml`, and the OS hostname here is `strong`, not
`ludo-mini`. Renaming the OS hostname was ruled out (already a cluster
member — Proxmox doesn't support in-place node rename, only leave+rejoin).
Renamed the wiki/inventory side instead: `inventory.yaml` key, this page
(`hosts/ludo-mini.md` → `hosts/strong.md`), README, ssh-access.md all now
say `strong`. "Ludo"/"ludo-mini" is still fine as a spoken nickname.
### 2026-07-01 — 2nd NVMe added; new LVM-thin pool `ludo-lvm`
User added a WD_BLACK SN7100 2TB (`nvme1n1`), previously used in a Mac
(arrived with an EFI + APFS partition table — confirmed disposable, wiped).
Used Proxmox's own disk-management API rather than raw LVM commands:
`pvesh set nodes/strong/disks/wipedisk --disk /dev/nvme1n1` to clear the old
partition table/signatures, then `pvesh create nodes/strong/disks/lvmthin
--name ludo-lvm --device /dev/disk/by-id/nvme-WD_BLACK_SN7100_2TB_251663803202
--add_storage 1` to create the PV/VG/thinpool and register it as Proxmox
storage in one step. Result: storage ID `ludo-lvm`, ~1.8 TiB, `content
rootdir,images`, `nodes strong` (mirrors hubris's `library` node-restriction
pattern — this pool only physically exists here). Empty — no VM/CT disks
placed on it yet.
### 2026-07-01 — Proxmox install; joined Homelab cluster
Reformatted from Linux workstation to Proxmox VE 9.2.3. SSH keys seeded from
hubris's trusted set (root password used once, then discarded). Joined the
existing `Homelab` cluster via `pvecm add 192.168.8.77 --use_ssh 1` from
this node's side, using key-based SSH pre-authorized in both directions —
no interactive password prompt needed for the join itself. Cluster now 2
nodes (`hubris`, `strong`), quorate, no QDevice. Decided to leave hostname
as `strong` and skip a QDevice for now — both revisitable later.

14
archive/knowledge/index.md Normal file
View File

@@ -0,0 +1,14 @@
# Knowledge
The durable, authoritative current-state documentation of the homelab: one page per node and per
cross-cutting system, synthesized from live state and evidence. Structure and rules are in
[the knowledge schema](../.agents/domains/knowledge/schema.md).
| Section | What it covers |
|---------|----------------|
| [wiki/hosts/](wiki/hosts/index.md) | Proxmox host narratives — `hubris`, `strong`. |
| [wiki/containers/](wiki/containers/index.md) | LXC fleet — one page per container, plus the master table and archaeology. |
| [wiki/vms/](wiki/vms/index.md) | Virtual machines — ZimaOS, Home Assistant OS. |
| [wiki/infrastructure/](wiki/infrastructure/index.md) | Cross-cutting systems — DNS, ingress, mesh, storage, auth, monitoring, generated topology. |
| [sources/](sources/index.md) | External reference docs and the pointer to incident evidence. |
| [GLOSSARY.md](GLOSSARY.md) | Term definitions. |

149
archive/knowledge/infrastructure/auto-deploy.md Normal file
View File

@@ -0,0 +1,149 @@
# Auto-deploy — gitea-webhook pipelines
Several configs and apps in the lab live in `dtoro/*` repos on [gitea (104)](../containers/104-gitea.md) and auto-redeploy on push. All pipelines follow one of two shapes.
## Two shapes
### Shape A — checkout IS the working tree (config repos)
`/etc/<thing>` or `/var/lib/<thing>/...` is itself a `git clone`. Push triggers `git pull` + a reload command. Used for pure-config repos where re-cloning is cheap.
### Shape B — receiver outside the app repo (compose stacks)
The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`webhook.py`, `deploy.sh`, systemd unit) lives in a sibling `/opt/<thing>-deploy/` so the app repo stays portable. Push triggers `git pull` + `docker compose up -d --build`. Returns 202 immediately and runs the build in a daemon thread because docker builds exceed gitea's request timeout.
## Common
- All receivers validate `X-Gitea-Signature` HMAC-SHA256 against a per-pipeline secret in `/etc/<thing>-deploy/secret`.
- All filter to `refs/heads/main` (or `master` for older repos). Gitea's "test delivery" button sends `ref=main` (without `refs/heads/`) — those will log "ignoring ref main" and 204. Real pushes work. **Don't "fix" the ref filter to accept both** — it'd also accept PR merges from side branches that got fast-forwarded.
- Gitea's `app.ini` `[webhook] ALLOWED_HOST_LIST` must include every receiver IP. Currently:
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
- ~~`192.168.8.190` ([plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology))~~ (destroyed 2026-06-28)
- `192.168.8.211` ([trmnl (128)](../containers/128-trmnl.md) — terminalito)
**Don't strip these when editing app.ini.**
- Git creds for root-run deploy services live in `/etc/<thing>-deploy/git-credentials` (mode 600) and are wired via `credential.helper = store --file=/etc/<thing>-deploy/git-credentials` in the repo's `.git/config`. Necessary because the unit typically runs with `ProtectHome=true`, which blocks `/root`.
## Pipelines
| Repo | Target | Shape | Receiver | Webhook id | Reload action |
| ------------------------------- | -------------------------------------------- | ----- | ------------------------------------- | ---------- | ------------- |
| `dtoro/caddy-conf` | [caddy (121)](../containers/121-caddy.md) `/etc/caddy/` | A | `http://192.168.8.175:9797/deploy` | 2 | `caddy validate` + `systemctl reload caddy` |
| `dtoro/gitea-customizations` | [gitea (104)](../containers/104-gitea.md) `/var/lib/gitea/custom/` | A | `http://127.0.0.1:9797/deploy` (loopback) | (orig) | `systemctl restart gitea` if templates changed |
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
| ~~`dtoro/Plato`~~ | ~~[plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology) `/opt/plato/app/`~~ (destroyed 2026-06-28) | ⊘ | `http://192.168.8.190:9799/deploy` (dead) | 8 (removed) | Repo archived — LXC destroyed |
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 (deprecated) | ~~reinstalls `homelab-mcp.service` + restart~~ → replaced by Go Docker stack on mac-mini |
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 (deprecated) | ~~reinstalls `secrets-issuance.service` + restart~~ → replaced by `internal/secrets/` Go package |
| `dtoro/terminalito` | [trmnl (128)](../containers/128-trmnl.md) `/opt/terminalito/` | B | `http://192.168.8.211:9797/deploy` | 12 | reinstalls units + `systemctl restart trmnl-plugins` |
| `dtoro/Homelab-Docs` → oikos-console | [apps (105)](../containers/105-apps.md) `/opt/oikos-console/` | B | `http://192.168.8.205:9831/deploy` | 14 | reinstalls `oikos-console.service` + restart — see [oikos/console/deploy/README.md](../../../oikos/console/deploy/README.md) |
> Note: `dtoro/Homelab-Docs` has **three webhooks** firing on the same push.
> Each owns its own clone on LXC 105. They don't conflict because each
> deploy.sh only touches its own service unit + venv.
> **Not yet wired:** `dtoro/claudio-monitor` (push, then `/opt/claudio-monitor/scripts/deploy.sh` manually). The former authentik LXC (124) is destroyed — Authentik runs on the [VPS](../../../hosts/netbird-vps.yaml). DNS moved to [Technitium on dns (107)](../containers/107-dns.md).
## When you change a tracked config
Always commit + push. Local-only edits drift. Common ones:
- `/etc/caddy/Caddyfile` ↔ `dtoro/caddy-conf` (auto-deploys)
- `/var/lib/gitea/custom/` ↔ `dtoro/gitea-customizations` (auto-deploys)
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
- ~~`/opt/plato/app/` ↔ `dtoro/Plato`~~ (destroyed 2026-06-28)
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
## Per-pipeline notes / gotchas
### caddy-conf
- Repo includes `scripts/webhook/install.sh`. Editing the systemd unit *inside the repo* does **not** auto-reinstall — re-run `install.sh` manually after unit edits.
- The unit has `ReadWritePaths=/etc/caddy` — load-bearing (`ProtectSystem=full` would otherwise block `git pull`).
### gitea-customizations
- Receiver is on **loopback** (`127.0.0.1:9797`), not the LXC IP.
- Online3DViewer binary assets are NOT tracked; `deploy.sh` fetches them on first run.
### mule-image / Artifacto
- Async deploy (returns 202) — gitea would otherwise time out the request. Logs: `pct exec <id> -- journalctl -u <thing>-deploy-webhook -f`.
- **Cloning from inside the LXC must use the internal gitea IP** (`http://192.168.8.121:3000/...`). `https://git.hubris.network` hits a connection reset from inside [apps (105)](../containers/105-apps.md) (Caddy routing / TLS hairpin not configured for this LXC). Configured `origin` on the in-LXC checkout is the internal URL.
- Manual deploy: `pct exec <id> -- /opt/<thing>-deploy/deploy.sh`.
- Health: `pct exec <id> -- curl -s http://127.0.0.1:<port>/health` → `ok`.
- Setup tokens used to register the webhook (e.g., `artifacto-deploy-setup`, `artifacto-deploy-setup-2`, `artifacto-cleanup` on user `dtoro`) need manual revocation in the Gitea UI → Settings → Applications → Manage Access Tokens. Gitea's `/users/{u}/tokens` endpoints require basic auth (not bearer), so cleanup couldn't be automated.
### backup-library
- Currently the only deploy that targets the host directly (`192.168.8.77:9798`).
- `deploy.sh` is careful to preserve admin edits to `/etc/restic/include-*.list` — canonical source is `config/` in the repo, but the install path is treated as authoritative once `deploy.sh` has run.
### homelab-mcp / secrets-issuance
- Both ride a single push to `dtoro/Homelab-Docs`. Two clones on LXC 105
(`/opt/homelab-mcp`, `/opt/secrets-issuance`) — each is an independent
Shape-B target with its own webhook receiver.
- The deploy script restarts the service it just updated. Because the
webhook receiver itself is a separate systemd unit (`*-deploy.service`),
it does NOT restart itself — but `deploy.sh` running `systemctl
restart homelab-mcp-deploy.service` (or the secrets-issuance one)
would create a kill-self loop. The current `deploy.sh` is careful
to only restart the main service.
- Both services consume `/opt/homelab-context` for their runtime data
(inventory, secret recipient lookup). That clone is **the same clone
every other client has** — kept fresh by `homelab-context-sync.timer`,
not by these webhooks.
## Custom-built binaries that overlap apt-managed paths
If a pipeline (or any out-of-band build) drops a binary into a path that an apt package also owns — most commonly `/usr/bin/<name>` — then the next `apt upgrade` of the corresponding package will silently clobber the custom build. That's exactly how [LXC 121 caddy](../containers/121-caddy.md) went down for ~10 min on 2026-05-21: an xcaddy build with `caddy-dns/ionos` lived at `/usr/bin/caddy` and Debian's caddy 2.11.2→2.11.3 apt upgrade replaced it with a vanilla 2.11.3 that couldn't parse the Caddyfile.
Two patterns are acceptable, pick one when authoring a pipeline that ships a non-apt binary:
1. **Ship the build as a `.deb` with an epoch-bumped version.** Use `dpkg-deb --build` (or `nfpm`) to package the binary as `Package: <name>`, `Version: 1:<upstream>-hubris<n>`. The epoch (`1:`) means it beats any non-epoch upstream version regardless of point bumps, so `apt upgrade` is a no-op for that package. Used by caddy: `caddy 1:2.11.3-hubris1` (see commit `2026-05-21` in [121-caddy.md](../containers/121-caddy.md)).
2. **Hold the apt package.** `apt-mark hold <pkg>` on the LXC during pipeline install; apt will refuse to upgrade it. Simpler than `.deb` packaging but: (a) the hold flag isn't preserved by `dpkg -i` of a new version, (b) it's invisible unless you check `apt-mark showhold`, (c) you have to remember to `apt-mark unhold` when you intentionally want a new version. The new `homelab apt-audit` subcommand surfaces holds across the fleet so they don't get forgotten.
If you're not sure what's already lurking, run `homelab apt-audit --fleet` and look at the `NONAPT` column — that's a count of binaries in `/usr/bin/{caddy,docker,jellyfin}` + `/usr/local/bin/*` that no apt package owns.
## Related
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
- [Backups (disabled)](backups.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
## Changelog
### 2026-06-28 — Plato pipeline decommissioned
LXC 126 destroyed, webhook id 8 on `dtoro/Plato` removed. `192.168.8.190` removed from gitea `app.ini` `ALLOWED_HOST_LIST`.
### 2026-06-24 — terminalito pipeline added
Webhook id 12 on `dtoro/terminalito` → `http://192.168.8.211:9797/deploy` on [trmnl (128)](../containers/128-trmnl.md). Shape B (`server/deploy/webhook.py` receiver, in-repo `server/deploy/deploy.sh`; secret `/etc/terminalito-deploy/secret`). `app.ini` `ALLOWED_HOST_LIST` extended with `192.168.8.211`. Verified end-to-end with a push. Repo-local `credential.helper` in `/opt/terminalito/.git/config` (the unit can't read root's global git config).
### 2026-05-20 — homelab-mcp + secrets-issuance pipelines added
Webhook ids 10 + 11 on `dtoro/Homelab-Docs` (ports `9811` + `9821` on [apps (105)](../containers/105-apps.md)). Two webhooks on one repo — each owns its own clone (`/opt/homelab-mcp`, `/opt/secrets-issuance`) and only restarts its own service. See [homelab-context](homelab-context.md) for why both services live in one repo.
### 2026-05-13 — Plato pipeline added
Webhook id 8 on `dtoro/Plato` (port `9799` on [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology)). `app.ini` `ALLOWED_HOST_LIST` extended to include `192.168.8.190`.
### 2026-04-28 — wiki entry created
Initial documentation. Six active pipelines.
### 2026-04-22 — Artifacto pipeline added
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
### 2026-06-04 — claudio-bot pipeline decommissioned
LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
### 2026-04-21 — mule-image + claudio-bot pipelines added
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
### 2026-04-20 — caddy-conf + gitea-customizations + backup-library pipelines shipped
Initial three. Set the conventions everything else follows.

142
archive/knowledge/infrastructure/backups.md Normal file
View File

@@ -0,0 +1,142 @@
# Backups — restic on external drive (DEPRECATED — superseded)
> **DEPRECATED 2026-07-01.** Superseded by the **rclone → Proton Drive** off-host mirror on
> [LXC 132 `rclone`](../containers/132-rclone.md). That job finally closes the off-host / 3-2-1 gap
> this page flagged for months. The restic-on-USB job below is kept for archaeology; it has been
> **DISABLED since 2026-04-22** and is not coming back in its old form.
## Current backup — rclone → Proton Drive (LXC 132)
- **Where:** [LXC 132 `rclone`](../containers/132-rclone.md) (`192.168.8.214`), `/mnt/library`
mounted **read-only**.
- **What:** plain `rclone sync` (Proton mirrors local; browsable files, no versioning) of the
folders listed in `/etc/rclone-backup/folders.list`, to `proton:library-backup/…`.
- **When:** monthly — `rclone-backup.timer` (`OnCalendar=*-*-01 03:00`).
- **UI:** rclone Web GUI on `192.168.8.214:5572` (LAN-only, no auth).
- **Encryption:** Proton's built-in E2E (no rclone `crypt` overlay).
- **Runs / logs:** `/var/log/rclone-backup/` + `runs.jsonl`.
- **Still a single off-host target** (Proton only). Not yet a full 3-2-1 (no second independent
copy), but strictly better than the previous "no off-host copy at all."
See [132-rclone](../containers/132-rclone.md) for the full design.
---
## Legacy — restic on external drive (DISABLED 2026-04-22)
Chunked monthly restic backup of `/mnt/library`'s irreplaceable subset. **Disabled 2026-04-22** as part of the [hubris crash-loop A/B test](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
## Status
**DISABLED 2026-04-22.** All four timers `systemctl disable --now`'d:
- `backup-library@homecloud.timer`
- `backup-library@images.timer`
- `backup-library@small.timer`
- `backup-library-check.timer`
Fstab entry commented out. USB drive de-authorized and physically removed. `backup-library-deploy.service` left enabled (harmless webhook receiver).
**Reason:** the host hang recurred 2026-04-22 18:42 after 30h despite the `cpu-epp` fix, the UAS blacklist, and mount-on-demand. User wants to confirm host stability without the drive at all (was stable 33 days before the drive arrived). See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
**To re-enable:** uncomment fstab line, `systemctl enable --now` the four timers, re-attach drive.
## Design
Monthly rolling snapshots onto a 2 TB external USB drive. Chunked across the month so no single run pushes the Samsung 990 EVO Plus 4 TB (which backs `/mnt/library`) into thermal danger.
Retention per tag: `--keep-last 3 --keep-monthly 12 --keep-yearly 3` with `--group-by host,tags,paths`.
## Components
- **Repo:** `dtoro/backup-library`
- **Checkout:** `/opt/backup-library` on the [hubris host](../hosts/hubris.md)
- **Auto-deploys** via gitea webhook → `http://192.168.8.77:9798/deploy`. See [auto-deploy](auto-deploy.md). [Gitea (104)](../containers/104-gitea.md) `app.ini` `ALLOWED_HOST_LIST` includes `192.168.8.77` for this.
- **Restic repo:** `/mnt/backup/restic-library`. Passphrase `/etc/restic/passphrase` (mode 600). **Escrow in password manager — loss = permanent data loss.**
- **External drive:** `/dev/sda1` ext4 label `backup-library` UUID `ff46e775-1ba1-4892-82c9-e5cac5be933a`. Fstab uses `noauto` + `nofail,x-systemd.device-timeout=10s,errors=remount-ro`.
## Mount-on-demand
`/usr/local/sbin/backup-usb.sh attach|detach|status`. All three backup units (`backup-library.service`, `backup-library@.service`, `backup-library-check.service`) have `ExecStartPre=backup-usb.sh attach` and `ExecStopPost=backup-usb.sh detach`. The helper toggles `/sys/bus/usb/devices/*/authorized` by matching vendor:product `090c:2320`, then mounts/unmounts `/mnt/backup`. Drive is de-authorized when not backing up — no UAS keepalive, no kernel error-recovery paths firing against a flaky bridge.
## UAS blacklist
`/etc/modprobe.d/usb-storage-quirks.conf`:
```
options usb-storage quirks=090c:2320:u
```
Forces Bulk-Only Transport (BOT) instead of UAS for the SMI bridge. Confirm with `dmesg | grep "UAS is ignored"`.
## Schedule
Three timers, one per chunk, staggered ~10 days apart so each disk zone gets a long cooldown:
| Timer | When | Include list | Approx size |
| ---------------------------------- | -------------- | ------------------------------------ | ----------- |
| `backup-library@homecloud.timer` | day 1 / month | `/etc/restic/include-homecloud.list` | ~315 G |
| `backup-library@images.timer` | day 10 / month | `/etc/restic/include-images.list` | ~103 G |
| `backup-library@small.timer` | day 20 / month | `/etc/restic/include-small.list` (docs / books / music / notes / repos / marimo / heaper) | ~11 G |
Snapshots tagged `chunk-<group>` so forget/prune treats each series independently.
Ad-hoc full run (kept for manual use): `systemctl start backup-library.service` (no arg → uses `/etc/restic/include.list`, tag `monthly`).
Yearly integrity: `backup-library-check.timer` (`OnCalendar=yearly`) runs full `restic check --read-data`.
## Thermal caps
Baked into the systemd units:
- `IOReadBandwidthMax=/mnt/library 50M`
- `IOWriteBandwidthMax=/mnt/backup 30M`
- `--read-concurrency=1` on restic.
## Wrapper
`/usr/local/sbin/backup-library.sh` — preflight → unlock → backup → forget/prune (`--group-by host,tags,paths`) → `check --read-data-subset=5%` → notify. Takes optional `<group>` arg or `GROUP=` env.
## Notifications
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
## Recovery
Runbook at `/usr/share/doc/backup-library/RECOVERY.md` (or in the repo at `doc/RECOVERY.md`). Covers `restic snapshots/ls/find/restore/mount`, uid/gid gotcha, cross-host recovery.
## Known SPOF
Single drive. RECOVERY.md flags the 3-2-1 gap. Mitigations (second drive, cloud repo via `restic copy`) were not implemented before this job was retired — the **off-host copy is now provided by [rclone → Proton Drive (LXC 132)](../containers/132-rclone.md)** instead. A second independent copy is still outstanding.
## Drive history
The `Silicon Motion Portable SSD` (vid:pid `090c:2320`) drops under sustained heavy writes through a hub chain. Bypass all hubs / use a rear motherboard USB 3 port if attaching it again.
After it was first attached on 2026-04-19, hubris crashed twice in 2.5 days (46h then 12h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1. The UAS blacklist + mount-on-demand mitigations didn't fully eliminate it (recurrence 2026-04-22), prompting drive removal as the cleaner test. See [investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md).
## Thermal monitoring
Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213`). See [monitoring](monitoring.md).
## Related
- [Hubris host](../hosts/hubris.md)
- ~~[claudio-bot (123)](../containers/archive/123-claudio-bot.md)~~ (destroyed 2026-06-04)
- [Monitoring](monitoring.md)
- [Auto-deploy](auto-deploy.md)
- [Investigation: 2026-04-21 crash loop](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md)
## Changelog
### 2026-07-01 — DEPRECATED; superseded by rclone → Proton Drive (LXC 132)
Off-host backup moved to a plain `rclone sync` mirror on the new [LXC 132 `rclone`](../containers/132-rclone.md) (`/mnt/library` → Proton Drive, monthly, LAN Web GUI). This finally provides the off-host copy the "Known SPOF" note wanted. The restic-on-USB units on hubris remain `disabled` (drive already removed 2026-04-22); page restructured to lead with the current job and demote restic to "Legacy".
### 2026-04-28 — wiki entry created
Initial documentation. Status remains DISABLED.
### 2026-04-22 — DISABLED
Drive removed as the A/B test in the [crash investigation](../../sources/investigations/archive/2026-04-21-hubris-crash-loop.md). Timers disabled, fstab commented, drive de-authorized.
### 2026-04-21 — UAS blacklist + mount-on-demand shipped; root-caused host hangs to drive
Drive identified as the source of the hangs after hubris crashed twice in 2.5 days. UAS blacklist forces BOT; helper script toggles `/sys/bus/usb/.../authorized` so the drive is de-authorized when not backing up. Recovery drill (restore 188KB PDF + hash compare) had passed earlier. Bug fixed in `backup-library.sh`: `python3 -c '…' KEY=VAL` does NOT pass env vars — env-var prefix must precede the command. Caused false-failure even after successful backups.
### 2026-04-20 — deployed; redesigned for thermal-gentleness
Initial deploy. First backup attempt died at 18:56 (USB drive dropped off the bus during heavy writes); after re-plugging, restic resumed and completed at 21:21.

137
archive/knowledge/infrastructure/dns.md Normal file
View File

@@ -0,0 +1,137 @@
# DNS — split-horizon `*.hubris.network`
LAN clients resolve `*.hubris.network` to the [Caddy reverse proxy](../containers/121-caddy.md) (`192.168.8.175`). Public clients resolve to the IONOS VPS (`82.165.190.79`) via an IONOS wildcard, where they hit the [VPS traefik public ingress](ingress.md).
There is **no wildcard on the LAN side**. Every subdomain needs an explicit entry.
## Components
- **Authoritative public DNS:** IONOS. `*.hubris.network → 82.165.190.79` (was `74.118.126.4` until 2026-04-22).
- **LAN authoritative for `hubris.network` records:** [Technitium DNS](https://technitium.com) on [dns (107)](../containers/107-dns.md) at `192.168.8.2:53`. Syncs A records to the NetBird managed DNS zone via cron (see [dns-sync.py](../../../scripts/dns-sync.py)). Formerly dnsmasq on [authentik (124)](../containers/106-auth-outpost.md) (decommissioned 2026-06-04).
- **PVE host** (`192.168.8.77`): resolver is the local Netbird daemon at `100.122.38.109:53`, which forwards to the LAN/upstream and learns hubris.network answers via that path. `netbird status` says "Nameservers: 0/0 Available" — confirming netbird does NOT manage a hubris.network zone; it just caches whatever the system resolver returns.
- **Some LXCs** keep router DNS (`192.168.8.1`) or Tailscale MagicDNS (`100.100.100.100`), both of which return the public IONOS A record. Those LXCs need either a `/etc/hosts` override or local dnsmasq — see [mesh migration](mesh.md) for which technique applies where.
## Live entries (as of 2026-06-04)
```
address=/auth.hubris.network/82.165.190.79 # → VPS, not Caddy (Authentik migrated 2026-05-31)
address=/sso.hubris.network/192.168.8.175 # → Caddy → LAN forward-auth outpost (106); added 2026-06-01
address=/git.hubris.network/192.168.8.175
address=/media.hubris.network/192.168.8.175
address=/paperless.hubris.network/192.168.8.175
address=/books.hubris.network/192.168.8.175
address=/home.hubris.network/192.168.8.175
address=/cloud.hubris.network/192.168.8.175
address=/matrix.hubris.network/192.168.8.175
address=/proxmox.hubris.network/192.168.8.175
address=/docker.hubris.network/192.168.8.175
address=/jellyseerr.hubris.network/192.168.8.175
address=/qbit.hubris.network/192.168.8.175
address=/sab.hubris.network/192.168.8.175
address=/blog.hubris.network/192.168.8.175
address=/photos.hubris.network/192.168.8.175
address=/photos-new.hubris.network/192.168.8.175
address=/artifacto.hubris.network/192.168.8.175
address=/zimaos.hubris.network/192.168.8.175
address=/nfs-export.hubris.network/192.168.8.200
```
Note: `nfs-export.hubris.network` is the only `.hubris.network` entry that points to a non-HTTP service (NFSv4 on port 2049). It bypasses [caddy (121)](../containers/121-caddy.md) because NFS is L4, not HTTP — Caddy has nothing to do.
## Why split-horizon
The IONOS wildcard points at the VPS for public ingress (per-host routers in [VPS traefik](ingress.md)). The VPS only routes hostnames it knows — anything else 404s. So LAN clients pointing at the public IP are a dead end for any service that isn't explicitly published. The [Technitium DNS](dns.md) override on `192.168.8.2` keeps LAN traffic on the home Caddy.
## The gotcha that cost a debug session (2026-04-22)
Creating a new Caddyfile site block is necessary but **not sufficient**. Without the Technitium entry on [dns (107)](../containers/107-dns.md), LAN queries fall through to upstream, get the public IONOS answer, and time out. Symptom: "subdomain doesn't load" even though Caddy config + cert are fine.
## Recipe — adding a new subdomain
1. Edit `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md), commit + push to `dtoro/caddy-conf`. Webhook reloads caddy. See [auto-deploy](auto-deploy.md).
2. Add the A record in the [Technitium UI](http://192.168.8.2) at `dns (107)` — the NetBird managed DNS zone sync picks it up within ~10 minutes via cron. Or add directly to the NetBird managed zone via API if you need it faster.
3. Verify: `dig @192.168.8.2 +short <new>.hubris.network` → `192.168.8.175`.
4. On macOS clients, flush: `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder`.
> The Technitium config on LXC 107 is the single source of truth. Never hand-edit the NetBird managed zone directly — the [`scripts/dns-sync.py`](../../../scripts/dns-sync.py) cron on 107 reconciles them and reaps stale records. See [dns.md changelog 2026-06-03](#2026-06-03--single-authoring-source-technitium--netbird-managed-zone-sync).
## Public path — what does and doesn't follow the LAN map
- Hostnames published in [VPS traefik dynamic config](ingress.md) (currently `artifacto.hubris.network`, `blog.hubris.network`) reach a real backend over the netbird mesh.
- Anything else with a `*.hubris.network` URL hits the VPS but isn't routed anywhere — returns 404.
- `netbird.hubris.network` is its own thing — TCP passthrough at the VPS, served by netbird-proxy. Doesn't follow the file-provider router pattern.
## Long-term plan
Either:
- Move split-horizon DNS to the LAN router so `*.hubris.network → 192.168.8.175` is answered for every LAN client. Eliminates per-LXC overrides.
- Or, once the [Tailscale → Netbird migration](mesh.md) completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain.
## Related
- [Caddy (121)](../containers/121-caddy.md) — every LAN entry points here
- [Ingress (VPS traefik)](ingress.md) — public-side counterpart
- [Mesh migration](mesh.md) — per-LXC DNS workarounds during the transition
- [DNS server (107)](../containers/107-dns.md) — Technitium, current DNS authority
## Changelog
### 2026-06-28 — `plato.hubris.network` removed
Plato (LXC 126) decommissioned. Technitium entry deleted; dns-sync cron reaped the NetBird managed zone record.
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium; old limitation resolved
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs — the limitation noted below is resolved. Configured at Fritz!Box Internet → Filter → DNS Server → DNSv4 Server = `192.168.8.2` (User-defined).
Authentik LXC 124 (192.168.8.180) destroyed — Authentik runs on VPS, DNS on Technitium (107).
- Caddy: `auth.hubris.network`, `authentik` snippet, and `sso.hubris.network` all proxied to VPS
- `header_up Host auth.hubris.network` added to strip `:443` from upstream Host header
- All 9 LXCs' /etc/hosts updated: `auth.hubris.network → 192.168.8.175` (Caddy proxy)
- Inventory: removed `hosts.authentik`, renamed `dnsmasq` service → `dns`
- Docs: `124-authentik.md` deleted; dns.md references updated to Technitium (107)
The "delete NetBird managed zone → forward everything to Technitium" plan was **abandoned** — NetBird's DNS defeats it: it **won't apply a nameserver group that contains the peer's own mesh IP** (the Mac's `100.122.234.17` → `Nameservers: 0/0 Available`), and nameserver-group forwarding to Technitium never actually took effect for mesh peers (the **managed zone was doing all the real work**; disabling it broke all mesh resolution). So the model is now:
- **Technitium (`192.168.8.2`) is the single place you author DNS** (UI/API, MX/SPF/CAA, full zone).
- A **sync job on [dns (107)](../containers/107-dns.md)** (`/opt/dns-sync/sync.py`, cron */10) reconciles Technitium's named A-records → the **NetBird managed DNS zone** via the NetBird API (`/api/dns/zones/{id}/records`, PAT in sops `secrets/netbird-pat.yaml`). Mesh peers keep using the managed zone (which works); non-mesh LAN clients query Technitium directly; undefined names fall to the public IONOS wildcard (`.79`) — correct.
- This **killed the manual drift** that caused the whole `auth`/`sso`/`nfs-export` saga. Never hand-edit the NetBird managed zone again — edit Technitium; the sync propagates.
**Cleanup done same day:** removed the inert Mac-Mini Technitium secondary (mesh-only, served nobody); reverted the primary's `zoneTransfer=Allow`; fixed `home-lab-dns` group → `[192.168.8.2]` (dropped the self-referencing Mac IP → now `1/1 Available`); deleted the vestigial `Proxmox Names` group.
> Reference: [scripts/dns-sync.py](../../../scripts/dns-sync.py). The sync's source of truth is Technitium; it **deletes** NetBird records absent from Technitium (so obsolete names like `files`, `photos-new` get reaped).
### 2026-06-06 — dns-sync cron finally installed (had been dormant since 2026-06-04 deployment)
The `dns-sync.py` script on LXC 107 had been placed at `/opt/dns-sync/sync.py` on 2026-06-04 but **no crontab was configured** — the sync had never run automatically. The NetBird managed DNS zone was only in sync because manual runs happened during incident debugging.
**Fixed:** added `/etc/cron.d/dns-sync` (`*/10 * * * * root python3 /opt/dns-sync/sync.py >> /var/log/dns-sync.log 2>&1`).
Also added a Caddy backend health check cron on hubris (`/etc/cron.d/caddy-backend-health`) that runs `scripts/check-caddy-backends.sh` every 10 minutes.
### 2026-06-02 — 8 LXCs moved from DHCP to static IP
All LXCs that Caddy reverse-proxies to by IP were on `ip=dhcp` and could float on reboot (arriman got a different lease mid-session and broke). Fixed via `pct set` + in-LXC `/etc/network/interfaces`. Affected: 101 jellyfin, 103 paperless, 104 gitea, 105 apps, 114 nextcloud, 118 elementsynapse, 120 mule-images, 121 caddy, 122 arriman. See [arriman changelog](../containers/122-arriman.md#changelog).
### 2026-06-01 — dnsmasq replaced by Technitium on [dns (107)](../containers/107-dns.md); LXC 124 retired
Split-horizon DNS moved off [124](../containers/106-auth-outpost.md) to a dedicated **Technitium** LXC at **`192.168.8.2`** (zone: specific A overrides + wildcard→VPS + replicated MX/SPF/CAA). NetBird `home-lab-dns` nameserver group cut over to `192.168.8.2` (with `.180` as a now-dead fallback). dnsmasq stopped, all names verified via Technitium, **LXC 124 shut down**. **Caveat:** the [NetBird managed DNS zone](../containers/106-auth-outpost.md) still answers most app names *directly* (bypassing the nameserver group) — three overlapping DNS sources remain; see the single-source-of-truth decision (Phase 4). **Action needed:** update router DHCP DNS from the dead `.180` → `192.168.8.2` for any plain-LAN (non-mesh) clients.
### 2026-05-31 — `auth.hubris.network` re-pointed to the VPS (`82.165.190.79`)
Authentik migrated off LXC 124 onto the VPS (see [investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). The dnsmasq entry changed from `192.168.8.175` (home Caddy) to `82.165.190.79` (VPS traefik). This is the first LAN entry that intentionally points at the VPS rather than Caddy — `auth` is now a genuinely public service served directly from the VPS. **Gotcha logged:** the NetBird per-client resolver (`100.122.255.254`) caches dnsmasq answers and does **not** clear on `netbird down/up`; clients needed `/etc/hosts` overrides or `resolvectl flush-caches` to pick up the change. Since the service is now fully public, the long-term cleaner option is to drop the override entirely and let it fall through to the IONOS wildcard (which also points at the VPS).
### 2026-05-14 — `nfs-export.hubris.network` added (direct, non-HTTP)
NFSv4 export server [nfs-export (102)](../containers/102-nfs-export.md) at `192.168.8.200`. Direct entry, not Caddy-fronted — NFS is L4, no HTTP reverse-proxy meaningful.
### 2026-05-14 — `zimaos.hubris.network` added (Caddy-fronted, standard pattern)
New LAN entry for [100-zimaos](../vms/100-zimaos.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.195`. Briefly pointed direct-to-VM during install for the initial smoke-test, then re-pointed once a Caddyfile block was added (`reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS).
### 2026-05-13 — `plato.hubris.network` added; `files.hubris.network` removed
New LAN-only entry for [plato (126)](../containers/index.md#recently-destroyed-kept-for-archaeology). Same day, the `files.hubris.network` entry for the just-decommissioned seafile experiment was dropped; queries now fall through to the public IONOS answer (no LAN backend).
### 2026-05-12 — `files.hubris.network` added (since removed 2026-05-13)
Originally added for the seafile (LXC 125) Nextcloud-replacement evaluation. Pointed at 192.168.8.175 (Caddy reverse-proxied to 192.168.8.185:80). Entry removed when the experiment was torn down a day later.
### 2026-04-28 — wiki entry created
Initial documentation. 16 active entries.
### 2026-04-22 — IONOS wildcard moved 74.118.126.4 → 82.165.190.79
Public path now lands on the VPS traefik, not the old yunohost. Necessary for the [public ingress](ingress.md) pattern. The LAN dead-end semantics didn't change — public DNS still doesn't help LAN clients reach LAN-only services.
### 2026-04-22 — three caddy sites without DNS entries (jellyseerr, qbit, sab)
Caddy + certs were working but LAN resolution failed because the dnsmasq lines weren't added. Lesson recorded; entries added later that day.
### 2026-04-21 — dnsmasq stood up on LXC 124
Co-located with Authentik. Initial entries cover everything routed through Caddy.

143
archive/knowledge/infrastructure/homelab-context.md Normal file
View File

@@ -0,0 +1,143 @@
# Homelab context distribution
The cross-client context-and-secrets system that makes every agent (Claude
Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
self-locating and able to read the same source of truth.
Operational walkthrough for enrolling a new client lives in
[operations/agent-enrollment.md](../../../.agents/operations/agent-enrollment.md); this
page is the architecture reference.
## What's where
| Piece | Host | Path | Role |
| --- | --- | --- | --- |
| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab` → `/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
| Secrets-issuance | [apps (105)](../containers/105-apps.md) | `secrets-issuance.service` on port 9820 (https://secrets.hubris.network/issue) | Generates per-client age keypair on first bootstrap; idempotent; admin-token-gated `/revoke` |
| Sync timer | every enrolled client | `homelab-context-sync.timer` (Linux) / `network.hubris.homelab-context-sync.plist` (macOS) | `git pull --ff-only` every 5 min |
| Encrypted secrets | `dtoro/Homelab-Docs` | `secrets/*.yaml` (SOPS+age) | Recipients declared in `.sops.yaml` |
| Read-only context PAT | `dtoro` Gitea user, scope `read:repository` | given to operators out-of-band | Bootstrap-only — for the initial clone before SOPS works |
| Write-scoped PAT | `secrets/gitea-pat.yaml` (SOPS) | `homelab refresh-creds` swaps it into `/etc/homelab-context/git-credentials` | All post-bootstrap pushes (client lifecycle, wiki edits) |
## Data flow
```
dtoro/Homelab-Docs (gitea)
│
┌────────── push ────────┤ ◀── git push (write PAT or SSH)
│ │
│ ┌────── push ──────┘
│ │ │
│ │ ▼ webhook (push event)
│ │ ┌─── homelab-mcp-deploy ──── (LXC 105:9811)
│ │ └─── secrets-issuance-deploy (LXC 105:9821)
│ │ │
│ │ ▼
│ │ git pull → deploy.sh → restart service
│ │
│ └── on every client:
│ timer (5 min) → git pull --ff-only into /opt/homelab-context
│
▼
homelab CLI / MCP server reads /opt/homelab-context for everything
```
## Why two clones on LXC 105
The MCP server and secrets-issuance each have their own clone
(`/opt/homelab-mcp`, `/opt/secrets-issuance`) **in addition to**
`/opt/homelab-context`. Reasons:
- The deploy webhook for each service updates its own clone, runs
`deploy.sh` from there, and re-installs the systemd unit. Mixing this
with the client-context clone would create a circular dependency
(deploy reinstalls the unit that pulled it).
- The MCP server reads its data from `/opt/homelab-context` (the same path
every client uses) so changes to inventory propagate identically. Code
changes live in `/opt/homelab-mcp` and trigger a service restart.
## Mesh / network gates
- Both services bind `0.0.0.0:<port>`. The trust boundary is
`MESH_SUBNETS` in the service's environment + nftables (planned). Today
`MESH_SUBNETS=100.122.0.0/16,100.64.0.0/10,192.168.8.0/24` — Netbird +
Tailscale + the homelab LAN. Adjust if the LAN ever has untrusted
devices.
- Caddy fronts both with Let's Encrypt certs via the IONOS DNS challenge:
`mcp.hubris.network` → `192.168.8.205:9810`,
`secrets.hubris.network` → `192.168.8.205:9820`. Off-LAN clients on
Netbird reach them via the `192.168.8.0/24` network resource routed
through the PVE peer ([mesh.md](mesh.md)).
- Clients with default-public DNS (workstations not on Netbird, LXCs
using router DNS) need a `/etc/hosts` override pointing
`mcp.hubris.network` and `secrets.hubris.network` at the caddy LXC
(`192.168.8.175`) — same caveat as every other `*.hubris.network`
service, see [dns.md](dns.md).
## Secrets model
- Each enrolled client gets one **age private key** issued by
secrets-issuance on first bootstrap. The key file stays root-only on
the client; the public key is committed to `inventory.yaml` (and
becomes a recipient on SOPS-encrypted files via `.sops.yaml`).
- `secrets/*.yaml` are SOPS+age. Recipients per file are pinned in
`.sops.yaml` `creation_rules` by `path_regex`. Re-encrypting a file is
`sops updatekeys -y secrets/<name>.yaml`.
- The MCP server's `list_my_secrets(caller_pubkey)` tool returns only
secret *names* a given pubkey can decrypt — the server never sees
plaintext. Decryption is local-on-client (`homelab secret <name>`
shells out to `sops -d` with the client's key).
- The "all-clients" secrets (`hello.yaml` for the bootstrap decrypt
test, `gitea-pat.yaml` for the write-scoped PAT) are auto-granted to
every newly enrolled client by `homelab client add --finalize-pubkey`
(which appends the pubkey to the matching `.sops.yaml` rule and runs
`sops updatekeys`).
- **Removal does not erase past disclosure.** Revoking a client via
`homelab client remove` shreds the issuance-side key, denylists the
hostname, removes them from the recipient list, and re-keys all
shared secrets — but anything they already decrypted to disk is out of
your control. Rotate the underlying credential if compromise is
suspected.
## Why this design
- **One source of truth** keeps inventory, code, secrets, and docs
versioned together. A `git log` of `inventory.yaml` is the history of
the homelab.
- **Per-client age keys** scale better than a shared admin secret —
removing a client is a real revocation (for new ciphertext), not just
removing them from a wiki page.
- **MCP layer over the same clone** gives MCP-capable agents structured
query (`find_service`, `search_docs`) without forcing non-MCP tools to
go without — anything can still `cat` the markdown.
- **Sync timer rather than push fan-out** keeps the failure mode
contained: one client's webhook outage doesn't block a push from
landing on the others. Sub-5-min staleness is fine for docs and rare
enough for secrets that we don't need lower latency.
## Related
- [Operations: agent enrollment](../../../.agents/operations/agent-enrollment.md) — the
step-by-step for adding a new client
- [Auto-deploy](auto-deploy.md) — the `homelab-mcp` + `secrets-issuance`
pipelines (and the rest of the lab's webhook pipelines)
- [Mesh](mesh.md) — Netbird / Tailscale paths and the `192.168.8.0/24`
network resource
- [Apps (105)](../containers/105-apps.md) — where both services run
- [Gitea (104)](../containers/104-gitea.md) — the source of truth
## Changelog
### 2026-05-20 — system live across hubris, apps, republic-laptop
Phase 1 of the [cross-client context plan](../../../README.md) merged. Three
clients enrolled end-to-end: PAT-based bootstrap, age-key issuance, SOPS
decrypt verified on each. Webhook auto-deploy for both LXC 105 services
wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
`client add --finalize-pubkey` grant flow live so new clients are one
ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
Hermes Agent integration so the agent uses inventory at chat-time.

66
archive/knowledge/infrastructure/index.md Normal file
View File

@@ -0,0 +1,66 @@
# Infrastructure — cross-cutting systems
The homelab's shared infrastructure: systems that span multiple nodes and
are documented in their own pages. Each system below links to its full doc.
## Network
- **[Network](network.md)** — physical topology, subnets, routing, DHCP.
Homelab `192.168.8.0/24` isolated from household `192.168.178.0/24`.
Proxmox hubris acts as subnet router.
- **[DNS — split-horizon](dns.md)** — Technitium DNS on LXC 107,
`192.168.8.2:53`. `*.hubris.network` resolves to LAN IPs on the homelab
network and to mesh addresses off-LAN.
## Connectivity / mesh
- **[Mesh — Tailscale → Netbird migration](mesh.md)** — overlay networking.
Netbird is the preferred path; Tailscale is legacy.
- **[SSH access](ssh-access.md)** — dual-server SSH (OpenSSH + Netbird SSH)
on hubris, key distribution.
## Public ingress
- **[Public ingress — VPS traefik + cert mirror](ingress.md)** — how home
services reach the open internet. Two-stage: VPS traefik (IONOS) terminates
TLS, proxies over Netbird to home Caddy.
- **[Caddy reverse proxy](../containers/121-caddy.md)** — LAN endpoint.
Terminates TLS for every `*.hubris.network` hostname, forwards to backends.
## Storage
- **[Media permissions — GID 10000 standard](media-permissions.md)** — shared
group permission model across all LXCs that read/write the media library.
- **[Backups — rclone → Proton Drive](backups.md)** — off-host backup strategy.
LXC 132 handles rclone to Proton Drive; restic-on-USB deprecated.
## Identity & access
- **[Authentik SSO](../containers/106-auth-outpost.md)** — identity provider.
Core server runs on the VPS; LAN forward-auth outpost at LXC 106.
OIDC providers configured for Jellyfin, Jellyseerr, Sabnzbd, qBittorrent,
Yuvomi, and more.
## Management & automation
- **[Homelab context distribution](homelab-context.md)** — `/opt/homelab-context`
clone, MCP server, secrets issuance, cross-client sync.
- **[Auto-deploy — gitea-webhook pipelines](auto-deploy.md)** — push-to-deploy
for Caddy config, mule-image, and other tracked repos.
- **[Monitoring](monitoring.md)** — health checks, watchdogs, alerting
(migrated from claudio-bot to Hermes cron).
- **[VPS hardening](vps-hardening.md)** — IONOS netbird VPS: fail2ban,
nftables, OIDC SSH, security posture.
## Topology
- **[Topology diagram (generated)](topology.md)** — Mermaid graph of compute,
ingress routing, and storage mounts. Auto-generated from `inventory.yaml`
by `oikos/gen-topology.py` (Python — Go DB-native replacement planned).
## Related
- [README](../../../README.md) — entry point
- [Containers index](../containers/index.md)
- [Operations cheatsheet](../../../.agents/operations/commands.md)
- [OIKOS operating model](../../../.agents/OIKOS.md)

103
archive/knowledge/infrastructure/ingress.md Normal file
View File

@@ -0,0 +1,103 @@
# Public ingress — VPS traefik + cert mirror
How home services reach the open internet without exposing the home network. Two-stage pattern: traefik on the IONOS VPS terminates TLS at the public edge, then reverse-proxies over the netbird mesh to home Caddy / direct backends.
## The shape
```
Public client
│ *.hubris.network → 82.165.190.79 (IONOS wildcard)
▼
[VPS traefik] priority 1: HostSNI(*) → netbird-proxy:8443 ← netbird control plane
priority 10: per-host HTTP routers ← home services
│ HTTP over netbird mesh
▼
[Home backend on 192.168.8.x]
```
LAN clients resolve via the [Technitium DNS on dns (107)](dns.md) → `192.168.8.175` → home [Caddy (121)](../containers/121-caddy.md), unchanged. The two paths are independent.
## Why this shape
- VPS traefik already has a HostSNI(*) TCP passthrough at priority 1 so netbird's own ingress (`netbird.hubris.network`, future `*.proxy.hubris.network`) is unaffected.
- Per-hostname HTTP file-provider routers at priority 10 win over the passthrough for the listed hosts and let traefik terminate TLS itself for those.
- Traefik's own ACME (`letsencrypt` resolver) fails on this box: HostSNI(*) grabs TLS-ALPN-01 challenges before traefik's `allowACMEByPass=true` can respond. Solution: home Caddy obtains certs via IONOS DNS-01 (no conflict) and the VPS *mirrors* the result over.
## Components
### On the VPS (`82.165.190.79`)
- `/opt/traefik-dynamic.yaml` — file-watched dynamic config. One `http.routers.<name>-public` + one `http.services.<name>-public` per exposed service, plus one entry in the top-level `tls.certificates` list per hostname.
- Let's-Encrypt-effective directory: `/letsencrypt/` inside the traefik container, backed by the host-side docker volume `opt_netbird_traefik_letsencrypt`.
- Backups before edits: `cp /opt/traefik-dynamic.yaml /opt/traefik-dynamic.yaml.bak.$(date +%s)` — several bak files live alongside.
### On the PVE host (`192.168.8.77`)
- `/usr/local/bin/hubris-public-cert-sync.sh` — runs daily via `hubris-public-cert-sync.timer`. Maps source hostname → VPS cert filenames in a bash assoc array. For each mapping: `pct pull` cert+key from [Caddy (121)](../containers/121-caddy.md)'s store, diff against the VPS copy, scp only on change.
- Filenames are stable per host so the dynamic.yaml never needs editing on renewal — traefik file-watches and hot-reloads the cert.
## Services currently exposed
| Hostname | Path scope | Backend | Middlewares | Cert files on VPS |
| ------------------------------ | -------------------------------- | -------------------------------- | -------------------------------------------- | ------------------------------------------ |
| `artifacto.hubris.network` | `/p/*`, `/static/*`, `/healthz` | `192.168.8.205:3100` | `artifacto-strip-sso` + `artifacto-ratelimit` (50 rps / 100 burst) | `fullchain.crt` / `privkey.key` |
| `blog.hubris.network` | whole host | `192.168.8.205:8080` | `blog-ratelimit` (100 rps / 200 burst) | `blog.fullchain.crt` / `blog.privkey.key` |
| `trmnl.hubris.network` | whole host | `192.168.8.211:9851` ([trmnl 128](../containers/128-trmnl.md)) | `trmnl-ratelimit` (20 rps / 40 burst) | `trmnl.fullchain.crt` / `trmnl.privkey.key` |
| `house.hubris.network` | whole host | `192.168.8.212:3000` ([house 129](../containers/129-house.md)) | `house-ratelimit` (30 rps / 60 burst) | `house.fullchain.crt` / `house.privkey.key` |
`artifacto-strip-sso` blanks inbound `X-Authentik-*` and `X-Artifacto-Gateway` so external clients can't spoof the SSO auto-login header contract. Path split is enforced at the VPS router rule, not by home Caddy. See [Artifacto on apps (105)](../containers/105-apps.md).
### `auth.hubris.network` — different pattern (local container, not cert-mirror)
Since 2026-05-31 [Authentik runs on the VPS itself](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md), so `auth.hubris.network` is served by a **local Docker container**, not proxied to a home backend. It therefore does **not** use the file-provider + cert-mirror pattern above:
- Routed via traefik **Docker provider labels** on the `authentik-server` service (`/opt/docker-compose.yml`), not `traefik-dynamic.yaml`.
- TLS via traefik's own `letsencrypt` resolver (works here because it's a normal HTTP router, not the HostSNI passthrough).
- Traefik reaches it over the `auth` Docker network (`172.30.1.0/24`); Postgres/Redis on that net are isolated from the netbird containers.
- Admin UI is IP-gated: an `admin-allowlist` ipAllowList middleware on `PathPrefix(/if/admin/)` (currently `5.61.168.0/24`). Login/flow endpoints stay public.
No cert-mirror entry and no `hubris-public-cert-sync.sh` mapping is needed for `auth`.
## Recipe — exposing another service
1. Ensure home Caddy on [LXC 121](../containers/121-caddy.md) already serves the hostname (cert exists at `/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/<host>/`).
2. Add an entry to `HOSTS` in `/usr/local/bin/hubris-public-cert-sync.sh` mapping the hostname → VPS filenames. Run once: `systemctl start hubris-public-cert-sync.service`. Confirm the cert landed.
3. Edit `/opt/traefik-dynamic.yaml` on the VPS:
- Add to `tls.certificates`: paths `/letsencrypt/<crt>` and `/letsencrypt/<key>`.
- Add `http.routers.<host>-public`: `rule: 'Host(\`<host>\`)'` (or with path matchers if scope-gating), `entryPoints: [websecure]`, `priority: 10`, `tls: {}`, `service: <host>-public`, `middlewares: [...]`.
- Add a ratelimit middleware under `http.middlewares` if wanted.
- Add `http.services.<host>-public.loadBalancer.servers[0].url: 'http://<lan-ip>:<port>'`.
4. Verify:
```
ssh root@100.122.165.149 'curl -skI --resolve <host>:443:127.0.0.1 https://<host>/' # 2xx/3xx
curl -skI --resolve <host>:443:<vps-public-ip> https://<host>/ # same
```
5. **No DNS edit needed** — the IONOS wildcard already points at the VPS.
## What does NOT follow this pattern
- `netbird.hubris.network` (and any future `*.proxy.hubris.network`) uses the netbird-proxy / HostSNI passthrough path. Netbird handles its own cert via ACME cleanly because it *is* the passthrough target.
## Related
- [DNS split-horizon](dns.md)
- [Caddy (121)](../containers/121-caddy.md) — cert source, internal counterpart
- [Mesh migration](mesh.md) — netbird is the transport between VPS and home
- [VPS hardening](vps-hardening.md) — fail2ban / nftables that the access logs feed
- [Artifacto on apps (105)](../containers/105-apps.md) — first publicly-exposed service
## Changelog
### 2026-06-24 — `trmnl.hubris.network` exposed
TRMNL plugins middleware on [trmnl (128)](../containers/128-trmnl.md). File-provider router `trmnl-public` → `192.168.8.211:9851`, `trmnl-ratelimit` (20 rps / 40 burst), cert mirrored as `trmnl.fullchain.crt`/`trmnl.privkey.key`. Verified live from the internet (200 with token / 401 without). It was provisioned during a mesh outage — the `home-lab-network` (192.168.8.0/24) route had no active routing peer because the **mac-mini routing peer's netbird was down** (all home-backed public services 504'd). Bringing netbird up on mac-mini restored the route; no traefik change was needed.
### 2026-05-31 — `auth.hubris.network` now served locally on the VPS
Authentik migrated onto the VPS ([investigation](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md)). Unlike the home-backed services above, `auth` is a local container routed via traefik Docker-provider labels with traefik-managed Let's Encrypt — no cert-mirror, no `traefik-dynamic.yaml` router. Admin UI gated by an ipAllowList middleware. Traefik gained a second Docker network (`auth`, `172.30.1.0/24`) to reach it while keeping its DB/Redis isolated from the netbird stack.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — `blog.hubris.network` exposed
WriteFreely on [apps (105)](../containers/105-apps.md). Whole host is public.
### 2026-04-22 — pattern established with Artifacto
First service through the file-provider router. IONOS wildcard moved to the VPS this day. Cert mirror script + timer deployed on the PVE host.

101
archive/knowledge/infrastructure/media-permissions.md Normal file
View File

@@ -0,0 +1,101 @@
# Media permissions — `media` GID 10000
Standard for any LXC reading/writing `/mnt/library` on [hubris](../hosts/hubris.md). Applied 2026-04-20.
## Standard
Every LXC that mounts `/mnt/library` participates in a shared `media` group with **GID 10000**. Shared subtrees are owned by that group with the setgid bit (`drwxrwsr-x`, mode `2775`), so new files auto-inherit the right group regardless of which container wrote them.
## Why
`/mnt/library` is a cross-container storage pool. \*arr writes, jellyfin reads, mulita scans, paperless ingests. Without a shared group, each container sees files as `nobody:nogroup` (unprivileged) or `www-data` (privileged 1:1) and the permission web collapses into one-off chmods. GID 10000 bridges privileged and unprivileged containers.
## Onboarding a new LXC
1. `pct set <id> -mp0 /mnt/library,mp=/mnt/library` (if not already mounted).
2. Inside the container:
```
groupadd -g 10000 media
usermod -aG media <service-user> # for every user that needs library access
```
3. If the container is **unprivileged** (check `pct config <id> | grep unprivileged`), append this idmap block to `/etc/pve/lxc/<id>.conf` (back up first):
```
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 10000
lxc.idmap: g 10000 10000 1
lxc.idmap: g 10001 110001 55535
```
Then `pct stop <id> && pct start <id>`.
4. For systemd services running with `User=root` (not typical), add a drop-in with `SupplementaryGroups=media`. Systemd skips `initgroups()` for `User=root`.
5. `pct exec` sessions don't get supplementary groups (no initgroups). Use `sudo -i` or `su - <user>` inside the container to verify membership interactively. Real services use `initgroups` and work correctly.
## State snapshot
### Host
- Group `media` GID 10000 exists.
- `/etc/subgid` has `root:100000:65536` AND `root:10000:1` (second line required for unprivileged LXCs to receive GID 10000).
- Shared subtrees owned `:media` mode `2775` (drwxrwsr-x, setgid):
- `movies`, `tv`, `music`, `anime`, `podcasts` — jellyfin libraries
- `audiobooks`, `audiobookshelf-metadata`, `books`, `comics` — audiobookshelf / grimmory
- `downloads` — \*arr stack output
- `images` — photoprism / immich / mulita
- `roms` — emu frontends
- `syncthing` — empty subtree, retained for archaeology (LXC 109 destroyed 2026-05-14)
- Container-specific subtrees intentionally **not** migrated (keep their own owner:group):
- `documents` (paperless, `www-data:www-data 750`)
- `homecloud` (nextcloud — its own permission model, easy to break)
- `marimo` (marimo venv) — *LXC since destroyed; review whether subtree still serves a purpose*
- `notes`, `sophia` (single-container use); `heaper` — orphaned data subtree (LXC since destroyed 2026-05-14, 224 MiB retained)
- `repos` (owner UID 102 GID 105 from inside [gitea](../containers/104-gitea.md) — don't touch)
### LXCs with media-group membership
| ID | Name | Priv | Media-group members |
| --- | --------------------------------------------- | ---- | --------------------------------------------- |
| 101 | [jellyfin](../containers/101-jellyfin.md) | **unpriv + idmap** | jellyfin |
| 103 | [paperless](../containers/103-paperless.md) | priv | www-data |
| 104 | [gitea](../containers/104-gitea.md) | priv | www-data, gitea |
| 105 | [apps](../containers/105-apps.md) | priv | www-data |
| 114 | [nextcloud](../containers/114-nextcloud.md) | priv | www-data |
| 119 | [sophia](../containers/119-sophia.md) | priv | www-data |
| 120 | [mule-images](../containers/120-mule-images.md) | priv | www-data |
| 122 | [arriman](../containers/122-arriman.md) | priv | www-data, audiobookshelf, radarr, sonarr, lidarr, prowlarr, qbittorrent, bazarr, jellyseerr, mylar, jackett, overseerr, plex, arr |
| 130 | [grimmory](../containers/130-grimmory.md) | priv | Docker container uses `GROUP_ID=10000` env var (linuxserver pattern) — no in-LXC group needed |
| 132 | [rclone](../containers/132-rclone.md) | priv | **read-only** mount; runs as root → reads all subtrees. No media group needed |
> Some entries from earlier snapshots — 100 (arr-yunohost), 107 (marimo), 109 (syncthing), 110 (photoprism), 112 (immich), 116 (heaper) — referenced LXCs that have since been destroyed. See [containers/index](../containers/index.md#recently-destroyed-kept-for-archaeology).
Config backups: `/root/101.conf.bak.*`, `/root/109.conf.bak.*` (109 destroyed 2026-05-14).
## Gotchas
- **[apps (105)](../containers/105-apps.md) and [grimmory (130)](../containers/130-grimmory.md) are Docker hosts.** Adding `media` to the LXC alone is *not* enough for Docker containers inside. Each Docker container needs its GID passed in explicitly: `--group-add 10000`, `user: "<uid>:10000"`, or `GROUP_ID=10000` (linuxserver images) in compose. Grimmory, audiobookshelf-in-docker, etc. need this per-container.
- **`pct exec` does NOT run initgroups.** So `pct exec <id> -- id` shows only the primary group. For interactive verification, use `pct exec <id> -- sudo -i -u root id` or `su - <user> -c id`. Real systemd services work fine.
- **systemd `User=root`** skips initgroups — explicit `SupplementaryGroups=media` drop-in needed.
- **`pct restore`** or template rebuilds wipe in-container group membership and unprivileged-LXC idmap blocks. Re-apply from this page.
- **`/etc/subgid`** must retain both `root:100000:65536` AND `root:10000:1`. Dropping the second breaks startup of any unprivileged LXC with the idmap block.
- **\*arr "Set Permissions" options** can override the setgid inheritance by explicitly chown'ing files. Leave those off, or set the group to `media`. Relevant to Sonarr/Radarr/qBittorrent on [arriman (122)](../containers/122-arriman.md).
- **Nextcloud** files under `/mnt/library/homecloud` are deliberately NOT in the media group. NC manages its own permission model. See [nextcloud (114)](../containers/114-nextcloud.md).
- **\*arr stack on arriman** required `MEDIACENTER_GID=10000` (not 13000) in `.env` because s6-setuidgid only honors the primary PGID; `group_add:` doesn't propagate. See [arriman (122)](../containers/122-arriman.md#changelog).
## Related
- [Hubris host](../hosts/hubris.md)
- All container pages list whether they're in the standard
## Changelog
### 2026-05-14 — LXC 109 (syncthing) destroyed
Removed the syncthing row from the membership table and the syncthing-as-`User=root` example from the onboarding section. `/mnt/library/syncthing` subtree was already empty and retained as an empty dir.
### 2026-05-14 — LXC 116 (heaper) destroyed
Removed the heaper row from the LXC membership table and noted the orphaned `/mnt/library/heaper` subtree (224 MiB retained). See [host changelog](../hosts/hubris.md#changelog).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-26 — `MEDIACENTER_GID` fix on [arriman (122)](../containers/122-arriman.md)
qBit was erroring every torrent with "Permission denied" because `MEDIACENTER_GID=13000` was set as a supplementary GID via `group_add:`. Changed to 10000 (primary GID); fix described above is now standard.
### 2026-04-20 — standard rolled out
GID 10000 hostgroup, idmap blocks for unprivileged LXCs, setgid 2775 on shared subtrees, `media` membership for service users in every participating LXC.

194
archive/knowledge/infrastructure/mesh.md Normal file
View File

@@ -0,0 +1,194 @@
# Mesh — Tailscale → Netbird migration
The hubris fleet is migrating from Tailscale to Netbird. Netbird is the target end-state. In-progress as of 2026-04-21.
## Current state
- **PVE host** uses Netbird (`wt0`, `100.122.38.109/16`). Its resolver is the local netbird daemon, which forwards to LAN/upstream — so the PVE host gets `*.hubris.network → 192.168.8.175` via the system resolver chain.
- **Netbird mgmt host** (`82.165.190.79`, FQDN `inspiring-ramanujan.netbird.selfhosted`, NB IP `100.122.165.149`) is now itself a peer on the mesh (joined 2026-04-22 via setup key, netbird 0.69.0). Routes the homelab network (`192.168.8.0/24`) via the PVE peer. This gives the mgmt host LAN access *and* split-horizon DNS for `*.hubris.network`. Useful independently of any Authentik integration.
- **Most LXCs** still run Tailscale or use router DNS (`192.168.8.1`) / Tailscale MagicDNS (`100.100.100.100`), both of which return the *public* IONOS A record `*.hubris.network → 82.165.190.79`. The VPS only routes hostnames it actually publishes (today, `artifacto` + `blog`), so this path is a dead end for any LAN-only service.
## ICE / STUN / TURN
**Today** (post-2026-05-21 migration):
- External STUN servers (Google + Cloudflare) declared under `server.Stuns` in `/opt/management.json`. Embedded STUN is no longer in use.
- coturn runs on the VPS host (apt-installed, systemd-managed). TURN URL `turn:netbird.hubris.network:3478?transport=tcp` is advertised to peers via mgmt's `TURNConfig.Turns` block. Long-term credentials at `user=netbird:<pwd from /root/turn-pass.txt>`.
- For non-symmetric peers, ICE picks direct `srflx/srflx` (P2P). For symmetric-NAT peers, ICE falls through to TURN-relay before falling back to the WSS relay (`rels://netbird.hubris.network:443`).
**IONOS port-3478 caveat** (load-bearing — undocumented before 2026-05-21):
IONOS upstream filters STUN-class traffic on port 3478 **for both UDP and TCP** by default. The UDP block was already known (verified 2026-05-10 with `tcpdump -ni any udp port 3478`). The TCP block was discovered 2026-05-21 when TURN-allocate retries timed out from a symmetric-NAT laptop: TCP handshake completed (carried by kernel-only SYN exchange) but data packets never reached the VPS's `ens6` interface.
**Resolution**: operator must add an inbound firewall exception in the IONOS dashboard (Cloud Panel → Networks → Firewall policy on the VPS) for inbound TCP 3478 (and UDP 3478 if STUN-via-VPS-listener is ever needed; we use external STUN so this is optional). Without that exception, coturn is unreachable from outside even though host nftables + listener look fine.
**Verifying TURN works** end-to-end from an outside peer:
```python
# python3
import socket, struct, secrets
s = socket.create_connection(("netbird.hubris.network", 3478), timeout=10)
tid = secrets.token_bytes(12)
attrs = struct.pack("!HHI", 0x0019, 4, 17 << 24) # REQUESTED-TRANSPORT = UDP
msg = struct.pack("!HHI", 0x0003, len(attrs), 0x2112A442) + tid + attrs # Allocate
s.sendall(msg)
print(s.recv(4096).hex()) # expect 120 bytes starting 0x0113 (Allocate Error = auth challenge)
```
A 120-byte `0x0113`-typed response means coturn is reachable and responding. An indefinite TCP timeout (after `connected from ...` prints) means the IONOS rule was reverted or narrowed.
**If a peer is still on the WSS relay after this** — verify in `netbird status -d`: `Connection type: P2P` with srflx/srflx is the cone-NAT happy path; `Connection type: P2P` with relay/srflx (or relay/prflx) means the peer's network is symmetric-NAT and it's using TURN. Falling back to `Relayed (rels://...)` only happens if TURN allocate also fails — investigate the IONOS exception first.
**Old combined-server note (history, kept for context):**
Pre-migration, the bundled `netbirdio/netbird-server` combined image silently ignored both `server.turns:` and top-level `TURNConfig:` YAML, so TURN was non-functional. That's why the architectural migration to the canonical multi-container stack (`mgmt + signal + relay + coturn`) happened. See the 2026-05-21 changelog entry below and the [Netbird-combined-no-TURN finding](https://github.com/netbirdio/netbird/issues) in homelab memory for the full discovery.
## Consequence — every LXC wired to Authentik needs an internal override
Until each LXC is migrated to Netbird, anything that needs to reach `auth.hubris.network` (Authentik), `cloud.hubris.network` (Nextcloud), etc., must override the public answer with `192.168.8.175`.
Two techniques. Pick by HTTP-client behavior.
### A) `/etc/hosts` override
Works for libc `getaddrinfo` clients: curl, wget, most Go/Python/Ruby apps, gitea.
- Place the line **outside** the `# --- BEGIN PVE ---` / `# --- END PVE ---` markers. Proxmox rewrites everything inside that block on every container start.
- Belt-and-suspenders: `/etc/systemd/system/hubris-hosts-override.service` (oneshot, enabled, idempotent).
### B) Local dnsmasq
Required for clients that bypass `/etc/hosts`. **Nextcloud (PHP Guzzle + `OC\Http\Client\DnsPinMiddleware`) is one** — uses `dns_get_record()`, not `getaddrinfo`. Likely candidates for the same treatment: Python/Ruby/Java apps that do explicit DNS resolution before connecting.
Recipe:
```
apt install dnsmasq
cat > /etc/dnsmasq.d/hubris-internal.conf <<EOF
address=/auth.hubris.network/192.168.8.175
server=192.168.8.1
server=1.1.1.1
interface=lo
bind-interfaces
no-hosts
no-resolv
EOF
# Set LXC default nameservers and live resolv.conf
pct set <id> --nameserver "127.0.0.1 192.168.8.1 1.1.1.1"
# Then update /etc/resolv.conf inside the LXC too.
```
### Known overrides applied
| LXC | Technique | Notes |
| ------------------------------------------ | ---------------------------------------- | ----- |
| [104 (gitea)](../containers/104-gitea.md) | `/etc/hosts` + `hubris-hosts-override.service` | Standard |
| [114 (nextcloud)](../containers/114-nextcloud.md) | local dnsmasq @ 127.0.0.1 + resolv.conf reorder | Guzzle bypasses /etc/hosts |
| [105 (apps)](../containers/105-apps.md), inner containers | `extra_hosts:` in compose | Booklore, mulita, WriteFreely each ship with this |
## Adding new LXCs
- Don't add new LXCs to Tailscale; add them to Netbird. Tailscale is being decommissioned on hubris.
- When wiring a new app into Authentik: `cat /etc/resolv.conf` on the target LXC. If nameserver is `192.168.8.1` or `100.100.100.100`, add the hosts override. If it's the netbird daemon IP, skip.
## Long-term fix
Either:
- Split-horizon DNS at LAN/router level so `*.hubris.network → 192.168.8.175` for every LAN client. Eliminates all per-LXC overrides.
- Once Netbird migration completes, every LXC's resolver becomes the netbird daemon, which already learns hubris.network answers via the system resolver chain on the PVE host.
## CRITICAL — never `docker compose up` Portainer-managed stacks
[apps (105)](../containers/105-apps.md) runs multiple stacks deployed via the Portainer UI (`/var/lib/docker/volumes/portainer_data/_data/compose/<N>/`). Running `docker compose up -d <svc>` from the host shell on these triggers a recreate of OTHER services in the stack — labels drift, compose treats the project as reconciling, containers get destroyed and remade. **This wiped Booklore's mariadb data on 2026-04-22** (bind mount `./mariadb/config` re-initialized fresh).
Recipe for container-config changes (e.g. adding `extra_hosts`) on Portainer-managed stacks:
1. Edit the compose in Portainer UI → **Stacks → <stack> → Editor → Update stack**. Portainer handles the recreate cleanly with its own state tracking.
2. Do NOT edit `/var/lib/docker/volumes/portainer_data/_data/compose/<N>/docker-compose.yml` directly.
3. For data-preserving DNS overrides on stateful services, prefer Caddy-side handling (forward-auth) or the app's built-in OIDC over container-level `extra_hosts` when possible.
## Related
- [DNS split-horizon](dns.md)
- [Authentik (124)](../containers/106-auth-outpost.md) — the IdP that triggers most of these overrides
- [Nextcloud (114)](../containers/114-nextcloud.md) — example of Technique B
- [Gitea (104)](../containers/104-gitea.md) — example of Technique A
- [Public ingress (VPS traefik)](ingress.md) — uses the same mesh as transport
## Changelog
### 2026-05-31 (later) — Authentik moved to the VPS; mesh-dependency for auth eliminated (supersedes the band-aid below)
The earlier same-day fix routed `auth.hubris.network` through VPS Traefik → Caddy → LXC 124 **over the mesh**. That restored service but re-created the original fragility: if the mesh is dark when management restarts, the `192.168.8.175` backend is unreachable and management crash-loops again (the "Bootstrap note" in the entry below). That note is now **obsolete** — Authentik was migrated onto the VPS itself, so OIDC no longer touches the mesh. The `auth-authentik` → `192.168.8.175` route and its `skip-verify` transport were removed from `/opt/traefik-dynamic.yaml`; `auth.hubris.network` is now served by a local `authentik-server` container via Traefik Docker-provider labels, and netbird-mgmt has `depends_on: authentik-server: condition: service_healthy`. The socat / reverse-SSH bootstrap dance is no longer needed. Full detail: [2026-05-31 Authentik VPS migration](../../sources/investigations/archive/2026-05-31-authentik-vps-migration.md).
### 2026-05-31 — Netbird mesh recovered; auth.hubris.network exposed via VPS Traefik
**Symptom:** `netbird-mgmt` crash-looped for ~9 days (since 2026-05-21 migration). All peers showed `Connecting`, management returned `404 (Not Found)` for gRPC → `EOF` on startup.
**Root cause:** The 2026-05-21 migration configured `management.json` with `OIDCConfigEndpoint: https://auth.hubris.network/...`. The public DNS for `auth.hubris.network` already pointed to `82.165.190.79` (VPS), but the VPS Traefik had no route for that host → management got EOF on every boot, crash-looped.
**Fix:**
1. Added `auth-hubris` router to `/opt/traefik-dynamic.yaml` — `Host(auth.hubris.network)` with `certResolver: letsencrypt` → service `auth-authentik`.
2. Service backend: `https://192.168.8.175` (Caddy on hubris LAN) via the Netbird mesh.
3. ServersTransport `skip-verify` with `serverName: auth.hubris.network` + `insecureSkipVerify: true` so Traefik sends the correct SNI to Caddy.
4. Upgraded VPS Netbird client `0.69.0 → 0.71.3` via apt (`apt install netbird=0.71.3`).
5. Mesh fully recovered; management connected to peers within ~1 min.
**Bootstrap note (if mesh is dark and management must restart):** If the WireGuard tunnels are fully dead AND management needs to restart (after e.g. a VPS reboot with no peer handshakes), the Traefik backend `192.168.8.175` will be unreachable and management will crash again. Recovery: temporarily open VPS port 22 via IONOS console (`nft insert rule inet hubris-fw input iifname "ens6" tcp dport 22 accept`), SSH in, run `socat TCP-LISTEN:8443,bind=172.30.0.1,reuseaddr,fork TCP:127.0.0.1:8443 &`, then from hubris `ssh -f -N -R 127.0.0.1:8443:192.168.8.175:443 root@82.165.190.79` — this bootstraps one management start, after which the mesh self-heals.
**Architecture after this change:** `auth.hubris.network` is publicly accessible (HTTPS via VPS Traefik → Caddy on hubris → Authentik LXC 124). External devices authenticating to Netbird now hit this public path. Phase 6 (Authentik as Netbird IdP) is complete and live.
### 2026-05-21 — VPS migrated combined → vanilla netbird stack with external TURN
The combined `netbirdio/netbird-server` image was replaced with the canonical multi-container deploy (`netbirdio/management:0.71.3` + `signal:0.71.3` + `relay:0.71.3` + `dashboard:latest` + host coturn) on `/opt/docker-compose.yml`. Driver: combined image silently ignored external `TURNConfig` so symmetric-NAT peers couldn't use TURN.
Same migration also swapped OIDC from the combined image's embedded Dex IdP to Authentik on [LXC 124](../containers/106-auth-outpost.md), upgrading mgmt to 0.71.3. The `store.db` schema auto-migrated cleanly from 0.68.3 (copy-not-move from the old `opt_netbird_data` volume into the new `mgmt_data` volume). Pre-cutover backups at `/root/netbird-*.tgz` on the VPS, ~857 MB, retained for ~7d.
Also during this work: IONOS upstream was found to filter TCP 3478 in addition to UDP 3478. Added a TCP-3478 inbound exception in the IONOS firewall (see ICE/STUN section above for the verification probe).
The new Authentik provider for NetBird is `Client type: Public` (PKCE-only). Confidential would break the dashboard SPA's token exchange. The Device Code grant flow is wired (see [containers/124-authentik.md](../containers/106-auth-outpost.md#device-code-grant--configured-2026-05-21)) so interactive `netbird up` works — `--setup-key` is no longer required for new peers.
**Post-migration JWT-issuer gotcha on existing peers** (cost ~30 min to diagnose 2026-05-21):
Existing peers — registered against the old combined image's embedded Dex IdP at `https://netbird.hubris.network/oauth2` — cache the OLD expected SSH-JWT issuer in the netbird daemon's in-memory state. After the migration, incoming `netbird ssh` connections were rejected with:
```
JWT authentication failed: validate token (
expected issuer=https://netbird.hubris.network/oauth2,
audiences=[netbird-dashboard netbird-cli],
actual issuer=https://auth.hubris.network/application/o/netbird/,
audience=netbird-dashboard
)
```
Neither `systemctl restart netbird` nor `netbird down && netbird up` clears the cache. Root cause: in `client/internal/engine_ssh.go`, `updateSSH()` bails out with `if e.sshServer != nil { return nil }` whenever the SSH server is already running, so mgmt-pushed JWT config updates are silently ignored. Only a full daemon-process tear-down lets the SSH server re-initialize with the new validator config:
```
sudo systemctl stop netbird
sleep 3
sudo systemctl start netbird
```
After that, `grep -iE "issuer|audience" /var/log/netbird/client.log | tail` shows the new Authentik issuer. Run this on every existing peer (PVE host + every LXC + every workstation) once after a future IdP swap.
**Username gotcha (related):** `netbird ssh` defaults the remote username to the LOCAL one (e.g. `dtoro` from the operator's laptop). Hubris + the LXCs only have `root`, so the JWT is accepted but the session immediately fails with `user dtoro not found`. Always use the explicit `root@` prefix when invoking netbird-ssh manually:
```
netbird ssh -p 22022 root@proxmox-server.netbird.selfhosted
```
The `homelab` CLI handles this automatically via the per-host `ssh.user` field in `inventory.yaml` (defaults to `root`; set explicitly only for workstations whose login user isn't `root`).
Open follow-up: TURN-over-TLS on TCP 5349 (cert via certbot or extract Traefik's acme.json) for hostile-middlebox networks; plain TCP 3478 is sufficient for current usage.
### 2026-05-10 — ICE direct p2p restored (external STUN swap)
All peers were `Connection type: Relayed` because the embedded STUN on the VPS was unreachable from outside (IONOS drops UDP 3478 inbound). Swapped to Google + Cloudflare STUN in `/opt/config.yaml`. After `netbird down/up`, peers now report `P2P` with srflx/host candidates. Backup of pre-change config at `/opt/config.yaml.bak-20260510-185050`. Fixes a real-world Nextcloud upload bottleneck (~366 kB/s through the relay → LAN-direct on same-network peers).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-22 — Booklore mariadb data wiped (lesson recorded)
The "never docker compose up Portainer-managed stacks" rule comes from this. See [apps (105)](../containers/105-apps.md#changelog).
### 2026-04-22 — netbird mgmt host joined its own mesh
`82.165.190.79` is now a peer (`100.122.165.149`). LAN access + split-horizon DNS via PVE peer. See [VPS hardening](vps-hardening.md).
### 2026-04-21 — overrides applied to gitea (104) and nextcloud (114)
Two techniques documented; Nextcloud forced the dnsmasq route because of Guzzle.

60
archive/knowledge/infrastructure/monitoring.md Normal file
View File

@@ -0,0 +1,60 @@
# Monitoring — Hermes health watchdog
Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
## Current approach
Two layers:
1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
service reachability, and apt/docker drift across all hosts.
2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
cron. Silent when healthy. When thresholds breach, sends an actionable alert
to Matrix (`@dtoro:avispero`) with options the user can reply to directly
(e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
the selected option via SSH.
Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
apt >10/>50 upgradable, services down.
Home Assistant pulls PVE metrics independently via its Proxmox VE integration
(unaffected by this change).
## Legacy: claudio-monitor (deprecated 2026-06-04)
The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
on LXC 123:9090. All components decommissioned:
| Component | Fate |
|-----------|------|
| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
| `claudio-monitor.timer` | Disabled on hubris |
| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
## Related pages
- [Hubris host](../hosts/hubris.md)
- [HAOS VM (108)](../vms/108-haos.md)
- [Backups (disabled)](backups.md)
- [Homelab context distribution](homelab-context.md)
## Changelog
### 2026-06-04 — migrated to Hermes health watchdog
claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
Matrix alerts. LXC 123 destroyed, repos archived.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
General health monitor with per-LXC checks. MQTT/REST push paths ripped out.

88
archive/knowledge/infrastructure/network.md Normal file
View File

@@ -0,0 +1,88 @@
# Network
Physical and logical network topology for the homelab.
## Why
The homelab runs on a dedicated internal subnet (`192.168.8.0/24`) isolated from the main household LAN (`192.168.178.0/24`). Isolation is enforced at Proxmox: LXC/VM traffic is bridged only on the internal `vmbr0` bridge; Proxmox routes packets out to Fritz!Box via `vmbr1`. The main LAN cannot reach homelab services directly without a Fritz!Box static route (which is configured to allow inbound).
Fritz!OS 8.x does not support second IP networks on LAN ports, so Proxmox (`hubris`) acts as the subnet router rather than the Fritz!Box.
## Hardware
| Device | Role |
|---|---|
| Fritz!Box 7590 | Main router / ISP gateway (`192.168.178.1`) |
| SODOLA 5-Port 2.5Gbit Managed | Homelab switch — flat L2, all ports native |
| hubris (Proxmox) | Subnet router — routes between `192.168.8.0/24` and `192.168.178.0/24` |
## Topology
```
ISP
└── Fritz!Box 7590 (192.168.178.1)
│ static route: 192.168.8.0/24 → 192.168.178.10
│
└── SODOLA 5-Port 2.5Gbit
├── Port 1 uplink → Fritz!Box LAN
├── Port 2 hubris eno1 → vmbr1 (192.168.178.10)
├── Port 3 [device]
├── Port 4 [device]
└── Port 5 spare
hubris internal bridges:
vmbr1 192.168.178.10/24 eno1 (uplink, DHCP-reserved) gateway 192.168.178.1
vmbr0 192.168.8.77/24 no physical port (internal)
192.168.8.1/24 alias — LXC default gateway
├── all 16 LXCs
└── HAOS VM
```
## Subnets
| Subnet | Gateway | Purpose |
|---|---|---|
| `192.168.178.0/24` | `192.168.178.1` | Household LAN — laptops, phones, Fritz!Box DHCP |
| `192.168.8.0/24` | `192.168.8.1` (Proxmox `vmbr0` alias) | Homelab — all LXCs and VMs |
## DHCP
- **Household (`192.168.178.x`)**: Fritz!Box built-in DHCP. Proxmox `vmbr1` has a reservation: MAC `84:47:09:6b:e7:58` → `192.168.178.10`.
- **Homelab (`192.168.8.x`)**: Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2`. Range `192.168.8.241–192.168.8.254`, gateway `192.168.8.1`, DNS `192.168.8.2`.
Static IPs span `.101–.239` (all LXCs, VMs, and workstations). DHCP pool narrowed to `.241–.254` (2026-06-03) to avoid overlap and IP conflicts.
## DNS
Split-horizon DNS for `*.hubris.network` served by Technitium on [CT 107](../containers/107-dns.md) at `192.168.8.2:53`. See [dns.md](dns.md) for full detail.
## Routing
Proxmox has `net.ipv4.ip_forward=1` (already enabled by PVE). Packets from LXCs on `vmbr0` destined for the internet exit via `vmbr1` → Fritz!Box. Fritz!Box masquerades all outbound WAN traffic. Fritz!Box has a static route (`192.168.8.0/24 → 192.168.178.10`) so return traffic reaches the LXCs.
No NAT on Proxmox — traffic flows without double-NAT.
## Remote access
- **NetBird mesh** — primary path for remote administration. Authenticated via [Authentik on the VPS](../../../vps/).
- **Tailscale** — legacy, being phased out. See [mesh.md](mesh.md).
## Related
- [DNS](dns.md) — split-horizon config and entry list
- [Ingress](ingress.md) — public entry points via VPS traefik
- [Mesh](mesh.md) — NetBird / Tailscale VPN overlay
- [hosts/hubris.md](../hosts/hubris.md) — Proxmox host (vmbr0/vmbr1 config)
- [CT 107 — dns](../containers/107-dns.md) — Technitium DNS + DHCP server
## Changelog
### 2026-06-17 — Fritz!Box DNSv4 server set to Technitium (192.168.8.2)
Household LAN clients (192.168.178.x) now resolve `*.hubris.network` to LAN IPs. Configured in Fritz!Box at Internet → Filter → DNS Server → DNSv4 Server → "Use other DNSv4 servers" → Preferred = `192.168.8.2`. No per-device or Netbird setup needed.
Previous pool `.100–.240` overlapped with all static LXCs/VMs (` .101–.239`), creating IP conflict risk (DHCP could hand out an IP that a static service expects). Shrunk pool to `.241–.254` via Technitium API. No services re-IP'd. 11 stale DHCP leases in `.101–.110` will expire naturally. **Open:** ZimaOS (VM 100) holds DHCP lease `.103` but inventory expects `.195` — needs static IP set inside VM. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-06-02 — Executed migration; Proxmox as subnet router
Fritz!OS 8.x does not support second IP networks on LAN ports, so the final design uses Proxmox as the router: `vmbr1` (eno1 → SODOLA → Fritz!Box) is the uplink at `192.168.178.10`; `vmbr0` is a portless internal bridge with `192.168.8.1` alias as the LXC gateway. Technitium DHCP enabled for `192.168.8.100–240`. Caddy service unit was missing and recreated. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).
### 2026-06-01 — Initial network doc; Slate AX retired; SODOLA switch added
Replaced the GL.iNet Slate AX sub-router with the SODOLA 5-Port 2.5Gbit managed switch. Eliminated double-NAT. See [migration plan](../../../plans/done/2026-06-01-slate-ax-to-sodola-migration.md).

206
archive/knowledge/infrastructure/ssh-access.md Normal file
View File

@@ -0,0 +1,206 @@
# SSH access
How to reach every host in the fleet from any workstation, with LAN as
the primary path and Netbird as the automatic backup.
## Architecture
SSH access relies on three layers:
1. **Homelab inventory (`inventory.yaml`)** — the single source of truth
for every host's LAN IP, Netbird addresses, SSH user, and port.
2. **Key distribution (`ssh/deploy-keys.sh`)** — deploys workstation SSH
public keys to hubris and every running LXC, so any key-authorized
workstation can log in anywhere.
3. **Config generation (`homelab ssh-config --install`)** — generates
`~/.ssh/config.d/homelab` with short hostname aliases for every host,
using LAN IPs (routed via Netbird's `192.168.8.0/24` subnet route when
off-LAN) with Netbird FQDN fallbacks (`<name>-mesh`) for roaming
workstations.
### How it works
- **From on-LAN:** `ssh gitea` resolves to `192.168.8.121` directly.
- **From off-LAN (Netbird):** The same `192.168.8.121` works because
hubris routes the `192.168.8.0/24` subnet through Netbird.
- **Roaming workstations:** `ssh mac-mini-mesh` or `ssh republic-laptop-mesh`
uses the Netbird FQDN as a fallback when the workstation is off its
home subnet.
The `homelab ssh <host>` CLI command also has built-in LAN probing:
it tries a 1.5s TCP connect to the LAN IP, and if that fails, falls
back to the Netbird FQDN.
## Key distribution
Each workstation's SSH public key lives in the repo at:
`ssh/authorized_keys/<hostname>.pub`
To deploy or re-deploy all workstation keys to hubris + every running LXC:
```bash
# From hubris (or via homelab pct):
sudo bash /opt/homelab-context/ssh/deploy-keys.sh
# Or from any workstation:
ssh root@192.168.8.77 "bash /opt/homelab-context/ssh/deploy-keys.sh"
```
This script:
- Reads all `.pub` files from `ssh/authorized_keys/`
- Adds any missing keys to `/etc/pve/priv/authorized_keys` on hubris
- For each running LXC, appends keys to `/root/.ssh/authorized_keys`
- Is idempotent — skips keys already present
## Config generation
To generate the SSH config on any workstation:
```bash
homelab ssh-config --install
```
This writes to `~/.ssh/config.d/homelab` and ensures
`Include ~/.ssh/config.d/homelab` is present in `~/.ssh/config`.
The config is regenerated automatically on every `homelab sync` (which
kicks the 5-minute context sync timer).
## Adding a new workstation
When onboarding a new machine:
1. Hostname must match an entry in `inventory.yaml`.
2. If the workstation will be on the LAN, add its `lan_ip` to
`inventory.yaml` and push. This gives it a primary LAN entry in the
generated SSH config.
3. Enable SSH Remote Login:
- **macOS:** `sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist`
- **Linux:** `sudo systemctl enable --now sshd`
4. Generate an SSH keypair if one doesn't exist:
```bash
ssh-keygen -t ed25519 -a 100
```
5. Publish the public key to the repo:
```bash
cp ~/.ssh/id_ed25519.pub /opt/homelab-context/ssh/authorized_keys/<hostname>.pub
cd /opt/homelab-context && git add ssh/authorized_keys/ && git commit -m 'ssh: add <hostname> pubkey' && git push
```
6. Deploy the key to all hosts:
```bash
ssh root@192.168.8.77 "cd /opt/homelab-context && git pull --ff-only && bash ssh/deploy-keys.sh"
```
7. Generate the local SSH config:
```bash
homelab ssh-config --install
```
## Hosts
### Hubris + strong (PVE cluster: `Homelab`)
Both nodes share `/etc/pve/priv/authorized_keys` — it's Proxmox
cluster-synced, so a key added on either node is authorized on both.
| Detail | hubris | strong |
|--------|--------|-----------|
| LAN IP | `192.168.8.77` | `192.168.178.181` |
| Cluster node name | `hubris` | `strong` (OS hostname kept as-is from install) |
| Netbird | `100.122.38.109` (`proxmox-server.netbird.selfhosted`) | not enrolled yet |
| Netbird SSH port | `22022` (mesh-only, OIDC auth) | n/a |
| SSH user | `root` | `root` |
Authorized root keys currently deployed (cluster-wide):
- `root@hubris` (self, RSA)
- `d.toro.v@pm.me` (ed25519) — mac-mini
- `root@strong` (RSA) — strong's own key, added 2026-07-01 for the cluster join
### LXCs
Every LXC at `192.168.8.x` accepts root SSH via authorized_keys. Keys
are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
| LXC | Name | LAN IP | Role |
|-----|------|--------|------|
| 101 | jellyfin | `192.168.8.206` | media-server |
| 102 | nfs-export | `192.168.8.200` | storage-export |
| 103 | paperless | `192.168.8.130` | document-archive |
| 104 | gitea | `192.168.8.121` | git-server |
| 105 | apps | `192.168.8.205` | docker-apps |
| 106 | auth-outpost | `192.168.8.184` | authentik-outpost |
| 107 | dns | `192.168.8.185` | dns-helper |
| 114 | nextcloud | `192.168.8.224` | file-sync |
| 118 | elementsynapse | `192.168.8.239` | matrix-server |
| 119 | sophia | `192.168.8.157` | workshop |
| 120 | mule-images | `192.168.8.136` | photo-management |
| 121 | caddy | `192.168.8.175` | reverse-proxy |
| 122 | arriman | `192.168.8.132` | arr-stack |
### Workstations
| Name | OS | LAN IP | Netbird FQDN | SSH user |
|------|----|--------|--------------|----------|
| mac-mini | macOS | `192.168.8.174` | `mac-mini-234-17.netbird.selfhosted` | `dtoro` |
| republic-laptop | Linux | TBD | `republic-laptop.netbird.selfhosted` | `dtoro` |
strong moved out of this table 2026-07-01 — it's a Proxmox host now, see the cluster table above.
### VPS (external)
| Detail | Value |
|--------|-------|
| Public IP | `82.165.190.79` |
| Netbird | `100.122.165.149` (FQDN: `netbird-ionos.netbird.selfhosted`) |
| SSH user | `root` |
| Access | Mesh-only — public port 22 is blocked by nftables. Key-only auth. |
## VPS
Access is mesh-only. From a mesh-connected peer:
```bash
ssh root@100.122.165.149
ssh root@netbird-ionos.netbird.selfhosted
# or via homelab:
homelab ssh netbird-vps
```
## Verification
```bash
# From any workstation after running homelab ssh-config --install:
for name in hubris gitea apps sophia paperless caddy jellyfin nextcloud; do
ssh -o BatchMode=yes "$name" "hostname" && echo "$name OK"
done
```
## Related
- [Mesh migration](mesh.md)
- [VPS hardening](vps-hardening.md)
- [Agent enrollment](../../../.agents/operations/agent-enrollment.md)
- [Homelab CLI](../../../bin/homelab)
## Changelog
### 2026-07-01 — strong reformatted to Proxmox, joined cluster; table corrected
strong moved from the Workstations table to the PVE-cluster table (was showing a stale `192.168.8.133`, never actually reachable — the real LAN IP has always been `192.168.178.181`, matching hosts/strong.yaml). Root key access bootstrapped via one-time console password, then key-only going forward. See [hosts/hubris.md#cluster](../hosts/hubris.md#cluster) and [hosts/strong.md](../hosts/strong.md).
### 2026-06-02 — universal SSH reachability
Replaced ad-hoc per-workstation SSH configs with inventory-generated
configs (`ssh/gen-config.py`, `homelab ssh-config`). Added centralized
key distribution (`ssh/deploy-keys.sh`, `ssh/authorized_keys/`). All
LXCs now accept root SSH from any workstation whose pubkey is in the
repo. mac-mini Remote Login enabled. Netbird subnet route
(192.168.8.0/24 via hubris) provides off-LAN reachability for all LAN
IPs.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — VPS SSH hardened to mesh-only
Public `:22` blocked at nftables. Key-only sshd.
### 2026-04-22 — iMac key authorized on hubris
`d.toro.v@pm.me` added to `/etc/pve/priv/authorized_keys`.

96
archive/knowledge/infrastructure/topology.md Normal file
View File

@@ -0,0 +1,96 @@
<!-- Generated by oikos/gen-topology.py from inventory.yaml. -->
<!-- Do NOT edit by hand - your changes will be overwritten. -->
# Topology (generated)
Source: [inventory.yaml](../../../inventory.yaml) — 2 hypervisors, 20 LXCs, 2 VMs, 2 workstations, 18 services.
Edge semantics: [oikos/ontology.yaml](../../../oikos/ontology.yaml). Operating model: [OIKOS.md](../../../.agents/OIKOS.md).
## Compute & ingress
```mermaid
flowchart LR
subgraph hubris_sub["hubris (Proxmox)"]
trmnl["trmnl<br/>LXC 128<br/>trmnl-middleware<br/>192.168.8.211"]
nfs_export["nfs-export<br/>LXC 102<br/>storage-export<br/>192.168.8.200"]
paperless["paperless<br/>LXC 103<br/>document-archive<br/>192.168.8.130"]
gitea["gitea<br/>LXC 104<br/>git-server<br/>192.168.8.121"]
apps["apps<br/>LXC 105<br/>docker-apps<br/>192.168.8.205"]
auth_outpost["auth-outpost<br/>LXC 106<br/>authentik-gateway<br/>192.168.8.6"]
dns["dns<br/>LXC 107<br/>dns-server<br/>192.168.8.2"]
nextcloud["nextcloud<br/>LXC 114<br/>file-sync<br/>192.168.8.224"]
sophia["sophia<br/>LXC 119<br/>workshop<br/>192.168.8.109"]
mule_images["mule-images<br/>LXC 120<br/>photo-management<br/>192.168.8.136"]
caddy["caddy<br/>LXC 121<br/>reverse-proxy<br/>192.168.8.175"]
teddycloud["teddycloud<br/>LXC 131<br/>teddycloud<br/>192.168.8.150"]
zimaos["zimaos<br/>VM 100<br/>nas-frontend-eval<br/>192.168.8.195"]
haos["haos<br/>VM 108<br/>home-automation<br/>192.168.8.101"]
end
subgraph strong_sub["strong (Proxmox)"]
house["house<br/>LXC 129<br/>family-planner<br/>192.168.8.244"]
jellyfin["jellyfin<br/>LXC 101<br/>media-server<br/>192.168.8.246"]
elementsynapse["elementsynapse<br/>LXC 118<br/>matrix-server<br/>192.168.8.242"]
arriman["arriman<br/>LXC 122<br/>arr-stack<br/>192.168.8.245"]
grimmory["grimmory<br/>LXC 130<br/>book-library<br/>192.168.8.247"]
seanime["seanime<br/>LXC 133<br/>anime-media-server<br/>192.168.8.248"]
romm["romm<br/>LXC 134<br/>rom-manager<br/>192.168.8.249"]
end
rclone["rclone<br/>lxc<br/>backup"]
republic_laptop([republic-laptop<br/>workstation<br/>primary-dev])
mac_mini([mac-mini<br/>workstation<br/>dev<br/>192.168.178.182])
netbird_vps[[netbird-vps<br/>external<br/>netbird-mgmt]]
url_artifacto(["artifacto.hubris.network"]) -->|routes-to| apps
url_authentik(["auth.hubris.network"]) -->|routes-to| netbird_vps
url_gitea(["git.hubris.network"]) -->|routes-to| gitea
url_homelab_mcp(["mcp.hubris.network"]) -->|routes-to| apps
url_jellyfin(["media.hubris.network"]) -->|routes-to| jellyfin
url_matrix(["matrix.hubris.network"]) -->|routes-to| elementsynapse
url_nextcloud(["cloud.hubris.network"]) -->|routes-to| nextcloud
url_paperless(["paperless.hubris.network"]) -->|routes-to| paperless
url_photos(["photos.hubris.network"]) -->|routes-to| mule_images
url_proxmox_ui(["proxmox.hubris.network"]) -->|routes-to| hubris_sub
url_secrets_issuance(["secrets.hubris.network"]) -->|routes-to| apps
url_teddycloud(["teddy.hubris.network"]) -->|routes-to| teddycloud
url_trmnl(["trmnl.hubris.network"]) -->|routes-to| trmnl
url_zimaos(["zimaos.hubris.network"]) -->|routes-to| zimaos
```
## Storage (mounts)
```mermaid
flowchart LR
mnt_library[("/mnt/library")]
mnt_media_local[("/mnt/media_local")]
mnt_media_local_anime[("/mnt/media_local/anime")]
apps["apps"] -->|mounts| mnt_library
arriman["arriman"] -->|mounts| mnt_media_local
gitea["gitea"] -->|mounts| mnt_library
grimmory["grimmory"] -->|mounts| mnt_media_local
hubris["hubris"] -->|mounts| mnt_library
jellyfin["jellyfin"] -->|mounts| mnt_media_local
mule_images["mule-images"] -->|mounts| mnt_library
nextcloud["nextcloud"] -->|mounts| mnt_library
paperless["paperless"] -->|mounts| mnt_library
romm["romm"] -->|mounts| mnt_media_local
seanime["seanime"] -->|mounts| mnt_media_local_anime
sophia["sophia"] -->|mounts| mnt_library
teddycloud["teddycloud"] -->|mounts| mnt_library
```
## Archaeology (destroyed nodes)
| Node | ID | Destroyed | Reason |
|---|---|---|---|
| plato | 126 | 2026-06-28 | notes workspace decommissioned; data retained at /mnt/library/documents/plato |
| claudio-bot | 123 | 2026-06-04 | replaced by Hermes Agent on mac-mini; monitoring moved to homelab-health-watchdog cron |
| mule-photos-new | 127 | 2026-05-22 | PhotoPrism test stack promoted to LXC 120 (Mulimage 2.0 merge) |
| heaper | 116 | 2026-05-14 | decommissioned; data retained at /mnt/library/heaper |
| syncthing | 109 | 2026-05-14 | decommissioned; library subtree was empty |
| seafile | 125 | 2026-05-13 | Seafile Pro evaluation rejected; files.hubris.network removed from caddy + dns |
| arr-yunohost | 100 | 2026-04-28 | migrated to docker stack on arriman (LXC 122) |
| flaresolverr | 106 | 2026-04-28 | folded into the arriman docker compose |
| marimo | 107 | 2026-04-28 | decommissioned |
| photoprism | 110 | 2026-04-28 | replaced by mule-images (LXC 120) |
| karakeep | 111 | 2026-04-28 | decommissioned |
| immich | 112 | 2026-04-28 | replaced by mule-images (LXC 120) |
| reticulum | 115 | 2026-04-28 | decommissioned |

91
archive/knowledge/infrastructure/vps-hardening.md Normal file
View File

@@ -0,0 +1,91 @@
# VPS hardening — `82.165.190.79` / `100.122.165.149`
IONOS VPS that runs the Netbird control plane and the [public ingress traefik](ingress.md). Hardened 2026-04-23 from its stock-Plesk state.
## At a glance
- **Hostname:** `inspiring-ramanujan.82-165-190-79.plesk.page`
- **OS:** Debian 13
- **Mesh:** netbird `100.122.165.149` (peer of the lab mesh; routes `192.168.8.0/24` via [hubris](../hosts/hubris.md)).
- **Public:** `82.165.190.79` (`ens6`).
- **Public DNS:** IONOS wildcard `*.hubris.network → 82.165.190.79`.
- **Docker stack** at `/opt/docker-compose.yml`: `traefik` (TLS/ACME) + `dashboard` + `mgmt` + `signal` + `relay` + `proxy` — netbird-mgmt 0.71.3 vanilla deploy since 2026-05-21 (see [mesh.md changelog](mesh.md#changelog)).
- **Host services (outside docker):** `coturn` (TURN-TCP on :3478, long-term creds rendered into `/etc/turnserver.conf` by `homelab render-vps-configs` from sops-encrypted `secrets/turn-shared-secret.yaml`).
- **Config rendering:** `/etc/turnserver.conf` + `/opt/management.json` are generated from templates in `vps/*.tmpl` on this repo by `homelab render-vps-configs`. Secret placeholders (`{{TURN_PASSWORD}}`, `{{AUTHENTIK_CLIENT_SECRET}}`) are substituted from sops-encrypted secrets decrypted on hubris and pushed over ssh. **Do not hand-edit those two files on the VPS** — the next render will overwrite them.
## SSH
- Key-only (`PasswordAuthentication no`, `PermitRootLogin prohibit-password`) via drop-in at `/etc/ssh/sshd_config.d/10-hubris-hardening.conf`. Original config backed up at `/etc/ssh/sshd_config.bak.<ts>`.
- **Mesh-only**: public `:22` is dropped by the nftables firewall. SSH reaches the VPS only over `wt0`. `ListenAddress` itself is still `0.0.0.0` — gating is firewall-layer.
- Authorized root keys: PVE (`root@hubris`), Mac Mini (`d.toro.v@pm.me`). Add a new device with `ssh-copy-id root@100.122.165.149` from a mesh peer **before** disabling its access paths.
## Firewall — nftables (`inet hubris-fw`)
Config at `/etc/nftables.conf`, service enabled.
- Public iface `ens6`. Wireguard iface `wt0`.
- **INPUT on `ens6`** allow-list: DHCP (67→68), rate-limited ICMP/ICMPv6, **TCP 3478** (coturn TURN-TCP, added 2026-05-21). Everything else drops.
- `wt0` fully accepted in INPUT. `lo` accepted.
- **IONOS upstream firewall** also gates inbound traffic before it reaches `ens6`. Open ports today: TCP 80/443 (traefik), UDP 51820 (netbird-proxy), TCP 3478 (coturn, added 2026-05-21). UDP 3478 is dropped by IONOS upstream regardless of local nftables. See [mesh.md ICE/STUN](mesh.md) for the STUN/TURN port matrix.
- **FORWARD chain at priority `filter-10`** (runs before Docker's FORWARD) hosts the fail2ban ban enforcement — see below.
- Set `banned4` (typed `ipv4_addr`, flag `timeout`) holds fail2ban's drops.
- Coexists with Docker's `ip nat` / `ip filter` tables (iptables-nft compat). **Do NOT `flush ruleset`** in this config — it'll wipe Docker's state too.
## fail2ban
- **Jail `traefik-4xx`** tails `/var/log/traefik/access.log` (bind-mounted from container). Filter at `/etc/fail2ban/filter.d/traefik-4xx.conf` matches 401/403/404/429 from `blog-public@file` or `artifacto-public@file` routers only — netbird-grpc traffic isn't considered.
- Tunables: `findtime=600, maxretry=30, bantime=3600`.
- **Action** at `/etc/fail2ban/action.d/nft-hubris.conf` adds/removes elements from `inet hubris-fw banned4` with per-element timeout.
### CRITICAL invariant — wireguard / fail2ban
**Bans must never affect `wt0` or wireguard UDP.** The FORWARD chain explicitly `accept`s the following *before* the ban check:
- `udp 51820` (wireguard)
- `udp 3478` (STUN)
- `ct state established,related`
The INPUT ban rule is scoped to `iifname "ens6"`.
Violating this takes the mesh down for every home device (they share one public IP) and the only recovery is IONOS console → `nft flush set inet hubris-fw banned4`.
## Traefik access log
- Written to `/var/log/traefik/access.log` on the host via a bind mount added to `/opt/docker-compose.yml` (traefik volumes include `/var/log/traefik:/logs`) plus `--accesslog.filepath=/logs/access.log`.
- CLF format. Real client IP arrives correctly because docker userland-proxy is off — see [public ingress](ingress.md).
## Plesk / mail / FTP / Dr.Web
Stopped and disabled (not uninstalled). All of:
`dovecot`, `dovecot.socket`, `postfix`, `postfix@-`, `pc-remote`, `xinetd`, `plesk-task-manager`, `plesk-web-socket`, `sw-cp-server`, `sw-engine`, `plesk-repaird`, `plesk-repaird.socket`, `drwebd`.
`psa.service` is masked (was a one-shot boot bootstrap). `/etc/cron.d/plesk-backup-manager-task` renamed to `.disabled`.
Reverse: `systemctl unmask psa; systemctl enable --now <svc>`.
## Auto-patching
- `unattended-upgrades` enabled (stock).
- Drop-in at `/etc/apt/apt.conf.d/52hubris-reboot.conf` sets auto-reboot at **04:00 UTC** when `/var/run/reboot-required` is set.
- Runs inside the stock `apt-daily-upgrade.timer`.
## Recovery paths
Ordered by preference:
1. **SSH via mesh** — primary. Any mesh peer with an authorized key.
2. **IONOS web console** (my.ionos.com → VPS → Console) — uses the system password, not SSH keys. Bypasses any firewall misconfig.
3. **IONOS rescue mode** — boot rescue, mount rootfs, edit `/etc/nftables.conf` or `/etc/ssh/sshd_config.d/10-hubris-hardening.conf` to a known-good state, reboot.
## Related
- [Public ingress (VPS traefik)](ingress.md)
- [Mesh migration](mesh.md) — VPS as a mesh peer
- [SSH access](ssh-access.md)
## Changelog
### 2026-05-21 — netbird stack migrated combined → vanilla; coturn added
Replaced the `netbirdio/netbird-server` combined image with the canonical `mgmt + signal + relay + dashboard` containers (0.71.3). Added host-side `coturn` for external TURN, with nftables rule `iifname "ens6" tcp dport 3478 accept` and an IONOS upstream firewall exception. Authentik on LXC 124 now provides OIDC for the netbird dashboard. Full context in [mesh.md changelog](mesh.md#changelog).
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-23 — hardened
nftables firewall, mesh-only SSH, fail2ban traefik jail, Plesk disabled, auto-reboot 04:00 UTC, wireguard/fail2ban invariant established.

89
archive/knowledge/investigations/2026-04-21-hubris-crash-loop.md Normal file
View File

@@ -0,0 +1,89 @@
# 2026-04-21 — Hubris crash loop (thermal + USB drive)
## Summary
[`hubris`](../../../wiki/hosts/hubris.md) hard-locked repeatedly on 2026-04-21 (silent CPU hangs, no panic, no OOM, no MCE). Two contributors identified: idle CPU sitting at ~95 °C on the `performance` governor, and a USB-attached external SSD whose UAS interaction with the AMD USB4/Thunderbolt PCIe tunnel triggered hard locks. CPU thermal addressed via `cpu-epp.service`; drive removed 2026-04-22 as an A/B test. As of 2026-04-28 the host has 3+ days uptime — the drive looks like the primary contributor; `cpu-epp` remains as belt-and-suspenders.
## Timeline
### 2026-04-19 — drive attached
External `Silicon Motion Portable SSD` (vid:pid `090c:2320`) attached for the new restic [backup pipeline](../../../wiki/infrastructure/backups.md). Pre-attach uptime had been 33 days stable.
### 2026-04-19 → 2026-04-21 — first crashes
Two hard crashes in 2.5 days (46 h then 12 h uptime). Kernel logs ended abruptly with routine apparmor entries — no panic, OOM, or MCE — the classic hard-lock signature. Preceded by `uas_eh_abort_handler` storms and xHCI resets on port 6-1.
### 2026-04-21 — crash cadence escalates
23:02 (previous night) → 12:01 the next day. Cadence: 12 h → 43 m → 8 m → 5 m. Stable since the 12:03 boot once `cpu-epp.service` started applying `EPP=balance_power`.
### 2026-04-21 — first set of mitigations applied
- **`cpu-epp.service`** deployed. Sets `scaling_governor=powersave` + EPP=`balance_power` at boot. Drops idle Tctl from ~95 °C → ~60 °C, clocks from 4.4 GHz pinned → 1.1–2 GHz idle.
- **Crash capture**: `/etc/sysctl.d/60-crash-capture.conf` (panic on oops/hardlockup/softlockup/rcu, auto-reboot 10 s), `/etc/modprobe.d/softdog.conf` (`soft_panic=1 soft_margin=60`), `/etc/systemd/system.conf.d/watchdog.conf` (`RuntimeWatchdogSec=15s`).
- **`rasdaemon`** installed (Debian pkg; mcelog is retired) — logs MCE / memory / PCIe AER / thermal events to `/var/lib/rasdaemon/ras-mc_event.db`. Query `ras-mc-ctl --summary`.
- **UAS blacklist** for the drive: `/etc/modprobe.d/usb-storage-quirks.conf` → `options usb-storage quirks=090c:2320:u`. Forces BOT instead of UAS for the SMI bridge.
- **Mount-on-demand** for the drive: `/usr/local/sbin/backup-usb.sh attach|detach|status` toggles `/sys/bus/usb/devices/*/authorized` so the drive is de-authorized when no backup is running.
### 2026-04-22 — recurrence after 30 h 37 m
Same silent-cutoff signature at 18:42:08. Much longer than any pre-`cpu-epp` crash (12 h max), so `cpu-epp` helps but is not sufficient on its own. [claudio-monitor](../../../wiki/infrastructure/monitoring.md) showed healthy runtimes up to 43 s before the hang (no pre-crash degradation). No MCE / no RAS / pstore empty.
### 2026-04-22 — `cpu-epp.service` design bug fixed
Was `After=multi-user.target` + `WantedBy=multi-user.target` — queued behind `pve-guests.service`. The hottest window of every boot (20 LXCs + 1 VM coming up) ran on the `performance` governor. Fixed: now `After=sysinit.target` + `Before=pve-guests.service`.
### 2026-04-22 — drive removed (A/B test)
User physically removed the external USB drive. [Backup timers disabled](../../../wiki/infrastructure/backups.md#status), fstab entry commented, drive de-authorized. Goal: confirm whether the drive + UAS + AMD USB4 PCIe-tunnel interaction is the dominant root cause.
### 2026-04-23 — SSD cooling + thermal pads installed
Cold-boot baseline (3 min uptime): nvme0n1 35 °C composite / sensor1 (controller) **53 °C**; nvme1n1 36 °C composite / both sensors ≤36 °C. Lifetime warning-time counters at install: nvme0n1 709 min warn + 5 min crit; nvme1n1 778 min warn + 45 min crit — both drives had spent real time in thermal warning historically.
#### 2026-04-23 thermal-pad verdict
Resolved (3 h self-paced watch). Steady-state across 17:03 → 19:56 stayed pinned at nvme0 composite 47 °C / sensor1 60–61 °C, nvme1 38–40 °C. Zero new warning-time minutes on either drive. Controller sensor1 at 60-ish is normal idle for M.2 NVMe. **The load-bearing signal is the lifetime warning-time counter, not absolute sensor1** — watch that going forward, not sensor1 threshold.
### 2026-04-28 — A/B test passing so far
3+ days continuous uptime with the drive removed. If uptime now exceeds ~3 days without crash, the drive/PCIe-tunnel interaction is confirmed as the dominant root cause. Will revisit options (different drive, different USB port, different bridge) once stability is firmly established.
### 2026-04-29 — physical relocation
User moved the host to a spot with better airflow. Post-move idle baseline (45 min uptime, light load): Tctl **47.2 °C**, GPU 42 °C, nvme0 composite 34.9 °C / sensor1 32.9 °C, nvme1 composite 38.9 °C / sensor1 52.9 °C, DRAM 34–35.5 °C. Notably nvme0 sensor1 is **27 °C cooler** than the 2026-04-23 thermal-pad steady-state of 60–61 °C. Watch lifetime warning-time counters (`nvme smart-log` → "Warning Comp. Temperature Time") over the next week — flat trend confirms the move helped. Doesn't change the drive-removal A/B test, but reduces residual thermal headroom risk independently.
## Root cause
Two-contributor:
1. **Thermal**: Ryzen 5 7640HS (Phoenix APU) in a passively-cooled GMKtec NucBox sitting at ~95 °C idle under `performance` governor. Mitigated by `cpu-epp.service`. In `amd-pstate=active` mode the governor MUST be `powersave` for EPP values to take effect — `performance` silently ignores them.
2. **Storage**: external USB SSD's UAS keepalive interacts badly with the AMD USB4/Thunderbolt PCIe tunnel. UAS blacklist + mount-on-demand reduced but did not eliminate it. Removing the drive eliminates it (so far).
Background: BIOS 1.02 (latest publicly visible from GMKtec). Vendor not on LVFS so `fwupdmgr` can't update. Non-ECC RAM means memory faults are silent and undetectable except by memtest86+.
## Diagnostic notes
- **Pure hardware hangs leave no trace.** `pstore` stays empty across reboots despite `panic_on_oops/hardlockup/softlockup/rcu` + softdog `soft_panic=1` being configured. The CPU is gone before the kernel can react. Don't waste time on panic traces — use crash cadence + uptime pattern as the diagnostic.
- **First check** if hubris crash-loops again with the same silent-cutoff signature: `cpu-epp.service` is active and `/sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference` is `balance_power`.
- **Non-ECC RAM means memory faults are silent.** If `cpu-epp` is on and crashes still happen, suspect DIMM next (run memtest86+) and/or try the commented-out `pcie_aspm=off pci=nomsi` workarounds in `/etc/default/grub`.
- **Grub escalation** (`pcie_aspm=off pcie_port_pm=off pci=nomsi`) **deliberately NOT applied** yet to keep the drive-removal test clean. Enable only if crashes continue without the drive.
- `mcelog` is not in Debian 13 repos; `rasdaemon` is the replacement.
## BIOS update path
Checked 2026-04-21. GMKtec is **not on LVFS**, so `fwupdmgr` can't update the NucBox M6 Ultra. BIOS 1.02 (2025-08-06) is the latest publicly visible — GMKtec's download center only publishes Windows drivers, no BIOS listing or changelog. Update path is a `.exe` flasher that needs Windows (would require Windows-To-Go USB on hubris). Before committing to that, email `service@gmktec.com` with the S/N to confirm a newer BIOS / AGESA level actually exists. `fwupd` is now installed, so if GMKtec ever onboards to LVFS, updates appear automatically.
## Mitigations applied
| Mitigation | Status | Notes |
| ----------------------------------------- | -------- | ---------------------------------------------------- |
| `cpu-epp.service` (powersave + balance_power) | Active | Unit ordering fixed 2026-04-22 to run before guests |
| Crash capture (sysctl + softdog + watchdog) | Active | Catches everything but pure silicon hangs |
| `rasdaemon` | Active | Persistent telemetry for MCE / memory / PCIe AER |
| UAS blacklist for `090c:2320` | Active (drive currently absent) | `/etc/modprobe.d/usb-storage-quirks.conf` |
| Mount-on-demand (`backup-usb.sh`) | Active (drive currently absent) | Backup unit `ExecStartPre`/`ExecStopPost` hooks |
| Drive physically removed | Active | A/B test in progress |
| Thermal pads on NVMes | Active 2026-04-23 | Steady-state composite 47 °C / sensor1 60 °C |
| `pcie_aspm=off pci=nomsi` | NOT applied | Reserved for if crashes recur without the drive |
## Affected nodes
- [Hubris host](../../../wiki/hosts/hubris.md)
- [Backups (disabled)](../../../wiki/infrastructure/backups.md)
- [Monitoring](../../../wiki/infrastructure/monitoring.md)
## Open questions
- Will the host stay up indefinitely without the drive? (Test ongoing — 3+ days as of 2026-04-28.)
- If yes: which mitigation lets the drive come back? (different drive? different USB port bypassing the hub chain? rear motherboard USB 3 port? alternative bus path?)
- Newer BIOS / AGESA from GMKtec — answered email pending. Without LVFS we need explicit vendor confirmation.
- Memtest86+ pass on the DIMMs — not yet run; deferred until the drive case is closed.

109
archive/knowledge/investigations/2026-05-31-authentik-vps-migration.md Normal file
View File

@@ -0,0 +1,109 @@
# 2026-05-31 — Authentik migrated from LXC 124 to the VPS
## Summary
The NetBird management server (on the [VPS](../../../wiki/infrastructure/ingress.md)) crash-looped 1200+ times because it fetches the Authentik OIDC discovery document on startup, and Authentik was only reachable via the NetBird mesh — which was down *because* mgmt couldn't start. A classic bootstrap deadlock: **mgmt needs OIDC → OIDC needs the mesh → the mesh needs mgmt.**
Resolved by moving Authentik off [LXC 124](../../../wiki/containers/106-auth-outpost.md) onto the VPS itself, so `auth.hubris.network` resolves to a container co-located with netbird-mgmt — no mesh dependency. A `depends_on: condition: service_healthy` on the mgmt service makes the deadlock structurally impossible to recur.
The full Authentik Postgres DB (all users, apps, passwords, groups) was migrated, so every gated app keeps working with no per-app reconfiguration.
## Timeline
### Trigger
`netbird status` on clients showed `Management: Disconnected`; `netbird.hubris.network` unreachable. VPS `docker logs netbird-mgmt`:
```
Error: failed reading provided config file: /etc/netbird/management.json:
OIDC configuration request returned status 504 with response: Gateway Timeout
```
The 504 came from VPS traefik trying to reach the old `auth-authentik` backend (`https://192.168.8.175`, home Caddy) over the down mesh.
### Fix — Phase A (restore the mesh)
1. Stood up Authentik on the VPS in `/opt/docker-compose.yml`: `authentik-postgres`, `authentik-redis`, `authentik-server`, `authentik-worker` (image `ghcr.io/goauthentik/server:2026.5.2`), on a dedicated `auth` Docker network (`172.30.1.0/24`). Traefik joined both `netbird` and `auth` networks.
2. Removed the `auth-authentik` → `192.168.8.175` backend from `/opt/traefik-dynamic.yaml`; Authentik now routed via Docker provider labels.
3. Added `depends_on: authentik-server: condition: service_healthy` to the mgmt service.
4. Created the NetBird OAuth2 provider/app manually (only manual step), then mgmt auto-recovered within 30 s.
### Fix — Phase B (full data migration)
Once the mesh was back, the homelab DB was reachable:
1. `ssh proxmox 'pct exec 124 -- docker exec authentik-postgresql-1 pg_dump -U authentik authentik'` → gzip.
2. Stopped VPS authentik server/worker, dropped + recreated the VPS DB (connect to the `postgres` DB to drop `authentik`), restored the dump.
3. **Swapped `AUTHENTIK_SECRET_KEY` in `/opt/authentik.env` to the homelab's key** — Authentik encrypts OAuth2 client secrets in the DB with this key; a mismatch makes every client secret unreadable and silently breaks all SSO.
4. Restarted Authentik. All users / apps / passwords / groups present.
Scripts left on the VPS: `/opt/migrate-authentik.sh` (re-runnable), `/opt/backup-authentik.sh` (daily cron, 14-day retention).
### Fix — DNS cutover (the long tail)
The real reason the browser kept hitting the *old* Authentik even after the VPS one was healthy: split-horizon dnsmasq on LXC 124 still had `address=/auth.hubris.network/192.168.8.175`, and the NetBird per-client resolver (`100.122.255.254`) cached it. Symptom: the login page console logged `authentik(early): version 2026.2.2` (the old LXC) instead of `2026.5.2` (the VPS).
- Updated dnsmasq: `address=/auth.hubris.network/82.165.190.79`, `systemctl restart dnsmasq`.
- `netbird down && netbird up` does **not** reliably flush the NetBird resolver cache. Immediate per-client fixes: `/etc/hosts` override (mac), `sudo resolvectl flush-caches` (Linux), `sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder` (mac).
## Issues hit during the migration (and fixes)
1. **Redirect URI error.** The restored DB had redirect URIs in `REGEX` matching mode; in Authentik 2026.5.x they failed to match. Fixed by switching to `STRICT` exact matching (Django ORM, `RedirectURIMatchingMode.STRICT`). Set all four: `http://localhost:53000/` (CLI), `https://netbird.hubris.network/{peers,nb-auth,nb-silent-auth}`.
2. **Only the password field showed (no username).** NetBird passes `login_hint=<email>` in the OAuth2 URL → Authentik pre-identifies and skips the identification stage. Expected behavior; not a bug.
3. **"Request has been denied. Unknown error."** Several overlapping causes: wrong password (reset via Django shell), reputation lockout after repeated failures (`Reputation.objects.all().delete()` — see [124-authentik](../../../wiki/containers/106-auth-outpost.md)), and **broken default expression policies**. The restored DB carried 8 default policies authored in old `return`-style syntax incompatible with 2026.5.x's eval context; `ak apply_blueprints` re-applied the current defaults.
4. **Browser ran stale frontend JS.** Console showed `version 2026.2.2` while the backend was `2026.5.2` — because DNS still pointed at the old LXC (see DNS cutover above), not a cache issue.
5. **WebAuthn devices dead post-migration.** Passkeys are device/origin-bound and don't survive a host move. Deleted all WebAuthn devices via Django ORM; users must re-register MFA.
## Architecture delta
| | Before | After |
|---|---|---|
| Authentik host | [LXC 124](../../../wiki/containers/106-auth-outpost.md) `192.168.8.180` | VPS `82.165.190.79`, `auth` Docker net `172.30.1.0/24` |
| Version | `2026.2.2` | `2026.5.2` |
| `auth.hubris.network` (LAN) | dnsmasq → `192.168.8.175` (Caddy) | dnsmasq → `82.165.190.79` (VPS traefik) |
| `auth.hubris.network` (public) | IONOS wildcard → VPS → mesh → LXC 124 | IONOS wildcard → VPS → local container |
| Reachability | mesh-only (split-horizon) | **public** (auth UI now exposed on the open internet) |
| Redis | none (docs said 2026.x dropped it) | present (this deploy uses Redis) |
| DB engine | sqlite? (`StoreConfig` in mgmt) / Postgres for Authentik | Postgres, separate from netbird |
> **Security note:** the Authentik admin UI is now publicly reachable. Mitigated with a traefik `admin-allowlist` ipAllowList middleware on `/if/admin/` (currently `5.61.168.0/24`). Login/flow endpoints are public by necessity. The `auth` Docker network isolates Postgres/Redis from the netbird containers.
## Lessons learned
1. **Never put the IdP behind the VPN it authenticates.** The circular dependency is invisible until a restart, then it's total.
2. **`depends_on: condition: service_healthy` is the structural fix.** Docker refuses to start mgmt until Authentik is healthy, regardless of boot order.
3. **Migrating an Authentik DB requires carrying its `AUTHENTIK_SECRET_KEY`.** It encrypts client secrets at rest; a mismatch breaks SSO silently with no useful error.
4. **Default expression policies break across major Authentik versions.** Always run `ak apply_blueprints` after restoring an older DB into a newer Authentik.
5. **Split-horizon DNS + a VPN resolver cache = an invisible stale answer.** The NetBird resolver (`100.122.255.254`) outlives `netbird down/up`. Removing the dnsmasq override (fall through to public DNS) is the durable fix once a service is genuinely public.
6. **Redirect URIs: prefer `STRICT` over `REGEX`** in Authentik 2026.x.
7. **Frontend/backend version skew shows as "Unknown error."** Confirm which Authentik the browser actually loaded (`version` line in the console) before chasing backend logs.
8. **WebAuthn/passkeys do not survive host migrations** — delete and re-enroll.
## Forward-auth outpost — RESOLVED 2026-06-01
Forward-auth apps (Paperless, qBittorrent, Artifacto) initially still validated against LXC 124's *embedded* outpost (Caddy → `192.168.8.180:9000`) — split-brain against the frozen DB. Pointing Caddy at `https://auth.hubris.network` instead fails: VPS Traefik rewrites `X-Forwarded-Host` → outpost can't match the app → 404 (tested + reverted).
Fixed with a **dedicated LAN outpost** ([106 — auth-outpost](../../../wiki/containers/106-auth-outpost.md), `192.168.8.6`): `goauthentik/proxy` connects outbound to the VPS core and serves forward-auth locally; Caddy → outpost over the LAN, no Traefik, header preserved. Outpost `hubris-lan-outpost` carries the 3 proxy providers. Verified with 124-Authentik **stopped**. This was Phase 1 of the broader architecture migration (plan: VPS edge / hubris LAN core / Mac Mini redundancy).
### 2026-06-05 — identification stage skip: broken "Trust me" reputation policy
The default authentication flow was skipping the identification stage (showing only a password field with `pending_user: ""`), making login impossible. Root cause: a **"Trust me" reputation policy** (`threshold: -3`, `negate: true`) attached to the identification stage binding (#10). The `negate: true` + broken policy evaluation caused the stage binding to fail re-evaluation, and Authentik's `ReevaluateMarker` skipped the stage entirely.
**Fix:** Deleted from Postgres:
```sql
DELETE FROM authentik_policies_policybinding WHERE policy_id = '<uuid>';
DELETE FROM authentik_policies_reputation_reputationpolicy WHERE policy_ptr_id = '<uuid>';
DELETE FROM authentik_policies_policy WHERE policy_uuid = '<uuid>';
```
The policy was orphaned (no matched type data or had incompatible evaluation). Removing it restores normal two-stage login (identification → password).
**Note:** This is different from the 2026-05-31 issue where NetBird's `login_hint` OAuth2 param intentionally pre-identifies users. That was expected behavior; this was a genuine bug.
## Outstanding
- **NetBird client version skew.** Mac client is `0.68.3` vs mgmt `0.71.3`; NetBird-SSH cert exchange fails between them. Upgrade all clients toward `0.71.x`.
- **republic-laptop** regular SSH publickey auth still rejected (correct `authorized_keys`/perms/SELinux context, yet denied — `sshd_config` `AuthorizedKeysFile` suspect). DNS not yet flushed there either.
- **muli-laptop** needs `netbird down && netbird up` + `resolvectl flush-caches`.
- **VPS port 22** opened for this repair; close once remote access is otherwise stable.
- **Decommission LXC 124 Authentik** after a ~2-week dual-run validation. dnsmasq stays on 124 regardless (separate service).
- **Reconcile [124-authentik](../../../wiki/containers/106-auth-outpost.md) provider notes** — docs describe a `Public`/PKCE provider; the migrated DB carries the `Confidential` `netbird-dashboard` client. Verify which is live and correct the page.
- **sops-encrypt** the VPS secrets (`/opt/authentik.env`) into the `secrets/` tree.
## Related
- [124 — authentik](../../../wiki/containers/106-auth-outpost.md)
- [DNS split-horizon](../../../wiki/infrastructure/dns.md)
- [Public ingress (VPS traefik)](../../../wiki/infrastructure/ingress.md)
- [Mesh migration](../../../wiki/infrastructure/mesh.md)

191
archive/knowledge/investigations/2026-06-01-mac-mini-onboarding.md Normal file
View File

@@ -0,0 +1,191 @@
# mac-mini onboarding — post-mortem & lessons learned
Onboarded the `mac-mini` workstation (macOS Sequoia, arm64) into the hubris
homelab context system with the `--with-hermes` profile. What follows is a
chronological recap of every hitch, the fix, and the systemic improvements
needed so the next workstation takes 5 min instead of an hour.
## Session log
### Step 1 — clone + symlink
- Manually cloned `git.hubris.network/dtoro/Homelab-Docs` to `/Users/dtoro/Homelab-Docs`.
- Created `/opt/homelab-context` → `/Users/dtoro/Homelab-Docs` symlink.
- **Lesson:** bootstrap.sh was designed to do this from scratch, but we'd
already cloned by hand. The bootstrap's `clone exists; pulling` path handled
it gracefully.
### Step 2 — hostname mismatch
- `scutil --get LocalHostName` → `Davids-Mac-mini`
- `hostname -s` → `Mac`
- Inventory file: `hosts/mac-mini.yaml`
- **Fix:** `sudo scutil --set LocalHostName mac-mini && sudo scutil --set HostName mac-mini`
- **Lesson:** The bootstrap and `homelab whoami` use different hostname
resolution. Bootstrap uses `scutil --get LocalHostName` (correct on macOS),
but the `homelab` CLI binary uses `hostname -s`. Both need to match the
inventory key. On a fresh macOS machine, neither does.
### Step 3 — bootstrap dependencies
- pyyaml was missing → `pip install pyyaml`
- age and sops were missing → `brew install age sops`
- Netbird was already installed and connected ✓
- **Lesson:** The bootstrap preflight handles these, but only if running
`bootstrap.sh` from the start. Since we ran it after manual setup, some
steps (netbird install) were correctly skipped as already-present.
### Step 4 — full bootstrap with `--with-mcp --with-hermes`
- Ran `sudo HOMELAB_GITEA_TOKEN=... bash bootstrap.sh --with-mcp --with-hermes`
- Age key issued ✓
- Launchd sync timer installed ✓
- Goose binary installed ✓
- Hermes CLI linked ✓
- MCP config merged ✓
- `refresh-creds` skipped (not yet a recipient) ⚠️
- Cosmetics: `chown: dtoro: illegal group name` at the end (benign, macOS
group-naming quirk)
### Step 5 — finalize from hubris
- Ran `homelab client add mac-mini --finalize-pubkey <age...> --with-hermes` on
hubris
- Push failed: `[rejected] main -> main (fetch first)` — hubris clone was
stale, bootstrap had already pushed from mac-mini
- **Fix:** `git pull --rebase && git push` on hubris
- **Lesson:** bootstrap pushes remote changes before hubris can finalize,
creating a race. The `homelab client add --finalize-pubkey` command should
pull before committing/pushing.
### Step 6 — sops couldn't find the age key
- `homelab secret hello` failed because sops looks in
`/Users/dtoro/.ssh/id_rsa` etc. by default, not `/etc/age/key.txt`
- The `homelab` CLI re-execs via `sudo -E env SOPS_AGE_KEY_FILE=... sops ...`,
but this requires passwordless sudo and the correct env var passthrough
- **Fix:**
1. Added NOPASSWD sudo rules
2. Eventually `SOPS_AGE_KEY` env with the raw key content worked directly
- **Lesson:** Document the explicit `SOPS_AGE_KEY_FILE` incantation in
agent-enrollment troubleshooting. New clients can't assume `homelab secret`
works out of the gate — the sudo re-exec chain has permission pitfall.
### Step 7 — OpenRouter key was a placeholder
- `secrets/openrouter-api-key.yaml` contained
`api_key: PLACEHOLDER_REPLACE_WITH_REAL_OPENROUTER_KEY`
- User ran `sops` on hubris, but got the same error (age key not found on
hubris either — `/root/.config/sops/age/keys.txt` didn't exist)
- **Fix:** `SOPS_AGE_KEY_FILE=/etc/age/key.txt sops ...` on hubris.
Later: the user pasted the real key, but the sops file showed
`sk-or-...5c55` — the literal content was truncated with ellipsis.
### Step 8 — editor loaded the wrong data
- Neovim on the system is configured with `clipboard+=unnamedplus`, which
points `*` and `+` registers to the macOS clipboard manager rather than
X11. When editing SOPS files, this caused the **system clipboard** to be
pasted instead of the actual ciphertext.
- This wasn't diagnosed during the session — the sops file would load empty
or show the wrong content because the editor's idea of "paste" was
disconnected from what sops expected.
- **Fix:** Run `sops` with `EDITOR=nano` or another editor that doesn't
hijack OS clipboards:
```bash
EDITOR=nano SOPS_AGE_KEY_FILE=/etc/age/key.txt sops secrets/openrouter-api-key.yaml
```
- **Lesson:** Add a strong warning to `hermes-agent.md` / `agent-enrollment.md`:
macOS neovim with `clipboard+=unnamedplus` silently breaks sops editing
because the paste register reads from the system clipboard instead of the
sops-managed buffer. Use `EDITOR=nano` or `EDITOR=vim` when running sops
interactively. Alternatively, override the clipboard option with
`EDITOR='nvim -c "set clipboard="'`.
- Also useful for the troubleshooting table in `agent-enrollment.md` under a
new row: "sops file loads empty / wrong content on macOS"
### Step 9 — model doesn't support tool use
- Goose config pinned `nousresearch/hermes-4-405b` via OpenRouter
- Error: `No endpoints found that support tool use`
- **Fix:** Switched to `deepseek/deepseek-v4-flash` in
`~/.config/goose/config.yaml`
- Also updated `operations/hermes-agent.md` with the correct model
- **Lesson:** The default model in `bootstrap.sh` and `hermes-agent.md` was
never validated on OpenRouter for tool-use capability. Need to either:
(a) Pin a model known to work (`deepseek/deepseek-v4-flash`)
(b) Or have the bootstrap probe OpenRouter at install time
### Step 10 — MCP homelab extension returns 404
- Goose's homelab extension configured as `streamable_http` at
`https://mcp.hubris.network/mcp` returns HTTP 404
- The actual MCP server runs on SSE (`/sse`), not streamable_http
- **Known follow-up** from `hermes-agent.md` — server needs migration
- **Workaround:** The agent works fine via the built-in `developer` extension
(shell + file) and the `homelab` CLI
### Step 11 — git credential dance
Several layers of issues with git authentication:
1. **Directory permissions:** `/etc/homelab-context` was `0700 root` — dtoro
couldn't access the credential file, and git's `store` helper couldn't
create its `.lock` file.
- **Fix 1a:** `sudo chmod 755 /etc/homelab-context`
- **Fix 1b:** `sudo chown -R dtoro:staff /etc/homelab-context`
2. **macOS built-in osxkeychain:** Apple's git has `osxkeychain` compiled in
as a default credential helper. Even after configuring the `store` helper,
git calls osxkeychain after every successful `get`, which prompts for
macOS keychain access.
- `credential.helper = ""` does NOT disable the compiled-in default
- The `store` helper's `.lock` file in `/etc/homelab-context/` also failed
silently, causing fallback behaviour
- **Fix:** Embed the credential in the remote URL directly:
`https://dtoro:TOKEN@git.hubris.network/dtoro/Homelab-Docs.git`
- This bypasses all credential helpers entirely
3. **refresh-creds:** The credential is now managed by the remote URL.
If the PAT is rotated, `homelab refresh-creds` won't update the URL.
- **Workaround:** Run `git remote set-url origin` after `refresh-creds`
### Step 12 — `homelab` CLI hostname detection
- `homelab whoami` failed with `no hosts/Mac.yaml` because the CLI uses
`hostname -s` (returning `Mac`) while the bootstrap uses
`scutil --get LocalHostName` (which we fixed to `mac-mini`)
- **Fix:** Set both `LocalHostName` and `HostName` via `scutil`
- **Lesson:** The homelab CLI on macOS should prefer `scutil --get
LocalHostName` like the bootstrap does, or at least try both and pick the
one that matches an inventory key. This is a CLI bug.
## Improvement backlog
### High priority
| # | Issue | Fix |
|---|-------|-----|
| 1 | Homelab CLI uses `hostname -s` on macOS; should use `scutil --get LocalHostName` to match bootstrap | Patch `bin/homelab` to try `scutil` first, fall back to `hostname -s` |
| 2 | `homelab client add --finalize-pubkey` on hubris fails to push when remote is ahead (bootstrap pushes first) | `pull --rebase` before committing in `cmd_client_add` |
| 3 | Default Goose model (`nousresearch/hermes-4-405b`) doesn't support tool use on OpenRouter | Update `bootstrap.sh` default and `hermes-agent.md` to `deepseek/deepseek-v4-flash` |
| 4 | MCP server needs streamable_http migration | Follow-up #1 from `hermes-agent.md` — FastMCP `transport="sse"` → `transport="streamable_http"` |
| 5 | `homelab refresh-creds` should also update embedded remote URLs | In `cmd_refresh_creds`, after writing the credential file, update any remote URL that has an embedded credential |
### Medium priority
| # | Issue | Fix |
|---|-------|-----|
| 6 | macOS neovim with `clipboard+=unnamedplus` breaks sops editing | Add troubleshooting row to `agent-enrollment.md`; recommend `EDITOR=nano` |
| 7 | `/etc/homelab-context` directory with 0700 root permissions blocks non-root git | Change to 755 in `bootstrap.sh` on all OSes (or at least macOS) |
| 8 | Bootstrap `chown` at end gives `illegal group name` on macOS | Fix group detection in bootstrap.sh for macOS |
| 9 | `homelab secret` re-exec via sudo chain is fragile — needs passwordless sudo or TTY | Consider `SOPS_AGE_KEY` env-var fallback path in the CLI itself |
### Low priority
| # | Issue | Fix |
|---|-------|-----|
| 10 | Onboarding this machine revealed no `HERMES.md` file in the repo | Create the persona file for the Goose hints symlink |
| 11 | Git credential `store` helper lock mechanism is OS-sensitive and fragile | Document embedded-URL pattern as the preferred approach for workstations |
| 12 | AGENTS.md references `/opt/homelab-context/hosts/<hostname>.yaml` but the canonical path is now a symlink | Verify the path resolution edge cases |
## Changelog
### 2026-06-01 — initial post-mortem
Captured the full onboarding of mac-mini into the hubris homelab context
system, including the hostname mismatch, sudo/sops credential chain, model
selection, git credential gymnastics, and editor clipboard interference.
### 2026-06-02 — MCP server migrated to streamable-http
Item #4 resolved. FastMCP server now uses `transport="streamable-http"` (hyphen,
not underscore), serving at `https://mcp.hubris.network/mcp`. All doc references
and client configs (Goose, bootstrap.sh, agent-enrollment.md) updated. Service
re-deployed on apps LXC (105).

84
archive/knowledge/investigations/2026-06-03-moonlight-sunshine-wifi-jitter.md Normal file
View File

@@ -0,0 +1,84 @@
# 2026-06-03 — Moonlight/Sunshine game streaming unstable over WiFi
## Summary
[`ludo-mini`](../../../hosts/strong.yaml) runs Sunshine as the game-streaming server; [`mac-mini`](../../../hosts/mac-mini.yaml) runs Moonlight as the client. Despite both machines being on the same physical subnet (192.168.178.0/24), streaming was unstable — stuttering, dropouts, and high latency. Root cause: **mac-mini is connected only via WiFi**, while ludo-mini is wired Ethernet (2.5 Gbps). WiFi throughput shows 1-second UDP dropouts and high jitter (28 ms stddev), which breaks real-time video streaming.
## Timeline
### Pre-existing state
- ludo-mini: wired Ethernet (eno1, 2.5 Gbps), 192.168.178.181/24
- mac-mini: WiFi (en1, 802.11ac, 234 Mbps Tx rate, 1 stream, -60 dBm RSSI), 192.168.178.99/24
- Both on the same subnet via a consumer router at 192.168.178.1
- Sunshine configured: bitrate=80 Mbps, encoder=amf (AMD GPU), fec_percentage=5, hevc_mode=2
- Moonlight configured: bitrate=57 Mbps, fps=60, localaddr=192.168.178.181 (correct)
### 2026-06-03 — investigation
**Network tests (iperf3 between mac-mini ↔ ludo-mini):**
- TCP throughput: 49–86 Mbps, average ~65 Mbps — highly variable
- UDP throughput: 1-second complete dropout during a 5-second test; 0% loss otherwise
- Ping: 4.7 ms avg, but **28 ms stddev**, max **138 ms** — WiFi-level jitter
- Netbird P2P connection also showed 7 ms latency over the tunnel (unnecessary given same-subnet direct connection)
**Additional findings:**
- mac-mini has a built-in Ethernet port (en0) but it is inactive — no cable connected
- mac-mini also has Thunderbolt Ethernet adapters (en5, en6, en7) all inactive
- ludo-mini's en0 shows `speed 2500` (2.5 GbE)
- Netbird routes `192.168.8.0/24` via `utun100`, but 192.168.178.0/24 traffic stays on `en1`
## Root cause
mac-mini is on WiFi. WiFi introduces:
1. **Throughput variance** — 49–86 Mbps TCP fluctuation
2. **Dropouts** — 1-second UDP blackouts from radio retransmissions
3. **Jitter** — 28 ms stddev with spikes to 138 ms
Moonlight/Sunshine streams real-time video over UDP. Any jitter spike or dropout causes visual stutter or frame drops. The 57 Mbps Moonlight bitrate was too aggressive for the WiFi link's consistency, even though the average throughput is ~65 Mbps.
## Mitigations applied
### mac-mini — Moonlight (via `defaults write`)
| Setting | Before | After | Rationale |
||---------|--------|-------|-----------|
|| bitrate | 57 Mbps | 25 Mbps | Reduce to a level WiFi can sustain consistently |
|| framepacing | off (0) | on (1) | Smooths frame timing over variable latency |
|| fps | 30 | 60 | Restored to game-appropriate value |
### mac-mini — Moonlight (current, wired — 2026-06-04)
| Setting | WiFi value | Wired value | Rationale |
||---------|-----------|-------------|-----------|
|| bitrate | 60 Mbps | 80 Mbps | Wired 1 GbE can sustain comfortably |
|| framepacing | on (1) | off (0) | Wired latency is consistent, no smoothing needed |
|| fps | 60 | 60 | Unchanged |
### ludo-mini — Sunshine (via SSH)
| Setting | WiFi value | Wired value | Rationale |
||---------|-----------|-------------|-----------|
|| bitrate | 40 Mbps | 80 Mbps | Wired can handle full bandwidth |
|| fec_percentage | 2 | 5 | Restored to default — wired has no dropout concern |
|| packetsize | 1024 | 1316 | Restored to default (Ethernet MTU) |
Sunshine restarted after changes; confirmed active.
## Resolution
### 2026-06-04 — mac-mini wired to LAN
mac-mini plugged into Ethernet (en0, 192.168.178.182). This definitively resolves the WiFi jitter that caused streaming instability. All WiFi-era conservative settings reverted to appropriate wired values on both ends.
## Open questions
~~1. **Ethernet wiring** — mac-mini has an active en0 port. Plugging it into the same switch/router as ludo-mini would eliminate WiFi jitter entirely. This is the definitive fix.~~ **RESOLVED** — mac-mini now wired via en0.
2. **WiFi alternatives** — If wiring is impractical, a WiFi 6 (802.11ax) USB adapter or Thunderbolt-to-Ethernet adapter would help, but wired Ethernet is the only reliable solution for game streaming.
3. **Netbird route** — `192.168.8.0/24` is routed over Netbird `utun100`. If Moonlight ever falls back to that subnet (ludo-mini's 192.168.8.133 LAN IP), traffic tunnels unnecessarily. Moonlight correctly uses 192.168.178.181, so this is not currently an issue.
## Changelog
### 2026-06-04 — mac-mini wired to LAN, configs reverted from WiFi mitigations
- mac-mini connected via en0 (wired Ethernet, 192.168.178.182), WiFi mitigations no longer needed
- Moonlight: bitrate 60→80 Mbps, framepacing disabled
- Sunshine: bitrate 40→80 Mbps, fec_percentage 2→5, packetsize restored to default (1316)
- First root cause in investigation (Open questions #1) resolved

101
archive/knowledge/investigations/2026-06-06-authentik-session-lifetime.md Normal file
View File

@@ -0,0 +1,101 @@
# 2026-06-06 — Frequent Authentik login prompts fixed (session duration)
## Summary
User needed to re-authenticate to Authentik several times per day. Root cause was the Django session being configured as a **session cookie** (cleared on browser close) with a short 24-hour lifetime. Fixed by extending both the session cookie lifetime and the user_login stage session duration to 30 days.
## Timeline
- **2026-06-06 ~23:00** — User reports "having to login to authentik several times per day"
- **~23:10** — Investigation begins: check Authentik OAuth2 provider settings, proxy session table, Django session config
- **~23:20** — Found `SESSION_EXPIRE_AT_BROWSER_CLOSE=True` — the `authentik_session` cookie has no Max-Age, cleared on browser close
- **~23:25** — Found the `user_login` stage has `session_duration=seconds=0`, which calls `session.set_expiry(0)` → `_session_expire_at_browser_close=True`
- **~23:30** — Applied DB fix: updated `authentik_stages_user_login_userloginstage` → `session_duration='days=30'`
- **~23:35** — Added `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30` to `/opt/authentik.env` (maps to `sessions.unauthenticated_age` in Authentik config, setting `SESSION_COOKIE_AGE` to 30 days)
- **~23:40** — Recreated container with `docker compose up -d` (restart doesn't re-read env file)
- **~23:45** — Verified both changes took effect
## Root cause
Two independent but compounding issues:
1. **`SESSION_EXPIRE_AT_BROWSER_CLOSE=True`** — hardcoded in `/authentik/root/settings.py`. Makes the `authentik_session` cookie a session cookie (no `Max-Age`), cleared when the browser closes. Cannot be changed via env vars or config files (hardcoded in Python source).
2. **`user_login` stage `session_duration=seconds=0`** — the default-authentication-login stage called `session.set_expiry(0)`, which in Django sets `_session_expire_at_browser_close=True` on the session object, overriding any session cookie lifetime set via `SESSION_COOKIE_AGE`.
3. **`sessions.unauthenticated_age=days=1`** (default) — `SESSION_COOKIE_AGE` was 86,400 seconds (24 hours), so even with the browser left open, the server-side session data expired after 24 hours with no way to extend it (`SESSION_SAVE_EVERY_REQUEST=False`).
### Why forward-auth worked but OAuth2 didn't
- **Forward-auth (Caddy-gated services)** — The `authentik_proxy_*` cookie is a *persistent* cookie with 24-hour validity, set by the Authentik proxy outpost. It survives browser restart. Caddy's forward-auth validates this cookie directly with the outpost API — no Django session needed.
- **OAuth2 (Gitea, Jellyfin, etc.)** — These services redirect to `auth.hubris.network/application/o/authorize/`, which checks the Django session (`authentik_session` cookie). If the browser was closed, this cookie is gone → user is redirected to the login form.
## Changes applied
### 1. Database — `authentik_stages_user_login_userloginstage`
```sql
UPDATE authentik_stages_user_login_userloginstage
SET session_duration = 'days=30'
WHERE stage_ptr_id = (
SELECT stage_uuid FROM authentik_flows_stage
WHERE name = 'default-authentication-login'
);
```
This causes the login stage to call `session.set_expiry(timedelta(days=30))`, which:
- Sets `_session_expire_at_browser_close=False` for that session
- Sets the session cookie `Max-Age` to 30 days (persistent cookie)
- Sets the server-side session expiry to 30 days
### 2. Environment — `AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE`
Added to `/opt/authentik.env`:
```
AUTHENTIK_SESSIONS__UNAUTHENTICATED_AGE=days=30
```
This maps to config key `sessions.unauthenticated_age`, which Authentik's `settings.py` reads via:
```python
SESSION_COOKIE_AGE = timedelta_from_string(
CONFIG.get("sessions.unauthenticated_age", "days=1")
).total_seconds()
```
Result: `SESSION_COOKIE_AGE` = 2,592,000 seconds (30 days). This is a fallback for sessions that don't go through the `user_login` stage or when the stage's explicit `set_expiry` doesn't apply.
## Verification
```python
# Inside authentik-server container:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "authentik.root.settings")
import django
django.setup()
from django.conf import settings
print("SESSION_COOKIE_AGE:", settings.SESSION_COOKIE_AGE) # → 2592000.0
print("SESSION_EXPIRE_AT_BROWSER_CLOSE:", settings.SESSION_EXPIRE_AT_BROWSER_CLOSE) # → True
from authentik.stages.user_login.models import UserLoginStage
stage = UserLoginStage.objects.filter(name="default-authentication-login").first()
print("session_duration:", stage.session_duration) # → "days=30"
```
## Open questions
- `SESSION_COOKIE_SECURE=False` — Should be `True` since Authentik is HTTPS-only, but the custom `SessionMiddleware.is_secure()` method dynamically sets the cookie's `secure` flag based on the request, so it's fine.
- 30 days is generous. Could be reduced to 7-14 days if desired. Change the DB value and env var accordingly.
## Related
- [Container 106 — auth-outpost](../../wiki/containers/106-auth-outpost.md)
- [Authentik VPS migration](archive/2026-05-31-authentik-vps-migration.md)
- [Ingress (VPS Traefik)](../../wiki/infrastructure/ingress.md)
- `.hermes/plans/2026-06-06_232200-authentik-frequent-login-fix.md` — original plan
## Changelog
### 2026-06-06 — created
Documented the session lifetime investigation, root cause, and applied fixes.

61
archive/knowledge/investigations/2026-06-06-caddyfile-truncation.md Normal file
View File

@@ -0,0 +1,61 @@
# Investigation: Caddyfile truncation — all LAN services down (2026-06-06)
**Date:** 2026-06-06
**Status:** resolved
**Duration:** ~10 hours (from last known good state ~12:39 UTC to restoration ~22:40 UTC)
## Symptom
All `*.hubris.network` URLs except `photos.hubris.network` and `auth.hubris.network` returned `tlsv1 alert internal error` or TCP timeouts from LAN/mesh clients. `dig @192.168.8.2` and `dig @100.122.255.254` both resolved to `192.168.8.175` correctly — DNS was fine. The issue was at the Caddy level.
## Root cause
The Caddyfile on LXC 121 was manually edited directly on the filesystem (not via the `dtoro/caddy-conf` git repo), reducing it from 260 lines/30+ site blocks to 43 lines with only 3 photo-related site blocks: `photos.hubris.network`, `prism.hubris.network`, and `photos2.hubris.network`.
### Timeline
| Time (UTC+2) | Event |
|---|---|
| Jun 04 23:43 | Last successful git-push deploy — full Caddyfile (260 lines) |
| Jun 06 ~12:00 | Caddyfile manually edited locally, truncating to 3 sites |
| Jun 06 12:39 | Deploy webhook triggered → `git pull --ff-only` failed: "Your local changes would be overwritten" |
| Jun 06 14:13 | Deploy webhook triggered again → `deploy ok` (the truncated file was committed or merged somehow) |
| Jun 06 22:34 | Investigation began |
| Jun 06 22:43 | Caddyfile restored from `origin/master`, `systemctl reload caddy` |
### Evidence
- `git diff HEAD -- Caddyfile` on LXC 121: `+3 / -159` lines
- Git reflog: HEAD at `32575ce` (fix: sab port 8081→8082), working tree diverged
- Backup file `Caddyfile.bak.1780263919`: 225 lines, full original config
- `git stash list` shows one auto-stash entry
- `origin/master` at `1b977aa`: 260 lines, all site blocks present
### Secondary root cause found during investigation
**elementsynapse (LXC 118)** had `iface eth0 inet dhcp` internally despite `pct set 118 --net0 ... ip=192.168.8.239/24`. On DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool. Caddy's `reverse_proxy 192.168.8.239:8008` was hitting a dead IP.
This is the same class of drift as the June 5th incidents (paperless, HAOS, apps, mule-images). Elementsynapse was missed during the 2026-06-02 static-IP migration.
## Fix applied
1. **Caddyfile** → `git checkout --force origin/master -- Caddyfile` + `systemctl reload caddy`
2. **elementsynapse** → replaced `iface eth0 inet dhcp` with static, killed dhclient, verified connectivity
## Permanent safeguards (all deployed)
| Safeguard | Location | What it does |
|---|---|---|
| Site-count guard | `/etc/caddy/scripts/deploy.sh` | Refuses reload if <20 `hubris.network` site blocks |
| Dirty-tree auto-stash | `/etc/caddy/scripts/deploy.sh` | Stashes local edits before git pull |
| Auto-backup | `/etc/caddy/scripts/deploy.sh` | Saves Caddyfile.bak.<timestamp> before any change, keeps 5 |
| Caddy backend health | `/etc/cron.d/caddy-backend-health` on hubris | Runs `check-caddy-backends.sh` every 10 min |
| DNS sync | `/etc/cron.d/dns-sync` on LXC 107 | Runs `dns-sync.py` every 10 min (was missing since 2026-06-04) |
## Related
- DHCP drift investigation (previous incident) — not filed as its own investigation; see the [DNS sync fix](../../../.hermes/plans/2026-06-05_170000-prevent-dhcp-ip-drift.md)
- [Caddy (121)](../../wiki/containers/121-caddy.md)
- [elementsynapse (118)](../../wiki/containers/118-elementsynapse.md)
- [dns-sync script](../../../scripts/dns-sync.py)
- [check-caddy-backends script](../../../scripts/check-caddy-backends.sh)

29
archive/knowledge/investigations/index.md Normal file
View File

@@ -0,0 +1,29 @@
# Investigations
Time-stamped incident reports and experiments. One entry per incident; the entry is the canonical source. Per-node changelog entries link back here.
## Active / recent
| Date | Title | Status |
| ------------ | ------------------------------------------------------------------ | ------------- |
| 2026-06-06 | [Caddyfile truncation incident](2026-06-06-caddyfile-truncation.md) | Resolved — permanent safeguards deployed (site-count guard, auto-stash, auto-backup) |
| 2026-06-06 | [Frequent Authentik login prompts — session lifetime fix](2026-06-06-authentik-session-lifetime.md) | Resolved — `session_duration=days=30`, `SESSION_COOKIE_AGE=30d` |
| 2026-06-03 | [Moonlight/Sunshine streaming — WiFi jitter](2026-06-03-moonlight-sunshine-wifi-jitter.md) | Mitigations applied; definitive fix requires wiring mac-mini via Ethernet |
| 2026-06-01 | [Mac-mini onboarding](2026-06-01-mac-mini-onboarding.md) | Onboarded |
## Resolved (archived)
See [`archive/`](archive/):
| Date | Title |
| ------------ | ------------------------------------------------------------------ |
| 2026-04-21 | [Hubris crash loop — thermal + USB drive](archive/2026-04-21-hubris-crash-loop.md) |
| 2026-05-31 | [Authentik migrated from LXC 124 to the VPS](archive/2026-05-31-authentik-vps-migration.md) |
## Conventions
- File name: `YYYY-MM-DD-<slug>.md`. Use the *first* date if the incident spans multiple days.
- Mandatory sections: Summary, Timeline, Root cause, Mitigations applied, Open questions.
- Update the entry as the situation evolves; never rewrite history. Add new dated sections at the bottom.
- Link back from every node's changelog that's affected.
- Move to `archive/` when the incident is fully resolved and no longer actively referenced.

10
archive/knowledge/log.md Normal file
View File

@@ -0,0 +1,10 @@
# Knowledge — operations log
Append-only record of documentation-maintenance operations on the knowledge wiki (restructures,
source ingests, lint sweeps). One line per operation, newest last. Infrastructure changes belong in
each page's `## Changelog` and the Oikos change ledger, not here.
## [2026-07-06] restructure | moved node/infrastructure narratives under knowledge/wiki/; references under knowledge/sources/; repointed inventory doc_page fields and gen-topology.py output.
## [2026-07-06] lint | banned-vocabulary scan of knowledge/ clean; added .agents/skills/docs-lint and knowledge/wiki/hosts/index.md.
## [2026-07-06] lint | fixed 126 pre-existing broken links (124-authentik.md rename, investigations/plans moved to archive/done, archive/ sibling depth, destroyed-node delinks); 2 remaining are an intentional cross-repo reference.
## [2026-07-06] restructure | Phase 6 consolidation: investigations/ -> knowledge/sources/investigations/; operations/ -> .agents/operations/; HERMES.md -> .agents/; deleted root OIKOS/CAVEMAN/CONTRIBUTING stubs.

351
archive/knowledge/references/cert-sync-and-traefik-config.md Normal file
View File

@@ -0,0 +1,351 @@
# Current cert sync script + traefik dynamic config
Snapshot of the two artifacts that control public service exposure as of
2026-07-05. Updated 2026-07-05: fixed Jellyfin backend from dead hubris IP
(192.168.8.206) to new strong IP (192.168.8.246).
## hubris-public-cert-sync.sh (PVE host, `/usr/local/bin/`)
```bash
#!/bin/bash
# Mirrors home caddy's LE certs for publicly-exposed hubris.network hostnames
# into the VPS traefik's /letsencrypt volume. Traefik file-watches the volume
# and hot-reloads.
#
# Why: netbird-proxy's HostSNI(*) TCP passthrough intercepts ACME TLS-ALPN-01
# challenges before traefik's allowACMEByPass can respond, so traefik can't
# obtain its own cert. Home caddy uses IONOS DNS-01 (no such conflict);
# we just mirror what it already has.
#
# Runs daily via hubris-public-cert-sync.timer.
set -euo pipefail
CADDY_LXC=121
CADDY_BASE=/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory
VPS_HOST=root@100.122.165.149
VPS_DEST=/var/lib/docker/volumes/opt_netbird_traefik_letsencrypt/_data
# Map: source hostname -> "crt_filename key_filename" on the VPS.
# Stable names so traefik dynamic.yaml doesn't need edits on renewal.
declare -A HOSTS=(
[artifacto.hubris.network]="fullchain.crt privkey.key"
[blog.hubris.network]="blog.fullchain.crt blog.privkey.key"
[trmnl.hubris.network]="trmnl.fullchain.crt trmnl.privkey.key"
[sso.hubris.network]="sso.fullchain.crt sso.privkey.key"
[media.hubris.network]="media.fullchain.crt media.privkey.key"
[paperless.hubris.network]="paperless.fullchain.crt paperless.privkey.key"
)
TMP=$(mktemp -d)
trap 'rm -rf "$TMP"' EXIT
for host in "${!HOSTS[@]}"; do
read -r crt_name key_name <<< "${HOSTS[$host]}"
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.crt" "$TMP/$crt_name"
pct pull "$CADDY_LXC" "$CADDY_BASE/$host/$host.key" "$TMP/$key_name"
if ssh -o BatchMode=yes "$VPS_HOST" "test -f $VPS_DEST/$crt_name && diff -q - $VPS_DEST/$crt_name" < "$TMP/$crt_name" >/dev/null 2>&1; then
echo "hubris-public-cert-sync: $host unchanged"
continue
fi
scp -q -o BatchMode=yes "$TMP/$crt_name" "$TMP/$key_name" "$VPS_HOST:$VPS_DEST/"
echo "hubris-public-cert-sync: shipped $host ($(openssl x509 -in "$TMP/$crt_name" -noout -enddate))"
done
```
### Adding a new host
1. Caddy must already have the cert (verify `pct exec 121 -- ls "$CADDY_BASE/$host/"`)
2. Add a line to the HOSTS array: `[new-host.hubris.network]="nickname.fullchain.crt nickname.privkey.key"`
3. `systemctl start hubris-public-cert-sync.service` to sync immediately
4. Verify certs landed: `ssh "$VPS_HOST" "ls -la $VPS_DEST/nickname.*"`
5. Add matching `tls.certificates` entry in traefik dynamic config
---
## traefik-dynamic.yaml (VPS, `/opt/`)
```yaml
tcp:
serversTransports:
pp-v2:
proxyProtocol:
version: 2
tls:
certificates:
- certFile: /letsencrypt/fullchain.crt
keyFile: /letsencrypt/privkey.key
- certFile: /letsencrypt/blog.fullchain.crt
keyFile: /letsencrypt/blog.privkey.key
- certFile: /letsencrypt/trmnl.fullchain.crt
keyFile: /letsencrypt/trmnl.privkey.key
- certFile: /letsencrypt/sso.fullchain.crt
keyFile: /letsencrypt/sso.privkey.key
- certFile: /letsencrypt/media.fullchain.crt
keyFile: /letsencrypt/media.privkey.key
- certFile: /letsencrypt/paperless.fullchain.crt
keyFile: /letsencrypt/paperless.privkey.key
http:
routers:
artifacto-public:
rule: 'Host(`artifacto.hubris.network`) && (PathPrefix(`/p/`) || PathPrefix(`/static/`) || Path(`/healthz`))'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- artifacto-strip-sso
- artifacto-ratelimit
service: artifacto-public
blog-public:
rule: 'Host(`blog.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- blog-ratelimit
service: blog-public
trmnl-public:
rule: 'Host(`trmnl.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- trmnl-ratelimit
service: trmnl-public
matrix-public:
rule: 'Host(`matrix.hubris.network`) && !PathPrefix(`/.well-known/matrix/`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- matrix-ratelimit
service: matrix-public
matrix-wellknown:
rule: 'Host(`matrix.hubris.network`) && (PathPrefix(`/.well-known/matrix/`) || PathPrefix(`/.well-known/acme-challenge/`))'
entryPoints:
- websecure
priority: 20
tls:
certResolver: letsencrypt
service: matrix-wellknown-svc
house-public:
rule: Host(`house.hubris.network`)
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- house-ratelimit
service: house-public
sso-public:
rule: 'Host(`sso.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- sso-ratelimit
service: sso-public
media-public:
rule: 'Host(`media.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- media-ratelimit
service: media-public
paperless-api-public:
rule: 'Host(`paperless.hubris.network`) && PathPrefix(`/api/`)'
entryPoints:
- websecure
priority: 20
tls:
certResolver: letsencrypt
middlewares:
- paperless-ratelimit
service: paperless-public
paperless-public:
rule: 'Host(`paperless.hubris.network`)'
entryPoints:
- websecure
priority: 10
tls:
certResolver: letsencrypt
middlewares:
- authentik-forwardauth
- paperless-ratelimit
service: paperless-public
middlewares:
artifacto-strip-sso:
headers:
customRequestHeaders:
X-Artifacto-Gateway: ""
X-Authentik-Username: ""
X-Authentik-Groups: ""
X-Authentik-Email: ""
X-Authentik-Name: ""
X-Authentik-Uid: ""
X-Authentik-Jwt: ""
X-Authentik-Meta-Jwks: ""
X-Authentik-Meta-Outpost: ""
X-Authentik-Meta-Provider: ""
X-Authentik-Meta-App: ""
X-Authentik-Meta-Version: ""
admin-allowlist:
ipAllowList:
sourceRange:
- "5.61.168.0/24"
artifacto-ratelimit:
rateLimit:
average: 50
period: 1s
burst: 100
blog-ratelimit:
rateLimit:
average: 100
period: 1s
burst: 200
trmnl-ratelimit:
rateLimit:
average: 20
period: 1s
burst: 40
matrix-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
house-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
sso-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
media-ratelimit:
rateLimit:
average: 30
period: 1s
burst: 60
paperless-ratelimit:
rateLimit:
average: 20
period: 1s
burst: 40
authentik-forwardauth:
forwardAuth:
address: "http://192.168.8.6:9000/outpost.goauthentik.io/auth/traefik"
trustForwardHeader: true
authResponseHeaders:
- X-authentik-username
- X-authentik-groups
- X-authentik-email
- X-authentik-name
- X-authentik-uid
- X-authentik-jwt
- X-authentik-meta-jwks
- X-authentik-meta-outpost
- X-authentik-meta-provider
- X-authentik-meta-app
- X-authentik-meta-version
services:
artifacto-public:
loadBalancer:
servers:
- url: 'http://192.168.8.205:3100'
blog-public:
loadBalancer:
servers:
- url: 'http://192.168.8.205:8080'
trmnl-public:
loadBalancer:
servers:
- url: 'http://192.168.8.211:9851'
matrix-public:
loadBalancer:
servers:
- url: 'http://192.168.8.242:8008'
matrix-wellknown-svc:
loadBalancer:
servers:
- url: 'http://matrix-wellknown:80'
house-public:
loadBalancer:
servers:
- url: 'http://192.168.8.244:3000'
sso-public:
loadBalancer:
servers:
- url: 'http://192.168.8.6:9000'
media-public:
loadBalancer:
servers:
- url: 'http://192.168.8.246:8096'
paperless-public:
loadBalancer:
servers:
- url: 'http://192.168.8.130:8000'
```
### Adding a new service — four blocks needed
1. **Router** — `http.routers.<name>-public` with `tls: {}` (not
`certResolver`)
2. **Middleware** — rate limit, one per service
3. **Service** — `http.services.<name>-public` with the backend URL
4. **tls.certificates** — add a new `- certFile/keryFile` pair matching the
cert sync HOSTS entry
### Key file naming convention
| Cert name | Host | Convention |
|-----------|------|------------|
| `fullchain.crt` + `privkey.key` | `artifacto.hubris.network` | First service — no prefix |
| `blog.fullchain.crt` + `blog.privkey.key` | `blog.hubris.network` | `{nickname}.fullchain.crt` |
| `trmnl.fullchain.crt` + `trmnl.privkey.key` | `trmnl.hubris.network` | `{nickname}.fullchain.crt` |
| `sso.fullchain.crt` + `sso.privkey.key` | `sso.hubris.network` | `{nickname}.fullchain.crt` |
| `media.fullchain.crt` + `media.privkey.key` | `media.hubris.network` | `{nickname}.fullchain.crt` |
| `paperless.fullchain.crt` + `paperless.privkey.key` | `paperless.hubris.network` | `{nickname}.fullchain.crt` |
### ⚠️ Critical — keep backends in sync after LXC migrations
When moving an LXC between Proxmox nodes, update **both**:
1. **Caddy** (`/etc/caddy/Caddyfile` on LXC 121)
2. **VPS traefik** (`/opt/traefik-dynamic.yaml` — via hubris bridge SSH)
Jellyfin migration from hubris to strong (2026-07-05) was fixed in Caddy
but **missed** in VPS traefik — old IP `192.168.8.206` remained. This caused
Bad Gateway for off-LAN users. Use Python-based editing (see
`references/traefik-config-editing.md`) for accurate surgical fixes.

9
archive/knowledge/sources/index.md Normal file
View File

@@ -0,0 +1,9 @@
# Sources
Immutable evidence the wiki synthesizes from. External reference docs live under `references/`;
incident evidence lives in [`investigations/`](investigations/index.md) (written once at
incident time, then linked from the changelogs of the nodes they implicate).
| Slug | Reference | Summary |
|------|-----------|---------|
| cert-sync-and-traefik-config | [references/cert-sync-and-traefik-config.md](references/cert-sync-and-traefik-config.md) | VPS traefik config and the LAN↔VPS certificate mirror. |

90
archive/knowledge/vms/100-zimaos.md Normal file
View File

@@ -0,0 +1,90 @@
# 100 — `zimaos`
ZimaOS (IceWhale / CasaOS-family NAS distro), installed as a Proxmox VM to evaluate it as a potential primary NAS frontend in front of `/mnt/library` — alongside the existing fleet ([nextcloud (114)](../containers/114-nextcloud.md), [jellyfin (101)](../containers/101-jellyfin.md), [mule-images (120)](../containers/120-mule-images.md)).
## At a glance
- **Type:** QEMU VM
- **ZimaOS version:** `v1.6.1` (build 2026-04-21, kernel 6.12.25)
- **IP:** `192.168.8.195` (DHCP lease from the LAN router — see "Open items" below)
- **Hostname (LAN DNS):** [`zimaos.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) `192.168.8.175` → VM `192.168.8.195:80`. TLS terminates at Caddy (Let's Encrypt via IONOS DNS-01).
- **Resources:** 4 vCPU, 8 GiB RAM (balloon min 2 GiB), 64 GiB OS disk on `local-lvm`
- **BIOS / machine:** OVMF (UEFI), `q35`, `virtio-scsi-pci`
- **EFI disk:** none — ZimaOS installer writes a removable-media fallback (`/EFI/BOOT/BOOTX64.EFI`), so a persistent EFI vars disk isn't required. PVE warns "no efidisk configured! Using temporary efivars disk" on start; harmless.
- **ISO:** `/var/lib/vz/template/iso/zimaos-x86_64-1.6.1_installer.iso` (1.48 GB, from `IceWhaleTech/ZimaOS` GitHub releases; upstream publishes SHA256 only for the `.img`/`.raucb` variants, not the `.iso`)
## Web UI
- **LAN:** <https://zimaos.hubris.network> (via [caddy (121)](../containers/121-caddy.md))
- **Netbird mesh (off-LAN):** same URL — works because hubris advertises `192.168.8.0/24` into the mesh, and once a peer resolves `zimaos.hubris.network → 192.168.8.175`, the request tunnels home. DNS resolution from a mesh peer needs **either** a netbird Management nameserver group (`hubris.network` match domain → `192.168.8.180`), or an `/etc/hosts` override on the peer (`192.168.8.175 zimaos.hubris.network`). See [mesh](../infrastructure/mesh.md).
- **Direct fallback:** <http://192.168.8.195> — backend ZimaOS bundles its own Caddy (`Server: Caddy`, `Via: ZimaOS-Gateway`), which the front-end Caddy on LXC 121 reverse-proxies to. Two Caddys in series.
## SSH
Disabled by default. To enable:
1. Toggle SSH on in the ZimaOS web UI ("Settings → SSH"), OR
2. From the VM console, switch to TTY2 with **Alt+F2** and run `passwd` to set a root password first.
## Storage strategy
**Initial assumption (didn't work).** PVE 9's native virtiofs share with the host's existing `library` dir mapping (used by [108-haos](108-haos.md)) was the obvious path. Attaching it took one line — but ZimaOS's kernel ships *without* the virtiofs module (`modinfo virtiofs` → `Module virtiofs not found`). The kvm-side device was attached fine; the guest just can't mount it. Detached on 2026-05-14.
**What we ended up with.** A dedicated NFS re-export LXC [102-nfs-export](../containers/102-nfs-export.md) bind-mounts `/mnt/library` from the host and serves it over NFSv4 to ZimaOS. Performance is within ~2% of host-served NFS (LXC adds namespace isolation only, not an IO hop), with the upside that no NFS/RPC daemons run on the bare-metal host. Read-only during evaluation; promote to rw once the UI test confirms behavior.
**Filesystem layout inside ZimaOS.** `/etc/fstab` lives on a writable overlay so persistent mounts via fstab DO survive reboots, but the *standard* path to add a network share is the ZimaOS Files UI (Connect Network Storage → NFS). UID/GID mapping is squashed at the server to `www-data:media` (33:10000) — the same identity Nextcloud and mule-images use — so any rw activity from ZimaOS records as a normal `media`-group write.
**Caveats discovered during install.**
- Root filesystem is **squashfs / read-only**; only `/DATA` is writable.
- `dtoro` identity is `uid 999, gid 1000(samba), groups samba+wheel`. Primary group is **`samba`** (not `dtoro` — that group doesn't exist). SSH key needs `chown dtoro:samba` to satisfy sshd StrictModes.
- `dtoro`'s home directory is literally `/DATA` (not `/home/dtoro`). Authorized keys go at `/DATA/.ssh/authorized_keys`.
- `/etc` is a writable overlay rooted at `/mnt/overlay/upper_etc`. Mounts added to `/etc/fstab` persist across reboots.
- `mount.nfs4` is a stripped busybox variant — rejects `_netdev`, `nofail`, `actimeo=N`. The working minimal fstab line: `192.168.8.200:/mnt/library /media/library nfs rw,vers=4 0 0`. **Mount path matters** — see the 2026-05-15 Changelog entry on why the NFS export is mounted at `/media/library` and not `/DATA/library`.
- No `qemu-guest-agent` is bundled — `qm guest cmd` calls fail. IP discovery via `qm monitor` + `screendump` instead.
- No package manager at runtime — no `apt`, no `opkg`. Apps come via ZimaOS's CasaOS-style appstore only.
The alternative (dedicated virtual data disk on the `library` lvmthin pool, e.g. `qm set 100 --scsi1 library:1024`) was rejected because it would start empty and force content migration; the NFS path keeps `/mnt/library` as the single source of truth shared with the existing 8 LXCs.
## Open items
- **DHCP → static IP fixed (2026-06-03).** ZimaOS IP drifted from `.195` (Slate AX) → `.103` (Technitium) after the DHCP migration, causing Caddy 502s. Fixed by injecting a static systemd-networkd config and restarting the VM. IP now pinned at `192.168.8.195`. See [changelog](#2026-06-03--static-ip-set-to-195-dhcp-drift-fixed).
- **No Authentik wiring.** [authentik (124)](../containers/106-auth-outpost.md) isn't enforcing auth in front of ZimaOS yet — ZimaOS handles its own first-run wizard. The Caddyfile block uses bare `reverse_proxy` rather than the `import authentik` pattern used by e.g. artifacto; layer it in once the wizard is complete and a static admin user exists.
- **No PBS backup.** No Proxmox Backup Server configured on hubris today; this VM is not backed up.
- **qemu-guest-agent not installed.** ZimaOS's installer doesn't bundle it, so `qm guest cmd 100 ...` returns "QEMU guest agent is not running". IP discovery during this install was done via console screendump → `qm monitor` → `screendump`.
## Related
- [108 — `haos-16.3`](108-haos.md) — the existing VM, same q35/OVMF pattern; also a virtiofs `library` consumer
- [DNS](../infrastructure/dns.md) — split-horizon entry lives here
- [Caddy (121)](../containers/121-caddy.md) — would front this if/when promoted
- [Media permissions](../infrastructure/media-permissions.md) — relevant when wiring `/mnt/library`
## Changelog
### 2026-06-03 — Static IP set to `.195`; DHCP drift fixed
ZimaOS had drifted from `.195` (Slate AX DHCP) → `.103` (Technitium DHCP), causing Caddy 502s. Injected `/etc/systemd/network/10-static.network` into overlay (match `en*/eth*`, address `192.168.8.195/24`, gateway `.1`, DNS `.2`). VM restarted; verified reachable at `.195`. Caddy (`zimaos.hubris.network`) now returns 200. See [plan](../../../.hermes/plans/2026-06-03_223218-dhcp-pool-exclude-static-ips.md).
### 2026-05-15 — NFS mount relocated to `/media/library` (UI delete fix)
Symptom: deleting any file or folder inside the NFS-served `library` tree from the ZimaOS Files UI failed with `rename ... invalid cross-device link`. Root cause: ZimaOS's "Move to Trash" is implemented as `rename(2)` into a `.trash/` directory derived from the *drive root*, and icewhale-files identifies the drive root with the regex `^/media/([^/]+)`. With the NFS mounted at `/DATA/library`, the visible UI path was `/media/ZimaOS-HD/library/foo` → drive `ZimaOS-HD` → trash at `/media/ZimaOS-HD/.trash` (which lives on the local ext4 `/dev/sda8`). NFS → ext4 rename = `EXDEV`.
Fix: remount the NFS export at `/media/library` so it sits as its own `/media/<name>` segment. Now icewhale-files extracts drive=`library`, computes trash at `/media/library/.trash`, and the rename stays within the NFS — no cross-device. The local-storage daemon (LSBLK-based) doesn't surface the NFS in the Files-app *sidebar* (drives sidebar only lists block devices), but the icewhale-files daemon picks the mount up as its own "storage" tile in the Files view, and crucially exposes a direct **Delete** action there (no trash flow, since ZimaOS treats non-LSBLK mounts as foreign and skips trash for them). For a 4 TB NFS share, immediate delete is arguably the right behavior anyway.
Concrete change:
- `/etc/fstab` line changed from `… /DATA/library nfs rw,vers=4 0 0` to `… /media/library nfs rw,vers=4 0 0`.
- New mountpoint dir `/var/lib/casaos_data/.media/library` (= `/media/library` via the existing `/media` bind).
- Old `/DATA/library` mountpoint dir removed.
Side effect: navigating via `ZimaOS-HD → library` in the UI now shows an empty/missing folder. Users access the library via the separate `library` tile in the Files app instead.
### 2026-05-14 — Mesh-reachable from netbird peers
Verified `https://zimaos.hubris.network` reachable from `republic-laptop` (100.122.78.140) through the existing hubris→netbird subnet advertisement (`192.168.8.0/24`). No new infrastructure required — the path was already wired by [mesh](../infrastructure/mesh.md). Open: configure a netbird Management nameserver group for `hubris.network` so peers don't need per-device `/etc/hosts` overrides.
### 2026-05-14 — Phase 2 promoted to rw
NFS export on [nfs-export (102)](../containers/102-nfs-export.md) flipped from `ro` to `rw` after Files UI evaluation passed. Verified the squash: a write from ZimaOS appears on hubris's `/mnt/library` as `www-data:media` (uid 33, gid 10000), matching the existing tree convention.
### 2026-05-14 — Phase 2: NFS mount via new LXC 102 (read-only)
Virtiofs path abandoned — ZimaOS kernel 6.12.25 ships without the virtiofs module (verified by `modinfo virtiofs` returning "Module not found"). Pivoted to NFSv4: built [nfs-export (102)](../containers/102-nfs-export.md) as a dedicated 512-MiB privileged Debian LXC bind-mounting `/mnt/library` and serving it on `192.168.8.200:2049` with `ro,all_squash,anonuid=33,anongid=10000` to the LAN subnet. Detached the now-useless `virtiofs0` from VM 100. ZimaOS-side mount via Files UI (NFS protocol, server `192.168.8.200`, path `/mnt/library`). Promote export to `rw` after Files UI evaluation passes.
### 2026-05-14 — Wired through Caddy (HTTPS)
Added `zimaos.hubris.network` site block to `/etc/caddy/Caddyfile` on [caddy (121)](../containers/121-caddy.md): bare `reverse_proxy 192.168.8.195` + IONOS DNS-01 TLS, same pattern as plato/jellyfin. dnsmasq entry repointed from `192.168.8.195` to `192.168.8.175`. Let's Encrypt cert issued on first request. Caddy commit `a219176` pending push to `dtoro/caddy-conf`.
### 2026-05-14 — VM created, ZimaOS 1.6.1 installed (Phase 1)
`qm create 100` with q35/OVMF, no EFI disk, 4 vCPU / 8 GiB / 64 GiB on `local-lvm`. Installed via the official ISO (manual console install). Web UI verified at `http://192.168.8.195`. `onboot=1`, `startup order=20`. dnsmasq entry `zimaos.hubris.network → 192.168.8.195` initially added direct-to-VM on [authentik (124)](../containers/106-auth-outpost.md) (later repointed — see above). `/mnt/library` is **not** yet shared into the VM; Phase 2 (virtiofs) is gated on UI evaluation.

44
archive/knowledge/vms/108-haos.md Normal file
View File

@@ -0,0 +1,44 @@
# 108 — `haos-16.3`
Home Assistant OS — the only VM on hubris (HAOS doesn't run cleanly in an LXC, hence the qm tenant).
## At a glance
- **Type:** QEMU VM
- **HAOS version:** 16.3 (last verified)
- **IP:** `192.168.8.101`
- **Resources:** 4 GiB RAM, 32 GiB boot disk
- **Public hostname:** [`home.hubris.network`](../infrastructure/dns.md) → [caddy (121)](../containers/121-caddy.md) → `192.168.8.101:8123`
## Auth
Native OIDC via the HACS integration `christiaangoossens/hass-oidc-auth` (repo `https://github.com/christiaangoossens/hass-oidc-auth`).
Key gotchas:
- HAOS containers don't honor the Network-panel DNS. Set Supervisor DNS via:
```
ha dns options --servers "dns://192.168.8.180" --servers "dns://1.1.1.1"
```
so OIDC discovery resolves internally to [authentik (124)](../containers/106-auth-outpost.md).
- Authentik app slug in the discovery URL is whatever was set in Authentik — confirm via the DB rather than guessing. User set `home-assistant` (with hyphen).
- YAML config:
- `features.automatic_user_linking: true` — link to existing HA users by `preferred_username` match (otherwise a duplicate is created).
- `features.default_redirect: true` — skip the welcome-splash so users land on the normal HA login page.
- Run `ha core restart` after each config change — HA caches DNS and OIDC discovery across frontend reloads.
## Telemetry
HA pulls Proxmox metrics via the official Proxmox VE integration. As of 2026-04-21 [claudio-monitor](../infrastructure/monitoring.md) stopped publishing to MQTT/REST (commit `82f0596`) — HA gets metrics from PVE directly; claudio-monitor focuses on alerting.
## Related
- [Authentik (124)](../containers/106-auth-outpost.md)
- [Caddy (121)](../containers/121-caddy.md)
- [DNS](../infrastructure/dns.md)
- [Monitoring](../infrastructure/monitoring.md)
## Changelog
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-21 — wired into Authentik via HACS hass-oidc-auth
Supervisor DNS pointed at LXC 124 dnsmasq; YAML features for auto-linking + default redirect.

14
archive/knowledge/vms/index.md Normal file
View File

@@ -0,0 +1,14 @@
# VMs — index
Two QEMU VMs running on [hubris](../hosts/hubris.md):
| ID | Name | Role | IP | Public hostname |
|----|------|------|----|-----------------|
| 100 | [zimaos](100-zimaos.md) | NAS frontend eval (ZimaOS) | `192.168.8.195` | [`zimaos.hubris.network`](../infrastructure/dns.md) |
| 108 | [haos-16.3](108-haos.md) | Home automation (HAOS) | `192.168.8.101` | [`home.hubris.network`](../infrastructure/dns.md) |
## Related
- [Hubris host](../hosts/hubris.md) — both VMs run here
- [Containers index](../containers/index.md) — LXCs on both nodes
- [README](../../../README.md)