docs: move narrative wiki under knowledge/wiki/ (phase 3)

Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-06 14:35:23 +02:00
parent bb5c0717a1
commit 8a6422bd7d
129 changed files with 249 additions and 264 deletions

View File

@@ -0,0 +1,90 @@
# 123 — `claudio-bot` (DEPRECATED — destroyed 2026-06-04)
> **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini.
> Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob.
> Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea.
> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details.
Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications.
## At a glance
- **Hostname:** `claudio-bot`
- **IP:** `192.168.8.230`
- **Privilege:** **unprivileged**
- **Resources:** 1 core / 512 MiB RAM / 8 GiB rootfs
- **Mounts:** none from `/mnt/library`
- **Public hostname:** none
## Stack
Repo `dtoro/claudio-bot`, checkout at `/opt/claudio-bot`, systemd unit `claudio-bot.service`. Connects to Matrix at `http://192.168.8.239:8008` (direct LAN to [synapse (118)](118-elementsynapse.md), avoids hairpin-NAT TLS issue on `matrix.hubris.network`).
Room: `!dEUVJArVKPorxHHJZK:avispero` (invite-only, `@dtoro:avispero` allowed).
## Configuration
Secrets at `/etc/claudio-bot/` (mode 600):
- `config.yaml`
- `matrix.token`
- `anthropic.key`
- `lmstudio.key`
- `ipc.token`
Deploy git creds at `/etc/claudio-deploy/git-credentials`.
### LLM backend (pluggable)
`llm.backend` in `config.yaml`:
- `anthropic` — Claude API via `anthropic.key`
- `lmstudio` — OpenAI-compat HTTP via `lmstudio.key` (bearer)
Currently set to `lmstudio``google/gemma-4-e4b` on the Mac mini at `192.168.8.174:1234` (since 2026-04-25). Switch back with `backend:` + `systemctl restart claudio-bot`. Original config saved at `/etc/claudio-bot/config.yaml.bak`. The LM Studio agent in `bot_core/lmstudio.py` translates Anthropic → OpenAI tool definitions.
## IPC
`http://192.168.8.230:9090/{notify,propose,status}`. Header `X-Bot-Token` must match `/etc/claudio-bot/ipc.token`. Used by:
- [claudio-monitor on hubris](../infrastructure/monitoring.md) for edge-triggered alerts (token in `/etc/claudio-monitor/bot.token`)
- The (currently disabled) [restic backup wrapper](../infrastructure/backups.md) (token in `/etc/restic/bot.token`)
> Token files at the source side **must hold the same value as `/etc/claudio-bot/ipc.token`** — rotate together.
## Plugins
Module under `plugins/<name>.py` exposing a `Plugin` class; add `<name>` to `plugins:` in `config.yaml`. Plugins can register slash commands, Claude tools, and `on_notify` / `on_proposal_response` hooks.
Active plugins:
- `system``ping`, `help`, `status`, `list_plugins`
- `backup` — ingests `/notify` from the backup wrapper (currently silent — backups disabled)
- `monitor``/monitor status`, `/monitor history [N]`, `/monitor clear <key>`. Tools: `get_alerts`, `get_monitor_history` for NL queries via Claude.
## Auto-deploy
Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/deploy` → pull + `pip install` + restart. Same shape as caddy-conf.
`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.230`.
## Related
- [elementsynapse (118)](118-elementsynapse.md)
- [Monitoring (claudio-monitor)](../infrastructure/monitoring.md)
- [Backups (disabled)](../infrastructure/backups.md)
- [Auto-deploy](../infrastructure/auto-deploy.md)
## Changelog
### 2026-06-04 — LXC destroyed; replaced by Hermes Agent
LXC 123 destroyed via `pct destroy 123 --purge`. Bot service stopped, systemd
units disabled. `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived on
Gitea. Monitoring replaced by Hermes `homelab-health-watchdog` cron job.
`@claudio:avispero` Matrix account decommissioned.
### 2026-04-28 — wiki entry created
Initial documentation.
### 2026-04-25 — LLM backend switched to LM Studio
`backend: lmstudio``google/gemma-4-e4b` on the Mac mini. Anthropic key still present so the swap is reversible by flipping the config field.
### 2026-04-21 — `monitor` plugin added
Receives events from [claudio-monitor](../infrastructure/monitoring.md). Slash commands + tools registered. See `dtoro/claudio-bot` commit `e56da25`.
### 2026-04-20 — claudio-bot deployed
LXC 123 provisioned. Repo, systemd unit, Matrix wiring, `system` + `backup` plugins, IPC server.

View File

@@ -0,0 +1,313 @@
# 127 — `mule-photos-new`
Side-by-side **PhotoPrism M0 test** of the `dtoro/mule-image` `new` branch
at `photos-new.hubris.network`. Production [LXC 120](120-mule-images.md) keeps
running on the legacy stack at `photos.hubris.network` until M5 cutover.
## At a glance
- **Hostname:** `mule-photos-new`
- **IP:** `192.168.8.181`
- **Privilege:** unpriv
- **Resources:** 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap
- **Features:** `nesting=1,fuse=1,keyctl=1`
- **Mounts:** *(none — see scratch copy below)*
- **Public hostname:** [`photos-new.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`)
## Stack (`/opt/mule-image`)
`/opt/mule-image` is the working tree of `dtoro/mule-image` on branch
`new`. Compose lives at `docker-compose.photoprism.yml`; LXC-127-only
overrides at `docker-compose.photoprism.override.yml` (untracked — see
[Why an override exists](#why-an-override-exists)). Invoked with
`--env-file .env.photoprism`.
| Service | Container | Port | Notes |
| ----------- | ------------- | ----------------- | -------------------------------------------------- |
| mariadb | `pp-mariadb` | `127.0.0.1:3306` | MariaDB 11; named volume `pp_mariadb_data`; init SQL provisions `mule_sidecar.marks` table |
| photoprism | `pp-app` | `:2342` | `docker.io/photoprism/photoprism:latest`, runs as uid 33 (PP_UID/GID env) |
| sidecar | `pp-sidecar` | `:8000` (overridden) | Go + Gin service for rename / folder mutations / heap convert / dup detect |
| vite (host) | systemd unit | `:5173` | SvelteKit dev server (`mule-vite.service`), `npm run dev` in `/opt/mule-image/web` |
PhotoPrism is enterprise-tier (`tier: 1`) per the build tag (`-Plus`). TF
vision pipeline and EXIF backwrite are **disabled** in M0 — `PP_READONLY=true`
keeps the originals view read-only as the M0 safety net.
## Library — writable rsync scratch copy (NOT the real admin Photos)
Unlike LXC 120 (which mounts `/mnt/library` directly), this LXC has **no
bind-mount of the production library**. Instead a one-shot rsync of the
admin's Photos lives on the LXC's own rootfs:
- **Host source (read-only reference):** `/mnt/library/homecloud/admin/files/Photos` (~4.9 GB)
- **LXC scratch (writable):** `/srv/photos-scratch` (owner `www-data:media`, mode 0775)
This means **sidecar rename / folder mutation operations land in the
scratch copy, not the real admin library**. The scratch is not
auto-synced — it's a snapshot from `2026-05-17`. To refresh from
production:
```bash
# on hubris (LXC 127 must be stopped to mount its rootfs)
pct stop 127
pct mount 127
rsync -aHAX --info=stats2 --chown=100033:110000 --no-perms \
--chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r \
/mnt/library/homecloud/admin/files/Photos/ \
/var/lib/lxc/127/rootfs/srv/photos-scratch/
pct unmount 127
pct start 127
```
`--chown=100033:110000` accounts for the unprivileged-LXC ID shift
(host 100033 = LXC `www-data`, host 110000 = LXC `media`). Don't try a
bind-mount of `/mnt/library/...` — the admin Photos tree is 0750 and
unprivileged LXCs can't see through.
## Auth — Authentik OIDC
PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](124-authentik.md).
- **Provider/Application slug:** `mule-photos-new`
- **Issuer:** `https://auth.hubris.network/application/o/mule-photos-new/`
- **Redirect URI:** `https://photos-new.hubris.network/api/v1/oidc/redirect`
- **Scopes:** `openid profile email`
- **Initiated by clicking the OIDC button** at `/library/login``GET /api/v1/oidc/login` → 302 to Authentik authorize.
- `OIDC_REGISTER=true` + `OIDC_ROLE=admin` so the first SSO login auto-creates a PhotoPrism admin account.
Local PhotoPrism admin (username `admin`, password in
`/root/mule-photos-new-secrets.txt` on hubris) stays available as a
fallback.
## Why an override exists
`docker-compose.photoprism.override.yml` is **only on LXC 127** (not in
the git repo). After the 2026-05-17/18 fixes landed upstream
(commits `cce1d87` for OIDC env names and `3d8e050` for worker caps),
the override has shrunk to just one stanza — the cross-host sidecar
bind:
```yaml
services:
sidecar:
ports: !override
- "0.0.0.0:8000:8000"
```
Upstream binds the sidecar to `127.0.0.1:8000` because the M4 design
colocates Caddy with the sidecar. On this test LXC Caddy lives on a
different host (LXC 121), so the port has to be reachable from the
LAN. Delete this file at M4 cutover.
## Auto-deploy
Mirrors the LXC 120 pattern.
- **Webhook listener:** `mule-deploy-webhook.service``python3 /opt/mule-deploy/webhook.py` on `0.0.0.0:9797`.
- **Branch filter:** `refs/heads/new` (LXC 120 still owns `main`).
- **HMAC secret:** `/etc/mule-deploy/secret` (mode 0600).
- **Deploy script:** `/opt/mule-deploy/deploy.sh``git fetch && git reset --hard origin/new`, fix `pp/{storage,import}` ownership to `33:10000` (PP container uid), `docker compose ... up -d --build --force-recreate` with both compose files, `systemctl restart mule-vite`.
- **Gitea webhook id 9** on `dtoro/mule-image` pointed at `http://192.168.8.181:9797/deploy`.
Push to the `new` branch on [git.hubris.network/dtoro/mule-image](http://git.hubris.network/dtoro/mule-image) → webhook fires → rebuild. The legacy LXC 120 watches `main` and is unaffected.
**Gitea gotcha:** the receiver IP must be in `[webhook] ALLOWED_HOST_LIST`
in `/etc/gitea/app.ini` on [LXC 104](104-gitea.md). LXC 127's
`192.168.8.181` was missing on first bring-up; every push delivered
status 0 with the message `webhook can only call allowed HTTP servers`.
Adding the IP and `systemctl restart gitea` is enough — same list is
also the gating mechanism for the LXC 120 webhook. Verify any future
test instance is added before relying on auto-deploy.
## Bootstrap secrets
Saved on hubris at `/root/mule-photos-new-secrets.txt` (mode 0600):
- `PP_ADMIN_PASSWORD` — initial PhotoPrism `admin` login
- `PP_DB_PASSWORD` — MariaDB `photoprism` user
- `PP_DB_ROOT_PASSWORD` — MariaDB root
- `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` — generated by `ak shell` against Authentik
`SIDECAR_DB_PASSWORD` is still the literal placeholder
`replace-at-m4-bringup` because `mariadb/init/01-sidecar.sql` hardcodes
it; rotate before this stack ever goes public.
## Health checks
```bash
# from hubris
pct exec 127 -- curl -sf http://127.0.0.1:2342/api/v1/status # PP
pct exec 127 -- curl -sf http://127.0.0.1:8000/api/sidecar/healthz # sidecar
pct exec 127 -- curl -sf http://127.0.0.1:5173/ # Vite
# through Caddy
curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \
https://photos-new.hubris.network/api/v1/oidc/login -i | head -2 # 302 → auth.hubris.network
```
> **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack
> validated here was promoted into production on [LXC 120](120-mule-images.md)
> via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This
> page is retained for archaeology; everything below is historic. See the
> 2026-05-22 entry in [120-mule-images.md](120-mule-images.md#changelog) for
> the cutover detail.
## Changelog
### 2026-05-22 — Destroyed
`pct destroy 127` after curl verification of the new 120 stack passed
end-to-end. dnsmasq `photos-new.hubris.network` line removed; gitea
webhook id 9 + `192.168.8.181` ALLOWED_HOST_LIST entry removed; caddy
`photos-new.hubris.network` site block dropped from `dtoro/caddy-conf`.
Authentik `mule-photos-new` app + provider deleted.
### 2026-05-18 (pm) — OIDC state-cookie fix: deploy.sh no longer recreates pp-app
User reported `failed to get state: securecookie: the value is not valid` after
authenticating at Authentik. Root cause: PhotoPrism rotates the `Session:` HMAC
key in `pp/storage/config/hub.yml` on **every container start** (it's the hub
auto-refresh; no env flag disables it, and with `Status: ""` it regenerates
each boot). The key signs OIDC state cookies, so every restart invalidated
every in-flight login.
`/opt/mule-deploy/deploy.sh` was doing `docker compose up -d --build
--force-recreate` after a `pull --ignore-buildable`. Both moves recreate
pp-app: `--force-recreate` unconditionally, and the pull bumps the
`photoprism:latest` digest which makes plain `up -d` recreate too. Every
auto-deploy on a code push therefore broke every OIDC login.
Fixed by:
1. Dropping the `pull` step from the auto-deploy. Image refreshes for pp-app
/ mariadb are now an admin operation (run `docker compose pull` manually
when you want a new PhotoPrism build).
2. Force-recreating only the **sidecar** (its image rebuilds on every push
anyway). `up -d photoprism mariadb` reconciles in place — only restarts
if their compose declaration actually changes.
Verified by running `deploy.sh` twice and watching `hub.yml`'s `Session:`
field and pp-app's container PID; both stayed stable across the deploy.
The sidecar's PID changed as expected.
The fix lives in `/opt/mule-deploy/deploy.sh` on LXC 127. `/opt/mule-deploy`
is not a git checkout — keep this Changelog entry as the source of truth.
### 2026-05-18 — OIDC bridge + indexer + folder fixes
The `new` branch's PhotoPrism stack required several iterations to be
actually usable. Fixes pushed upstream so they apply to anyone running
the M0 compose; the LXC override file shrank to just the cross-host
sidecar port (see [Why an override exists](#why-an-override-exists)).
**OIDC end-to-end** (upstream commits `4abe6d7`, `9a3ad3e`, `cce1d87`,
plus Caddy `/library/* → /` bounce on LXC 121).
- The SvelteKit `/login` had a `// OIDC SSO ships in M4` placeholder
but no button. Added a "Sign in with {provider}" button conditional
on `/api/v1/config.ext.oidc.enabled`; click sends the browser to
`/api/v1/oidc/login`.
- The compose file passed OIDC values through `PHOTOPRISM_OIDC_ISSUER_URL`
/ `_CLIENT_ID` / `_CLIENT_SECRET` / `_PROVIDER_NAME` — names PhotoPrism
silently ignores. The actual env-var names are `PHOTOPRISM_OIDC_URI`
/ `_CLIENT` / `_SECRET` / `_PROVIDER` (see `photoprism show config`).
Renamed upstream; user-facing keys in `.env.photoprism` (OIDC_ISSUER_URL,
OIDC_CLIENT_ID, …) are unchanged.
- PhotoPrism's OIDC callback does **not** set `auth_token` / `auth_session`
cookies. It returns an HTML page that writes the session into
`localStorage` under `pp:<storageNamespace>:session.{id,token,user,provider}`
and then runs `window.location.href = "/library/login"`. Caddy on this
test instance bounces `/library/*` back to `/`, and the SvelteKit root
layout (`bootstrapSessionFromPhotoPrism()`) reads those localStorage
entries on mount, fetches `/api/v1/session/<id>` with the cookied
token, and adopts the session into the SPA store.
**Indexer caps** (upstream commit `3d8e050`). A fresh index of ~1.2k
photos pushed the LXC load average above 50 with the default
`PHOTOPRISM_INDEX_WORKERS` (NumCPU/2 = 3 here, each forking TF +
ffmpeg + libvips). Compose now reads `PP_WORKERS` / `PP_INDEX_WORKERS`
from `.env.photoprism`, defaulting to 2. Both set explicitly on this
LXC to keep sibling containers happy.
**Library went RW.** Flipped `PP_READONLY=false` and `PP_ORIGINALS_MODE=rw`
in `.env.photoprism` so the indexer can actually run — `READONLY=true`
disables it entirely. Safe because `/srv/photos-scratch` is the
rsync scratch copy on the LXC rootfs, not the real admin Photos tree.
**Admin role.** OIDC creates users with `OIDC_ROLE` ONLY on first
registration. The `dtoro` user was created in an earlier flow before
the env-var-name fix, so it landed as `guest` and saw no photos in
the UI. Promoted manually:
```sql
UPDATE auth_users SET user_role='admin', super_admin=1, can_invite=1
WHERE user_name='dtoro';
DELETE FROM auth_sessions WHERE user_name='dtoro';
```
Stale sessions are dropped so a fresh OIDC login mints an admin token.
**Video pre-transcode pass** (no upstream change — operational fix on
this LXC). Only 11/45 `.mov` originals had a `.avc` sidecar; the rest
forced 1221 s inline libx264 transcodes on first playback, serialised
one ffmpeg at a time. Measured cold vs warm:
| Path | TTFB |
| --------------------------------- | ------- |
| Thumbnail `fit_1280` (warm) | ~2 ms |
| Video playback with `.avc` sidecar | ~2 ms |
| Video playback without sidecar | 1221 s |
| Thumbnail GET *during* a transcode | ~2 ms (no sibling slowdown — 6-core LXC, ffmpeg ~6%/core, nvme util 0.01%) |
Mitigation:
```bash
pct exec 127 -- docker exec -d pp-app /opt/photoprism/bin/photoprism convert
```
Walks the library, builds every missing `.avc` next to its original
(`/photoprism/storage/sidecar/<year>/<month>/<name>.mov.avc`),
idempotent on re-run, two ffmpeg processes in parallel. Took ~8 min
to bring coverage to 45/45. Previously-cold videos verified to serve
at ~2 ms TTFB after the pass. Future imports get AVC sidecars
automatically as part of indexing; a re-run is only needed if videos
ever land outside the indexer's path.
**Folder tree** (upstream commits `8083328`, `505fef5`, `cfd85a1`).
PhotoPrism's `path:` operator is exact-match by default but supports
a `*` wildcard. Without it, every internal tree node (year folders,
since photos always nest under YYYY/MM) returned zero hits — both in
the timeline and in the sidecar's folder-count fan-out. Fixed both
to emit `path:"<folder>*"`. Also relaxed the root-folder client-side
filter (was clipping to `Path === ''`, which is always empty) so `/`
shows the whole library. Root badge in the sidebar now reads
`config.count.all` directly instead of subtracting Σ(folderCounts) —
the subtraction double-counted after the recursive switch.
### 2026-05-17 — Bring-up
LXC 127 created from `debian-13-standard_13.1-2`, joined to vmbr0 with
static IP `192.168.8.181`. Docker engine + Node 20 installed.
`dtoro/mule-image` cloned at branch `new`, compose stack
(`docker-compose.photoprism.yml`) brought up: MariaDB 11 + PhotoPrism
`:latest` (`-Plus` build) + Go sidecar (built locally). Vite dev server
running as `mule-deploy` via `mule-vite.service` on port 5173.
Authentik OIDC application `mule-photos-new` provisioned via `ak shell`
(`OAuth2Provider` + `Application` + STRICT `RedirectURI`). PhotoPrism's
OIDC button delegates to Authentik; `OIDC_REGISTER=true` /
`OIDC_ROLE=admin` so the first SSO login becomes admin.
Admin's Photos library rsynced (~4.9 GB, 1206 files) into
`/srv/photos-scratch` on the LXC rootfs (no bind-mount). Sidecar
mutations land in the scratch copy, not the real library.
Caddy site `photos-new.hubris.network` added in `dtoro/caddy-conf`;
dnsmasq entry on LXC 124 → `192.168.8.175`. dnsmasq required a `restart`
(not `reload`) for the new `address=` line to take effect.
`docker-compose.photoprism.override.yml` (LXC-only, untracked) pins two
upstream issues: sidecar bound to `127.0.0.1` (cross-host Caddy can't
reach), and OIDC env-var name mismatch
(`PHOTOPRISM_OIDC_ISSUER_URL` vs `PHOTOPRISM_OIDC_URI` and friends).
Should land upstream on `new` next iteration.