oikos/containers/127-mule-photos-new.md

127 — mule-photos-new

Side-by-side PhotoPrism M0 test of the dtoro/mule-image new branch at photos-new.hubris.network. Production LXC 120 keeps running on the legacy stack at photos.hubris.network until M5 cutover.

At a glance

• Hostname: mule-photos-new
• IP: 192.168.8.181
• Privilege: unpriv
• Resources: 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap
• Features: nesting=1,fuse=1,keyctl=1
• Mounts: (none — see scratch copy below)
• Public hostname: photos-new.hubris.network → caddy (121) → split (PhotoPrism :2342, sidecar :8000, Vite :5173)

Stack (/opt/mule-image)

/opt/mule-image is the working tree of dtoro/mule-image on branch new. Compose lives at docker-compose.photoprism.yml; LXC-127-only overrides at docker-compose.photoprism.override.yml (untracked — see Why an override exists). Invoked with --env-file .env.photoprism.

Service	Container	Port	Notes
mariadb	pp-mariadb	127.0.0.1:3306	MariaDB 11; named volume pp_mariadb_data; init SQL provisions mule_sidecar.marks table
photoprism	pp-app	:2342	docker.io/photoprism/photoprism:latest, runs as uid 33 (PP_UID/GID env)
sidecar	pp-sidecar	:8000 (overridden)	Go + Gin service for rename / folder mutations / heap convert / dup detect
vite (host)	systemd unit	:5173	SvelteKit dev server (mule-vite.service), npm run dev in /opt/mule-image/web

PhotoPrism is enterprise-tier (tier: 1) per the build tag (-Plus). TF vision pipeline and EXIF backwrite are disabled in M0 — PP_READONLY=true keeps the originals view read-only as the M0 safety net.

Library — writable rsync scratch copy (NOT the real admin Photos)

Unlike LXC 120 (which mounts /mnt/library directly), this LXC has no bind-mount of the production library. Instead a one-shot rsync of the admin's Photos lives on the LXC's own rootfs:

• Host source (read-only reference): /mnt/library/homecloud/admin/files/Photos (~4.9 GB)
• LXC scratch (writable): /srv/photos-scratch (owner www-data:media, mode 0775)

This means sidecar rename / folder mutation operations land in the scratch copy, not the real admin library. The scratch is not auto-synced — it's a snapshot from 2026-05-17. To refresh from production:

# on hubris (LXC 127 must be stopped to mount its rootfs)
pct stop 127
pct mount 127
rsync -aHAX --info=stats2 --chown=100033:110000 --no-perms \
  --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r \
  /mnt/library/homecloud/admin/files/Photos/ \
  /var/lib/lxc/127/rootfs/srv/photos-scratch/
pct unmount 127
pct start 127

--chown=100033:110000 accounts for the unprivileged-LXC ID shift (host 100033 = LXC www-data, host 110000 = LXC media). Don't try a bind-mount of /mnt/library/... — the admin Photos tree is 0750 and unprivileged LXCs can't see through.

Auth — Authentik OIDC

PhotoPrism's "Sign in with OIDC" button delegates to Authentik (124).

• Provider/Application slug: mule-photos-new
• Issuer: https://auth.hubris.network/application/o/mule-photos-new/
• Redirect URI: https://photos-new.hubris.network/api/v1/oidc/redirect
• Scopes: openid profile email
• Initiated by clicking the OIDC button at /library/login → GET /api/v1/oidc/login → 302 to Authentik authorize.
• OIDC_REGISTER=true + OIDC_ROLE=admin so the first SSO login auto-creates a PhotoPrism admin account.

Local PhotoPrism admin (username admin, password in /root/mule-photos-new-secrets.txt on hubris) stays available as a fallback.

Why an override exists

docker-compose.photoprism.override.yml is only on LXC 127 (not in the git repo). After the 2026-05-17/18 fixes landed upstream (commits cce1d87 for OIDC env names and 3d8e050 for worker caps), the override has shrunk to just one stanza — the cross-host sidecar bind:

services:
  sidecar:
    ports: !override
      - "0.0.0.0:8000:8000"

Upstream binds the sidecar to 127.0.0.1:8000 because the M4 design colocates Caddy with the sidecar. On this test LXC Caddy lives on a different host (LXC 121), so the port has to be reachable from the LAN. Delete this file at M4 cutover.

Auto-deploy

Mirrors the LXC 120 pattern.

• Webhook listener: mule-deploy-webhook.service → python3 /opt/mule-deploy/webhook.py on 0.0.0.0:9797.
• Branch filter: refs/heads/new (LXC 120 still owns main).
• HMAC secret: /etc/mule-deploy/secret (mode 0600).
• Deploy script: /opt/mule-deploy/deploy.sh — git fetch && git reset --hard origin/new, fix pp/{storage,import} ownership to 33:10000 (PP container uid), docker compose ... up -d --build --force-recreate with both compose files, systemctl restart mule-vite.
• Gitea webhook id 9 on dtoro/mule-image pointed at http://192.168.8.181:9797/deploy.

Push to the new branch on git.hubris.network/dtoro/mule-image → webhook fires → rebuild. The legacy LXC 120 watches main and is unaffected.

Gitea gotcha: the receiver IP must be in [webhook] ALLOWED_HOST_LIST in /etc/gitea/app.ini on LXC 104. LXC 127's 192.168.8.181 was missing on first bring-up; every push delivered status 0 with the message webhook can only call allowed HTTP servers. Adding the IP and systemctl restart gitea is enough — same list is also the gating mechanism for the LXC 120 webhook. Verify any future test instance is added before relying on auto-deploy.

Bootstrap secrets

Saved on hubris at /root/mule-photos-new-secrets.txt (mode 0600):

• PP_ADMIN_PASSWORD — initial PhotoPrism admin login
• PP_DB_PASSWORD — MariaDB photoprism user
• PP_DB_ROOT_PASSWORD — MariaDB root
• OIDC_CLIENT_ID / OIDC_CLIENT_SECRET — generated by ak shell against Authentik

SIDECAR_DB_PASSWORD is still the literal placeholder replace-at-m4-bringup because mariadb/init/01-sidecar.sql hardcodes it; rotate before this stack ever goes public.

Health checks

# from hubris
pct exec 127 -- curl -sf http://127.0.0.1:2342/api/v1/status        # PP
pct exec 127 -- curl -sf http://127.0.0.1:8000/api/sidecar/healthz  # sidecar
pct exec 127 -- curl -sf http://127.0.0.1:5173/                     # Vite

# through Caddy
curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \
  https://photos-new.hubris.network/api/v1/oidc/login -i | head -2   # 302 → auth.hubris.network

Decommissioned 2026-05-22. The PhotoPrism + sidecar + SvelteKit stack validated here was promoted into production on LXC 120 via the Mulimage 2.0 merge (dtoro/mule-image commit 70dc1b6). This page is retained for archaeology; everything below is historic. See the 2026-05-22 entry in 120-mule-images.md for the cutover detail.

Changelog

2026-05-22 — Destroyed

pct destroy 127 after curl verification of the new 120 stack passed end-to-end. dnsmasq photos-new.hubris.network line removed; gitea webhook id 9 + 192.168.8.181 ALLOWED_HOST_LIST entry removed; caddy photos-new.hubris.network site block dropped from dtoro/caddy-conf. Authentik mule-photos-new app + provider deleted.

2026-05-18 (pm) — OIDC state-cookie fix: deploy.sh no longer recreates pp-app

User reported failed to get state: securecookie: the value is not valid after authenticating at Authentik. Root cause: PhotoPrism rotates the Session: HMAC key in pp/storage/config/hub.yml on every container start (it's the hub auto-refresh; no env flag disables it, and with Status: "" it regenerates each boot). The key signs OIDC state cookies, so every restart invalidated every in-flight login.

/opt/mule-deploy/deploy.sh was doing docker compose up -d --build --force-recreate after a pull --ignore-buildable. Both moves recreate pp-app: --force-recreate unconditionally, and the pull bumps the photoprism:latest digest which makes plain up -d recreate too. Every auto-deploy on a code push therefore broke every OIDC login.

Fixed by:

1. Dropping the pull step from the auto-deploy. Image refreshes for pp-app / mariadb are now an admin operation (run docker compose pull manually when you want a new PhotoPrism build).
2. Force-recreating only the sidecar (its image rebuilds on every push anyway). up -d photoprism mariadb reconciles in place — only restarts if their compose declaration actually changes.

Verified by running deploy.sh twice and watching hub.yml's Session: field and pp-app's container PID; both stayed stable across the deploy. The sidecar's PID changed as expected.

The fix lives in /opt/mule-deploy/deploy.sh on LXC 127. /opt/mule-deploy is not a git checkout — keep this Changelog entry as the source of truth.

2026-05-18 — OIDC bridge + indexer + folder fixes

The new branch's PhotoPrism stack required several iterations to be actually usable. Fixes pushed upstream so they apply to anyone running the M0 compose; the LXC override file shrank to just the cross-host sidecar port (see Why an override exists).

OIDC end-to-end (upstream commits 4abe6d7, 9a3ad3e, cce1d87, plus Caddy /library/* → / bounce on LXC 121).

• The SvelteKit /login had a // OIDC SSO ships in M4 placeholder but no button. Added a "Sign in with {provider}" button conditional on /api/v1/config.ext.oidc.enabled; click sends the browser to /api/v1/oidc/login.
• The compose file passed OIDC values through PHOTOPRISM_OIDC_ISSUER_URL / _CLIENT_ID / _CLIENT_SECRET / _PROVIDER_NAME — names PhotoPrism silently ignores. The actual env-var names are PHOTOPRISM_OIDC_URI / _CLIENT / _SECRET / _PROVIDER (see photoprism show config). Renamed upstream; user-facing keys in .env.photoprism (OIDC_ISSUER_URL, OIDC_CLIENT_ID, …) are unchanged.
• PhotoPrism's OIDC callback does not set auth_token / auth_session cookies. It returns an HTML page that writes the session into localStorage under pp:<storageNamespace>:session.{id,token,user,provider} and then runs window.location.href = "/library/login". Caddy on this test instance bounces /library/* back to /, and the SvelteKit root layout (bootstrapSessionFromPhotoPrism()) reads those localStorage entries on mount, fetches /api/v1/session/<id> with the cookied token, and adopts the session into the SPA store.

Indexer caps (upstream commit 3d8e050). A fresh index of ~1.2k photos pushed the LXC load average above 50 with the default PHOTOPRISM_INDEX_WORKERS (NumCPU/2 = 3 here, each forking TF + ffmpeg + libvips). Compose now reads PP_WORKERS / PP_INDEX_WORKERS from .env.photoprism, defaulting to 2. Both set explicitly on this LXC to keep sibling containers happy.

Library went RW. Flipped PP_READONLY=false and PP_ORIGINALS_MODE=rw in .env.photoprism so the indexer can actually run — READONLY=true disables it entirely. Safe because /srv/photos-scratch is the rsync scratch copy on the LXC rootfs, not the real admin Photos tree.

Admin role. OIDC creates users with OIDC_ROLE ONLY on first registration. The dtoro user was created in an earlier flow before the env-var-name fix, so it landed as guest and saw no photos in the UI. Promoted manually:

UPDATE auth_users SET user_role='admin', super_admin=1, can_invite=1
  WHERE user_name='dtoro';
DELETE FROM auth_sessions WHERE user_name='dtoro';

Stale sessions are dropped so a fresh OIDC login mints an admin token.

Video pre-transcode pass (no upstream change — operational fix on this LXC). Only 11/45 .mov originals had a .avc sidecar; the rest forced 12–21 s inline libx264 transcodes on first playback, serialised one ffmpeg at a time. Measured cold vs warm:

Path	TTFB
Thumbnail fit_1280 (warm)	~2 ms
Video playback with .avc sidecar	~2 ms
Video playback without sidecar	12–21 s
Thumbnail GET during a transcode	~2 ms (no sibling slowdown — 6-core LXC, ffmpeg ~6%/core, nvme util 0.01%)

Mitigation:

pct exec 127 -- docker exec -d pp-app /opt/photoprism/bin/photoprism convert

Walks the library, builds every missing .avc next to its original (/photoprism/storage/sidecar/<year>/<month>/<name>.mov.avc), idempotent on re-run, two ffmpeg processes in parallel. Took ~8 min to bring coverage to 45/45. Previously-cold videos verified to serve at ~2 ms TTFB after the pass. Future imports get AVC sidecars automatically as part of indexing; a re-run is only needed if videos ever land outside the indexer's path.

Folder tree (upstream commits 8083328, 505fef5, cfd85a1). PhotoPrism's path: operator is exact-match by default but supports a * wildcard. Without it, every internal tree node (year folders, since photos always nest under YYYY/MM) returned zero hits — both in the timeline and in the sidecar's folder-count fan-out. Fixed both to emit path:"<folder>*". Also relaxed the root-folder client-side filter (was clipping to Path === '', which is always empty) so / shows the whole library. Root badge in the sidebar now reads config.count.all directly instead of subtracting Σ(folderCounts) — the subtraction double-counted after the recursive switch.

2026-05-17 — Bring-up

LXC 127 created from debian-13-standard_13.1-2, joined to vmbr0 with static IP 192.168.8.181. Docker engine + Node 20 installed.

dtoro/mule-image cloned at branch new, compose stack (docker-compose.photoprism.yml) brought up: MariaDB 11 + PhotoPrism :latest (-Plus build) + Go sidecar (built locally). Vite dev server running as mule-deploy via mule-vite.service on port 5173.

Authentik OIDC application mule-photos-new provisioned via ak shell (OAuth2Provider + Application + STRICT RedirectURI). PhotoPrism's OIDC button delegates to Authentik; OIDC_REGISTER=true / OIDC_ROLE=admin so the first SSO login becomes admin.

Admin's Photos library rsynced (~4.9 GB, 1206 files) into /srv/photos-scratch on the LXC rootfs (no bind-mount). Sidecar mutations land in the scratch copy, not the real library.

Caddy site photos-new.hubris.network added in dtoro/caddy-conf; dnsmasq entry on LXC 124 → 192.168.8.175. dnsmasq required a restart (not reload) for the new address= line to take effect.

docker-compose.photoprism.override.yml (LXC-only, untracked) pins two upstream issues: sidecar bound to 127.0.0.1 (cross-host Caddy can't reach), and OIDC env-var name mismatch (PHOTOPRISM_OIDC_ISSUER_URL vs PHOTOPRISM_OIDC_URI and friends). Should land upstream on new next iteration.