oikos/containers/127-mule-photos-new.md

# 127 — `mule-photos-new`

Side-by-side **PhotoPrism M0 test** of the `dtoro/mule-image` `new` branch
at `photos-new.hubris.network`. Production [LXC 120](120-mule-images.md) keeps
running on the legacy stack at `photos.hubris.network` until M5 cutover.

## At a glance
- **Hostname:** `mule-photos-new`
- **IP:** `192.168.8.181`
- **Privilege:** unpriv
- **Resources:** 6 cores / 8 GiB RAM / 40 GiB rootfs / 1 GiB swap
- **Features:** `nesting=1,fuse=1,keyctl=1`
- **Mounts:** *(none — see scratch copy below)*
- **Public hostname:** [`photos-new.hubris.network`](../infrastructure/dns.md) → [caddy (121)](121-caddy.md) → split (PhotoPrism `:2342`, sidecar `:8000`, Vite `:5173`)

## Stack (`/opt/mule-image`)

`/opt/mule-image` is the working tree of `dtoro/mule-image` on branch
`new`. Compose lives at `docker-compose.photoprism.yml`; LXC-127-only
overrides at `docker-compose.photoprism.override.yml` (untracked — see
[Why an override exists](#why-an-override-exists)). Invoked with
`--env-file .env.photoprism`.

| Service     | Container     | Port              | Notes                                              |
| ----------- | ------------- | ----------------- | -------------------------------------------------- |
| mariadb     | `pp-mariadb`  | `127.0.0.1:3306`  | MariaDB 11; named volume `pp_mariadb_data`; init SQL provisions `mule_sidecar.marks` table |
| photoprism  | `pp-app`      | `:2342`           | `docker.io/photoprism/photoprism:latest`, runs as uid 33 (PP_UID/GID env) |
| sidecar     | `pp-sidecar`  | `:8000` (overridden) | Go + Gin service for rename / folder mutations / heap convert / dup detect |
| vite (host) | systemd unit  | `:5173`           | SvelteKit dev server (`mule-vite.service`), `npm run dev` in `/opt/mule-image/web` |

PhotoPrism is enterprise-tier (`tier: 1`) per the build tag (`-Plus`). TF
vision pipeline and EXIF backwrite are **disabled** in M0 — `PP_READONLY=true`
keeps the originals view read-only as the M0 safety net.

## Library — writable rsync scratch copy (NOT the real admin Photos)

Unlike LXC 120 (which mounts `/mnt/library` directly), this LXC has **no
bind-mount of the production library**. Instead a one-shot rsync of the
admin's Photos lives on the LXC's own rootfs:

- **Host source (read-only reference):** `/mnt/library/homecloud/admin/files/Photos` (~4.9 GB)
- **LXC scratch (writable):** `/srv/photos-scratch` (owner `www-data:media`, mode 0775)

This means **sidecar rename / folder mutation operations land in the
scratch copy, not the real admin library**. The scratch is not
auto-synced — it's a snapshot from `2026-05-17`. To refresh from
production:

```bash
# on hubris (LXC 127 must be stopped to mount its rootfs)
pct stop 127
pct mount 127
rsync -aHAX --info=stats2 --chown=100033:110000 --no-perms \
  --chmod=Du=rwx,Dg=rx,Do=rx,Fu=rw,Fg=r,Fo=r \
  /mnt/library/homecloud/admin/files/Photos/ \
  /var/lib/lxc/127/rootfs/srv/photos-scratch/
pct unmount 127
pct start 127
```

`--chown=100033:110000` accounts for the unprivileged-LXC ID shift
(host 100033 = LXC `www-data`, host 110000 = LXC `media`). Don't try a
bind-mount of `/mnt/library/...` — the admin Photos tree is 0750 and
unprivileged LXCs can't see through.

## Auth — Authentik OIDC

PhotoPrism's "Sign in with OIDC" button delegates to [Authentik (124)](124-authentik.md).

- **Provider/Application slug:** `mule-photos-new`
- **Issuer:** `https://auth.hubris.network/application/o/mule-photos-new/`
- **Redirect URI:** `https://photos-new.hubris.network/api/v1/oidc/redirect`
- **Scopes:** `openid profile email`
- **Initiated by clicking the OIDC button** at `/library/login` → `GET /api/v1/oidc/login` → 302 to Authentik authorize.
- `OIDC_REGISTER=true` + `OIDC_ROLE=admin` so the first SSO login auto-creates a PhotoPrism admin account.

Local PhotoPrism admin (username `admin`, password in
`/root/mule-photos-new-secrets.txt` on hubris) stays available as a
fallback.

## Why an override exists

`docker-compose.photoprism.override.yml` is **only on LXC 127** (not in
the git repo). After the 2026-05-17/18 fixes landed upstream
(commits `cce1d87` for OIDC env names and `3d8e050` for worker caps),
the override has shrunk to just one stanza — the cross-host sidecar
bind:

```yaml
services:
  sidecar:
    ports: !override
      - "0.0.0.0:8000:8000"
```

Upstream binds the sidecar to `127.0.0.1:8000` because the M4 design
colocates Caddy with the sidecar. On this test LXC Caddy lives on a
different host (LXC 121), so the port has to be reachable from the
LAN. Delete this file at M4 cutover.

## Auto-deploy

Mirrors the LXC 120 pattern.

- **Webhook listener:** `mule-deploy-webhook.service` → `python3 /opt/mule-deploy/webhook.py` on `0.0.0.0:9797`.
- **Branch filter:** `refs/heads/new` (LXC 120 still owns `main`).
- **HMAC secret:** `/etc/mule-deploy/secret` (mode 0600).
- **Deploy script:** `/opt/mule-deploy/deploy.sh` — `git fetch && git reset --hard origin/new`, fix `pp/{storage,import}` ownership to `33:10000` (PP container uid), `docker compose ... up -d --build --force-recreate` with both compose files, `systemctl restart mule-vite`.
- **Gitea webhook id 9** on `dtoro/mule-image` pointed at `http://192.168.8.181:9797/deploy`.

Push to the `new` branch on [git.hubris.network/dtoro/mule-image](http://git.hubris.network/dtoro/mule-image) → webhook fires → rebuild. The legacy LXC 120 watches `main` and is unaffected.

**Gitea gotcha:** the receiver IP must be in `[webhook] ALLOWED_HOST_LIST`
in `/etc/gitea/app.ini` on [LXC 104](104-gitea.md). LXC 127's
`192.168.8.181` was missing on first bring-up; every push delivered
status 0 with the message `webhook can only call allowed HTTP servers`.
Adding the IP and `systemctl restart gitea` is enough — same list is
also the gating mechanism for the LXC 120 webhook. Verify any future
test instance is added before relying on auto-deploy.

## Bootstrap secrets

Saved on hubris at `/root/mule-photos-new-secrets.txt` (mode 0600):

- `PP_ADMIN_PASSWORD` — initial PhotoPrism `admin` login
- `PP_DB_PASSWORD` — MariaDB `photoprism` user
- `PP_DB_ROOT_PASSWORD` — MariaDB root
- `OIDC_CLIENT_ID` / `OIDC_CLIENT_SECRET` — generated by `ak shell` against Authentik

`SIDECAR_DB_PASSWORD` is still the literal placeholder
`replace-at-m4-bringup` because `mariadb/init/01-sidecar.sql` hardcodes
it; rotate before this stack ever goes public.

## Health checks

```bash
# from hubris
pct exec 127 -- curl -sf http://127.0.0.1:2342/api/v1/status        # PP
pct exec 127 -- curl -sf http://127.0.0.1:8000/api/sidecar/healthz  # sidecar
pct exec 127 -- curl -sf http://127.0.0.1:5173/                     # Vite

# through Caddy
curl -sk --resolve photos-new.hubris.network:443:192.168.8.175 \
  https://photos-new.hubris.network/api/v1/oidc/login -i | head -2   # 302 → auth.hubris.network
```

> **Decommissioned 2026-05-22.** The PhotoPrism + sidecar + SvelteKit stack
> validated here was promoted into production on [LXC 120](120-mule-images.md)
> via the `Mulimage 2.0` merge (`dtoro/mule-image` commit `70dc1b6`). This
> page is retained for archaeology; everything below is historic. See the
> 2026-05-22 entry in [120-mule-images.md](120-mule-images.md#changelog) for
> the cutover detail.

## Changelog

### 2026-05-22 — Destroyed

`pct destroy 127` after curl verification of the new 120 stack passed
end-to-end. dnsmasq `photos-new.hubris.network` line removed; gitea
webhook id 9 + `192.168.8.181` ALLOWED_HOST_LIST entry removed; caddy
`photos-new.hubris.network` site block dropped from `dtoro/caddy-conf`.
Authentik `mule-photos-new` app + provider deleted.

### 2026-05-18 (pm) — OIDC state-cookie fix: deploy.sh no longer recreates pp-app

User reported `failed to get state: securecookie: the value is not valid` after
authenticating at Authentik. Root cause: PhotoPrism rotates the `Session:` HMAC
key in `pp/storage/config/hub.yml` on **every container start** (it's the hub
auto-refresh; no env flag disables it, and with `Status: ""` it regenerates
each boot). The key signs OIDC state cookies, so every restart invalidated
every in-flight login.

`/opt/mule-deploy/deploy.sh` was doing `docker compose up -d --build
--force-recreate` after a `pull --ignore-buildable`. Both moves recreate
pp-app: `--force-recreate` unconditionally, and the pull bumps the
`photoprism:latest` digest which makes plain `up -d` recreate too. Every
auto-deploy on a code push therefore broke every OIDC login.

Fixed by:

1. Dropping the `pull` step from the auto-deploy. Image refreshes for pp-app
   / mariadb are now an admin operation (run `docker compose pull` manually
   when you want a new PhotoPrism build).
2. Force-recreating only the **sidecar** (its image rebuilds on every push
   anyway). `up -d photoprism mariadb` reconciles in place — only restarts
   if their compose declaration actually changes.

Verified by running `deploy.sh` twice and watching `hub.yml`'s `Session:`
field and pp-app's container PID; both stayed stable across the deploy.
The sidecar's PID changed as expected.

The fix lives in `/opt/mule-deploy/deploy.sh` on LXC 127. `/opt/mule-deploy`
is not a git checkout — keep this Changelog entry as the source of truth.

### 2026-05-18 — OIDC bridge + indexer + folder fixes

The `new` branch's PhotoPrism stack required several iterations to be
actually usable. Fixes pushed upstream so they apply to anyone running
the M0 compose; the LXC override file shrank to just the cross-host
sidecar port (see [Why an override exists](#why-an-override-exists)).

**OIDC end-to-end** (upstream commits `4abe6d7`, `9a3ad3e`, `cce1d87`,
plus Caddy `/library/* → /` bounce on LXC 121).

- The SvelteKit `/login` had a `// OIDC SSO ships in M4` placeholder
  but no button. Added a "Sign in with {provider}" button conditional
  on `/api/v1/config.ext.oidc.enabled`; click sends the browser to
  `/api/v1/oidc/login`.
- The compose file passed OIDC values through `PHOTOPRISM_OIDC_ISSUER_URL`
  / `_CLIENT_ID` / `_CLIENT_SECRET` / `_PROVIDER_NAME` — names PhotoPrism
  silently ignores. The actual env-var names are `PHOTOPRISM_OIDC_URI`
  / `_CLIENT` / `_SECRET` / `_PROVIDER` (see `photoprism show config`).
  Renamed upstream; user-facing keys in `.env.photoprism` (OIDC_ISSUER_URL,
  OIDC_CLIENT_ID, …) are unchanged.
- PhotoPrism's OIDC callback does **not** set `auth_token` / `auth_session`
  cookies. It returns an HTML page that writes the session into
  `localStorage` under `pp:<storageNamespace>:session.{id,token,user,provider}`
  and then runs `window.location.href = "/library/login"`. Caddy on this
  test instance bounces `/library/*` back to `/`, and the SvelteKit root
  layout (`bootstrapSessionFromPhotoPrism()`) reads those localStorage
  entries on mount, fetches `/api/v1/session/<id>` with the cookied
  token, and adopts the session into the SPA store.

**Indexer caps** (upstream commit `3d8e050`). A fresh index of ~1.2k
photos pushed the LXC load average above 50 with the default
`PHOTOPRISM_INDEX_WORKERS` (NumCPU/2 = 3 here, each forking TF +
ffmpeg + libvips). Compose now reads `PP_WORKERS` / `PP_INDEX_WORKERS`
from `.env.photoprism`, defaulting to 2. Both set explicitly on this
LXC to keep sibling containers happy.

**Library went RW.** Flipped `PP_READONLY=false` and `PP_ORIGINALS_MODE=rw`
in `.env.photoprism` so the indexer can actually run — `READONLY=true`
disables it entirely. Safe because `/srv/photos-scratch` is the
rsync scratch copy on the LXC rootfs, not the real admin Photos tree.

**Admin role.** OIDC creates users with `OIDC_ROLE` ONLY on first
registration. The `dtoro` user was created in an earlier flow before
the env-var-name fix, so it landed as `guest` and saw no photos in
the UI. Promoted manually:

```sql
UPDATE auth_users SET user_role='admin', super_admin=1, can_invite=1
  WHERE user_name='dtoro';
DELETE FROM auth_sessions WHERE user_name='dtoro';
```

Stale sessions are dropped so a fresh OIDC login mints an admin token.

**Video pre-transcode pass** (no upstream change — operational fix on
this LXC). Only 11/45 `.mov` originals had a `.avc` sidecar; the rest
forced 12–21 s inline libx264 transcodes on first playback, serialised
one ffmpeg at a time. Measured cold vs warm:

| Path                              | TTFB    |
| --------------------------------- | ------- |
| Thumbnail `fit_1280` (warm)       | ~2 ms   |
| Video playback with `.avc` sidecar | ~2 ms   |
| Video playback without sidecar     | 12–21 s |
| Thumbnail GET *during* a transcode | ~2 ms (no sibling slowdown — 6-core LXC, ffmpeg ~6%/core, nvme util 0.01%) |

Mitigation:

```bash
pct exec 127 -- docker exec -d pp-app /opt/photoprism/bin/photoprism convert
```

Walks the library, builds every missing `.avc` next to its original
(`/photoprism/storage/sidecar/<year>/<month>/<name>.mov.avc`),
idempotent on re-run, two ffmpeg processes in parallel. Took ~8 min
to bring coverage to 45/45. Previously-cold videos verified to serve
at ~2 ms TTFB after the pass. Future imports get AVC sidecars
automatically as part of indexing; a re-run is only needed if videos
ever land outside the indexer's path.

**Folder tree** (upstream commits `8083328`, `505fef5`, `cfd85a1`).
PhotoPrism's `path:` operator is exact-match by default but supports
a `*` wildcard. Without it, every internal tree node (year folders,
since photos always nest under YYYY/MM) returned zero hits — both in
the timeline and in the sidecar's folder-count fan-out. Fixed both
to emit `path:"<folder>*"`. Also relaxed the root-folder client-side
filter (was clipping to `Path === ''`, which is always empty) so `/`
shows the whole library. Root badge in the sidebar now reads
`config.count.all` directly instead of subtracting Σ(folderCounts) —
the subtraction double-counted after the recursive switch.

### 2026-05-17 — Bring-up

LXC 127 created from `debian-13-standard_13.1-2`, joined to vmbr0 with
static IP `192.168.8.181`. Docker engine + Node 20 installed.

`dtoro/mule-image` cloned at branch `new`, compose stack
(`docker-compose.photoprism.yml`) brought up: MariaDB 11 + PhotoPrism
`:latest` (`-Plus` build) + Go sidecar (built locally). Vite dev server
running as `mule-deploy` via `mule-vite.service` on port 5173.

Authentik OIDC application `mule-photos-new` provisioned via `ak shell`
(`OAuth2Provider` + `Application` + STRICT `RedirectURI`). PhotoPrism's
OIDC button delegates to Authentik; `OIDC_REGISTER=true` /
`OIDC_ROLE=admin` so the first SSO login becomes admin.

Admin's Photos library rsynced (~4.9 GB, 1206 files) into
`/srv/photos-scratch` on the LXC rootfs (no bind-mount). Sidecar
mutations land in the scratch copy, not the real library.

Caddy site `photos-new.hubris.network` added in `dtoro/caddy-conf`;
dnsmasq entry on LXC 124 → `192.168.8.175`. dnsmasq required a `restart`
(not `reload`) for the new `address=` line to take effect.

`docker-compose.photoprism.override.yml` (LXC-only, untracked) pins two
upstream issues: sidecar bound to `127.0.0.1` (cross-host Caddy can't
reach), and OIDC env-var name mismatch
(`PHOTOPRISM_OIDC_ISSUER_URL` vs `PHOTOPRISM_OIDC_URI` and friends).
Should land upstream on `new` next iteration.