|
|
|
@@ -1,68 +1,46 @@
|
|
|
|
# 120 — `mule-images`
|
|
|
|
# 120 — `mule-images`
|
|
|
|
|
|
|
|
|
|
|
|
Hosts `mule-image` / "mulita" — the photos app at `photos.hubris.network`. Auto-deploys from `dtoro/mule-image` on `git push origin main`.
|
|
|
|
Hosts `mule-image` — the photos app at `photos.hubris.network`. PhotoPrism + Go sidecar + SvelteKit, replacing the legacy FastAPI/Celery stack as of 2026-05-22 (see Changelog). Auto-deploys from `dtoro/mule-image` on `git push origin main`.
|
|
|
|
|
|
|
|
|
|
|
|
## At a glance
|
|
|
|
## At a glance
|
|
|
|
- **Hostname:** `mule-images`
|
|
|
|
- **Hostname:** `mule-images`
|
|
|
|
- **IP:** `192.168.8.136`
|
|
|
|
- **IP:** `192.168.8.136`
|
|
|
|
- **Privilege:** privileged
|
|
|
|
- **Privilege:** privileged
|
|
|
|
- **Resources:** 4 cores / 8 GiB RAM / 60 GiB rootfs
|
|
|
|
- **Resources:** 6 cores / 12 GiB RAM / 60 GiB rootfs / 2 GiB swap
|
|
|
|
- **Mounts:** `/mnt/library` ↔ `/mnt/library`
|
|
|
|
- **Mounts:** `/mnt/library` ↔ `/mnt/library`; `/dev/dri/{card0,renderD128}` passed through for VA-API video accel on the AMD Phoenix1 iGPU.
|
|
|
|
- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:3000` (frontend)
|
|
|
|
- **Public hostname:** [`photos.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → path-routed to PhotoPrism `:2342` / sidecar `:8000` / nginx-static `:3000`.
|
|
|
|
|
|
|
|
|
|
|
|
## Stack (`/opt/mule-image`)
|
|
|
|
## Stack (`/opt/mule-image`)
|
|
|
|
|
|
|
|
|
|
|
|
`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose at `/opt/mule-image/docker-compose.yml`. Services:
|
|
|
|
`/opt/mule-image` IS the working tree of `dtoro/mule-image`. Compose stack: base `docker-compose.yml` + LAN-port-rebind `docker-compose.override.yml` (untracked) + VA-API `docker-compose.gpu.yml`. The SvelteKit frontend is built on the host and served as a static bundle by nginx — no `vite dev` in prod.
|
|
|
|
|
|
|
|
|
|
|
|
| Service | Port | Notes |
|
|
|
|
| Service / process | Port (LAN) | Notes |
|
|
|
|
| ----------------- | ------ | ----- |
|
|
|
|
| ------------------------ | --------------------- | ----- |
|
|
|
|
| frontend | 3000 | Reverse-proxied by Caddy |
|
|
|
|
| pp-mariadb | (internal 3306) | MariaDB 11; holds PhotoPrism's `photoprism` DB + sidecar's `mule_sidecar.*` |
|
|
|
|
| backend | 8001 | FastAPI |
|
|
|
|
| pp-app | `0.0.0.0:2342` | PhotoPrism `:latest`; `PHOTOPRISM_FFMPEG_ENCODER=h264_vaapi` |
|
|
|
|
| worker-light | — | Sole celery worker — runs default/high/low queues + beat scheduler |
|
|
|
|
| pp-sidecar | `0.0.0.0:8000` | Go service (file rename / folder ops / heap convert / dup scan / per-photo marks); also reconciles `USER_BASEPATHS` into PhotoPrism's `auth_users` every 60 s |
|
|
|
|
| db | (pg) | postgres:16 (was pgvector/pg16 until 2026-05-14) |
|
|
|
|
| nginx (host process) | `0.0.0.0:3000` | Serves the SvelteKit static bundle from `web/build/` |
|
|
|
|
| redis | (rd) | queue |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**No vision / ML services.** The OpenCLIP ONNX classifier and its dedicated `worker-vision` were ripped out on 2026-05-14 — see Changelog.
|
|
|
|
**Per-user scoping** is driven by `USER_BASEPATHS=admin:admin/files, muli:muli/files` in `.env`. Sidecar applies it to PhotoPrism's `auth_users` table on boot + every 60s, `mkdir -p`s each target so PhotoPrism's ACL filter has somewhere to point.
|
|
|
|
|
|
|
|
|
|
|
|
`worker-watcher` was retired 2026-05-11 — file events come from NC webhook_listeners instead of watchfiles inotify. See "Nextcloud integration" below.
|
|
|
|
`.env` is **untracked** — `git reset --hard` won't touch it, but `git clean -fdx` would. Holds PP_*, MariaDB passwords, SIDECAR_DB_PASSWORD, OIDC_* (existing `mule-image` Authentik app, redirect URI now `/api/v1/oidc/redirect`), `USER_BASEPATHS`, and `PP_FFMPEG_ENCODER=vaapi`.
|
|
|
|
|
|
|
|
|
|
|
|
`.env` is **untracked** — `git checkout .env` will wipe it. Holds:
|
|
|
|
`docker-compose.override.yml` is also untracked — it `!override`s the upstream `127.0.0.1:port` mappings to `0.0.0.0:port` so cross-host Caddy on [LXC 121](121-caddy.md) can reach pp-app + sidecar.
|
|
|
|
- `PHOTO_DIRS=/mnt/library/images/`
|
|
|
|
|
|
|
|
- `NEXTCLOUD_USERS_HOST_PATH=/mnt/library/homecloud`
|
|
|
|
|
|
|
|
- `NEXTCLOUD_BASE_URL=https://cloud.hubris.network`
|
|
|
|
|
|
|
|
- OIDC client secret + scopes
|
|
|
|
|
|
|
|
- `SECRET_KEY` (generated)
|
|
|
|
|
|
|
|
- `NEXTCLOUD_WEBHOOK_SECRET` (since 2026-05-11; bearer token NC presents on every webhook call to `/api/v1/internal/nc-webhook`)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Nextcloud integration (since 2026-05-11)
|
|
|
|
## Library access
|
|
|
|
|
|
|
|
|
|
|
|
Phase 1 + 2 of dropping the duplicated indexing work between mule and Nextcloud — see `~/.claude/projects/-root/memory/project_mule_nc_integration.md` for the full detail.
|
|
|
|
PhotoPrism reads the library **directly off the bind-mounted filesystem** — no Nextcloud webhook integration in the new stack. The base path is `/mnt/library/homecloud` and per-user scoping comes from `USER_BASEPATHS` (see Stack section above).
|
|
|
|
|
|
|
|
|
|
|
|
**Thumbnails (Phase 1, commit 576b0c2):** `GET /api/v1/photos/{id}/thumb/{size}` proxies `https://cloud.hubris.network/index.php/core/preview` keyed by `photos.nextcloud_fileid` (alembic 0018), authenticated with the photo owner's encrypted NC app password. NC handles JPEG/HEIC/TIFF/RAW/DNG/Movie previews natively. `X-Mule-Thumb-Source: nextcloud|disk` on every response. Worker now only writes the `medium` size to `/data/thumbs/{user}/{photo}/medium.webp` (the vision worker still loads it from disk); `small` + `large` are NC-proxied. Backfill script: `backend/scripts/backfill_nextcloud_fileid.py` — 17,591/17,591 photos filled.
|
|
|
|
- `dtoro` is mapped to NC user `admin` → `/mnt/library/homecloud/admin/files/`
|
|
|
|
|
|
|
|
- `muli` is mapped to NC user `muli` → `/mnt/library/homecloud/muli/files/`
|
|
|
|
|
|
|
|
|
|
|
|
**File events (Phase 2, commits 362fbc6 + f657e2c):** mule no longer runs `watchfiles`. NC's `webhook_listeners` app (already installed, registered via `backend/scripts/register_nc_webhooks.py`) POSTs every `NodeCreated|NodeWritten|NodeDeleted|NodeRenamed` event to `http://192.168.8.136:8001/api/v1/internal/nc-webhook` with `Authorization: Bearer $NEXTCLOUD_WEBHOOK_SECRET`. The endpoint dispatches the same `scan_folder` / `handle_file_deletion` machinery the watcher used. **NC dispatches webhooks asynchronously via its background-job queue** — `crontab -u www-data` on LXC 114 runs `cron.php` every minute (`*/1`), so end-to-end latency is ≲60 s. Re-running the registrar script is idempotent (it deletes existing webhooks pointing at the same URL first).
|
|
|
|
PhotoPrism's container user is `33:10000` (`www-data:media`), matching the host ownership of the NC tree. The legacy `oc_webhook_listeners` rows + `NEXTCLOUD_WEBHOOK_SECRET` are gone — when NC writes via WebDAV (still its own primary surface), PhotoPrism picks up the new file on its next index pass.
|
|
|
|
|
|
|
|
|
|
|
|
## Nextcloud-rooted libraries (since 2026-04-26)
|
|
|
|
## Authentication
|
|
|
|
|
|
|
|
|
|
|
|
Photo libraries live under each user's Nextcloud `files/` tree, NOT in `/mnt/library/images/*`.
|
|
|
|
OIDC via Authentik. App slug `mule-image`, redirect URI `https://photos.hubris.network/api/v1/oidc/redirect` (PhotoPrism's auto-derived path; PhotoPrism builds it from `PHOTOPRISM_SITE_URL`). `OIDC_REGISTER=true` auto-creates a PhotoPrism user at role `user` on first SSO login. The sidecar's basepath reconciler then assigns their scoped folder.
|
|
|
|
- `/mnt/library/homecloud` is bind-mounted into `backend`, `worker-light`, `worker-watcher`, `worker-vision` as `/nextcloud-users`. Each NC user is `/nextcloud-users/<nc_user>/files/`.
|
|
|
|
|
|
|
|
- Reads use that bind directly.
|
|
|
|
|
|
|
|
- Mutations (upload, delete, rename, move) dispatch through `services/nextcloud_dav.py` (HTTP Basic auth, per-user app password Fernet-encrypted in `users.nextcloud_app_password_enc`) so Nextcloud's `oc_filecache`, trashbin, comments, and desktop-sync clients stay coherent.
|
|
|
|
|
|
|
|
- Photo copy + cross-system moves return 501 with a "use Nextcloud's web UI" hint — defer until needed.
|
|
|
|
|
|
|
|
- `users.nextcloud_username` overrides the default OIDC `preferred_username`. **`dtoro` (mule-image) maps to `admin` (Nextcloud)** — don't assume username equality.
|
|
|
|
|
|
|
|
- Surviving SourceRoots in DB: `Photos` → `/nextcloud-users/admin/files/Photos`; `Memories` → `/nextcloud-users/admin/files/Memories` (both owned by `dtoro`). User `muli` has `nextcloud_username=muli` backfilled but no SourceRoot yet.
|
|
|
|
|
|
|
|
- Pre-migration DB dump: `/root/snapshots/mulita-pre-nc-migration-20260426-075132.dump` (11 MB) on the host.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Authentication (since 2026-04-22)
|
|
|
|
The `mule-image` Authentik app's redirect URI was migrated from the legacy FastAPI `/api/v1/auth/oidc/callback` path on 2026-05-22 — same client ID/secret were reused. The separate `mule-photos-new` Authentik app was deleted in the same operation.
|
|
|
|
|
|
|
|
|
|
|
|
Native OIDC via Authentik. Code in `backend/app/auth_oidc.py`, routes `/api/v1/auth/oidc/{login,callback}`. Authentik side:
|
|
|
|
|
|
|
|
- OAuth2/OIDC Provider, client ID `fCuHew48ONTskDjUKnMTZjFbVXuHwvQqTScQRNQ1`
|
|
|
|
|
|
|
|
- App slug `mule-image`
|
|
|
|
|
|
|
|
- Redirect URI: `https://photos.hubris.network/api/v1/auth/oidc/callback`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Backend container needs `extra_hosts: auth.hubris.network:192.168.8.175` via `docker-compose.override.yml` (gitignored). Otherwise Authlib's metadata fetch fails with `SSL: CERTIFICATE_VERIFY_FAILED: self-signed certificate` (it ends up at a random public host because LXC DNS resolves the public IONOS A record).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Caddyfile stays plain `reverse_proxy 192.168.8.136:3000` — no forward-auth, no `/api/*` bypass needed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Auto-deploy
|
|
|
|
## Auto-deploy
|
|
|
|
|
|
|
|
|
|
|
|
@@ -70,11 +48,10 @@ Push to `dtoro/mule-image` `main` → gitea webhook → `http://192.168.8.136:97
|
|
|
|
- Validates HMAC against `/etc/mule-deploy/secret`
|
|
|
|
- Validates HMAC against `/etc/mule-deploy/secret`
|
|
|
|
- Filters to `refs/heads/main`
|
|
|
|
- Filters to `refs/heads/main`
|
|
|
|
- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
|
|
|
|
- Runs `/opt/mule-deploy/deploy.sh` in a daemon thread (returns 202 immediately — docker builds exceed gitea's request timeout)
|
|
|
|
- `git pull --ff-only` + `docker compose up -d --build` + `docker image prune -f`
|
|
|
|
- `git fetch && git reset --hard origin/main`, force-recreates **only the sidecar** image (so PhotoPrism's `Session` HMAC key in `pp/storage/config/hub.yml` doesn't rotate and invalidate every in-flight OIDC state cookie), reconciles `pp-app` + `mariadb` in place, `cd web && npm ci && npm run build`, `systemctl reload nginx`.
|
|
|
|
|
|
|
|
- Conditionally layers `docker-compose.gpu.yml` when `/dev/dri/renderD128` exists, and always layers `docker-compose.override.yml` (the LAN-port rebind) when present.
|
|
|
|
|
|
|
|
|
|
|
|
Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`, unit at `/etc/systemd/system/mule-deploy-webhook.service`. Same shape as the Caddy + Artifacto pipelines. Gitea webhook id 6.
|
|
|
|
Deploy tooling is **outside** the app repo: `/opt/mule-deploy/{deploy.sh,webhook.py}`, secret at `/etc/mule-deploy/secret`. Gitea webhook id 6. `app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
|
|
|
|
|
|
|
|
|
|
|
|
`app.ini` `ALLOWED_HOST_LIST` on [gitea](104-gitea.md) includes `192.168.8.136`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`.
|
|
|
|
Logs: `pct exec 120 -- journalctl -u mule-deploy-webhook -f`.
|
|
|
|
Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`.
|
|
|
|
Manual deploy: `pct exec 120 -- /opt/mule-deploy/deploy.sh`.
|
|
|
|
@@ -91,6 +68,53 @@ For pushes from inside the LXC, gitea creds at `/etc/mule-deploy/git-credentials
|
|
|
|
|
|
|
|
|
|
|
|
## Changelog
|
|
|
|
## Changelog
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 2026-05-22 — Cutover to PhotoPrism stack (Mulimage 2.0)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The `new` branch of `dtoro/mule-image` was merged into `main` as commit
|
|
|
|
|
|
|
|
`70dc1b6`. The merge replaces the legacy FastAPI + PostgreSQL + Celery
|
|
|
|
|
|
|
|
+ React stack with PhotoPrism + Go sidecar + MariaDB + SvelteKit, lifted
|
|
|
|
|
|
|
|
in from the M0 evaluation on (now-destroyed) [LXC 127](#).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Cutover on 120 (this session):
|
|
|
|
|
|
|
|
- Bumped to 6 cores / 2 GB swap. `/dev/dri/{card0,renderD128}` already
|
|
|
|
|
|
|
|
passed through; added an opt-in `docker-compose.gpu.yml` overlay that
|
|
|
|
|
|
|
|
the deploy script layers in when the device is present. PhotoPrism now
|
|
|
|
|
|
|
|
uses `h264_vaapi` instead of `libx264`.
|
|
|
|
|
|
|
|
- Installed host nginx + a small `photos.conf` vhost serving the SvelteKit
|
|
|
|
|
|
|
|
static bundle from `/opt/mule-image/web/build/` on `:3000` — no Vite
|
|
|
|
|
|
|
|
dev server in prod. SvelteKit's `adapter-static` produces a real SPA
|
|
|
|
|
|
|
|
bundle.
|
|
|
|
|
|
|
|
- Replaced `/opt/mule-deploy/deploy.sh` with the 127-style multi-compose
|
|
|
|
|
|
|
|
deploy (force-recreates only sidecar to preserve PhotoPrism's Session
|
|
|
|
|
|
|
|
HMAC key; reconciles pp-app + mariadb in place; rebuilds web/ and
|
|
|
|
|
|
|
|
reloads nginx).
|
|
|
|
|
|
|
|
- `/opt/mule-image/.env` was rewritten to the PhotoPrism shape. Reused
|
|
|
|
|
|
|
|
passwords from `/root/mule-photos-new-secrets.txt`. `USER_BASEPATHS=
|
|
|
|
|
|
|
|
admin:admin/files, muli:muli/files` exposes both libraries.
|
|
|
|
|
|
|
|
- `docker-compose.override.yml` (untracked) `!override`s pp-app +
|
|
|
|
|
|
|
|
sidecar ports to `0.0.0.0` so cross-LXC caddy on [121](121-caddy.md)
|
|
|
|
|
|
|
|
can reach them.
|
|
|
|
|
|
|
|
- Authentik: `mule-image` app's redirect URI updated to `/api/v1/oidc/redirect`;
|
|
|
|
|
|
|
|
the separate `mule-photos-new` app deleted.
|
|
|
|
|
|
|
|
- Caddyfile (`dtoro/caddy-conf`): `photos.hubris.network` switched from
|
|
|
|
|
|
|
|
plain passthrough to path-matchers (PhotoPrism API + sidecar API +
|
|
|
|
|
|
|
|
nginx static catch-all), and `photos-new.hubris.network` was removed
|
|
|
|
|
|
|
|
entirely.
|
|
|
|
|
|
|
|
- Cleanups: 4 Nextcloud `oc_webhook_listeners` rows for the legacy
|
|
|
|
|
|
|
|
`/api/v1/internal/nc-webhook` endpoint deleted; gitea webhook id 9
|
|
|
|
|
|
|
|
(the `refs/heads/new` deploy hook for 127) deleted; `192.168.8.181`
|
|
|
|
|
|
|
|
removed from gitea's `ALLOWED_HOST_LIST`; `photos-new.hubris.network`
|
|
|
|
|
|
|
|
removed from dnsmasq.
|
|
|
|
|
|
|
|
- LXC 127 destroyed via `pct destroy 127` after the new stack passed
|
|
|
|
|
|
|
|
curl verification end-to-end (PP `/api/v1/status` returns
|
|
|
|
|
|
|
|
`{"status":"operational"}` through caddy; sidecar `/api/sidecar/healthz`
|
|
|
|
|
|
|
|
returns `{"ok":true,...}`).
|
|
|
|
|
|
|
|
- Rollback safety net: legacy `mulita` postgres dump at
|
|
|
|
|
|
|
|
`/root/backups/mulita-legacy-final-20260521-212036.sql.gz` on hubris
|
|
|
|
|
|
|
|
(10 MB, 16,155 photo rows). The `mule-image_mule_db_data` docker
|
|
|
|
|
|
|
|
volume on 120 retains the on-disk postgres data for at least 24 h
|
|
|
|
|
|
|
|
before housekeeping removes it.
|
|
|
|
|
|
|
|
|
|
|
|
### 2026-05-14 — Drop AI/vision pipeline, plain Postgres, DB↔FS refresh
|
|
|
|
### 2026-05-14 — Drop AI/vision pipeline, plain Postgres, DB↔FS refresh
|
|
|
|
|
|
|
|
|
|
|
|
**AI removal (commits `a27267f` + `423a73a`).** The OpenCLIP-on-ONNX
|
|
|
|
**AI removal (commits `a27267f` + `423a73a`).** The OpenCLIP-on-ONNX
|
|
|
|
|