07b1e5e02a
Three overlapping fixes so the ingestion pipeline actually runs and the
user can see what it's doing:
Pipeline recovery
- app/database.py: use NullPool when MULITA_CELERY_WORKER=1 so each
Celery task opens a fresh asyncpg connection on its own event loop.
Fixes "another operation in progress" and "Future attached to a
different loop" errors that were dropping ~every thumbnail +
extract_metadata task on the floor.
- app/tasks/thumbs.py: initialize photo=None before the try and rollback
on error so a transport failure in the initial SELECT doesn't raise
UnboundLocalError in the except block and leak rows stuck in 'pending'.
- app/services/vision/bootstrap_models.py: on missing model files,
invoke export_models automatically instead of just warning. First
boot of a fresh install now self-heals.
- app/services/vision/export_models.py: shutil.move instead of
Path.rename so the YOLO export survives the /app → /data/models
cross-volume hop.
- requirements.txt: add ultralytics so export works in a stock image.
Worker topology
- docker-compose.yml: replace the single worker with worker-light
(default/high/low queues, c=2, IO-bound) and worker-vision (vision
queue, c=5, OMP_NUM_THREADS=1 to avoid oversubscription on 6 cores).
Vision is pinned to ≤5 parallel inferences so ONNX doesn't each
spawn an all-cores intra-op pool.
- .env / .env.example: CELERYD_CONCURRENCY replaced with
CELERY_LIGHT_CONCURRENCY + CELERY_VISION_CONCURRENCY.
- Backfill queries in thumbs / scan / vision now ORDER BY taken_at
DESC NULLS LAST so newest photos finish first — the library fills
in top-down in the UI instead of arbitrary insertion order.
Settings visibility
- routers/library.py: new GET /maintenance/pipeline-stats returning
done/total per stage (thumbnails, exif, gps, phash, embeddings,
tags, ocr, faces, face clusters, duplicate groups). Worker-status
now also reports the `vision` queue depth, which was missing.
- services/api.ts: PipelineStats / PipelineStage / ScanStatus types
and the matching client call.
- components/dialogs/SettingsDialog.tsx:
- new Pipeline Progress card with one progress bar per stage
- inline scan banner (processed/total/current folder) inside the
Library section while a scan is running
- Tasks/min throughput computed by diffing worker processed counters
between polls
- Workers section calls out the vision queue and documents the
CELERY_LIGHT/VISION_CONCURRENCY + docker compose up -d scale path
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
100 lines
4.9 KiB
Plaintext
100 lines
4.9 KiB
Plaintext
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Mulita / PhotoVault — example environment file
|
||
#
|
||
# Copy this file to `.env` and adjust the values for your setup. Every key
|
||
# below has a sensible default in docker-compose.yml, so you only need to
|
||
# uncomment the ones you actually want to change.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
|
||
# ── REQUIRED ─────────────────────────────────────────────────────────────────
|
||
|
||
# Host path to your photo library. The compose file mounts this at /photos
|
||
# inside the backend + worker containers. The backend creates a default
|
||
# source root pointing at /photos on first boot, so once this is set the
|
||
# library is scanned with zero further configuration.
|
||
#
|
||
# Examples:
|
||
# macOS / Linux: PHOTO_DIRS=/Users/you/Pictures
|
||
# Network share: PHOTO_DIRS=/mnt/nas/photos
|
||
# Windows (WSL): PHOTO_DIRS=/mnt/c/Users/you/Pictures
|
||
PHOTO_DIRS=./photos
|
||
|
||
|
||
# ── PORTS ────────────────────────────────────────────────────────────────────
|
||
|
||
# Host port the SPA is served on. Browse to http://<host>:<FRONTEND_PORT>/.
|
||
FRONTEND_PORT=3000
|
||
|
||
# Host port for the backend API. Almost never needed directly — the frontend
|
||
# nginx proxies /api/ to the backend over the internal compose network. Kept
|
||
# exposed for debugging / curl.
|
||
BACKEND_PORT=8001
|
||
|
||
# Redis host port. Internal services reach Redis on its container name; this
|
||
# is just for local debugging.
|
||
REDIS_PORT=6379
|
||
|
||
|
||
# ── CORS ─────────────────────────────────────────────────────────────────────
|
||
|
||
# Comma-separated list of allowed origins for direct browser access to the
|
||
# backend. Same-origin requests through the nginx / vite proxy never trip
|
||
# CORS, so this only matters when something hits the backend port directly
|
||
# from a different origin (e.g. another machine, dev tools, a reverse proxy
|
||
# under a different hostname).
|
||
#
|
||
# Default "*" is permissive, fine for a single-user homelab. Lock it down in
|
||
# real deployments:
|
||
# ALLOWED_ORIGINS=https://photos.example.com
|
||
# ALLOWED_ORIGINS=https://photos.example.com,http://192.168.1.10:3000
|
||
ALLOWED_ORIGINS=*
|
||
|
||
|
||
# ── LOGGING / TIMEZONE ───────────────────────────────────────────────────────
|
||
|
||
# Python log level for the backend and Celery worker. Bump to DEBUG when
|
||
# chasing scan / thumbnail issues.
|
||
LOG_LEVEL=INFO
|
||
|
||
# Container timezone. Affects the timestamps in logs and the "added at"
|
||
# field on newly imported photos. Defaults to UTC.
|
||
# TZ=Europe/Berlin
|
||
# TZ=America/New_York
|
||
TZ=UTC
|
||
|
||
|
||
# ── WORKER CONCURRENCY ───────────────────────────────────────────────────────
|
||
#
|
||
# The ingestion pipeline runs on two Celery worker services with separate
|
||
# concurrency knobs so heavy vision tasks can't starve cheap IO tasks:
|
||
#
|
||
# worker-light (default / high / low queues)
|
||
# Runs: scan, thumbnails, EXIF, pHash, duplicate regrouping.
|
||
# Mostly IO-bound — 2 prefork children keep a library streaming in.
|
||
#
|
||
# worker-vision (vision queue)
|
||
# Runs: embeddings, object detection, OCR, face extraction, content
|
||
# classification. Each prefork child loads ~2 GB of ONNX model weights,
|
||
# so set this to roughly (physical_cores − 1) and watch RAM.
|
||
#
|
||
# Defaults target a ~6 core / 16 GB host. Raise these, then
|
||
# docker compose up -d worker-light worker-vision
|
||
# to pick them up. Lower for a Pi; go higher on a workstation.
|
||
#
|
||
# The old `CELERYD_CONCURRENCY=N` single-worker variable is no longer
|
||
# read — delete it from your .env if it's set.
|
||
CELERY_LIGHT_CONCURRENCY=2
|
||
CELERY_VISION_CONCURRENCY=5
|
||
|
||
|
||
# ── INTERNAL (rarely overridden) ─────────────────────────────────────────────
|
||
|
||
# These point at the in-compose Redis and the bind-mounted SQLite db. Override
|
||
# only if you're running Mulita without docker-compose or against an external
|
||
# Redis.
|
||
# REDIS_URL=redis://redis:6379
|
||
# CELERY_BROKER_URL=redis://redis:6379
|
||
# CELERY_RESULT_BACKEND=redis://redis:6379
|
||
# DATABASE_URL=sqlite+aiosqlite:////data/db/mulita.db
|