Three overlapping fixes so the ingestion pipeline actually runs and the
user can see what it's doing:
Pipeline recovery
- app/database.py: use NullPool when MULITA_CELERY_WORKER=1 so each
Celery task opens a fresh asyncpg connection on its own event loop.
Fixes "another operation in progress" and "Future attached to a
different loop" errors that were dropping ~every thumbnail +
extract_metadata task on the floor.
- app/tasks/thumbs.py: initialize photo=None before the try and rollback
on error so a transport failure in the initial SELECT doesn't raise
UnboundLocalError in the except block and leak rows stuck in 'pending'.
- app/services/vision/bootstrap_models.py: on missing model files,
invoke export_models automatically instead of just warning. First
boot of a fresh install now self-heals.
- app/services/vision/export_models.py: shutil.move instead of
Path.rename so the YOLO export survives the /app → /data/models
cross-volume hop.
- requirements.txt: add ultralytics so export works in a stock image.
Worker topology
- docker-compose.yml: replace the single worker with worker-light
(default/high/low queues, c=2, IO-bound) and worker-vision (vision
queue, c=5, OMP_NUM_THREADS=1 to avoid oversubscription on 6 cores).
Vision is pinned to ≤5 parallel inferences so ONNX doesn't each
spawn an all-cores intra-op pool.
- .env / .env.example: CELERYD_CONCURRENCY replaced with
CELERY_LIGHT_CONCURRENCY + CELERY_VISION_CONCURRENCY.
- Backfill queries in thumbs / scan / vision now ORDER BY taken_at
DESC NULLS LAST so newest photos finish first — the library fills
in top-down in the UI instead of arbitrary insertion order.
Settings visibility
- routers/library.py: new GET /maintenance/pipeline-stats returning
done/total per stage (thumbnails, exif, gps, phash, embeddings,
tags, ocr, faces, face clusters, duplicate groups). Worker-status
now also reports the `vision` queue depth, which was missing.
- services/api.ts: PipelineStats / PipelineStage / ScanStatus types
and the matching client call.
- components/dialogs/SettingsDialog.tsx:
- new Pipeline Progress card with one progress bar per stage
- inline scan banner (processed/total/current folder) inside the
Library section while a scan is running
- Tasks/min throughput computed by diffing worker processed counters
between polls
- Workers section calls out the vision queue and documents the
CELERY_LIGHT/VISION_CONCURRENCY + docker compose up -d scale path
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
200 lines
7.2 KiB
YAML
200 lines
7.2 KiB
YAML
services:
|
||
frontend:
|
||
build:
|
||
context: ./frontend
|
||
dockerfile: Dockerfile
|
||
container_name: mulita-frontend
|
||
ports:
|
||
# Host port is configurable via FRONTEND_PORT in .env so multiple
|
||
# instances / other services on the same host don't collide.
|
||
- "${FRONTEND_PORT:-3000}:80"
|
||
depends_on:
|
||
- backend
|
||
networks:
|
||
- mulita-network
|
||
restart: unless-stopped
|
||
|
||
backend:
|
||
build:
|
||
context: ./backend
|
||
dockerfile: Dockerfile
|
||
container_name: mulita-backend
|
||
ports:
|
||
# Direct backend access on the host is rarely needed (the frontend
|
||
# talks to it through the nginx /api proxy on the same network),
|
||
# but it's exposed for debugging / curl. Override with BACKEND_PORT.
|
||
- "${BACKEND_PORT:-8001}:8000"
|
||
volumes:
|
||
- ./mulita.yml:/app/config/mulita.yml:ro
|
||
# The single host → container mount for your photo library. Set
|
||
# PHOTO_DIRS in .env to your library root. Mounted :rw because file
|
||
# operations (rename, move, empty discard pile) need to mutate the
|
||
# filesystem; flip to :ro for a strict read-only library and the
|
||
# write endpoints will return EROFS.
|
||
- ${PHOTO_DIRS:-./photos}:/photos:rw
|
||
- thumbs_data:/data/thumbs
|
||
- proxies_data:/data/proxies
|
||
- db_data:/data/db # retained so the docker-compose.sqlite.yml override has somewhere to put mulita.db
|
||
# Run Alembic migrations before starting uvicorn. On a fresh Postgres
|
||
# the empty 0001 baseline is a no-op stamp; create_all in init_db then
|
||
# builds the schema.
|
||
command: sh -c "python -c \"import asyncio; from app.database import init_db; asyncio.run(init_db())\" && alembic upgrade head && uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload"
|
||
environment:
|
||
- DATABASE_URL=postgresql+asyncpg://mulita:mulita@db:5432/mulita
|
||
- REDIS_URL=redis://redis:6379
|
||
- CELERY_BROKER_URL=redis://redis:6379
|
||
- CELERY_RESULT_BACKEND=redis://redis:6379
|
||
- PHOTO_DIRS=${PHOTO_DIRS:-/photos}
|
||
- ALLOWED_ORIGINS=${ALLOWED_ORIGINS:-*}
|
||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||
- TZ=${TZ:-UTC}
|
||
depends_on:
|
||
redis:
|
||
condition: service_started
|
||
db:
|
||
condition: service_healthy
|
||
networks:
|
||
- mulita-network
|
||
restart: unless-stopped
|
||
|
||
# ── Celery workers ─────────────────────────────────────────────────────
|
||
#
|
||
# The ingestion pipeline is split across two worker services so CPU-heavy
|
||
# vision tasks (embed / detect / OCR / faces / classify) cannot starve
|
||
# the fast IO-bound tasks (scan / thumbnails / EXIF / phash / duplicates).
|
||
#
|
||
# worker-light listens on default,high,low — IO-bound, cheap
|
||
# worker-vision listens on vision — CPU-bound, loads ONNX
|
||
#
|
||
# Both share the same image, photo volume, and model cache, so there's
|
||
# no disk duplication and model weights are loaded lazily only by
|
||
# worker-vision. Each service has its own concurrency knob; both
|
||
# workers ship their heartbeat to the same Redis broker so the
|
||
# Settings > Workers panel lists them side-by-side.
|
||
#
|
||
# Sizing defaults target a 6-core / 16 GB host:
|
||
# CELERY_LIGHT_CONCURRENCY=2 (enough for parallel thumbnail + EXIF)
|
||
# CELERY_VISION_CONCURRENCY=5 (5 × ~2GB ONNX = ~10GB RAM, 5/6 cores)
|
||
# Raise these in .env and run `docker compose up -d worker-light worker-vision`
|
||
# to scale. Keep light under ~4 and vision under your physical core
|
||
# count; more just thrashes.
|
||
worker-light:
|
||
build:
|
||
context: ./backend
|
||
dockerfile: Dockerfile
|
||
image: mule-image-worker
|
||
container_name: mulita-worker-light
|
||
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_LIGHT_CONCURRENCY:-2} -Q default,high,low -n light@%h"
|
||
volumes:
|
||
- ./mulita.yml:/app/config/mulita.yml:ro
|
||
- ${PHOTO_DIRS:-./photos}:/photos:rw
|
||
- thumbs_data:/data/thumbs
|
||
- proxies_data:/data/proxies
|
||
- db_data:/data/db
|
||
- models_data:/data/models
|
||
environment:
|
||
- DATABASE_URL=postgresql+asyncpg://mulita:mulita@db:5432/mulita
|
||
- REDIS_URL=redis://redis:6379
|
||
- CELERY_BROKER_URL=redis://redis:6379
|
||
- CELERY_RESULT_BACKEND=redis://redis:6379
|
||
- PHOTO_DIRS=${PHOTO_DIRS:-/photos}
|
||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||
- TZ=${TZ:-UTC}
|
||
# NullPool — see app/database.py for rationale.
|
||
- MULITA_CELERY_WORKER=1
|
||
depends_on:
|
||
redis:
|
||
condition: service_started
|
||
backend:
|
||
condition: service_started
|
||
db:
|
||
condition: service_healthy
|
||
networks:
|
||
- mulita-network
|
||
restart: unless-stopped
|
||
|
||
worker-vision:
|
||
build:
|
||
context: ./backend
|
||
dockerfile: Dockerfile
|
||
image: mule-image-worker
|
||
container_name: mulita-worker-vision
|
||
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_VISION_CONCURRENCY:-5} -Q vision -n vision@%h"
|
||
volumes:
|
||
- ./mulita.yml:/app/config/mulita.yml:ro
|
||
- ${PHOTO_DIRS:-./photos}:/photos:rw
|
||
- thumbs_data:/data/thumbs
|
||
- proxies_data:/data/proxies
|
||
- db_data:/data/db
|
||
- models_data:/data/models
|
||
environment:
|
||
- DATABASE_URL=postgresql+asyncpg://mulita:mulita@db:5432/mulita
|
||
- REDIS_URL=redis://redis:6379
|
||
- CELERY_BROKER_URL=redis://redis:6379
|
||
- CELERY_RESULT_BACKEND=redis://redis:6379
|
||
- PHOTO_DIRS=${PHOTO_DIRS:-/photos}
|
||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||
- TZ=${TZ:-UTC}
|
||
- MULITA_CELERY_WORKER=1
|
||
# Pin each ONNX session to one intra-op thread so N prefork children
|
||
# × default-all-cores doesn't oversubscribe the box. With
|
||
# concurrency=5 and OMP=1, vision peaks at 5 busy cores, leaving
|
||
# one for worker-light + system. These env vars cover the three
|
||
# threading runtimes ONNX Runtime might pick up on first use.
|
||
- OMP_NUM_THREADS=1
|
||
- OPENBLAS_NUM_THREADS=1
|
||
- MKL_NUM_THREADS=1
|
||
depends_on:
|
||
redis:
|
||
condition: service_started
|
||
backend:
|
||
condition: service_started
|
||
db:
|
||
condition: service_healthy
|
||
networks:
|
||
- mulita-network
|
||
restart: unless-stopped
|
||
|
||
db:
|
||
image: pgvector/pgvector:pg16
|
||
container_name: mulita-db
|
||
environment:
|
||
POSTGRES_USER: mulita
|
||
POSTGRES_PASSWORD: mulita
|
||
POSTGRES_DB: mulita
|
||
volumes:
|
||
- pg_data:/var/lib/postgresql/data
|
||
networks:
|
||
- mulita-network
|
||
restart: unless-stopped
|
||
healthcheck:
|
||
test: ["CMD-SHELL", "pg_isready -U mulita -d mulita"]
|
||
interval: 5s
|
||
timeout: 5s
|
||
retries: 10
|
||
|
||
redis:
|
||
image: redis:7-alpine
|
||
container_name: mulita-redis
|
||
# Host port exposed only for local debugging; the backend / worker
|
||
# reach Redis via the internal mulita-network on its container name.
|
||
ports:
|
||
- "${REDIS_PORT:-6379}:6379"
|
||
volumes:
|
||
- redis_data:/data
|
||
networks:
|
||
- mulita-network
|
||
restart: unless-stopped
|
||
command: redis-server --appendonly yes
|
||
|
||
networks:
|
||
mulita-network:
|
||
driver: bridge
|
||
|
||
volumes:
|
||
thumbs_data:
|
||
proxies_data:
|
||
db_data:
|
||
redis_data:
|
||
pg_data:
|
||
models_data: |