feat: split celery workers, fix asyncpg-in-fork, add pipeline progress UI

Three overlapping fixes so the ingestion pipeline actually runs and the
user can see what it's doing:

Pipeline recovery
- app/database.py: use NullPool when MULITA_CELERY_WORKER=1 so each
  Celery task opens a fresh asyncpg connection on its own event loop.
  Fixes "another operation in progress" and "Future attached to a
  different loop" errors that were dropping ~every thumbnail +
  extract_metadata task on the floor.
- app/tasks/thumbs.py: initialize photo=None before the try and rollback
  on error so a transport failure in the initial SELECT doesn't raise
  UnboundLocalError in the except block and leak rows stuck in 'pending'.
- app/services/vision/bootstrap_models.py: on missing model files,
  invoke export_models automatically instead of just warning. First
  boot of a fresh install now self-heals.
- app/services/vision/export_models.py: shutil.move instead of
  Path.rename so the YOLO export survives the /app → /data/models
  cross-volume hop.
- requirements.txt: add ultralytics so export works in a stock image.

Worker topology
- docker-compose.yml: replace the single worker with worker-light
  (default/high/low queues, c=2, IO-bound) and worker-vision (vision
  queue, c=5, OMP_NUM_THREADS=1 to avoid oversubscription on 6 cores).
  Vision is pinned to ≤5 parallel inferences so ONNX doesn't each
  spawn an all-cores intra-op pool.
- .env / .env.example: CELERYD_CONCURRENCY replaced with
  CELERY_LIGHT_CONCURRENCY + CELERY_VISION_CONCURRENCY.
- Backfill queries in thumbs / scan / vision now ORDER BY taken_at
  DESC NULLS LAST so newest photos finish first — the library fills
  in top-down in the UI instead of arbitrary insertion order.

Settings visibility
- routers/library.py: new GET /maintenance/pipeline-stats returning
  done/total per stage (thumbnails, exif, gps, phash, embeddings,
  tags, ocr, faces, face clusters, duplicate groups). Worker-status
  now also reports the `vision` queue depth, which was missing.
- services/api.ts: PipelineStats / PipelineStage / ScanStatus types
  and the matching client call.
- components/dialogs/SettingsDialog.tsx:
  - new Pipeline Progress card with one progress bar per stage
  - inline scan banner (processed/total/current folder) inside the
    Library section while a scan is running
  - Tasks/min throughput computed by diffing worker processed counters
    between polls
  - Workers section calls out the vision queue and documents the
    CELERY_LIGHT/VISION_CONCURRENCY + docker compose up -d scale path

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-11 10:06:45 +02:00
parent afe420c620
commit 07b1e5e02a
13 changed files with 746 additions and 49 deletions

View File

@@ -57,12 +57,34 @@ services:
- mulita-network
restart: unless-stopped
worker:
# ── Celery workers ─────────────────────────────────────────────────────
#
# The ingestion pipeline is split across two worker services so CPU-heavy
# vision tasks (embed / detect / OCR / faces / classify) cannot starve
# the fast IO-bound tasks (scan / thumbnails / EXIF / phash / duplicates).
#
# worker-light listens on default,high,low — IO-bound, cheap
# worker-vision listens on vision — CPU-bound, loads ONNX
#
# Both share the same image, photo volume, and model cache, so there's
# no disk duplication and model weights are loaded lazily only by
# worker-vision. Each service has its own concurrency knob; both
# workers ship their heartbeat to the same Redis broker so the
# Settings > Workers panel lists them side-by-side.
#
# Sizing defaults target a 6-core / 16 GB host:
# CELERY_LIGHT_CONCURRENCY=2 (enough for parallel thumbnail + EXIF)
# CELERY_VISION_CONCURRENCY=5 (5 × ~2GB ONNX = ~10GB RAM, 5/6 cores)
# Raise these in .env and run `docker compose up -d worker-light worker-vision`
# to scale. Keep light under ~4 and vision under your physical core
# count; more just thrashes.
worker-light:
build:
context: ./backend
dockerfile: Dockerfile
container_name: mulita-worker
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERYD_CONCURRENCY:-4} -Q default,high,low,vision"
image: mule-image-worker
container_name: mulita-worker-light
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_LIGHT_CONCURRENCY:-2} -Q default,high,low -n light@%h"
volumes:
- ./mulita.yml:/app/config/mulita.yml:ro
- ${PHOTO_DIRS:-./photos}:/photos:rw
@@ -76,9 +98,52 @@ services:
- CELERY_BROKER_URL=redis://redis:6379
- CELERY_RESULT_BACKEND=redis://redis:6379
- PHOTO_DIRS=${PHOTO_DIRS:-/photos}
- CELERYD_CONCURRENCY=${CELERYD_CONCURRENCY:-4}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
- TZ=${TZ:-UTC}
# NullPool — see app/database.py for rationale.
- MULITA_CELERY_WORKER=1
depends_on:
redis:
condition: service_started
backend:
condition: service_started
db:
condition: service_healthy
networks:
- mulita-network
restart: unless-stopped
worker-vision:
build:
context: ./backend
dockerfile: Dockerfile
image: mule-image-worker
container_name: mulita-worker-vision
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_VISION_CONCURRENCY:-5} -Q vision -n vision@%h"
volumes:
- ./mulita.yml:/app/config/mulita.yml:ro
- ${PHOTO_DIRS:-./photos}:/photos:rw
- thumbs_data:/data/thumbs
- proxies_data:/data/proxies
- db_data:/data/db
- models_data:/data/models
environment:
- DATABASE_URL=postgresql+asyncpg://mulita:mulita@db:5432/mulita
- REDIS_URL=redis://redis:6379
- CELERY_BROKER_URL=redis://redis:6379
- CELERY_RESULT_BACKEND=redis://redis:6379
- PHOTO_DIRS=${PHOTO_DIRS:-/photos}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
- TZ=${TZ:-UTC}
- MULITA_CELERY_WORKER=1
# Pin each ONNX session to one intra-op thread so N prefork children
# × default-all-cores doesn't oversubscribe the box. With
# concurrency=5 and OMP=1, vision peaks at 5 busy cores, leaving
# one for worker-light + system. These env vars cover the three
# threading runtimes ONNX Runtime might pick up on first use.
- OMP_NUM_THREADS=1
- OPENBLAS_NUM_THREADS=1
- MKL_NUM_THREADS=1
depends_on:
redis:
condition: service_started