07b1e5e02a
Three overlapping fixes so the ingestion pipeline actually runs and the
user can see what it's doing:
Pipeline recovery
- app/database.py: use NullPool when MULITA_CELERY_WORKER=1 so each
Celery task opens a fresh asyncpg connection on its own event loop.
Fixes "another operation in progress" and "Future attached to a
different loop" errors that were dropping ~every thumbnail +
extract_metadata task on the floor.
- app/tasks/thumbs.py: initialize photo=None before the try and rollback
on error so a transport failure in the initial SELECT doesn't raise
UnboundLocalError in the except block and leak rows stuck in 'pending'.
- app/services/vision/bootstrap_models.py: on missing model files,
invoke export_models automatically instead of just warning. First
boot of a fresh install now self-heals.
- app/services/vision/export_models.py: shutil.move instead of
Path.rename so the YOLO export survives the /app → /data/models
cross-volume hop.
- requirements.txt: add ultralytics so export works in a stock image.
Worker topology
- docker-compose.yml: replace the single worker with worker-light
(default/high/low queues, c=2, IO-bound) and worker-vision (vision
queue, c=5, OMP_NUM_THREADS=1 to avoid oversubscription on 6 cores).
Vision is pinned to ≤5 parallel inferences so ONNX doesn't each
spawn an all-cores intra-op pool.
- .env / .env.example: CELERYD_CONCURRENCY replaced with
CELERY_LIGHT_CONCURRENCY + CELERY_VISION_CONCURRENCY.
- Backfill queries in thumbs / scan / vision now ORDER BY taken_at
DESC NULLS LAST so newest photos finish first — the library fills
in top-down in the UI instead of arbitrary insertion order.
Settings visibility
- routers/library.py: new GET /maintenance/pipeline-stats returning
done/total per stage (thumbnails, exif, gps, phash, embeddings,
tags, ocr, faces, face clusters, duplicate groups). Worker-status
now also reports the `vision` queue depth, which was missing.
- services/api.ts: PipelineStats / PipelineStage / ScanStatus types
and the matching client call.
- components/dialogs/SettingsDialog.tsx:
- new Pipeline Progress card with one progress bar per stage
- inline scan banner (processed/total/current folder) inside the
Library section while a scan is running
- Tasks/min throughput computed by diffing worker processed counters
between polls
- Workers section calls out the vision queue and documents the
CELERY_LIGHT/VISION_CONCURRENCY + docker compose up -d scale path
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
62 lines
1.6 KiB
Plaintext
62 lines
1.6 KiB
Plaintext
# Core dependencies
|
|
fastapi==0.109.0
|
|
uvicorn[standard]==0.27.0
|
|
python-multipart==0.0.6
|
|
|
|
# Database
|
|
sqlalchemy[asyncio]==2.0.25
|
|
aiosqlite==0.19.0 # SQLite escape hatch (docker-compose.sqlite.yml override)
|
|
asyncpg==0.29.0 # async Postgres driver (default)
|
|
psycopg2-binary==2.9.9 # sync Postgres driver, used by Alembic CLI
|
|
pgvector==0.2.5 # pgvector SQLAlchemy types
|
|
alembic==1.13.1
|
|
|
|
# Redis and Celery
|
|
redis==5.0.1
|
|
celery==5.3.6
|
|
flower==2.0.1
|
|
|
|
# Image processing
|
|
# pyvips==2.2.1 # Optional - having compatibility issues, using Pillow as fallback
|
|
# rawpy==0.19.0 # Optional - numpy compatibility issues, using Pillow as fallback
|
|
pillow==10.2.0
|
|
pillow-heif==0.15.0
|
|
imagehash==4.3.1 # perceptual hash for duplicate detection
|
|
imageio==2.33.1
|
|
imageio-ffmpeg==0.4.9
|
|
|
|
# Video processing
|
|
ffmpeg-python==0.2.0
|
|
|
|
# Metadata extraction
|
|
pyexiftool==0.5.6
|
|
|
|
# File watching
|
|
watchfiles==0.21.0
|
|
|
|
# Vision pipeline (ONNX Runtime CPU inference)
|
|
onnxruntime==1.18.1
|
|
open-clip-torch==2.24.0 # tokenizer + export helper; inference via ONNX
|
|
ultralytics==8.4.37 # YOLOv8n export helper; inference via ONNX
|
|
rapidocr-onnxruntime==1.3.22
|
|
scikit-learn==1.4.0 # DBSCAN for face clustering
|
|
insightface>=0.7.3 # RetinaFace + ArcFace face detection/recognition
|
|
numpy>=1.26.0,<2.0
|
|
|
|
# Utilities
|
|
pyyaml==6.0.1
|
|
pydantic==2.5.3
|
|
pydantic-settings==2.1.0
|
|
python-dotenv==1.0.0
|
|
httpx==0.26.0
|
|
aiofiles==23.2.1
|
|
|
|
# Security and authentication
|
|
python-jose[cryptography]==3.3.0
|
|
passlib[bcrypt]==1.7.4
|
|
|
|
# Development
|
|
pytest==7.4.4
|
|
pytest-asyncio==0.23.3
|
|
black==23.12.1
|
|
ruff==0.1.11 |