fix: harden pipeline — retries, acks_late, time limits, session safety

Addresses 16 robustness, transparency, and performance issues across
the Celery media processing pipeline:

Critical:
- Singleton DB engine in vision tasks (was leaking one per task call)
- acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks
- Global soft/hard time limits (5/10 min) to prevent hung worker slots
- Thumbnail copy-before-resize (in-place mutation degraded larger sizes)
- backfill_vision now checks each task type independently (OCR, faces, etc.)
- Parameterized LIMIT in backfill_vision (was f-string SQL injection)

High:
- try/except + retry(max=3) on all vision inference tasks
- extract_metadata writes processing_error on exiftool failure
- PIL Image handles closed in _load_thumb/_load_original
- Scan progress Redis keys auto-expire after 1 hour
- Watcher lock renewal is wall-clock based (30s) not event-count based
- worker_process_init signal warms up vision models on startup

Medium:
- Explicit task_routes for every task name (wildcards never matched)
- app.services.metadata added to Celery include list
- POST /maintenance/recover-stuck endpoint for photos stuck in processing
- Docker healthchecks for worker-light, worker-vision, and Redis
- Task ID in vision log lines for distributed tracing
- Bare except:pass narrowed to specific exceptions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

backend/app/tasks/celery.py

@@ -1,15 +1,26 @@
 """
 Celery configuration and app initialization
 """
+import logging
+import os
+
 from celery import Celery
+from celery.signals import worker_process_init
 from app.config import settings
 
+logger = logging.getLogger(__name__)
+
 # Create Celery app
 celery_app = Celery(
     'mulita',
     broker=settings.celery_broker_url,
     backend=settings.celery_result_backend,
-    include=['app.tasks.scan', 'app.tasks.thumbs', 'app.tasks.vision']
+    include=[
+        'app.tasks.scan',
+        'app.tasks.thumbs',
+        'app.tasks.vision',
+        'app.services.metadata',   # extract_metadata lives here
+    ]
 )
 
 # Configure Celery
@@ -19,16 +30,34 @@ celery_app.conf.update(
     result_serializer='json',
     timezone='UTC',
     enable_utc=True,
+    # Robust acknowledgment: keep message in broker until task succeeds.
+    task_acks_late=True,
+    task_reject_on_worker_lost=True,
+    # Global time limits — individual tasks can override via decorator.
+    task_soft_time_limit=300,   # 5 min — raises SoftTimeLimitExceeded
+    task_time_limit=600,        # 10 min — SIGKILL
+    # Explicit routes for every task name. Wildcard patterns don't match
+    # short names produced by @shared_task(name='...').
     task_routes={
-        'app.tasks.thumbs.*': {'queue': 'high'},
-        'app.tasks.scan.*': {'queue': 'low'},
-        'app.tasks.vision.*': {'queue': 'vision'},
+        # Vision queue — GPU/CPU-bound inference
         'embed_photo': {'queue': 'vision'},
         'ocr_photo': {'queue': 'vision'},
         'detect_objects': {'queue': 'vision'},
         'extract_faces': {'queue': 'vision'},
         'classify_content': {'queue': 'vision'},
         'vision_fanout': {'queue': 'vision'},
+        'recluster_faces': {'queue': 'vision'},
+        # High-priority queue — thumbnails & duplicates
+        'generate_thumbnails': {'queue': 'high'},
+        'regenerate_all_thumbnails': {'queue': 'high'},
+        'backfill_phashes': {'queue': 'high'},
+        'regroup_duplicates': {'queue': 'high'},
+        'incremental_regroup_duplicates': {'queue': 'high'},
+        # Low-priority queue — scans
+        'scan_folder': {'queue': 'low'},
+        'scan_all_source_roots': {'queue': 'low'},
+        'backfill_gps': {'queue': 'low'},
+        # Dedicated watcher queue
         'watch_folders': {'queue': 'watcher'},
     },
     task_default_queue='default',
@@ -36,4 +65,22 @@ celery_app.conf.update(
     task_default_exchange_type='direct',
     task_default_routing_key='default',
     broker_connection_retry_on_startup=True,
-)
+)
+
+
+@worker_process_init.connect
+def _warmup_vision_models(**kwargs):
+    """Pre-load vision models in the worker process so the first task
+    doesn't pay cold-start latency. Only runs on the vision queue."""
+    # The worker name contains the queue — only warm up vision workers.
+    worker_queues = os.environ.get("CELERY_QUEUES", "")
+    if "vision" not in worker_queues:
+        # Heuristic: check the celery command line for -Q vision
+        import sys
+        if "vision" not in " ".join(sys.argv):
+            return
+    try:
+        from app.services.vision.registry import registry
+        registry.warmup()
+    except Exception:
+        logger.exception("Vision model warmup failed")