fix: harden pipeline — retries, acks_late, time limits, session safety

Addresses 16 robustness, transparency, and performance issues across
the Celery media processing pipeline:

Critical:
- Singleton DB engine in vision tasks (was leaking one per task call)
- acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks
- Global soft/hard time limits (5/10 min) to prevent hung worker slots
- Thumbnail copy-before-resize (in-place mutation degraded larger sizes)
- backfill_vision now checks each task type independently (OCR, faces, etc.)
- Parameterized LIMIT in backfill_vision (was f-string SQL injection)

High:
- try/except + retry(max=3) on all vision inference tasks
- extract_metadata writes processing_error on exiftool failure
- PIL Image handles closed in _load_thumb/_load_original
- Scan progress Redis keys auto-expire after 1 hour
- Watcher lock renewal is wall-clock based (30s) not event-count based
- worker_process_init signal warms up vision models on startup

Medium:
- Explicit task_routes for every task name (wildcards never matched)
- app.services.metadata added to Celery include list
- POST /maintenance/recover-stuck endpoint for photos stuck in processing
- Docker healthchecks for worker-light, worker-vision, and Redis
- Task ID in vision log lines for distributed tracing
- Bare except:pass narrowed to specific exceptions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

backend/app/tasks/scan.py

@@ -95,11 +95,13 @@ async def _scan_folder_async(folder_path: str, source_root_id: Optional[str], ta
 
     r = _get_redis()
 
+    PROGRESS_TTL = 3600  # 1 hour — auto-expire if scan crashes
+
     def progress_set(key: str, value) -> None:
         if r is None:
             return
         try:
-            r.set(key, str(value))
+            r.set(key, str(value), ex=PROGRESS_TTL)
         except Exception as e:
             logger.debug(f"scan progress set failed: {e}")
 
@@ -467,12 +469,20 @@ async def _scan_all_source_roots_async():
         except Exception as e:
             logger.warning(f"Could not queue post-scan face recluster: {e}")
 
+        # Re-extract metadata for photos missing GPS coordinates.
+        # Runs on every startup so photos scanned before the GPS fix
+        # eventually get their coordinates populated.
+        try:
+            backfill_gps.apply_async(countdown=30)
+        except Exception as e:
+            logger.warning(f"Could not queue post-scan GPS backfill: {e}")
+
 
 WATCHER_LOCK_KEY = "mulita:watch_folders:lock"
-WATCHER_LOCK_TTL = 300  # 5 min — renewed every 60s
+WATCHER_LOCK_TTL = 60  # 1 min — renewed every event batch via wall-clock check
 
 
-@shared_task(name='watch_folders', bind=True)
+@shared_task(name='watch_folders', bind=True, soft_time_limit=None, time_limit=None)
 def watch_folders(self):
     """
     Watch folders for changes using watchfiles. Long-running task that
@@ -527,16 +537,20 @@ def watch_folders(self):
                     return root_id
             return None
 
-        renew_counter = 0
-        for changes in watch(*paths):
-            # Renew the Redis lock periodically so it doesn't expire
-            # while the watcher is idle between events.
-            renew_counter += 1
-            if renew_counter % 10 == 0:
+        import time
+        last_renew = time.monotonic()
+        for changes in watch(*paths, rust_timeout=30_000):
+            # Renew the Redis lock on a wall-clock schedule (every 30s)
+            # instead of every N events, so quiet directories don't let
+            # the lock expire. watchfiles' rust_timeout ensures we wake
+            # at least every 30s even with no FS events.
+            now = time.monotonic()
+            if now - last_renew >= 30:
                 try:
                     lock.extend(WATCHER_LOCK_TTL)
+                    last_renew = now
                 except Exception:
-                    pass
+                    logger.warning("watch_folders: failed to renew Redis lock")
 
             for change_type, filepath in changes:
                 filepath = str(filepath)
@@ -557,7 +571,7 @@ def watch_folders(self):
         try:
             lock.release()
         except Exception:
-            pass
+            logger.warning("watch_folders: could not release Redis lock (may have expired)")
 
 async def handle_file_deletion(filepath: str):
     """Handle deletion of a file from the filesystem"""