fix: harden pipeline — retries, acks_late, time limits, session safety

Addresses 16 robustness, transparency, and performance issues across
the Celery media processing pipeline:

Critical:
- Singleton DB engine in vision tasks (was leaking one per task call)
- acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks
- Global soft/hard time limits (5/10 min) to prevent hung worker slots
- Thumbnail copy-before-resize (in-place mutation degraded larger sizes)
- backfill_vision now checks each task type independently (OCR, faces, etc.)
- Parameterized LIMIT in backfill_vision (was f-string SQL injection)

High:
- try/except + retry(max=3) on all vision inference tasks
- extract_metadata writes processing_error on exiftool failure
- PIL Image handles closed in _load_thumb/_load_original
- Scan progress Redis keys auto-expire after 1 hour
- Watcher lock renewal is wall-clock based (30s) not event-count based
- worker_process_init signal warms up vision models on startup

Medium:
- Explicit task_routes for every task name (wildcards never matched)
- app.services.metadata added to Celery include list
- POST /maintenance/recover-stuck endpoint for photos stuck in processing
- Docker healthchecks for worker-light, worker-vision, and Redis
- Task ID in vision log lines for distributed tracing
- Bare except:pass narrowed to specific exceptions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

backend/app/services/metadata.py

@@ -209,21 +209,24 @@ async def _extract_metadata_async(photo_id: str):
                     cmd,
                     capture_output=True,
                     text=True,
-                    timeout=30
+                    timeout=30,
+                    stdin=subprocess.DEVNULL,
                 )
-                
+
                 if result.returncode != 0:
                     logger.error(f"ExifTool error: {result.stderr}")
+                    photo.processing_error = f"ExifTool: {result.stderr[:500]}"
+                    await session.commit()
                     return {'status': 'error', 'message': result.stderr}
-                
+
                 # Parse JSON output
                 metadata = json.loads(result.stdout)
                 if metadata and len(metadata) > 0:
                     exif_data = metadata[0]
-                    
+
                     # Store full metadata as JSON
                     photo.exif_json = json.dumps(exif_data)
-                    
+
                     # Extract taken_at date
                     date_fields = [
                         'EXIF:DateTimeOriginal',
@@ -231,7 +234,7 @@ async def _extract_metadata_async(photo_id: str):
                         'QuickTime:MediaCreateDate',
                         'EXIF:ModifyDate'
                     ]
-                    
+
                     for field in date_fields:
                         if field in exif_data:
                             taken_at = parse_exif_datetime(exif_data[field])
@@ -247,7 +250,7 @@ async def _extract_metadata_async(photo_id: str):
                     photo.has_date_warning = has_date_warning(
                         photo.filepath, photo.taken_at
                     )
-                    
+
                     # Extract dimensions if not already set
                     if not photo.width:
                         photo.width = exif_data.get('EXIF:ImageWidth') or exif_data.get('File:ImageWidth')
@@ -262,26 +265,27 @@ async def _extract_metadata_async(photo_id: str):
 
                     # Extract and store key metadata for search
                     key_metadata = extract_key_metadata(exif_data)
-                    
-                    # Update FTS table (would be done via trigger in production)
-                    # For now, we'll store it in a comment
-                    
+
                     await session.commit()
-                    
+
                     logger.info(f"Metadata extracted for photo {photo_id}")
                     return {
                         'status': 'success',
                         'photo_id': photo_id,
                         'taken_at': photo.taken_at.isoformat() if photo.taken_at else None
                     }
-                
+
             except subprocess.TimeoutExpired:
                 logger.error(f"ExifTool timeout for {photo.filepath}")
+                photo.processing_error = 'ExifTool timeout'
+                await session.commit()
                 return {'status': 'error', 'message': 'ExifTool timeout'}
             except json.JSONDecodeError as e:
                 logger.error(f"Failed to parse ExifTool output: {e}")
+                photo.processing_error = f"Invalid ExifTool output: {e}"
+                await session.commit()
                 return {'status': 'error', 'message': 'Invalid ExifTool output'}
-            
+
         except Exception as e:
             logger.error(f"Error extracting metadata for {photo_id}: {e}")
             return {'status': 'error', 'message': str(e)}

backend/app/services/vision/bootstrap_models.py

@@ -103,6 +103,16 @@ def bootstrap(models_dir: str | None = None):
     else:
         logger.info("All model files present in %s", base)
 
+    # Signal readiness via Redis so the scan pipeline knows the vision
+    # worker can accept tasks.
+    try:
+        import redis as _redis
+        r = _redis.from_url(settings.redis_url)
+        r.set("mulita:vision:ready", "1")
+        logger.info("Set mulita:vision:ready in Redis")
+    except Exception as e:
+        logger.warning("Could not set vision readiness flag in Redis: %s", e)
+
 
 if __name__ == "__main__":
     logging.basicConfig(level=logging.INFO)