fix: harden pipeline — retries, acks_late, time limits, session safety
Addresses 16 robustness, transparency, and performance issues across the Celery media processing pipeline: Critical: - Singleton DB engine in vision tasks (was leaking one per task call) - acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks - Global soft/hard time limits (5/10 min) to prevent hung worker slots - Thumbnail copy-before-resize (in-place mutation degraded larger sizes) - backfill_vision now checks each task type independently (OCR, faces, etc.) - Parameterized LIMIT in backfill_vision (was f-string SQL injection) High: - try/except + retry(max=3) on all vision inference tasks - extract_metadata writes processing_error on exiftool failure - PIL Image handles closed in _load_thumb/_load_original - Scan progress Redis keys auto-expire after 1 hour - Watcher lock renewal is wall-clock based (30s) not event-count based - worker_process_init signal warms up vision models on startup Medium: - Explicit task_routes for every task name (wildcards never matched) - app.services.metadata added to Celery include list - POST /maintenance/recover-stuck endpoint for photos stuck in processing - Docker healthchecks for worker-light, worker-vision, and Redis - Task ID in vision log lines for distributed tracing - Bare except:pass narrowed to specific exceptions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -137,6 +137,35 @@ async def trigger_scan(current_user: User = Depends(get_current_user)):
|
||||
return {"status": "success", "message": "Library scan started"}
|
||||
|
||||
|
||||
@router.post("/maintenance/recover-stuck")
|
||||
async def recover_stuck_photos(
|
||||
db: AsyncSession = Depends(get_db),
|
||||
current_user: User = Depends(get_current_user),
|
||||
):
|
||||
"""Reset photos stuck in 'processing' for more than 30 minutes back to
|
||||
'pending' so the pipeline can retry them. Returns the count of recovered
|
||||
photos."""
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(minutes=30)
|
||||
result = await db.execute(
|
||||
update(Photo)
|
||||
.where(
|
||||
Photo.processing_status == 'processing',
|
||||
Photo.updated_at < cutoff,
|
||||
)
|
||||
.values(
|
||||
processing_status='pending',
|
||||
processing_error='Auto-recovered from stuck processing state',
|
||||
)
|
||||
)
|
||||
await db.commit()
|
||||
count = result.rowcount
|
||||
if count:
|
||||
logger.info("Recovered %d stuck photos back to pending", count)
|
||||
return {"status": "success", "recovered": count}
|
||||
|
||||
|
||||
@router.post("/backfill-gps")
|
||||
async def trigger_backfill_gps(current_user: User = Depends(get_current_user)):
|
||||
"""Re-run EXIF metadata extraction on every photo that's still missing
|
||||
|
||||
Reference in New Issue
Block a user