fix: harden pipeline — retries, acks_late, time limits, session safety

Addresses 16 robustness, transparency, and performance issues across
the Celery media processing pipeline:

Critical:
- Singleton DB engine in vision tasks (was leaking one per task call)
- acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks
- Global soft/hard time limits (5/10 min) to prevent hung worker slots
- Thumbnail copy-before-resize (in-place mutation degraded larger sizes)
- backfill_vision now checks each task type independently (OCR, faces, etc.)
- Parameterized LIMIT in backfill_vision (was f-string SQL injection)

High:
- try/except + retry(max=3) on all vision inference tasks
- extract_metadata writes processing_error on exiftool failure
- PIL Image handles closed in _load_thumb/_load_original
- Scan progress Redis keys auto-expire after 1 hour
- Watcher lock renewal is wall-clock based (30s) not event-count based
- worker_process_init signal warms up vision models on startup

Medium:
- Explicit task_routes for every task name (wildcards never matched)
- app.services.metadata added to Celery include list
- POST /maintenance/recover-stuck endpoint for photos stuck in processing
- Docker healthchecks for worker-light, worker-vision, and Redis
- Task ID in vision log lines for distributed tracing
- Bare except:pass narrowed to specific exceptions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-13 08:57:10 +02:00
parent e974ffbfd2
commit f090a809a9
8 changed files with 366 additions and 110 deletions

View File

@@ -137,6 +137,35 @@ async def trigger_scan(current_user: User = Depends(get_current_user)):
return {"status": "success", "message": "Library scan started"}
@router.post("/maintenance/recover-stuck")
async def recover_stuck_photos(
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user),
):
"""Reset photos stuck in 'processing' for more than 30 minutes back to
'pending' so the pipeline can retry them. Returns the count of recovered
photos."""
from datetime import datetime, timedelta, timezone
cutoff = datetime.now(timezone.utc) - timedelta(minutes=30)
result = await db.execute(
update(Photo)
.where(
Photo.processing_status == 'processing',
Photo.updated_at < cutoff,
)
.values(
processing_status='pending',
processing_error='Auto-recovered from stuck processing state',
)
)
await db.commit()
count = result.rowcount
if count:
logger.info("Recovered %d stuck photos back to pending", count)
return {"status": "success", "recovered": count}
@router.post("/backfill-gps")
async def trigger_backfill_gps(current_user: User = Depends(get_current_user)):
"""Re-run EXIF metadata extraction on every photo that's still missing