fix: harden pipeline — retries, acks_late, time limits, session safety

Addresses 16 robustness, transparency, and performance issues across
the Celery media processing pipeline:

Critical:
- Singleton DB engine in vision tasks (was leaking one per task call)
- acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks
- Global soft/hard time limits (5/10 min) to prevent hung worker slots
- Thumbnail copy-before-resize (in-place mutation degraded larger sizes)
- backfill_vision now checks each task type independently (OCR, faces, etc.)
- Parameterized LIMIT in backfill_vision (was f-string SQL injection)

High:
- try/except + retry(max=3) on all vision inference tasks
- extract_metadata writes processing_error on exiftool failure
- PIL Image handles closed in _load_thumb/_load_original
- Scan progress Redis keys auto-expire after 1 hour
- Watcher lock renewal is wall-clock based (30s) not event-count based
- worker_process_init signal warms up vision models on startup

Medium:
- Explicit task_routes for every task name (wildcards never matched)
- app.services.metadata added to Celery include list
- POST /maintenance/recover-stuck endpoint for photos stuck in processing
- Docker healthchecks for worker-light, worker-vision, and Redis
- Task ID in vision log lines for distributed tracing
- Bare except:pass narrowed to specific exceptions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
root
2026-04-13 08:57:10 +02:00
parent e974ffbfd2
commit f090a809a9
8 changed files with 366 additions and 110 deletions

View File

@@ -103,6 +103,16 @@ def bootstrap(models_dir: str | None = None):
else:
logger.info("All model files present in %s", base)
# Signal readiness via Redis so the scan pipeline knows the vision
# worker can accept tasks.
try:
import redis as _redis
r = _redis.from_url(settings.redis_url)
r.set("mulita:vision:ready", "1")
logger.info("Set mulita:vision:ready in Redis")
except Exception as e:
logger.warning("Could not set vision readiness flag in Redis: %s", e)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)