fix: harden pipeline — retries, acks_late, time limits, session safety
Addresses 16 robustness, transparency, and performance issues across the Celery media processing pipeline: Critical: - Singleton DB engine in vision tasks (was leaking one per task call) - acks_late + task_reject_on_worker_lost so crashed workers don't lose tasks - Global soft/hard time limits (5/10 min) to prevent hung worker slots - Thumbnail copy-before-resize (in-place mutation degraded larger sizes) - backfill_vision now checks each task type independently (OCR, faces, etc.) - Parameterized LIMIT in backfill_vision (was f-string SQL injection) High: - try/except + retry(max=3) on all vision inference tasks - extract_metadata writes processing_error on exiftool failure - PIL Image handles closed in _load_thumb/_load_original - Scan progress Redis keys auto-expire after 1 hour - Watcher lock renewal is wall-clock based (30s) not event-count based - worker_process_init signal warms up vision models on startup Medium: - Explicit task_routes for every task name (wildcards never matched) - app.services.metadata added to Celery include list - POST /maintenance/recover-stuck endpoint for photos stuck in processing - Docker healthchecks for worker-light, worker-vision, and Redis - Task ID in vision log lines for distributed tracing - Bare except:pass narrowed to specific exceptions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -116,6 +116,12 @@ services:
|
||||
condition: service_started
|
||||
db:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "celery -A app.tasks.celery inspect ping -d light@$$HOSTNAME 2>/dev/null | grep -q OK"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 120s
|
||||
networks:
|
||||
- mulita-network
|
||||
restart: unless-stopped
|
||||
@@ -197,6 +203,12 @@ services:
|
||||
# - driver: nvidia
|
||||
# count: all
|
||||
# capabilities: [gpu]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "celery -A app.tasks.celery inspect ping -d vision@$$HOSTNAME 2>/dev/null | grep -q OK"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 300s
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_started
|
||||
@@ -239,6 +251,11 @@ services:
|
||||
- mulita-network
|
||||
restart: unless-stopped
|
||||
command: redis-server --appendonly yes
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
|
||||
networks:
|
||||
mulita-network:
|
||||
|
||||
Reference in New Issue
Block a user