feat: retire the watchfiles watcher in favour of NC webhooks
End-to-end webhook flow is proven on this NC instance (NodeCreated + NodeWritten both fired and dispatched scan_folder on a PUT test), so the watchfiles-based polling layer is no longer needed. - scanner.start_initial_scan no longer queues watch_folders on boot. - scan.watch_folders kept as a one-line no-op shim so any leftover apply_async in flight from the previous deploy doesn't crash a worker. Will be deleted entirely after the queue drains. - celery.py reroutes watch_folders to the `default` queue (worker-light) so the no-op shim actually completes — the `watcher` queue is dead. - docker-compose drops the mulita-worker-watcher service. Its celery --beat responsibility (firing discard_missing_photos_beat every 30 min) moves to worker-light's command. Latency note: NC dispatches webhook events through its background-job queue, currently run by cron */5. After this commit lands you'll want to tighten cron to */1 so new uploads land in mule within ~60s instead of up to 5 min.
This commit is contained in:
@@ -144,7 +144,7 @@ services:
|
||||
dockerfile: Dockerfile
|
||||
image: mule-image-worker
|
||||
container_name: mulita-worker-light
|
||||
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_LIGHT_CONCURRENCY:-2} -Q default,high,low -n light@%h"
|
||||
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --beat --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_LIGHT_CONCURRENCY:-2} -Q default,high,low -n light@%h"
|
||||
volumes:
|
||||
- ./mulita.yml:/app/config/mulita.yml:ro
|
||||
- ${PHOTO_DIRS:-./photos}:/photos:rw
|
||||
@@ -191,52 +191,13 @@ services:
|
||||
- "cloud.hubris.network:192.168.8.175"
|
||||
restart: unless-stopped
|
||||
|
||||
# Dedicated watcher worker — runs the long-lived watch_folders task
|
||||
# on its own queue so it never blocks scan/thumbnail workers.
|
||||
worker-watcher:
|
||||
build:
|
||||
context: ./backend
|
||||
dockerfile: Dockerfile
|
||||
image: mule-image-worker
|
||||
container_name: mulita-worker-watcher
|
||||
# --beat runs the celery beat scheduler in-process alongside the
|
||||
# watcher worker — there's only ever one watcher (Redis-locked
|
||||
# singleton) and we don't need a separate container just to fire a
|
||||
# 30-minute periodic task. Beat schedule lives in app/tasks/celery.py.
|
||||
command: sh -c "celery -A app.tasks.celery worker --beat --loglevel=${LOG_LEVEL:-info} --concurrency=1 -Q watcher -n watcher@%h"
|
||||
volumes:
|
||||
- ./mulita.yml:/app/config/mulita.yml:ro
|
||||
- ${PHOTO_DIRS:-./photos}:/photos:rw
|
||||
- ${NEXTCLOUD_USERS_HOST_PATH:-./photos}:/nextcloud-users:rw
|
||||
- db_data:/data/db
|
||||
environment:
|
||||
- DATABASE_URL=postgresql+asyncpg://mulita:mulita@db:5432/mulita
|
||||
- REDIS_URL=redis://redis:6379
|
||||
- CELERY_BROKER_URL=redis://redis:6379
|
||||
- CELERY_RESULT_BACKEND=redis://redis:6379
|
||||
- PHOTO_DIRS=/photos
|
||||
- NEXTCLOUD_USERS_ROOT=${NEXTCLOUD_USERS_ROOT:-/nextcloud-users}
|
||||
- NEXTCLOUD_BASE_URL=${NEXTCLOUD_BASE_URL:-}
|
||||
- NEXTCLOUD_WEBHOOK_SECRET=${NEXTCLOUD_WEBHOOK_SECRET:-}
|
||||
- LOG_LEVEL=${LOG_LEVEL:-INFO}
|
||||
- TZ=${TZ:-UTC}
|
||||
- MULITA_CELERY_WORKER=1
|
||||
depends_on:
|
||||
redis:
|
||||
condition: service_started
|
||||
db:
|
||||
condition: service_healthy
|
||||
networks:
|
||||
- mulita-network
|
||||
# Pin cloud.hubris.network to the LAN caddy IP. Without this, the
|
||||
# docker DNS forwards the lookup to the host's resolver, which
|
||||
# returns the public IONOS VPS IP — but cloud isn't in the VPS
|
||||
# traefik exposure list, so TLS handshakes against it die with
|
||||
# "unexpected eof while reading". Caddy on 192.168.8.175 holds the
|
||||
# cloud.hubris.network cert and proxies to the Nextcloud LXC.
|
||||
extra_hosts:
|
||||
- "cloud.hubris.network:192.168.8.175"
|
||||
restart: unless-stopped
|
||||
# worker-watcher used to live here — it ran the long-lived
|
||||
# watchfiles-based `watch_folders` task plus celery `--beat`. Both
|
||||
# responsibilities moved on Phase 2:
|
||||
# * file events now come from NC's webhook_listeners → POST
|
||||
# /api/v1/internal/nc-webhook (see backend/app/routers/nc_webhook.py)
|
||||
# * `--beat` was folded into worker-light's command so the
|
||||
# periodic discard_missing_photos_beat job still fires.
|
||||
|
||||
worker-vision:
|
||||
build:
|
||||
|
||||
Reference in New Issue
Block a user