feat: retire the watchfiles watcher in favour of NC webhooks

End-to-end webhook flow is proven on this NC instance (NodeCreated +
NodeWritten both fired and dispatched scan_folder on a PUT test), so
the watchfiles-based polling layer is no longer needed.

- scanner.start_initial_scan no longer queues watch_folders on boot.
- scan.watch_folders kept as a one-line no-op shim so any leftover
  apply_async in flight from the previous deploy doesn't crash a
  worker. Will be deleted entirely after the queue drains.
- celery.py reroutes watch_folders to the `default` queue (worker-light)
  so the no-op shim actually completes — the `watcher` queue is dead.
- docker-compose drops the mulita-worker-watcher service. Its celery
  --beat responsibility (firing discard_missing_photos_beat every 30
  min) moves to worker-light's command.

Latency note: NC dispatches webhook events through its background-job
queue, currently run by cron */5. After this commit lands you'll want
to tighten cron to */1 so new uploads land in mule within ~60s instead
of up to 5 min.
This commit is contained in:
Claudio
2026-05-11 12:28:36 +02:00
parent 362fbc6d83
commit f657e2c0ba
4 changed files with 31 additions and 157 deletions

View File

@@ -144,7 +144,7 @@ services:
dockerfile: Dockerfile
image: mule-image-worker
container_name: mulita-worker-light
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_LIGHT_CONCURRENCY:-2} -Q default,high,low -n light@%h"
command: sh -c "python -m app.services.vision.bootstrap_models && celery -A app.tasks.celery worker --beat --loglevel=${LOG_LEVEL:-info} --concurrency=${CELERY_LIGHT_CONCURRENCY:-2} -Q default,high,low -n light@%h"
volumes:
- ./mulita.yml:/app/config/mulita.yml:ro
- ${PHOTO_DIRS:-./photos}:/photos:rw
@@ -191,52 +191,13 @@ services:
- "cloud.hubris.network:192.168.8.175"
restart: unless-stopped
# Dedicated watcher worker — runs the long-lived watch_folders task
# on its own queue so it never blocks scan/thumbnail workers.
worker-watcher:
build:
context: ./backend
dockerfile: Dockerfile
image: mule-image-worker
container_name: mulita-worker-watcher
# --beat runs the celery beat scheduler in-process alongside the
# watcher worker — there's only ever one watcher (Redis-locked
# singleton) and we don't need a separate container just to fire a
# 30-minute periodic task. Beat schedule lives in app/tasks/celery.py.
command: sh -c "celery -A app.tasks.celery worker --beat --loglevel=${LOG_LEVEL:-info} --concurrency=1 -Q watcher -n watcher@%h"
volumes:
- ./mulita.yml:/app/config/mulita.yml:ro
- ${PHOTO_DIRS:-./photos}:/photos:rw
- ${NEXTCLOUD_USERS_HOST_PATH:-./photos}:/nextcloud-users:rw
- db_data:/data/db
environment:
- DATABASE_URL=postgresql+asyncpg://mulita:mulita@db:5432/mulita
- REDIS_URL=redis://redis:6379
- CELERY_BROKER_URL=redis://redis:6379
- CELERY_RESULT_BACKEND=redis://redis:6379
- PHOTO_DIRS=/photos
- NEXTCLOUD_USERS_ROOT=${NEXTCLOUD_USERS_ROOT:-/nextcloud-users}
- NEXTCLOUD_BASE_URL=${NEXTCLOUD_BASE_URL:-}
- NEXTCLOUD_WEBHOOK_SECRET=${NEXTCLOUD_WEBHOOK_SECRET:-}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
- TZ=${TZ:-UTC}
- MULITA_CELERY_WORKER=1
depends_on:
redis:
condition: service_started
db:
condition: service_healthy
networks:
- mulita-network
# Pin cloud.hubris.network to the LAN caddy IP. Without this, the
# docker DNS forwards the lookup to the host's resolver, which
# returns the public IONOS VPS IP — but cloud isn't in the VPS
# traefik exposure list, so TLS handshakes against it die with
# "unexpected eof while reading". Caddy on 192.168.8.175 holds the
# cloud.hubris.network cert and proxies to the Nextcloud LXC.
extra_hosts:
- "cloud.hubris.network:192.168.8.175"
restart: unless-stopped
# worker-watcher used to live hereit ran the long-lived
# watchfiles-based `watch_folders` task plus celery `--beat`. Both
# responsibilities moved on Phase 2:
# * file events now come from NC's webhook_listeners → POST
# /api/v1/internal/nc-webhook (see backend/app/routers/nc_webhook.py)
# * `--beat` was folded into worker-light's command so the
# periodic discard_missing_photos_beat job still fires.
worker-vision:
build: