Commit Graph

6 Commits

Author SHA1 Message Date
claudio
a27267f7ad refactor: drop AI/vision pipeline + plain Postgres + full-refresh script
Removes the OpenCLIP-on-ONNX classifier and everything that fed or
consumed it:
  - backend: app/services/vision/, app/tasks/vision.py,
    app/services/feature_flags.py, app/routers/features.py — all
    deleted; admin AI/feature-flag endpoints and the worker-vision
    bootstrap call gone. Photo.needs_review and its index dropped.
  - frontend: AI Settings tab, useFeaturesQuery hook, FeatureFlag
    types, "Needs Review" sidebar entry + filter, needs_review filter
    URL param all gone.
  - infra: worker-vision compose service + models_data volume deleted;
    worker-light command no longer runs bootstrap_models; the db
    image switches from pgvector/pgvector:pg16 to postgres:16; backend
    Dockerfile drops the dedicated torch RUN layer; requirements.txt
    drops torch/torchvision/open-clip-torch/onnxruntime.

Alembic 0019_drop_ai_remnants:
  - drops photos.needs_review + ix_photos_needs_review
  - DROP EXTENSION IF EXISTS vector (must run before the image swap;
    the new postgres:16 doesn't ship pgvector)

New scripts/full_refresh.py: one-shot DB ↔ filesystem reconciliation.
Runs cleanup_data_integrity, scans every active SourceRoot inline
(no celery dependency so the worker can be stopped), hard-prunes
photo + folder rows for files that are gone, removes orphan
/data/thumbs/{user}/{photo}/ directories. New helper
prune_orphan_thumbnails in cleanup.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 00:20:38 +02:00
claudio
6915c30911 build(backend): cache torch wheel + pip downloads across rebuilds
Splits torch+torchvision into its own RUN layer so edits to
requirements.txt don't invalidate the ~200MB CPU-only torch download.
Adds buildkit cache mounts on both pip-install layers so even when a
layer is invalidated (or buildkit evicts it) the wheel is reused from
the on-disk pip cache instead of refetching from download.pytorch.org.

Triggered by two consecutive deploy failures where pytorch.org timed
out mid-download (2026-05-13 ~21:41 and possibly ~22:47).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 23:08:10 +02:00
root
afe420c620 fix: use CPU-only PyTorch and run init_db before alembic migrations
PyTorch default install pulls ~7GB of CUDA libs, exceeding disk on small
VMs. Switching to CPU-only saves ~6GB. Also run create_all before alembic
so migrations find existing tables on a fresh Postgres.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 18:23:32 +02:00
9282a5c734 feat: add vision pipeline scaffolding with ONNX backend
Introduce the app/services/vision/ module with ABC interfaces, ONNX
Runtime backend, model registry, and per-task implementations:
- OpenCLIP ViT-B/32 embedder (image + text, 512-d)
- RapidOCR engine (PP-OCRv4 via ONNX, no PaddlePaddle)
- YOLOv8n object detector (raw ONNX, no ultralytics runtime)
- YuNet + SFace face processor (Apache 2.0, opencv_zoo, 128-d)
- DBSCAN face clustering helper

Add VisionSettings to config (mulita.yml + Pydantic), bootstrap_models.py
for first-boot weight downloads, models_data Docker volume, and ROCm
backend stub for future GPU acceleration.

No Celery tasks wired yet — models load but nothing invokes them.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 09:00:06 +02:00
204d2bf2a8 feat: simplify folder setup — single mount, auto bootstrap, browser dialog
Cleans up the maze of overlapping ways folders entered the app, plus
removes the dead trash plumbing left over from the soft-discard
refactor.

Setup model (now)
- ONE env var: PHOTO_DIRS in .env, set to the host path of your
  library. Compose mounts that at /photos. That's the entire setup.
- On first boot, the backend auto-creates a SourceRoot row named
  "Library" pointing at /photos so the user sees their photos
  immediately without configuring anything.
- Source roots and discard live in the database; mulita.yml only
  carries operational settings (thumbnails, scanner, performance).
- The "Add Source Folder" dialog is now a directory browser
  restricted server-side to /photos and any existing source root —
  the user clicks through actual mounted directories instead of
  typing container paths they can't possibly know.

Backend
- New services/scanner.bootstrap_default_source_root(): if no
  SourceRoot rows exist and /photos is mounted, create one. Wired
  into the lifespan handler before cleanup + initial scan.
- New GET /library/browse?path= returning the immediate child
  directories of `path`, validated to live under one of the allowed
  roots (default mount + every active SourceRoot). Hidden entries
  are filtered. Children are tagged with is_existing_root so the UI
  can show an "Added" badge. Returns parent path for up-nav, or
  null when at the top of the allowed scope.
- scan_all_source_roots now reads from the DB instead of the YAML
  config so DB-managed source roots are honoured by initial scan.
- Dropped the placeholder source_roots block from mulita.yml — the
  paths /photos/main and /photos/iphone never existed and just
  produced startup warnings.
- Dropped TrashSettings, settings.trash, settings.source_roots,
  and the SourceRoot pydantic model from config.py. Soft discard
  has owned this for a while; it was dead code.

Compose
- Single ${PHOTO_DIRS:-./photos}:/photos:rw mount in both backend
  and worker.
- Removed the hardcoded ~/Pictures:/host/Pictures:rw mount — the
  PHOTO_DIRS variable is the single source of truth now.
- Removed the trash_data named volume + mounts (no consumers).
- backend/Dockerfile no longer creates /data/trash; it now creates
  /data/proxies (which the proxy endpoint actually uses).

Frontend
- AddSourceFolderDialog rewritten as a directory tree picker:
  loads /library/browse on open, lets the user navigate up via a
  ChevronUp button or down by clicking subfolders, shows the
  current path inline, and adds whatever directory is currently
  shown. Existing source roots are tagged "Added" so the user
  knows what's already registered. Errors from the backend (e.g.
  trying to navigate outside the allowed scope) surface inline.
- New library.browse() helper + BrowseChild / BrowseResponse types
  in services/api.ts.

Docs
- README Quick Start rewritten around the single PHOTO_DIRS env
  var, with macOS/Linux/Windows examples.
- New "How mounted folders and source folders relate" section that
  spells out the two-layer model (mount = visibility, source root
  = scanning) so the most common confusion is addressed up front.
- Added a "Read-only libraries" subsection that lists exactly which
  endpoints fail under :ro.
- "Configuration" section reframed: source roots are managed by the
  UI/API now, mulita.yml is operational settings only.
- .env file now has examples for the common host paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 00:27:21 +02:00
46a0d7aba8 feat: structure 2026-04-06 23:30:19 +02:00