refactor: drop AI/vision pipeline + plain Postgres + full-refresh script

Removes the OpenCLIP-on-ONNX classifier and everything that fed or
consumed it:
  - backend: app/services/vision/, app/tasks/vision.py,
    app/services/feature_flags.py, app/routers/features.py — all
    deleted; admin AI/feature-flag endpoints and the worker-vision
    bootstrap call gone. Photo.needs_review and its index dropped.
  - frontend: AI Settings tab, useFeaturesQuery hook, FeatureFlag
    types, "Needs Review" sidebar entry + filter, needs_review filter
    URL param all gone.
  - infra: worker-vision compose service + models_data volume deleted;
    worker-light command no longer runs bootstrap_models; the db
    image switches from pgvector/pgvector:pg16 to postgres:16; backend
    Dockerfile drops the dedicated torch RUN layer; requirements.txt
    drops torch/torchvision/open-clip-torch/onnxruntime.

Alembic 0019_drop_ai_remnants:
  - drops photos.needs_review + ix_photos_needs_review
  - DROP EXTENSION IF EXISTS vector (must run before the image swap;
    the new postgres:16 doesn't ship pgvector)

New scripts/full_refresh.py: one-shot DB ↔ filesystem reconciliation.
Runs cleanup_data_integrity, scans every active SourceRoot inline
(no celery dependency so the worker can be stopped), hard-prunes
photo + folder rows for files that are gone, removes orphan
/data/thumbs/{user}/{photo}/ directories. New helper
prune_orphan_thumbnails in cleanup.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
claudio
2026-05-14 00:20:38 +02:00
parent 6915c30911
commit a27267f7ad
39 changed files with 265 additions and 1573 deletions

View File

@@ -30,19 +30,6 @@ class PerformanceSettings(BaseModel):
db_pool_max_overflow: int = 10
db_pool_recycle: int = 3600
class ClassifierSettings(BaseModel):
"""Binary content classifier (photography vs other)."""
min_confidence: float = 0.3
class VisionSettings(BaseModel):
"""Vision pipeline — one binary classifier (photography vs other)."""
enabled: bool = True
backend: str = "onnx"
models_dir: str = "/data/models"
execution_providers: list[str] = ["CPUExecutionProvider"]
classifier: ClassifierSettings = ClassifierSettings()
worker_concurrency: int = 2
class MulitaConfig(BaseModel):
"""Main configuration from YAML file. Source roots and the discard
workflow are owned by the database now — only operational settings
@@ -50,13 +37,12 @@ class MulitaConfig(BaseModel):
thumbnails: ThumbnailSettings = ThumbnailSettings()
scanner: ScannerSettings = ScannerSettings()
performance: PerformanceSettings = PerformanceSettings()
vision: VisionSettings = VisionSettings()
class Settings(BaseSettings):
"""Application settings"""
# Database — Postgres + pgvector by default. The SQLite escape hatch
# remains supported via the docker-compose.sqlite.yml override and by
# setting DATABASE_URL=sqlite+aiosqlite:///... in .env for local dev.
# Database — plain Postgres. The SQLite escape hatch remains
# supported via the docker-compose.sqlite.yml override and by setting
# DATABASE_URL=sqlite+aiosqlite:///... in .env for local dev.
database_url: str = Field(
default="postgresql+asyncpg://mulita:mulita@db:5432/mulita",
env="DATABASE_URL"
@@ -195,23 +181,6 @@ class Settings(BaseSettings):
def performance(self) -> PerformanceSettings:
return self.config.performance
# ONNX Runtime execution providers, overridable via env var.
# Comma-separated: "CUDAExecutionProvider,CPUExecutionProvider"
# or "auto" for GPU auto-detection.
vision_execution_providers: str = Field(
default="CPUExecutionProvider",
env="VISION_EXECUTION_PROVIDERS",
)
@property
def vision(self) -> VisionSettings:
v = self.config.vision
# Override execution_providers from env if set.
providers = [p.strip() for p in self.vision_execution_providers.split(",") if p.strip()]
if providers:
v.execution_providers = providers
return v
class Config:
env_file = ".env"
case_sensitive = False