feat: CLIP-powered incremental duplicate detection

Replace O(N²) pHash-only duplicate detection with a hybrid approach:
- pHash Hamming distance for exact/near-exact copies
- CLIP embedding cosine similarity via pgvector HNSW for visually
  similar photos (crops, format changes, screenshots)

Post-scan now uses incremental mode: only newly added photos are
compared against the full library — O(new × log N) via HNSW index
instead of O(N²). Full regroup remains available from Settings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-12 22:03:04 +02:00
parent 8f41a23c41
commit c7dd03ade2
3 changed files with 295 additions and 122 deletions

View File

@@ -467,10 +467,23 @@ async def _backfill_phashes_async():
@shared_task(name='regroup_duplicates')
def regroup_duplicates_task():
"""Celery wrapper around app.services.duplicates.regroup_duplicates.
"""Full recompute of duplicate groups (pHash + CLIP similarity).
Importing the service inside the task body avoids a circular import
at worker boot (the service uses AsyncSessionLocal which is also
imported here at module top)."""
Used by the Settings → Re-detect duplicates button."""
from app.services.duplicates import regroup_duplicates
return asyncio.run(regroup_duplicates())
return asyncio.run(regroup_duplicates())
@shared_task(name='incremental_regroup_duplicates')
def incremental_regroup_duplicates_task(since_iso: str | None = None):
"""Incremental duplicate detection for newly added photos.
Compares only photos added after `since_iso` against the full library
using CLIP vector similarity (O(new × log N) via HNSW) plus pHash.
Default post-scan path — much faster than a full regroup."""
from app.services.duplicates import incremental_regroup
from datetime import datetime, timezone
since = None
if since_iso:
since = datetime.fromisoformat(since_iso)
return asyncio.run(incremental_regroup(since=since))