feat: CLIP-powered incremental duplicate detection
Replace O(N²) pHash-only duplicate detection with a hybrid approach: - pHash Hamming distance for exact/near-exact copies - CLIP embedding cosine similarity via pgvector HNSW for visually similar photos (crops, format changes, screenshots) Post-scan now uses incremental mode: only newly added photos are compared against the full library — O(new × log N) via HNSW index instead of O(N²). Full regroup remains available from Settings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -467,10 +467,23 @@ async def _backfill_phashes_async():
|
||||
|
||||
@shared_task(name='regroup_duplicates')
|
||||
def regroup_duplicates_task():
|
||||
"""Celery wrapper around app.services.duplicates.regroup_duplicates.
|
||||
"""Full recompute of duplicate groups (pHash + CLIP similarity).
|
||||
|
||||
Importing the service inside the task body avoids a circular import
|
||||
at worker boot (the service uses AsyncSessionLocal which is also
|
||||
imported here at module top)."""
|
||||
Used by the Settings → Re-detect duplicates button."""
|
||||
from app.services.duplicates import regroup_duplicates
|
||||
return asyncio.run(regroup_duplicates())
|
||||
return asyncio.run(regroup_duplicates())
|
||||
|
||||
|
||||
@shared_task(name='incremental_regroup_duplicates')
|
||||
def incremental_regroup_duplicates_task(since_iso: str | None = None):
|
||||
"""Incremental duplicate detection for newly added photos.
|
||||
|
||||
Compares only photos added after `since_iso` against the full library
|
||||
using CLIP vector similarity (O(new × log N) via HNSW) plus pHash.
|
||||
Default post-scan path — much faster than a full regroup."""
|
||||
from app.services.duplicates import incremental_regroup
|
||||
from datetime import datetime, timezone
|
||||
since = None
|
||||
if since_iso:
|
||||
since = datetime.fromisoformat(since_iso)
|
||||
return asyncio.run(incremental_regroup(since=since))
|
||||
Reference in New Issue
Block a user