dtoro/mule-image
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: perceptual-hash duplicate detection + grouped picker view

Browse Source 
The Duplicates section was useless: SHA-256-only detection only caught
byte-identical files, not the actual duplicates a real library
accumulates (re-encoded JPEGs, screenshots, resized exports), and the
view was a flat date-sorted list with no grouping or actions. This
replaces the whole flow.

Detection
- New phash + duplicate_group_id columns on Photo, added via an
  idempotent ALTER TABLE pass in init_db (the project has no Alembic).
- Thumbs worker computes a 64-bit pHash from the original-resolution
  decoded frame just before the destructive thumbnail loop. Falls back
  silently — phash is nice-to-have, not a blocker for thumbnails.
- backfill_phashes Celery task fills in phashes for photos that
  predated the column, reading the existing thumb_large rather than
  re-decoding the original.
- regroup_duplicates service runs union-find over Hamming distance
  (threshold 6), persists duplicate_group_id, and maintains is_duplicate
  as derived state so existing badges/counts keep working. Chained
  after scan_all_source_roots with a 60s countdown.

API
- GET /library/duplicates/groups returns all groups with members,
  bucketed in Python from one query. Each group has a reason ("exact"
  iff every member shares a SHA-256, "similar" otherwise).
- POST /library/maintenance/{regroup-duplicates,backfill-phashes}.

Frontend
- New DuplicatesView (sectioned grid, one section per cluster) replaces
  the timeline when the user is in the duplicates section. Each section
  shows a "Keep best, discard N" button that picks the highest-pixel
  copy and reuses the existing undoable bulk-discard so Cmd+Z works.
- Manual best override: hover any non-best thumbnail and click "Keep
  this" (Crown icon, top-right) to override the auto-pick. The header
  annotates "(manual)" so it's obvious which copy will be kept.
- Keyboard nav within the duplicates view walks the flat member list,
  with ↑/↓ jumping by the measured column count and scrollIntoView on
  every move. Timeline's keyboard handler now early-returns in the
  duplicates section so the two don't fight.
- BEST pill / Keep-this button live at top-right with a ring outline so
  they don't collide visually with the cyan selection ring around a
  selected cell. Dimensions chip moved to bottom-left to free both
  right corners for the keep affordances.
- New "Duplicates" section in SettingsDialog: shows group/member counts
  and exposes both backfill + re-detect actions, sharing a query cache
  with DuplicatesView via DUPLICATE_GROUPS_QUERY_KEY.
- PhotoInfoPanel "Basic Info" section now shows the photo's full file
  path in monospace below the size/dimensions/date grid.
- New imagehash==4.3.1 dep in requirements.txt.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-04-09 17:19:08 +02:00
parent e51b93d59e
commit 733c16bf82
14 changed files with 1098 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
backend/app/tasks/scan.py
View File

@@ -350,18 +350,39 @@ def scan_all_source_roots():
async def _scan_all_source_roots_async():
"""Read every active SourceRoot from the DB and queue a scan_folder task
for each. Source roots whose path no longer exists on disk are skipped
with a warning (the cleanup service surfaces those at startup too)."""
with a warning (the cleanup service surfaces those at startup too).
After dispatching the scans, queue a delayed `regroup_duplicates`
pass so duplicate clusters are recomputed once the new photos have
finished thumbnailing (and therefore picked up phashes). The
countdown is a best-effort hint — on a big library the user can
still hit Settings → Re-detect duplicates to force a fresh pass.
"""
from app.tasks.thumbs import regroup_duplicates_task
async with AsyncSessionLocal() as session:
result = await session.execute(
select(SourceRoot).where(SourceRoot.is_active == True) # noqa: E712
)
source_roots = result.scalars().all()
dispatched = 0
for sr in source_roots:
if os.path.exists(sr.path):
scan_folder.delay(sr.path, sr.id)
dispatched += 1
else:
logger.warning(f"Source root path does not exist: {sr.path}")
if dispatched > 0:
# 60s gives the thumbs worker a window to compute phashes for
# the new photos before regrouping. The task is idempotent, so
# firing too early just means the next manual run picks up the
# late arrivals — no corrupted state.
try:
regroup_duplicates_task.apply_async(countdown=60)
except Exception as e:
logger.warning(f"Could not queue post-scan regroup: {e}")
@shared_task(name='watch_folders')
def watch_folders():

97
backend/app/tasks/thumbs.py
View File

@@ -280,11 +280,23 @@ async def _generate_thumbnails_async(photo_id: str, task):
# Auto-rotate based on EXIF
image = auto_rotate_image(image)
# Store original dimensions
photo.width = image.width
photo.height = image.height
# Perceptual hash from the original-resolution decoded frame.
# pHash is robust to resize/recompression but the thumbnail
# loop below mutates `image` in place, so this MUST run before
# the loop sees it. Failures are non-fatal — phash is a
# nice-to-have, not a blocker for thumbnail generation.
try:
import imagehash
photo.phash = str(imagehash.phash(image)) # 16-char hex
except Exception as e:
logger.warning(f"phash failed for {photo_id}: {e}")
photo.phash = None
# Generate thumbnails for each size
for size_name, size_value in THUMB_SIZES.items():
thumb_path = get_thumb_path(photo_id, size_name)
@@ -333,10 +345,83 @@ async def _regenerate_all_thumbnails_async():
)
)
photos = result.scalars().all()
logger.info(f"Regenerating thumbnails for {len(photos)} photos")
for photo in photos:
generate_thumbnails.delay(photo.id)
return {'status': 'queued', 'count': len(photos)}
return {'status': 'queued', 'count': len(photos)}
# ── Perceptual hash backfill ────────────────────────────────────────────
#
# When phash was added post-launch, every existing photo has phash=NULL.
# This task fills them in by reading the existing thumb_large (the cheap
# option — pHash is robust to scale, and the thumb is already on local
# disk so we avoid re-decoding the original RAW/HEIC). Falls back to the
# original filepath if the thumb isn't available for some reason. Runs
# in batches to keep memory bounded and to give the user incremental
# progress visible in the worker logs.
@shared_task(name='backfill_phashes')
def backfill_phashes():
"""Compute and persist phash for every photo currently missing one."""
return asyncio.run(_backfill_phashes_async())
async def _backfill_phashes_async():
import imagehash
from PIL import Image as _PILImage
BATCH = 100
total_done = 0
total_failed = 0
async with AsyncSessionLocal() as session:
while True:
result = await session.execute(
select(Photo)
.where(Photo.phash.is_(None))
.where(Photo.processing_status == 'completed')
.limit(BATCH)
)
batch = result.scalars().all()
if not batch:
break
for photo in batch:
source = photo.thumb_large or photo.filepath
try:
if not source or not os.path.exists(source):
photo.phash = None
total_failed += 1
continue
with _PILImage.open(source) as im:
photo.phash = str(imagehash.phash(im))
total_done += 1
except Exception as e:
logger.warning(f"phash backfill failed for {photo.id}: {e}")
total_failed += 1
await session.commit()
logger.info(
f"Backfilled phashes: {total_done} done, {total_failed} failed"
)
return {
'status': 'success',
'computed': total_done,
'failed': total_failed,
}
@shared_task(name='regroup_duplicates')
def regroup_duplicates_task():
"""Celery wrapper around app.services.duplicates.regroup_duplicates.
Importing the service inside the task body avoids a circular import
at worker boot (the service uses AsyncSessionLocal which is also
imported here at module top)."""
from app.services.duplicates import regroup_duplicates
return asyncio.run(regroup_duplicates())