feat: perceptual-hash duplicate detection + grouped picker view
The Duplicates section was useless: SHA-256-only detection only caught
byte-identical files, not the actual duplicates a real library
accumulates (re-encoded JPEGs, screenshots, resized exports), and the
view was a flat date-sorted list with no grouping or actions. This
replaces the whole flow.
Detection
- New phash + duplicate_group_id columns on Photo, added via an
idempotent ALTER TABLE pass in init_db (the project has no Alembic).
- Thumbs worker computes a 64-bit pHash from the original-resolution
decoded frame just before the destructive thumbnail loop. Falls back
silently — phash is nice-to-have, not a blocker for thumbnails.
- backfill_phashes Celery task fills in phashes for photos that
predated the column, reading the existing thumb_large rather than
re-decoding the original.
- regroup_duplicates service runs union-find over Hamming distance
(threshold 6), persists duplicate_group_id, and maintains is_duplicate
as derived state so existing badges/counts keep working. Chained
after scan_all_source_roots with a 60s countdown.
API
- GET /library/duplicates/groups returns all groups with members,
bucketed in Python from one query. Each group has a reason ("exact"
iff every member shares a SHA-256, "similar" otherwise).
- POST /library/maintenance/{regroup-duplicates,backfill-phashes}.
Frontend
- New DuplicatesView (sectioned grid, one section per cluster) replaces
the timeline when the user is in the duplicates section. Each section
shows a "Keep best, discard N" button that picks the highest-pixel
copy and reuses the existing undoable bulk-discard so Cmd+Z works.
- Manual best override: hover any non-best thumbnail and click "Keep
this" (Crown icon, top-right) to override the auto-pick. The header
annotates "(manual)" so it's obvious which copy will be kept.
- Keyboard nav within the duplicates view walks the flat member list,
with ↑/↓ jumping by the measured column count and scrollIntoView on
every move. Timeline's keyboard handler now early-returns in the
duplicates section so the two don't fight.
- BEST pill / Keep-this button live at top-right with a ring outline so
they don't collide visually with the cyan selection ring around a
selected cell. Dimensions chip moved to bottom-left to free both
right corners for the keep affordances.
- New "Duplicates" section in SettingsDialog: shows group/member counts
and exposes both backfill + re-detect actions, sharing a query cache
with DuplicatesView via DUPLICATE_GROUPS_QUERY_KEY.
- PhotoInfoPanel "Basic Info" section now shows the photo's full file
path in monospace below the size/dimensions/date grid.
- New imagehash==4.3.1 dep in requirements.txt.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -350,18 +350,39 @@ def scan_all_source_roots():
|
||||
async def _scan_all_source_roots_async():
|
||||
"""Read every active SourceRoot from the DB and queue a scan_folder task
|
||||
for each. Source roots whose path no longer exists on disk are skipped
|
||||
with a warning (the cleanup service surfaces those at startup too)."""
|
||||
with a warning (the cleanup service surfaces those at startup too).
|
||||
|
||||
After dispatching the scans, queue a delayed `regroup_duplicates`
|
||||
pass so duplicate clusters are recomputed once the new photos have
|
||||
finished thumbnailing (and therefore picked up phashes). The
|
||||
countdown is a best-effort hint — on a big library the user can
|
||||
still hit Settings → Re-detect duplicates to force a fresh pass.
|
||||
"""
|
||||
from app.tasks.thumbs import regroup_duplicates_task
|
||||
|
||||
async with AsyncSessionLocal() as session:
|
||||
result = await session.execute(
|
||||
select(SourceRoot).where(SourceRoot.is_active == True) # noqa: E712
|
||||
)
|
||||
source_roots = result.scalars().all()
|
||||
dispatched = 0
|
||||
for sr in source_roots:
|
||||
if os.path.exists(sr.path):
|
||||
scan_folder.delay(sr.path, sr.id)
|
||||
dispatched += 1
|
||||
else:
|
||||
logger.warning(f"Source root path does not exist: {sr.path}")
|
||||
|
||||
if dispatched > 0:
|
||||
# 60s gives the thumbs worker a window to compute phashes for
|
||||
# the new photos before regrouping. The task is idempotent, so
|
||||
# firing too early just means the next manual run picks up the
|
||||
# late arrivals — no corrupted state.
|
||||
try:
|
||||
regroup_duplicates_task.apply_async(countdown=60)
|
||||
except Exception as e:
|
||||
logger.warning(f"Could not queue post-scan regroup: {e}")
|
||||
|
||||
|
||||
@shared_task(name='watch_folders')
|
||||
def watch_folders():
|
||||
|
||||
Reference in New Issue
Block a user