dtoro/mule-image
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: perceptual-hash duplicate detection + grouped picker view

Browse Source 
The Duplicates section was useless: SHA-256-only detection only caught
byte-identical files, not the actual duplicates a real library
accumulates (re-encoded JPEGs, screenshots, resized exports), and the
view was a flat date-sorted list with no grouping or actions. This
replaces the whole flow.

Detection
- New phash + duplicate_group_id columns on Photo, added via an
  idempotent ALTER TABLE pass in init_db (the project has no Alembic).
- Thumbs worker computes a 64-bit pHash from the original-resolution
  decoded frame just before the destructive thumbnail loop. Falls back
  silently — phash is nice-to-have, not a blocker for thumbnails.
- backfill_phashes Celery task fills in phashes for photos that
  predated the column, reading the existing thumb_large rather than
  re-decoding the original.
- regroup_duplicates service runs union-find over Hamming distance
  (threshold 6), persists duplicate_group_id, and maintains is_duplicate
  as derived state so existing badges/counts keep working. Chained
  after scan_all_source_roots with a 60s countdown.

API
- GET /library/duplicates/groups returns all groups with members,
  bucketed in Python from one query. Each group has a reason ("exact"
  iff every member shares a SHA-256, "similar" otherwise).
- POST /library/maintenance/{regroup-duplicates,backfill-phashes}.

Frontend
- New DuplicatesView (sectioned grid, one section per cluster) replaces
  the timeline when the user is in the duplicates section. Each section
  shows a "Keep best, discard N" button that picks the highest-pixel
  copy and reuses the existing undoable bulk-discard so Cmd+Z works.
- Manual best override: hover any non-best thumbnail and click "Keep
  this" (Crown icon, top-right) to override the auto-pick. The header
  annotates "(manual)" so it's obvious which copy will be kept.
- Keyboard nav within the duplicates view walks the flat member list,
  with ↑/↓ jumping by the measured column count and scrollIntoView on
  every move. Timeline's keyboard handler now early-returns in the
  duplicates section so the two don't fight.
- BEST pill / Keep-this button live at top-right with a ring outline so
  they don't collide visually with the cyan selection ring around a
  selected cell. Dimensions chip moved to bottom-left to free both
  right corners for the keep affordances.
- New "Duplicates" section in SettingsDialog: shows group/member counts
  and exposes both backfill + re-detect actions, sharing a query cache
  with DuplicatesView via DUPLICATE_GROUPS_QUERY_KEY.
- PhotoInfoPanel "Basic Info" section now shows the photo's full file
  path in monospace below the size/dimensions/date grid.
- New imagehash==4.3.1 dep in requirements.txt.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-04-09 17:19:08 +02:00
parent e51b93d59e
commit 733c16bf82
14 changed files with 1098 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
backend/app/database.py
View File

@@ -58,17 +58,57 @@ async def init_db():
async with engine.begin() as conn:
# Import all models to register them with Base
from app.models import Photo, Folder, SourceRoot, Tag, PhotoTag, Heap, HeapPhoto, Embedding
# Create all tables
# Create all tables. Note: create_all only creates *missing* tables —
# it does NOT add new columns to existing tables when the model gains
# them. Anything new on an existing table needs an explicit ALTER
# below.
await conn.run_sync(Base.metadata.create_all)
# Enable WAL mode for SQLite (better concurrency)
if "sqlite" in settings.database_url:
await conn.execute(text("PRAGMA journal_mode=WAL"))
await conn.execute(text("PRAGMA synchronous=NORMAL"))
await conn.execute(text("PRAGMA cache_size=10000"))
await conn.execute(text("PRAGMA temp_store=MEMORY"))
# ── Idempotent column adds ────────────────────────────────────────
# The project does not use Alembic; we lean on create_all + a small
# set of inline ALTER TABLE statements for the columns we've added
# post-launch. SQLite supports ADD COLUMN but not "IF NOT EXISTS"
# for columns, so introspect via PRAGMA first. Each entry is
# (column_name, ALTER statement). Add new columns at the bottom.
if "sqlite" in settings.database_url:
existing_cols = {
row[1]
for row in (
await conn.execute(text("PRAGMA table_info(photos)"))
).fetchall()
}
pending_alters: list[tuple[str, str]] = [
("phash", "ALTER TABLE photos ADD COLUMN phash VARCHAR(16)"),
(
"duplicate_group_id",
"ALTER TABLE photos ADD COLUMN duplicate_group_id VARCHAR",
),
]
for col_name, alter_sql in pending_alters:
if col_name not in existing_cols:
logger.info(f"Adding photos.{col_name} column")
await conn.execute(text(alter_sql))
# Indexes for the new duplicate-detection columns. CREATE INDEX
# IF NOT EXISTS is supported on SQLite so this is safe to run
# every startup.
await conn.execute(
text("CREATE INDEX IF NOT EXISTS ix_photos_phash ON photos(phash)")
)
await conn.execute(
text(
"CREATE INDEX IF NOT EXISTS ix_photos_duplicate_group_id "
"ON photos(duplicate_group_id)"
)
)
logger.info("Database initialized successfully")
async def create_fts_table():

18
backend/app/models/photos.py
View File

@@ -60,8 +60,24 @@ class Photo(Base):
# photo just means adding it to the active heap. Both DB columns may still
# exist on legacy installs but are no longer read or written.
# Duplicate detection
# Duplicate detection.
#
# - file_hash (above): SHA-256 of the raw bytes. Catches byte-identical
# copies but not visually-identical re-encodes / resizes / screenshots.
# - phash: 16-char hex of a 64-bit perceptual hash, computed by the
# thumbs worker from the decoded original frame. Robust to resize and
# re-compression — this is what actually identifies "the same photo
# saved twice with different JPEG quality".
# - duplicate_group_id: shared by every photo in the same duplicate
# cluster. Maintained by app.services.duplicates.regroup_duplicates,
# not on individual writes — recomputed in batches after scans / on
# demand from the Settings panel.
# - is_duplicate: derived boolean (group_id IS NOT NULL). Kept as a real
# column so the existing PhotoThumbnail badge and /library/stats
# duplicates count don't have to change.
is_duplicate = Column(Boolean, default=False)
phash = Column(String(16), index=True)
duplicate_group_id = Column(String, index=True)
# Live photo support
live_photo_video_id = Column(String, ForeignKey('photos.id'))

124
backend/app/routers/library.py
View File

@@ -447,4 +447,128 @@ async def run_data_integrity_cleanup():
return {"status": "success"}
except Exception as e:
logger.error(f"Manual cleanup failed: {e}")
return {"status": "error", "message": str(e)}
# ─────────────────────────────────────────────────────────────────────────
# Duplicate detection
# ─────────────────────────────────────────────────────────────────────────
@router.get("/duplicates/groups")
async def get_duplicate_groups(db: AsyncSession = Depends(get_db)):
"""Return every duplicate group with its members.
Drives the frontend grouped grid view in the Duplicates section. One
SQL query, bucketed in Python — no N+1, no per-member fetch. Groups
are sorted by member_count DESC then earliest taken_at DESC so the
biggest / most recent clusters bubble to the top.
Each group also carries a `reason` field:
* "exact" — every member shares the same SHA-256 (true byte
duplicates that the perceptual hash trivially caught)
* "similar" — members differ at the byte level but match perceptually
"""
rows = (
await db.execute(
select(
Photo.id,
Photo.filename,
Photo.taken_at,
Photo.file_size,
Photo.width,
Photo.height,
Photo.thumb_small,
Photo.file_hash,
Photo.folder_id,
Photo.media_type,
Photo.duplicate_group_id,
)
.where(Photo.duplicate_group_id.is_not(None))
.where(Photo.is_discarded.is_(False))
.order_by(Photo.duplicate_group_id)
)
).all()
# Bucket members by group_id.
groups: dict[str, list[dict]] = {}
for row in rows:
member = {
"id": row[0],
"filename": row[1],
"taken_at": row[2].isoformat() if row[2] else None,
"file_size": row[3],
"width": row[4],
"height": row[5],
"thumb_small": row[6],
"file_hash": row[7],
"folder_id": row[8],
"media_type": row[9],
}
groups.setdefault(row[10], []).append(member)
def earliest(g: list[dict]) -> str:
# Used as a secondary sort key. Photos with no taken_at sort last
# by returning a far-future sentinel.
taken = [m["taken_at"] for m in g if m["taken_at"]]
return min(taken) if taken else "9999"
out = []
for group_id, members in groups.items():
if len(members) < 2:
# Defensive: a regroup race could leave a singleton briefly.
# Skip it so the UI never shows a "group of 1".
continue
# exact iff every member shares the same non-null file_hash
# (true byte-identical copies that pHash also caught). Anything
# else — different hashes, missing hashes — counts as "similar".
all_hashes = [m["file_hash"] for m in members]
reason = (
"exact"
if len(set(all_hashes)) == 1 and all_hashes[0] is not None
else "similar"
)
out.append({
"group_id": group_id,
"member_count": len(members),
"reason": reason,
"members": members,
})
out.sort(key=lambda g: (-g["member_count"], earliest(g["members"])))
return {
"groups": out,
"total_groups": len(out),
"total_members": sum(g["member_count"] for g in out),
}
@router.post("/maintenance/regroup-duplicates")
async def trigger_regroup_duplicates():
"""Recompute duplicate groups from current perceptual hashes.
Fires the celery `regroup_duplicates` task which walks every photo's
phash, clusters by Hamming distance, and rewrites duplicate_group_id /
is_duplicate columns. Idempotent."""
from app.tasks.thumbs import regroup_duplicates_task
try:
regroup_duplicates_task.delay()
return {"status": "queued"}
except Exception as e:
logger.error(f"Regroup queue failed: {e}")
return {"status": "error", "message": str(e)}
@router.post("/maintenance/backfill-phashes")
async def trigger_backfill_phashes():
"""Compute perceptual hashes for every photo currently missing one.
One-shot recovery path for libraries that existed before the phash
column was added — the thumbs worker computes phash for everything
new, but old rows need a backfill pass."""
from app.tasks.thumbs import backfill_phashes
try:
backfill_phashes.delay()
return {"status": "queued"}
except Exception as e:
logger.error(f"Backfill queue failed: {e}")
return {"status": "error", "message": str(e)}

227
backend/app/services/duplicates.py Normal file
View File

@@ -0,0 +1,227 @@
"""
Duplicate detection: group photos by perceptual-hash similarity.
Strategy
--------
Each photo carries a 16-char hex perceptual hash (`Photo.phash`) computed
by the thumbnail worker from the original-resolution decoded frame
(`app.tasks.thumbs._generate_thumbnails_async`). pHash is robust to
resize / re-encoding / mild edits, so two photos that are visually the
"same shot" land at small Hamming distance even when their bytes are
completely different.
This module turns those per-photo hashes into explicit *groups*. The
result is persisted in two columns:
* `Photo.duplicate_group_id` — shared by every member of a group
* `Photo.is_duplicate` — derived: True iff group_id IS NOT NULL
(kept as a column so the existing
PhotoThumbnail badge and /library/stats
count don't have to change).
The grouping is recomputed in batches by `regroup_duplicates`, NOT on
individual writes — that lets us use a single in-memory pass instead of
maintaining a per-row similarity index. Triggered automatically after
each scan and on demand from the Settings panel.
Complexity
----------
Pairwise O(N²) over photos with a non-null phash. At ~5 µs per Hamming
distance in CPython this is roughly:
1k photos → ~5 s
5k photos → ~125 s
10k photos → ~500 s
That's the wrong shape for libraries past ~5k. The drop-in replacement
is a BK-tree (e.g. `pybktree`) which gives O(log N) lookups for a fixed
Hamming threshold; swap it in here when someone trips the limit. The
public function signature stays the same.
Out of scope (deferred)
-----------------------
* Dismissing a group / "intentional duplicates" — would need a per-group
or per-pair flag plus a skip-set in this function so re-grouping
doesn't bring them back. Add when there's a real user need.
* Incremental updates on individual photo writes — currently we just
re-run the whole job after each scan, which is fine while the cost is
bounded.
"""
from __future__ import annotations
import logging
import uuid
from typing import Optional
from sqlalchemy import select, update
from app.database import AsyncSessionLocal
from app.models.photos import Photo
logger = logging.getLogger(__name__)
# Hamming distance threshold under which two phashes are considered
# "the same image". 6 bits out of 64 is the rule-of-thumb sweet spot for
# pHash — tight enough to avoid false positives between unrelated photos,
# loose enough to catch JPEG re-encodes, slight crops, and a screenshot
# of a screenshot.
DEFAULT_THRESHOLD = 6
def _hex_to_int(h: str) -> int:
"""Parse a 16-char hex pHash to a Python int. Returns -1 on bad input
so the pairwise loop can skip the row without raising."""
try:
return int(h, 16)
except (TypeError, ValueError):
return -1
def _hamming(a: int, b: int) -> int:
"""Population count of XOR — the canonical hash distance metric.
`int.bit_count()` is C-implemented in CPython 3.10+ and is by far
the fastest path; the `bin(...).count('1')` fallback is here only
so the function still works on older interpreters."""
x = a ^ b
try:
return x.bit_count() # type: ignore[attr-defined]
except AttributeError:
return bin(x).count('1')
class _UnionFind:
"""Tiny union-find / disjoint-set used to merge similar phashes into
connected components. Inlined here (rather than pulled from a dep)
because it's ~15 lines and we don't need anything fancy."""
def __init__(self, n: int) -> None:
self.parent = list(range(n))
self.rank = [0] * n
def find(self, x: int) -> int:
# Path compression — flattens the tree on lookup so subsequent
# finds are amortized O(α(N)) ≈ O(1).
while self.parent[x] != x:
self.parent[x] = self.parent[self.parent[x]]
x = self.parent[x]
return x
def union(self, a: int, b: int) -> None:
ra, rb = self.find(a), self.find(b)
if ra == rb:
return
if self.rank[ra] < self.rank[rb]:
ra, rb = rb, ra
self.parent[rb] = ra
if self.rank[ra] == self.rank[rb]:
self.rank[ra] += 1
async def regroup_duplicates(threshold: int = DEFAULT_THRESHOLD) -> dict:
"""Recompute every photo's duplicate_group_id from current phashes.
Idempotent — safe to call as often as you like. Returns a small
summary dict the maintenance endpoint surfaces back to the UI.
Photos that end up alone in a component (size 1) get
`duplicate_group_id=NULL` and `is_duplicate=False`. This is what
cleans up "dead" groups after the user discards N-1 members from
one.
"""
async with AsyncSessionLocal() as session:
# Pull (id, phash) for every non-discarded photo with a hash.
# Discarded photos are excluded so we don't keep showing groups
# made up of trashed copies.
rows = (
await session.execute(
select(Photo.id, Photo.phash)
.where(Photo.phash.is_not(None))
.where(Photo.is_discarded.is_(False))
)
).all()
n = len(rows)
if n == 0:
# Still need to clear stale group_ids in case the user just
# discarded the last surviving member of every group.
await _clear_all_groups(session)
await session.commit()
return {
'photos_considered': 0,
'groups': 0,
'members': 0,
}
ids: list[str] = [row[0] for row in rows]
hashes: list[int] = [_hex_to_int(row[1]) for row in rows]
uf = _UnionFind(n)
# O(N²) pairwise comparison. See module docstring for the
# scaling analysis and the BK-tree upgrade path.
for i in range(n):
hi = hashes[i]
if hi < 0:
continue
for j in range(i + 1, n):
hj = hashes[j]
if hj < 0:
continue
if _hamming(hi, hj) <= threshold:
uf.union(i, j)
# Collect components. Each connected component of size >= 2 gets
# a fresh group id; size-1 components are intentionally dropped.
components: dict[int, list[int]] = {}
for i in range(n):
root = uf.find(i)
components.setdefault(root, []).append(i)
# First pass: clear EVERY photo's group_id so survivors of an
# earlier grouping that no longer match anyone end up clean. This
# is one bulk UPDATE rather than per-photo to keep the cost low
# even on big libraries.
await _clear_all_groups(session)
# Second pass: write the new group ids for components of size 2+.
groups_created = 0
members_total = 0
for members in components.values():
if len(members) < 2:
continue
group_id = str(uuid.uuid4())
member_ids = [ids[i] for i in members]
await session.execute(
update(Photo)
.where(Photo.id.in_(member_ids))
.values(
duplicate_group_id=group_id,
is_duplicate=True,
)
)
groups_created += 1
members_total += len(member_ids)
await session.commit()
logger.info(
f"regroup_duplicates: considered {n} photos, "
f"created {groups_created} group(s) covering {members_total} member(s)"
)
return {
'photos_considered': n,
'groups': groups_created,
'members': members_total,
}
async def _clear_all_groups(session) -> None:
"""Reset duplicate_group_id / is_duplicate on every photo. Used as
the first half of a regroup pass so photos that no longer cluster
with anyone end up clean instead of carrying a stale group id."""
await session.execute(
update(Photo).values(duplicate_group_id=None, is_duplicate=False)
)

23
backend/app/tasks/scan.py
View File

@@ -350,18 +350,39 @@ def scan_all_source_roots():
async def _scan_all_source_roots_async():
"""Read every active SourceRoot from the DB and queue a scan_folder task
for each. Source roots whose path no longer exists on disk are skipped
with a warning (the cleanup service surfaces those at startup too)."""
with a warning (the cleanup service surfaces those at startup too).
After dispatching the scans, queue a delayed `regroup_duplicates`
pass so duplicate clusters are recomputed once the new photos have
finished thumbnailing (and therefore picked up phashes). The
countdown is a best-effort hint — on a big library the user can
still hit Settings → Re-detect duplicates to force a fresh pass.
"""
from app.tasks.thumbs import regroup_duplicates_task
async with AsyncSessionLocal() as session:
result = await session.execute(
select(SourceRoot).where(SourceRoot.is_active == True) # noqa: E712
)
source_roots = result.scalars().all()
dispatched = 0
for sr in source_roots:
if os.path.exists(sr.path):
scan_folder.delay(sr.path, sr.id)
dispatched += 1
else:
logger.warning(f"Source root path does not exist: {sr.path}")
if dispatched > 0:
# 60s gives the thumbs worker a window to compute phashes for
# the new photos before regrouping. The task is idempotent, so
# firing too early just means the next manual run picks up the
# late arrivals — no corrupted state.
try:
regroup_duplicates_task.apply_async(countdown=60)
except Exception as e:
logger.warning(f"Could not queue post-scan regroup: {e}")
@shared_task(name='watch_folders')
def watch_folders():

97
backend/app/tasks/thumbs.py
View File

@@ -280,11 +280,23 @@ async def _generate_thumbnails_async(photo_id: str, task):
# Auto-rotate based on EXIF
image = auto_rotate_image(image)
# Store original dimensions
photo.width = image.width
photo.height = image.height
# Perceptual hash from the original-resolution decoded frame.
# pHash is robust to resize/recompression but the thumbnail
# loop below mutates `image` in place, so this MUST run before
# the loop sees it. Failures are non-fatal — phash is a
# nice-to-have, not a blocker for thumbnail generation.
try:
import imagehash
photo.phash = str(imagehash.phash(image)) # 16-char hex
except Exception as e:
logger.warning(f"phash failed for {photo_id}: {e}")
photo.phash = None
# Generate thumbnails for each size
for size_name, size_value in THUMB_SIZES.items():
thumb_path = get_thumb_path(photo_id, size_name)
@@ -333,10 +345,83 @@ async def _regenerate_all_thumbnails_async():
)
)
photos = result.scalars().all()
logger.info(f"Regenerating thumbnails for {len(photos)} photos")
for photo in photos:
generate_thumbnails.delay(photo.id)
return {'status': 'queued', 'count': len(photos)}
return {'status': 'queued', 'count': len(photos)}
# ── Perceptual hash backfill ────────────────────────────────────────────
#
# When phash was added post-launch, every existing photo has phash=NULL.
# This task fills them in by reading the existing thumb_large (the cheap
# option — pHash is robust to scale, and the thumb is already on local
# disk so we avoid re-decoding the original RAW/HEIC). Falls back to the
# original filepath if the thumb isn't available for some reason. Runs
# in batches to keep memory bounded and to give the user incremental
# progress visible in the worker logs.
@shared_task(name='backfill_phashes')
def backfill_phashes():
"""Compute and persist phash for every photo currently missing one."""
return asyncio.run(_backfill_phashes_async())
async def _backfill_phashes_async():
import imagehash
from PIL import Image as _PILImage
BATCH = 100
total_done = 0
total_failed = 0
async with AsyncSessionLocal() as session:
while True:
result = await session.execute(
select(Photo)
.where(Photo.phash.is_(None))
.where(Photo.processing_status == 'completed')
.limit(BATCH)
)
batch = result.scalars().all()
if not batch:
break
for photo in batch:
source = photo.thumb_large or photo.filepath
try:
if not source or not os.path.exists(source):
photo.phash = None
total_failed += 1
continue
with _PILImage.open(source) as im:
photo.phash = str(imagehash.phash(im))
total_done += 1
except Exception as e:
logger.warning(f"phash backfill failed for {photo.id}: {e}")
total_failed += 1
await session.commit()
logger.info(
f"Backfilled phashes: {total_done} done, {total_failed} failed"
)
return {
'status': 'success',
'computed': total_done,
'failed': total_failed,
}
@shared_task(name='regroup_duplicates')
def regroup_duplicates_task():
"""Celery wrapper around app.services.duplicates.regroup_duplicates.
Importing the service inside the task body avoids a circular import
at worker boot (the service uses AsyncSessionLocal which is also
imported here at module top)."""
from app.services.duplicates import regroup_duplicates
return asyncio.run(regroup_duplicates())

1
backend/requirements.txt
View File

@@ -18,6 +18,7 @@ flower==2.0.1
# rawpy==0.19.0 # Optional - numpy compatibility issues, using Pillow as fallback
pillow==10.2.0
pillow-heif==0.15.0
imagehash==4.3.1 # perceptual hash for duplicate detection
imageio==2.33.1
imageio-ffmpeg==0.4.9