Replace O(N²) pHash-only duplicate detection with a hybrid approach:
- pHash Hamming distance for exact/near-exact copies
- CLIP embedding cosine similarity via pgvector HNSW for visually
similar photos (crops, format changes, screenshots)
Post-scan now uses incremental mode: only newly added photos are
compared against the full library — O(new × log N) via HNSW index
instead of O(N²). Full regroup remains available from Settings.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a per-folder "hide from views" toggle so noisy subtrees
(screenshots, WhatsApp dumps, work archives) can be excluded from
cross-cutting views without losing indexing. Photos under a hidden
folder are still scanned, thumbnailed, embedded, OCR'd, face-
extracted — they just stop appearing in All Photos, Rated, Colors,
Map, Tags, People, Search, Duplicates, and the sidebar counts.
Navigating directly into the folder still shows every photo.
Schema (migration 0007_folder_hidden):
- folders.is_hidden user-set toggle, default false
- photos.is_hidden denormalized effective flag (true iff any
ancestor folder is hidden), indexed so cross-
cutting queries stay on the existing planner
paths
The denorm is maintained by two paths:
- The scanner walks the ancestry chain on insert, with a per-scan
memoized cache so each folder is resolved once per scan.
- POST /api/v1/folders/{id}/hide flips folders.is_hidden and runs a
WITH RECURSIVE CTE to recompute every folder's effective state in
one query, then bulk-updates photos WHERE IS DISTINCT FROM. Runs
in ~10 ms on a 13k-photo library.
Filters added (cross-cutting queries):
- /library/stats — every sidebar badge via a shared `visible` filter
- /photos (list) — only when neither folder_id nor heap_id is set;
folder browse and heap browse always show everything
- /photos/map
- /library/duplicates/groups
- /folders/tree photo_count subquery
- /tags count_subq (drives Tags + People sidebar counts)
- services/duplicates.regroup_duplicates (so hidden dupes never
contaminate the Duplicates view)
- services/search.hybrid_search — both semantic (pgvector) and FTS
legs join photos so rankings don't include hidden results
Intentionally NOT filtered:
- /photos?folder_id=X and /photos?heap_id=X (user-intentional browse)
- /library/maintenance/pipeline-stats (tracks real worker state)
- cleanup service (disk-level ops, not views)
Frontend:
- sourceFolders.setHidden(id, hidden) API client method
- FolderTreeNode.is_hidden carried through the tree into TreeItem
- LeftSidebar kebab menu: "Hide from views" / "Show in views" with a
mutation that invalidates folders, photos, stats, and tags caches
- Hidden folder rows swap the Folder icon for EyeOff and render the
label italic/muted so the state is visible at a glance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Duplicates section was useless: SHA-256-only detection only caught
byte-identical files, not the actual duplicates a real library
accumulates (re-encoded JPEGs, screenshots, resized exports), and the
view was a flat date-sorted list with no grouping or actions. This
replaces the whole flow.
Detection
- New phash + duplicate_group_id columns on Photo, added via an
idempotent ALTER TABLE pass in init_db (the project has no Alembic).
- Thumbs worker computes a 64-bit pHash from the original-resolution
decoded frame just before the destructive thumbnail loop. Falls back
silently — phash is nice-to-have, not a blocker for thumbnails.
- backfill_phashes Celery task fills in phashes for photos that
predated the column, reading the existing thumb_large rather than
re-decoding the original.
- regroup_duplicates service runs union-find over Hamming distance
(threshold 6), persists duplicate_group_id, and maintains is_duplicate
as derived state so existing badges/counts keep working. Chained
after scan_all_source_roots with a 60s countdown.
API
- GET /library/duplicates/groups returns all groups with members,
bucketed in Python from one query. Each group has a reason ("exact"
iff every member shares a SHA-256, "similar" otherwise).
- POST /library/maintenance/{regroup-duplicates,backfill-phashes}.
Frontend
- New DuplicatesView (sectioned grid, one section per cluster) replaces
the timeline when the user is in the duplicates section. Each section
shows a "Keep best, discard N" button that picks the highest-pixel
copy and reuses the existing undoable bulk-discard so Cmd+Z works.
- Manual best override: hover any non-best thumbnail and click "Keep
this" (Crown icon, top-right) to override the auto-pick. The header
annotates "(manual)" so it's obvious which copy will be kept.
- Keyboard nav within the duplicates view walks the flat member list,
with ↑/↓ jumping by the measured column count and scrollIntoView on
every move. Timeline's keyboard handler now early-returns in the
duplicates section so the two don't fight.
- BEST pill / Keep-this button live at top-right with a ring outline so
they don't collide visually with the cyan selection ring around a
selected cell. Dimensions chip moved to bottom-left to free both
right corners for the keep affordances.
- New "Duplicates" section in SettingsDialog: shows group/member counts
and exposes both backfill + re-detect actions, sharing a query cache
with DuplicatesView via DUPLICATE_GROUPS_QUERY_KEY.
- PhotoInfoPanel "Basic Info" section now shows the photo's full file
path in monospace below the size/dimensions/date grid.
- New imagehash==4.3.1 dep in requirements.txt.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
