feat: perceptual-hash duplicate detection + grouped picker view
The Duplicates section was useless: SHA-256-only detection only caught
byte-identical files, not the actual duplicates a real library
accumulates (re-encoded JPEGs, screenshots, resized exports), and the
view was a flat date-sorted list with no grouping or actions. This
replaces the whole flow.
Detection
- New phash + duplicate_group_id columns on Photo, added via an
idempotent ALTER TABLE pass in init_db (the project has no Alembic).
- Thumbs worker computes a 64-bit pHash from the original-resolution
decoded frame just before the destructive thumbnail loop. Falls back
silently — phash is nice-to-have, not a blocker for thumbnails.
- backfill_phashes Celery task fills in phashes for photos that
predated the column, reading the existing thumb_large rather than
re-decoding the original.
- regroup_duplicates service runs union-find over Hamming distance
(threshold 6), persists duplicate_group_id, and maintains is_duplicate
as derived state so existing badges/counts keep working. Chained
after scan_all_source_roots with a 60s countdown.
API
- GET /library/duplicates/groups returns all groups with members,
bucketed in Python from one query. Each group has a reason ("exact"
iff every member shares a SHA-256, "similar" otherwise).
- POST /library/maintenance/{regroup-duplicates,backfill-phashes}.
Frontend
- New DuplicatesView (sectioned grid, one section per cluster) replaces
the timeline when the user is in the duplicates section. Each section
shows a "Keep best, discard N" button that picks the highest-pixel
copy and reuses the existing undoable bulk-discard so Cmd+Z works.
- Manual best override: hover any non-best thumbnail and click "Keep
this" (Crown icon, top-right) to override the auto-pick. The header
annotates "(manual)" so it's obvious which copy will be kept.
- Keyboard nav within the duplicates view walks the flat member list,
with ↑/↓ jumping by the measured column count and scrollIntoView on
every move. Timeline's keyboard handler now early-returns in the
duplicates section so the two don't fight.
- BEST pill / Keep-this button live at top-right with a ring outline so
they don't collide visually with the cyan selection ring around a
selected cell. Dimensions chip moved to bottom-left to free both
right corners for the keep affordances.
- New "Duplicates" section in SettingsDialog: shows group/member counts
and exposes both backfill + re-detect actions, sharing a query cache
with DuplicatesView via DUPLICATE_GROUPS_QUERY_KEY.
- PhotoInfoPanel "Basic Info" section now shows the photo's full file
path in monospace below the size/dimensions/date grid.
- New imagehash==4.3.1 dep in requirements.txt.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,6 +11,8 @@ import {
|
||||
Cpu,
|
||||
AlertCircle,
|
||||
CheckCircle2,
|
||||
Copy,
|
||||
Sparkles,
|
||||
} from 'lucide-react'
|
||||
import clsx from 'clsx'
|
||||
import { useQuery, useQueryClient } from '@tanstack/react-query'
|
||||
@@ -27,6 +29,9 @@ const SETTINGS_THUMB_STATS_KEY = ['settings', 'thumbnail-stats'] as const
|
||||
const SETTINGS_LIB_STATS_KEY = ['settings', 'library-stats'] as const
|
||||
const SETTINGS_WORKER_STATUS_KEY = ['settings', 'worker-status'] as const
|
||||
const SETTINGS_MISSING_STATS_KEY = ['settings', 'missing-stats'] as const
|
||||
// Shared with the DuplicatesView so a regroup invalidates the same cache
|
||||
// the grid renders from. Imported via the canonical hook key.
|
||||
import { DUPLICATE_GROUPS_QUERY_KEY } from '../../hooks/useDuplicateGroupsQuery'
|
||||
|
||||
interface SettingsDialogProps {
|
||||
isOpen: boolean
|
||||
@@ -86,6 +91,14 @@ export function SettingsDialog({ isOpen, onClose }: SettingsDialogProps) {
|
||||
refetchInterval: isOpen ? 5000 : false,
|
||||
staleTime: 0,
|
||||
})
|
||||
// Duplicates: shares the cache with DuplicatesView so a regroup
|
||||
// triggered from Settings updates the grid view immediately.
|
||||
const duplicatesQuery = useQuery({
|
||||
queryKey: DUPLICATE_GROUPS_QUERY_KEY,
|
||||
queryFn: library.duplicates.groups,
|
||||
enabled: isOpen,
|
||||
staleTime: 0,
|
||||
})
|
||||
|
||||
const thumbStats = thumbStatsQuery.data
|
||||
const libStats = libStatsQuery.data
|
||||
@@ -103,6 +116,7 @@ export function SettingsDialog({ isOpen, onClose }: SettingsDialogProps) {
|
||||
const refreshStats = useCallback(() => {
|
||||
queryClient.invalidateQueries({ queryKey: SETTINGS_THUMB_STATS_KEY })
|
||||
queryClient.invalidateQueries({ queryKey: SETTINGS_LIB_STATS_KEY })
|
||||
queryClient.invalidateQueries({ queryKey: DUPLICATE_GROUPS_QUERY_KEY })
|
||||
}, [queryClient])
|
||||
const refreshWorkers = useCallback(() => {
|
||||
queryClient.invalidateQueries({ queryKey: SETTINGS_WORKER_STATUS_KEY })
|
||||
@@ -249,6 +263,60 @@ export function SettingsDialog({ isOpen, onClose }: SettingsDialogProps) {
|
||||
</div>
|
||||
</Section>
|
||||
|
||||
{/* ----------------------------------------------------- */}
|
||||
{/* Duplicate detection */}
|
||||
{/* ----------------------------------------------------- */}
|
||||
<Section
|
||||
icon={<Copy className="h-4 w-4" />}
|
||||
title="Duplicates"
|
||||
>
|
||||
<div className="grid grid-cols-2 gap-2 text-xs">
|
||||
<Stat
|
||||
label="Groups"
|
||||
value={duplicatesQuery.data?.total_groups}
|
||||
/>
|
||||
<Stat
|
||||
label="Members"
|
||||
value={duplicatesQuery.data?.total_members}
|
||||
/>
|
||||
</div>
|
||||
<p className="mt-3 text-xs text-text-muted">
|
||||
Duplicates are detected by perceptual hash (pHash), which catches
|
||||
visually-identical photos even when their bytes differ — re-encoded
|
||||
JPEGs, screenshots, resized exports. Backfill computes hashes for
|
||||
photos that existed before pHash was added; Re-detect re-runs
|
||||
clustering across the whole library.
|
||||
</p>
|
||||
<div className="mt-2 flex flex-wrap gap-2">
|
||||
<ActionButton
|
||||
loading={busy['backfill-phashes']}
|
||||
onClick={() =>
|
||||
runAction(
|
||||
'backfill-phashes',
|
||||
() => library.maintenance.backfillPhashes(),
|
||||
'pHash backfill queued'
|
||||
)
|
||||
}
|
||||
>
|
||||
<Sparkles className="h-4 w-4" />
|
||||
Backfill perceptual hashes
|
||||
</ActionButton>
|
||||
<ActionButton
|
||||
loading={busy['regroup-duplicates']}
|
||||
onClick={() =>
|
||||
runAction(
|
||||
'regroup-duplicates',
|
||||
() => library.maintenance.regroupDuplicates(),
|
||||
'Duplicate detection queued'
|
||||
)
|
||||
}
|
||||
>
|
||||
<Copy className="h-4 w-4" />
|
||||
Re-detect duplicates
|
||||
</ActionButton>
|
||||
</div>
|
||||
</Section>
|
||||
|
||||
{/* ----------------------------------------------------- */}
|
||||
{/* Thumbnail maintenance */}
|
||||
{/* ----------------------------------------------------- */}
|
||||
|
||||
Reference in New Issue
Block a user