Pool was 20+10 overflow per engine; uvicorn --reload leaked pools until
Postgres hit max_connections=100. Reduced to 5+5 with pool_pre_ping.
Grouped views (tags, people, colors, rated) now sort cards largest-first.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Scan now automatically queues backfill_vision (+90s) and recluster_faces
(+300s) after dispatching folder scans. Face extraction also schedules a
debounced recluster via Redis so incremental file-watcher imports get
clustered without manual intervention.
The ScanProgress widget now tracks worker queue activity beyond the scan
phase, showing a "Processing Photos" indicator with vision queue counts
while background tasks (embeddings, faces, tags, OCR) are running.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Renders a thin top-of-cell banner on each photo while a text search
is active, labelling which metadata field matched (filename, title,
note, tag, or EXIF key name) and showing a short excerpt with the
exact matched substring highlighted in amber. Extends the list
endpoint's search to also match photos whose tag names contain the
query so tags show up alongside filename/title/notes/EXIF hits.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Tracks the scroll position and shows a small date chip pinned to the
right edge of the timeline, fading in while the user scrolls and out
700ms after they stop. Only active in date-sorted views — other sort
modes hide it since the label would be meaningless. A cached
row-offset/date index keeps the lookup to a single binary search per
scroll frame.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rolls the standalone "Date issues" toggle back into FlagFilter as a
third value ('any' | 'discarded' | 'date_warning') so the date-warning
control lives in the same popover as Discarded, where operators expect
all flag-style filters. Drops the redundant dateWarning boolean and
its URL param.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Lets operators fix corrupted capture dates at scale. Adds an editable
Date Taken field with a folder/filename-derived suggestion hint, a bulk
Date Taken section in the multi-select sidebar that either applies one
date to the whole selection or infers a per-photo date from each path,
a warning badge on thumbnails whose stored date disagrees with the
path, and a "Date issues" filter pill so suspicious photos can be
surfaced and fixed as a group. Edits are written back to EXIF on disk
so rescans don't clobber the fix.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a per-folder "hide from views" toggle so noisy subtrees
(screenshots, WhatsApp dumps, work archives) can be excluded from
cross-cutting views without losing indexing. Photos under a hidden
folder are still scanned, thumbnailed, embedded, OCR'd, face-
extracted — they just stop appearing in All Photos, Rated, Colors,
Map, Tags, People, Search, Duplicates, and the sidebar counts.
Navigating directly into the folder still shows every photo.
Schema (migration 0007_folder_hidden):
- folders.is_hidden user-set toggle, default false
- photos.is_hidden denormalized effective flag (true iff any
ancestor folder is hidden), indexed so cross-
cutting queries stay on the existing planner
paths
The denorm is maintained by two paths:
- The scanner walks the ancestry chain on insert, with a per-scan
memoized cache so each folder is resolved once per scan.
- POST /api/v1/folders/{id}/hide flips folders.is_hidden and runs a
WITH RECURSIVE CTE to recompute every folder's effective state in
one query, then bulk-updates photos WHERE IS DISTINCT FROM. Runs
in ~10 ms on a 13k-photo library.
Filters added (cross-cutting queries):
- /library/stats — every sidebar badge via a shared `visible` filter
- /photos (list) — only when neither folder_id nor heap_id is set;
folder browse and heap browse always show everything
- /photos/map
- /library/duplicates/groups
- /folders/tree photo_count subquery
- /tags count_subq (drives Tags + People sidebar counts)
- services/duplicates.regroup_duplicates (so hidden dupes never
contaminate the Duplicates view)
- services/search.hybrid_search — both semantic (pgvector) and FTS
legs join photos so rankings don't include hidden results
Intentionally NOT filtered:
- /photos?folder_id=X and /photos?heap_id=X (user-intentional browse)
- /library/maintenance/pipeline-stats (tracks real worker state)
- cleanup service (disk-level ops, not views)
Frontend:
- sourceFolders.setHidden(id, hidden) API client method
- FolderTreeNode.is_hidden carried through the tree into TreeItem
- LeftSidebar kebab menu: "Hide from views" / "Show in views" with a
mutation that invalidates folders, photos, stats, and tags caches
- Hidden folder rows swap the Folder icon for EyeOff and render the
label italic/muted so the state is visible at a glance
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three overlapping fixes so the ingestion pipeline actually runs and the
user can see what it's doing:
Pipeline recovery
- app/database.py: use NullPool when MULITA_CELERY_WORKER=1 so each
Celery task opens a fresh asyncpg connection on its own event loop.
Fixes "another operation in progress" and "Future attached to a
different loop" errors that were dropping ~every thumbnail +
extract_metadata task on the floor.
- app/tasks/thumbs.py: initialize photo=None before the try and rollback
on error so a transport failure in the initial SELECT doesn't raise
UnboundLocalError in the except block and leak rows stuck in 'pending'.
- app/services/vision/bootstrap_models.py: on missing model files,
invoke export_models automatically instead of just warning. First
boot of a fresh install now self-heals.
- app/services/vision/export_models.py: shutil.move instead of
Path.rename so the YOLO export survives the /app → /data/models
cross-volume hop.
- requirements.txt: add ultralytics so export works in a stock image.
Worker topology
- docker-compose.yml: replace the single worker with worker-light
(default/high/low queues, c=2, IO-bound) and worker-vision (vision
queue, c=5, OMP_NUM_THREADS=1 to avoid oversubscription on 6 cores).
Vision is pinned to ≤5 parallel inferences so ONNX doesn't each
spawn an all-cores intra-op pool.
- .env / .env.example: CELERYD_CONCURRENCY replaced with
CELERY_LIGHT_CONCURRENCY + CELERY_VISION_CONCURRENCY.
- Backfill queries in thumbs / scan / vision now ORDER BY taken_at
DESC NULLS LAST so newest photos finish first — the library fills
in top-down in the UI instead of arbitrary insertion order.
Settings visibility
- routers/library.py: new GET /maintenance/pipeline-stats returning
done/total per stage (thumbnails, exif, gps, phash, embeddings,
tags, ocr, faces, face clusters, duplicate groups). Worker-status
now also reports the `vision` queue depth, which was missing.
- services/api.ts: PipelineStats / PipelineStage / ScanStatus types
and the matching client call.
- components/dialogs/SettingsDialog.tsx:
- new Pipeline Progress card with one progress bar per stage
- inline scan banner (processed/total/current folder) inside the
Library section while a scan is running
- Tasks/min throughput computed by diffing worker processed counters
between polls
- Workers section calls out the vision queue and documents the
CELERY_LIGHT/VISION_CONCURRENCY + docker compose up -d scale path
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
PyTorch default install pulls ~7GB of CUDA libs, exceeding disk on small
VMs. Switching to CPU-only saves ~6GB. Also run create_all before alembic
so migrations find existing tables on a fresh Postgres.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace Info icon with braille mule art in the right sidebar empty
state. Lighten text-muted (#a8997d → #c4b599) and text-faint
(#5e5448 → #7a6e5e) for better readability across the app.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reflect current state: Postgres+pgvector replaces SQLite, vision
pipeline (YOLO, CLIP, InsightFace, OCR) is shipped, card-grid browse
views for tags/colors/ratings/people, map view, duplicate detection,
and semantic search are all live. Remove completed items from future
features.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace Timeline-based grouped views (tags, colors, rated) with
dedicated card-grid components that drill into Timeline detail views
on click/Enter. Adds shared useCardGridNav hook for arrow-key
navigation across all four card grids (tags, colors, rated, people).
- TagsView, ColorsView, RatedView: card grid → inline Timeline detail
- PeopleView: migrated to same pattern (Timeline replaces custom grid)
- Tags endpoint: fall back to first associated photo for representative
- Filter store: add ratingMax for exact rating filtering in RatedView
- Timeline: remove tag/rating/color grouping; skip date headers when
groupBy != 'date' so detail views render flat grids
- SettingsDialog: bump z-index above Leaflet map layers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cross-reference face detections with YOLO 'person' bounding boxes —
only keep faces that overlap >= 50% with a detected human body. This
eliminates false positives on dogs, paintings, and cartoons without
needing an aggressive score threshold.
Lower face detection threshold back to 0.6 since the person-overlap
check is now the primary precision filter.
Tested: 6 verified faces from 4 photos, zero false positives.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminates false positives (dog faces, painting faces) by requiring
score >= 0.85. Tighten cluster eps to 0.25 for better separation.
Tested: 4 real faces from 2 photos, no false positives.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix Photo.created_at → Photo.added_at (column doesn't exist)
- Fix Postgres DISTINCT + ORDER BY conflict by using a subquery for
tag_id filtering instead of JOIN + DISTINCT on the outer query
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Clicking a person card now opens a detail sub-view within the People
section showing their photo grid. Back arrow returns to the card grid.
Photos are clickable to open the preview. Rename is available in both
the card grid and the detail header.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- recluster_faces now writes photo_tags rows for each face cluster so
the tag count and tag_ids filter work (previously count was always 0)
- Old cluster tags and photo_tags are cleaned up before re-clustering
- Raise face detection threshold from 0.4 to 0.7 to reduce false
positives (was detecting dog faces as people)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rewrite faces.py to use cv2.FaceDetectorYN instead of raw ONNX
(handles multi-scale anchor decoding and NMS internally)
- Load original photo files at up to 4000px for face detection instead
of 240px thumbnails — faces were too small to detect at thumbnail res
- Falls back to thumbnail if original is unavailable
Tested: 33 faces extracted from 13 photos, clustered into 1 person.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace the tag-grouping people section with a dedicated PeopleView:
- Grid of face cluster cards showing representative photo thumbnail,
person name, and photo count
- Click a card → navigates to all-photos filtered by that person's tag
- Inline rename via pencil icon on hover
- Empty state when no faces have been clustered yet
Wired into App.tsx as a section-level route alongside Map and Duplicates.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rewrite all vision tasks to use sync psycopg2 sessions instead of
asyncpg — fixes 'another operation in progress' and event loop errors
when Celery forks workers sharing the async connection pool
- Letterbox-pad images to exactly 640x640 for YuNet face detector
(was crashing on non-square thumbnails)
- Deduplicate object detections per label per photo — keep highest
confidence only to avoid photo_tags PK violation on multiple
detections of the same class
- Add all queues (-Q default,high,low,vision) to worker command
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add export_models.py for OpenCLIP ViT-B/32 and YOLOv8n ONNX export
- Fix ArgMax(13) ORT ARM64 incompatibility by passing eot_indices as a
separate ONNX input (computed outside the graph in embed.py)
- Use legacy TorchScript exporter (dynamo=False) for IR version 9 compat
- Upgrade onnxruntime to 1.18.1
- Rewrite bootstrap_models.py with clear separation of auto-downloadable
models (YuNet, SFace) vs manually-exported ones (OpenCLIP, YOLOv8n)
- Wire bootstrap into worker CMD (runs before Celery)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add search API client (POST /photos/search) and useSearchQuery hook
for hybrid FTS + semantic search with RRF ranking
- Extend Tag type with kind, source, representative_photo_id fields
- Add tags.merge() API method
- Update useTagsQuery to accept optional kind filter
- Add People section to sidebar (face clusters from GET /tags?kind=face_cluster)
- Sidebar Tags count now shows user tags only; People shows face clusters
The existing GET /photos?q= flow is preserved for browsing; the new
search hook activates when the search box has a non-empty query.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create face_embeddings table with pgvector Vector(128) + HNSW index
- Implement extract_faces task (YuNet detection + SFace recognition)
- Implement recluster_faces task (DBSCAN clustering → Tag(kind=face_cluster))
- Clusters are named "Person N" and get representative_photo_id
- cluster_id FK → tags.id, SET NULL on delete for merge/rename support
Migration 0005 creates the face_embeddings table.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implement detect_objects Celery task:
- Runs YOLOv8n on 640px thumbnail via ONNX Runtime
- Creates Tag(kind=object) rows for each COCO class detected
- Writes photo_tags associations with confidence, bbox, and source
- Wipes previous detections per source model on re-run
No new tables/migrations — uses the unified Tag model from PR3.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Create ocr_text table for storing per-region OCR results
- Add tsvector search_vector column to photos with GIN index and
auto-update trigger on filename/user_title/user_notes
- Implement ocr_photo Celery task using rapidocr-onnxruntime
- Add FTS leg to hybrid search: queries photos.search_vector and
ocr_text via UNION, fused with semantic results via RRF (k=60)
Migration 0004 backfills search_vector for existing rows.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire the full embedding flow:
- Rewrite Embedding model to use pgvector Vector(512) with HNSW index
- Add embed_photo, vision_fanout, backfill_vision Celery tasks on
dedicated `vision` queue
- Hook vision_fanout into generate_thumbnails completion
- Add POST /api/v1/photos/search with hybrid RRF ranking (semantic-only
for now; FTS leg added in PR5)
- Stub ocr_photo, detect_objects, extract_faces tasks for later PRs
Migration 0003 drops/recreates the embeddings table (was never populated).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unify object detections, scene labels, and face clusters with user tags
via new columns on the existing Tag model:
- kind (user|object|scene|face_cluster), source, representative_photo_id
- photo_tags gains confidence, bbox (JSONB), source per-association
- Uniqueness moves from (name) to (name, kind) so ML labels coexist
with user tags without collision
Add Alembic migration 0002 with defensive IF NOT EXISTS guards.
Update tags router: kind filter on GET, merge endpoint for combining
auto-detected clusters/objects, include kind/source in list response.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch the default database from SQLite to Postgres + pgvector (via
pgvector/pgvector:pg16 Docker image) to support the upcoming vision
pipeline (embeddings, OCR, object detection, face clustering).
- Add `db` service to docker-compose.yml with healthcheck
- Wire `alembic upgrade head` into backend CMD before uvicorn
- Bootstrap empty 0001_baseline revision (schema still owned by create_all)
- Guard SQLite-only PRAGMAs and inline ALTERs behind _is_sqlite flag
- Run `CREATE EXTENSION IF NOT EXISTS vector` on Postgres init
- Add asyncpg, psycopg2-binary, pgvector to requirements
- Provide docker-compose.sqlite.yml escape hatch for legacy SQLite mode
Fresh DB + rescan assumed — no SQLite→Postgres data migration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Discard yanks photos from the grid optimistically (cache strip + active
cursor advance) instead of waiting for the mutation round-trip; wired
through the X hotkey, RightSidebar bulk discard, LeftSidebar discard
drop, and DiscardActionBar restore/delete.
- usePhotosQuery resolves on the first 500-photo page and streams the
remaining pages into the cache in the background, so the first
thumbnails paint immediately on large libraries.
- Closing preview restores the photo it was opened on (snapshot ref in
PreviewView, written directly to the store) and Timeline scrolls that
row back into view. Escape is handled on the dialog with
stopPropagation so Timeline's window-level Esc handler doesn't wipe
the restored selection.
- Preview overlay bumped to z-[1000] so it covers Leaflet map tiles,
and the right sidebar no longer collapses during preview — both fix
visible layout shifts on close.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a new Map sidebar entry that plots photos by their EXIF GPS
coordinates on a clustered Leaflet map. While wiring this up, the
metadata extractor was reading unprefixed GPS keys that never exist
in `exiftool -G -j` output AND assumed coordinates were already
floats — every photo silently lost its GPS. The new extract_gps
helper handles Composite/EXIF group prefixes and parses DMS strings,
and lat/lon are stored as first-class indexed columns so the map
can query them without parsing exif_json on every request.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a `colored` field to /library/stats counting non-discarded
photos with a color_label set, mirroring how `rated` is exposed.
The Colors sidebar entry now shows the live count and refreshes
through the existing LIBRARY_STATS_QUERY_KEY invalidations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a Colors entry under Views that buckets photos in canonical
R-O-Y-G-B-P order (plus an Uncolored tail), mirroring the rated/tags
grouping pattern. Surfaces the color label on each thumbnail as a
small swatch chip preceding the rating badge in the BL corner so
labels are visible everywhere, not just inside the new view. Also
types color_label on the Photo interface — the backend already
returned it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
LIBRARY_STATS_QUERY_KEY was never invalidated by the bulk/single
mutation paths in RightSidebar and PhotoInfoPanel, so the All Photos
/ Rated / Discarded / Tags badges only updated on reload. Add it to
both shared invalidation helpers and the inline updateMutation, and
correct the stale docstring on useLibraryStatsQuery.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mirrors the tags view: rated section now buckets photos 5★→1★ with
an "Unrated" tail group, instead of a flat ratingMin=1 stream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Switch tree and heap rows from py-0.5 to a fixed h-[24px] + leading-none
so the hover-only kebab buttons can't stretch the row vertically as the
pointer crosses them.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bump the ActiveHeapCard stack thumbnails to a layered drop shadow and
a darker ring so they lift cleanly off the softened desert backdrop.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace browser scrollbars with slim 6px overlay pills that tint to
primary on hover (transparent track, Firefox + WebKit).
- Rework the thumbnail badge family: drop the mismatched white ring
halo for a 1px dark frame + inset top highlight, and swap rounded-
full pills for rounded-sm chips so the ornaments match the pixel-art
aesthetic used elsewhere in the app.
- Soften the ActiveHeapCard desert backdrop so the fanned thumbnails
read on top.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Compact the left/right sidebars and the photo info panel: shrink panel
widths, drop header height, switch top-level tree groups and metadata
sections to small uppercase eyebrow labels, and tighten row padding,
icon sizes, and count badges throughout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the cool-grey neutral theme with a warm desert dusk palette
pulled from a new pixel-art TopBar: a tiled, right-to-left scrolling
desert under a sky gradient, a 6-frame walking mule sprite where the
logo used to sit, and an ASCII block-character title on a black plate.
The active heap card now uses a cactus/dune scene as its stack
backdrop, and the "Built with hubris" mark moves into the TopBar's
bottom-right corner (AppFooter component removed).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Every overlay on a thumbnail now composes the shared THUMB_BADGE_* classes
(one shape, one height, one ring, three semantic colour variants: primary
for user state, neutral for metadata, pick for auto-suggested best). RAW
badges, BEST pill, Keep-this button and the dimensions chip — previously
three different styles — join the family, is_duplicate moves to neutral
since it's file metadata not a user decision, and the selection check
shrinks to match the rest.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Arrow-key navigation required a click to establish an active photo
first. Now the grid auto-selects photos[0] on initial mount when no
active photo is set, so the user can land on the app and immediately
walk the grid with arrows. Guarded on viewMode === 'grid' and
!activePhotoId so we never clobber an existing selection or fight
with PreviewView.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The bottom-center hint bar used bg-surface/40 + text-text-muted, which
left busy photos leaking through and washed the text out — users
couldn't actually read the shortcut labels when a bright thumbnail sat
directly behind the pill. Switched to a near-opaque bg-black/80 with
white-alpha text + white/15 key caps. Backdrop blur stays for the
subtle glass feel, but the contrast is now unconditional on the
photo underneath.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
