feat(nextcloud): hard-delete SourceRoot + reliable delete sync
Two related fixes for the Nextcloud library lifecycle.
1. DELETE /api/v1/nextcloud/source-roots/{id} now actually deletes
the SourceRoot, every Folder under it, and every Photo in those
folders (Nextcloud files untouched). Was a soft-deactivate
(is_active=false) that left the rows around forever, so re-adding
the same path resurrected ghosts and prune-missing reported zero.
Returns {deleted_photos, deleted_folders}; the Settings UI toasts
the count and invalidates photos/folders/stats so cached lists
don't show ghosts. photo_tags and heap_photos already cascade via
ON DELETE CASCADE; FolderShare uses a stringly-typed folder_id
with no FK so cleaned up explicitly.
2. The watcher (watch_folders task) was getting killed every five
minutes by the global task_soft_time_limit=300 in app/tasks/celery.py
despite passing soft_time_limit=None on the decorator (None falls
back to the worker default in this Celery version). Override with
soft_time_limit=0, time_limit=0 (= unlimited) so the watch loop
actually stays alive. The 'Soft time limit (300s) exceeded' /
'Worker exited prematurely' lines should stop in worker-watcher
logs.
3. Added discard_missing_photos() in services/cleanup.py — a soft
variant of prune_missing_photos that walks every present source
root, checks os.path.exists for each non-discarded Photo, and
flips is_discarded=true on the missing ones (UPDATE not DELETE).
Wired as discard_missing_photos_beat in tasks/scan.py and
scheduled every 30 min via celery beat. Beat runs in-process on
worker-watcher (--beat flag in compose) — there's only ever one
watcher and we don't need a separate container.
Hard delete remains manual via prune-missing for users who want to
review before committing. The beat catch-up only soft-discards (file
gone -> mule-image trash, restorable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -299,6 +299,71 @@ async def prune_missing_photos(dry_run: bool = True) -> dict:
|
||||
raise
|
||||
|
||||
|
||||
async def discard_missing_photos() -> dict:
|
||||
"""Soft variant of prune_missing_photos for the periodic beat
|
||||
catch-up. Walks every active source root that is currently
|
||||
`present` (not 'renamed' — the user-driven manual flow handles
|
||||
those — and not 'unmounted'), and for each Photo whose file is
|
||||
gone from disk, sets is_discarded=True so it shows up in the
|
||||
in-app trash. Idempotent: skips photos that are already
|
||||
discarded.
|
||||
|
||||
Hard-deletion stays manual via prune_missing_photos so users
|
||||
can review the list before committing.
|
||||
"""
|
||||
async with AsyncSessionLocal() as session:
|
||||
try:
|
||||
sr_rows = (await session.execute(select(SourceRoot))).scalars().all()
|
||||
present_sr_ids = {
|
||||
sr.id for sr in sr_rows if _sr_state(sr.path) == 'present'
|
||||
}
|
||||
if not present_sr_ids:
|
||||
return {"discarded": 0, "checked": 0}
|
||||
|
||||
folder_to_sr = {
|
||||
fid: srid
|
||||
for fid, _path, srid in (await session.execute(
|
||||
select(Folder.id, Folder.path, Folder.source_root_id)
|
||||
)).all()
|
||||
}
|
||||
|
||||
photo_rows = (await session.execute(
|
||||
select(Photo.id, Photo.filepath, Photo.folder_id)
|
||||
.where(Photo.is_discarded.is_(False))
|
||||
)).all()
|
||||
|
||||
missing_ids: list[str] = []
|
||||
checked = 0
|
||||
for pid, fp, folder_id in photo_rows:
|
||||
sr_id = folder_to_sr.get(folder_id)
|
||||
if sr_id not in present_sr_ids:
|
||||
continue
|
||||
checked += 1
|
||||
if not os.path.exists(fp):
|
||||
missing_ids.append(pid)
|
||||
|
||||
if missing_ids:
|
||||
CHUNK = 500
|
||||
now = datetime.utcnow()
|
||||
for i in range(0, len(missing_ids), CHUNK):
|
||||
await session.execute(
|
||||
update(Photo)
|
||||
.where(Photo.id.in_(missing_ids[i:i + CHUNK]))
|
||||
.values(is_discarded=True, discarded_at=now)
|
||||
)
|
||||
await session.commit()
|
||||
logger.info(
|
||||
f"discard_missing_photos: discarded {len(missing_ids)} "
|
||||
f"of {checked} photos under {len(present_sr_ids)} present source roots"
|
||||
)
|
||||
|
||||
return {"discarded": len(missing_ids), "checked": checked}
|
||||
except Exception as e:
|
||||
logger.error(f"discard_missing_photos failed: {e}")
|
||||
await session.rollback()
|
||||
raise
|
||||
|
||||
|
||||
async def cleanup_data_integrity() -> dict:
|
||||
"""Top-level entry point. Runs the dedupe + count refresh in a single
|
||||
transaction. Returns a small summary dict for logging."""
|
||||
|
||||
Reference in New Issue
Block a user