dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

118-elementsynapse: phantom-notification cleanup + DB fact-fix

Browse Source 
Synapse runs on SQLite (not Postgres — Postgres only hosts the
mautrix bridge dbs). Documented the post-disk-full
event_push_actions cleanup that took @admin'\''s phantom count
from 125 to 4.
This commit is contained in:
Claudio
2026-05-15 21:56:35 +02:00
parent c9eb7fdab3
commit 565453773c
1 changed files with 13 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
containers/118-elementsynapse.md
View File

@@ -10,7 +10,7 @@ Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
- **Mounts:** none from `/mnt/library`
- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008`
- **`server_name`:** `avispero` (intentionally different from the public URL).
- **Database:** PostgreSQL on `127.0.0.1:5432` (the `homeserver.db` SQLite file in `/var/lib/matrix-synapse/` is a leftover, not in use).
- **Database:** Synapse runs on **SQLite** (`/var/lib/matrix-synapse/homeserver.db`, ~90 MiB). PostgreSQL on `127.0.0.1:5432` is dedicated to the mautrix bridge databases (one per bridge), not Synapse itself.
## Service / port map
@@ -43,6 +43,18 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
## Changelog
### 2026-05-15 — phantom-notification cleanup for `@admin`
After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.
2. **The noisiest room** (a Signal-bridged DM) was flooded with `m.room.redaction` events from `@signalbot:avispero` (disappearing messages being deleted). Synapse counts redactions as notifications by default, but Element X doesn't render them — so they were invisible "ghosts".
Fix (after stopping Synapse and backing up `homeserver.db`):
- `DELETE FROM event_push_actions WHERE stream_ordering <= max(receipt_stream)` per (user, room) — clears receipt-superseded rows.
- `DELETE FROM event_push_actions WHERE event_id IN (SELECT event_id FROM events WHERE type='m.room.redaction')` — removes redaction-driven notifications.
- Rebuild `event_push_summary` from the remaining rows (preserve `thread_id='main'` to satisfy the NOT-NULL check constraint).
Result: 125 → 4 (legitimate WhatsApp unreads). If this recurs, the durable fix is a push-rule to suppress `m.room.redaction` notifications (account data, per-user) or upstream a custom rule in `homeserver.yaml`.
### 2026-05-15 — disk-full incident, rootfs grown to 16 GiB, bridges restarted
Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed `OSError: [Errno 28] No space left on device` during flush. Fix: `pct resize 118 rootfs +8G` (host `pve` VG had plenty of headroom), `journalctl --vacuum-size=50M` to recover ~200 M, `systemctl restart matrix-synapse`, then `docker compose up -d` in each `/root/mautrix-*` directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).