oikos/archive/knowledge/containers/118-elementsynapse.md

118 — elementsynapse

Matrix homeserver (Synapse). Backs @dtoro:avispero.

At a glance

• Hostname: elementsynapse
• IP: 192.168.8.242
• Host: strong (migrated from hubris 2026-07-05)
• Privilege: unprivileged
• Resources: 1 core / 2 GiB RAM / 16 GiB rootfs (grown from 8 GiB on 2026-05-15 after disk-full incident)
• Mounts: none from /mnt/library
• Public hostname: matrix.hubris.network → caddy → :8008
• server_name: avispero (intentionally different from the public URL).
• Database: Synapse runs on SQLite (/var/lib/matrix-synapse/homeserver.db, ~90 MiB). PostgreSQL on 127.0.0.1:5432 is dedicated to the mautrix bridge databases (one per bridge), not Synapse itself.

Service / port map

Service	Listen	Notes
synapse	:8008	Matrix client-server / federation HTTP. Caddy terminates TLS for matrix.hubris.network.
synapse-admin	:5173	Web UI (vite dev server) for Synapse admin API.
mautrix-whatsapp	:29318	Docker, network_mode: host, compose in /root/mautrix-whatsapp/.
mautrix-meta	127.0.0.1:29319	Facebook/Instagram bridge, /root/mautrix-meta/.
mautrix-signal	127.0.0.1:29328	/root/mautrix-signal/.
mautrix-slack	127.0.0.1:29335	/root/mautrix-slack/.
mautrix-linkedin	127.0.0.1:29341	/root/mautrix-linkedin/.

Bridges (mautrix)

All five bridges run as plain docker compose stacks under /root/mautrix-<name>/ — no systemd unit, restart: unless-stopped. Their appservice registration YAMLs live in /etc/matrix-synapse/ and are listed in homeserver.yaml under app_service_config_files. If a bridge daemon is down, Synapse logs push_bulk … ConnectionRefusedError and retries every 512 s; restart the bridge with cd /root/mautrix-<name> && docker compose up -d.

Operational notes

• Rootfs is on pve LVM-thin, grow with pct resize 118 rootfs +<N>G from the host — resize2fs runs automatically.
• /var/lib/matrix-synapse/media_store is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.

Known issues

• Disk usage was 86.8% at last legacy monitor reading on 2026-04-21 — resolved by growing rootfs to 16 GiB on 2026-05-15.

Related

• claudio-bot (123) — decommissioned 2026-06-04, replaced by Hermes Agent
• Caddy
• DNS
• Monitoring

Changelog

2026-06-06 — DHCP drift fixed: internal /etc/network/interfaces was dhcp despite Proxmox static config

Symptom: Matrix was down. Caddy at 192.168.8.175 couldn't reach 192.168.8.239:8008 — the LXC was actually at 192.168.8.244 because the guest-side dhclient had overridden the PVE-assigned static IP.

Root cause: During the 2026-06-02 static-IP migration, pct set 118 --net0 ... ip=192.168.8.239/24 was applied to the Proxmox config, but the internal /etc/network/interfaces still had iface eth0 inet dhcp. On every DHCP lease renewal, dhclient grabbed .244 from Technitium's pool.

Fix:

• Replaced iface eth0 inet dhcp with iface eth0 inet static + address 192.168.8.239/24 + gateway 192.168.8.1
• ifdown eth0 && ifup eth0 applied the static IP
• Killed lingering dhclient process
• Verified: curl http://192.168.8.239:8008 returns 302 from Caddy's LXC

Prevention: The check-caddy-backends.sh cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.

2026-05-15 — phantom-notification cleanup for @admin

After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in event_push_actions that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:

1. 8 of 12 affected rooms had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.
2. The noisiest room (a Signal-bridged DM) was flooded with m.room.redaction events from @signalbot:avispero (disappearing messages being deleted). Synapse counts redactions as notifications by default, but Element X doesn't render them — so they were invisible "ghosts".

Fix (after stopping Synapse and backing up homeserver.db):

• DELETE FROM event_push_actions WHERE stream_ordering <= max(receipt_stream) per (user, room) — clears receipt-superseded rows.
• DELETE FROM event_push_actions WHERE event_id IN (SELECT event_id FROM events WHERE type='m.room.redaction') — removes redaction-driven notifications.
• Rebuild event_push_summary from the remaining rows (preserve thread_id='main' to satisfy the NOT-NULL check constraint).

Result: 125 → 4 (legitimate WhatsApp unreads).

Then, to stop this recurring, an override push-rule was installed for @admin via the client API:

PUT /_matrix/client/v3/pushrules/global/override/suppress_redactions
{ "actions": [], "conditions": [{ "kind": "event_match", "key": "type", "pattern": "m.room.redaction" }] }

With empty actions, all m.room.redaction events are silently ignored by the push system. Inspect/remove via the same endpoint. (Note: rule ids beginning with . are reserved for server defaults — custom ids cannot start with a dot.)

2026-05-15 — disk-full incident, rootfs grown to 16 GiB, bridges restarted

Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed OSError: [Errno 28] No space left on device during flush. Fix: pct resize 118 rootfs +8G (host pve VG had plenty of headroom), journalctl --vacuum-size=50M to recover ~200 M, systemctl restart matrix-synapse, then docker compose up -d in each /root/mautrix-* directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).

2026-04-28 — wiki entry created

Initial documentation.