118-elementsynapse: document disk-full fix + bridges
Rootfs grew to 16 GiB on 2026-05-15. Added bridges section (mautrix slack/signal/meta/linkedin/whatsapp), synapse-admin port, Postgres backend note, and operational tips for future disk growth.
This commit is contained in:
@@ -6,19 +6,34 @@ Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
|
||||
- **Hostname:** `elementsynapse`
|
||||
- **IP:** `192.168.8.239`
|
||||
- **Privilege:** **unprivileged**
|
||||
- **Resources:** 1 core / 2 GiB RAM / 8 GiB rootfs
|
||||
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
|
||||
- **Mounts:** none from `/mnt/library`
|
||||
- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008`
|
||||
- **`server_name`:** `avispero` (intentionally different from the public URL).
|
||||
- **Database:** PostgreSQL on `127.0.0.1:5432` (the `homeserver.db` SQLite file in `/var/lib/matrix-synapse/` is a leftover, not in use).
|
||||
|
||||
## Service / port map
|
||||
|
||||
| Service | Listen | Notes |
|
||||
| -------------- | ------- | -------------------------------- |
|
||||
| synapse | `:8008` | Matrix client-server / federation HTTP. Caddy terminates TLS for `matrix.hubris.network`. |
|
||||
| synapse-admin | `:5173` | Web UI (vite dev server) for Synapse admin API. |
|
||||
| mautrix-whatsapp | `:29318` | Docker, `network_mode: host`, compose in `/root/mautrix-whatsapp/`. |
|
||||
| mautrix-meta | `127.0.0.1:29319` | Facebook/Instagram bridge, `/root/mautrix-meta/`. |
|
||||
| mautrix-signal | `127.0.0.1:29328` | `/root/mautrix-signal/`. |
|
||||
| mautrix-slack | `127.0.0.1:29335` | `/root/mautrix-slack/`. |
|
||||
| mautrix-linkedin | `127.0.0.1:29341` | `/root/mautrix-linkedin/`. |
|
||||
|
||||
## Bridges (mautrix)
|
||||
|
||||
All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name>/` — no systemd unit, `restart: unless-stopped`. Their appservice registration YAMLs live in `/etc/matrix-synapse/` and are listed in `homeserver.yaml` under `app_service_config_files`. If a bridge daemon is down, Synapse logs `push_bulk … ConnectionRefusedError` and retries every 512 s; restart the bridge with `cd /root/mautrix-<name> && docker compose up -d`.
|
||||
|
||||
## Operational notes
|
||||
- Rootfs is on `pve` LVM-thin, grow with `pct resize 118 rootfs +<N>G` from the host — resize2fs runs automatically.
|
||||
- `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
|
||||
|
||||
## Known issues
|
||||
- Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21 — the Matrix DB grows steadily. Watch or grow rootfs.
|
||||
- ~~Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
||||
|
||||
## Related
|
||||
- [claudio-bot (123)](123-claudio-bot.md) — connects directly to `192.168.8.239:8008` (avoids hairpin-NAT TLS issue on the public URL)
|
||||
@@ -28,5 +43,8 @@ Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-05-15 — disk-full incident, rootfs grown to 16 GiB, bridges restarted
|
||||
Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed `OSError: [Errno 28] No space left on device` during flush. Fix: `pct resize 118 rootfs +8G` (host `pve` VG had plenty of headroom), `journalctl --vacuum-size=50M` to recover ~200 M, `systemctl restart matrix-synapse`, then `docker compose up -d` in each `/root/mautrix-*` directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
Reference in New Issue
Block a user