dtoro
fd35b48c8d
Phase 1 — fix stale state after strong migration (Phase 1+2, 2026-07-05)
- README: corrected IPs (jellyfin 206→246, arriman 132→245, etc.),
added missing containers (128 trmnl, 129 house, 133 seanime, 134 romm,
124 authentik), updated last-refreshed date, added strong host context
- containers/101-jellyfin.md: IP 206→246, host hubris→strong, mount
/mnt/library→/mnt/media_local, GPU 760M→680M+RX7600, privilege→priv
- containers/118-elementsynapse.md: IP 239→242, added Host: strong
- containers/122-arriman.md: IP 132→245, mount→/mnt/media_local, added Host
- containers/129-house.md: IP 212→244, added Host: strong
- containers/130-grimmory.md: IP 213→247, mount→/mnt/media_local, added Host
- containers/121-caddy.md: fixed site list (books→grimmory, removed auth→VPS,
added house, roms, teddy, trmnl)
- hosts/strong.md: updated At-a-glance to reflect 7 LXCs hosted
- containers/123-claudio-bot.md, 127-mule-photos-new.md: archived to
containers/archive/ (were destroyed LXCs with living pages)
- inventory.yaml: verified correct — no changes needed
Phase 2 — structural cleanup
- infrastructure/index.md: one-page overview of all cross-cutting systems
- runbooks/: moved runbook-budget-from-csv.md and runbook-dpkg-interrupted.md
from operations/ with YAML frontmatter added
- plans/done/: moved 4 completed plans out of active view; updated index
- vms/index.md: added VM index page
Phase 3 — navigation & discoverability
- GLOSSARY.md: term definitions (Authentik, Caddy, LXC, VAAPI, etc.)
- README: added table of contents, links to glossary + infrastructure index
- investigations/: archived 2 resolved cases (crash-loop, authentik-migration)
to investigations/archive/; updated index with active vs archived sections
Phase 4 — ongoing discipline
- CONTRIBUTING.md: documented same-session update rule with explicit checklist
- README: replaced full LXC table with summary + link to containers/index.md
(single source of truth; de-duplication)
84 lines
6.3 KiB
Markdown
84 lines
6.3 KiB
Markdown
# 118 — `elementsynapse`
|
|
|
|
Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
|
|
|
|
## At a glance
|
|
- **Hostname:** `elementsynapse`
|
|
- **IP:** `192.168.8.242`
|
|
- **Host:** **strong** (migrated from hubris 2026-07-05)
|
|
- **Privilege:** **unprivileged**
|
|
- **Resources:** 1 core / 2 GiB RAM / **16 GiB rootfs** (grown from 8 GiB on 2026-05-15 after disk-full incident)
|
|
- **Mounts:** none from `/mnt/library`
|
|
- **Public hostname:** [`matrix.hubris.network`](../infrastructure/dns.md) → [caddy](121-caddy.md) → `:8008`
|
|
- **`server_name`:** `avispero` (intentionally different from the public URL).
|
|
- **Database:** Synapse runs on **SQLite** (`/var/lib/matrix-synapse/homeserver.db`, ~90 MiB). PostgreSQL on `127.0.0.1:5432` is dedicated to the mautrix bridge databases (one per bridge), not Synapse itself.
|
|
|
|
## Service / port map
|
|
|
|
| Service | Listen | Notes |
|
|
| -------------- | ------- | -------------------------------- |
|
|
| synapse | `:8008` | Matrix client-server / federation HTTP. Caddy terminates TLS for `matrix.hubris.network`. |
|
|
| synapse-admin | `:5173` | Web UI (vite dev server) for Synapse admin API. |
|
|
| mautrix-whatsapp | `:29318` | Docker, `network_mode: host`, compose in `/root/mautrix-whatsapp/`. |
|
|
| mautrix-meta | `127.0.0.1:29319` | Facebook/Instagram bridge, `/root/mautrix-meta/`. |
|
|
| mautrix-signal | `127.0.0.1:29328` | `/root/mautrix-signal/`. |
|
|
| mautrix-slack | `127.0.0.1:29335` | `/root/mautrix-slack/`. |
|
|
| mautrix-linkedin | `127.0.0.1:29341` | `/root/mautrix-linkedin/`. |
|
|
|
|
## Bridges (mautrix)
|
|
|
|
All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name>/` — no systemd unit, `restart: unless-stopped`. Their appservice registration YAMLs live in `/etc/matrix-synapse/` and are listed in `homeserver.yaml` under `app_service_config_files`. If a bridge daemon is down, Synapse logs `push_bulk … ConnectionRefusedError` and retries every 512 s; restart the bridge with `cd /root/mautrix-<name> && docker compose up -d`.
|
|
|
|
## Operational notes
|
|
- Rootfs is on `pve` LVM-thin, grow with `pct resize 118 rootfs +<N>G` from the host — resize2fs runs automatically.
|
|
- `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
|
|
|
|
## Known issues
|
|
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
|
|
|
## Related
|
|
- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
|
|
- [Caddy](121-caddy.md)
|
|
- [DNS](../infrastructure/dns.md)
|
|
- [Monitoring](../infrastructure/monitoring.md)
|
|
|
|
## Changelog
|
|
|
|
### 2026-06-06 — DHCP drift fixed: internal `/etc/network/interfaces` was `dhcp` despite Proxmox static config
|
|
**Symptom:** Matrix was down. Caddy at `192.168.8.175` couldn't reach `192.168.8.239:8008` — the LXC was actually at `192.168.8.244` because the guest-side dhclient had overridden the PVE-assigned static IP.
|
|
|
|
**Root cause:** During the 2026-06-02 static-IP migration, `pct set 118 --net0 ... ip=192.168.8.239/24` was applied to the Proxmox config, but the internal `/etc/network/interfaces` still had `iface eth0 inet dhcp`. On every DHCP lease renewal, dhclient grabbed `.244` from Technitium's pool.
|
|
|
|
**Fix:**
|
|
- Replaced `iface eth0 inet dhcp` with `iface eth0 inet static` + `address 192.168.8.239/24` + `gateway 192.168.8.1`
|
|
- `ifdown eth0 && ifup eth0` applied the static IP
|
|
- Killed lingering dhclient process
|
|
- Verified: `curl http://192.168.8.239:8008` returns 302 from Caddy's LXC
|
|
|
|
**Prevention:** The `check-caddy-backends.sh` cron on hubris now runs every 10 minutes, which would have caught this drift within 10 minutes of occurrence.
|
|
|
|
### 2026-05-15 — phantom-notification cleanup for `@admin`
|
|
After the disk-full incident, the mobile (Element X) badge showed ~125 unread but every room read clean in the UI. Root cause: stale rows in `event_push_actions` that were never reaped — Synapse's read-receipt-driven cleanup didn't catch up. Two contributors:
|
|
1. **8 of 12 affected rooms** had read receipts past the "unread" stream_ordering — pure stale state, likely from the disk-full window stalling rotation/cleanup.
|
|
2. **The noisiest room** (a Signal-bridged DM) was flooded with `m.room.redaction` events from `@signalbot:avispero` (disappearing messages being deleted). Synapse counts redactions as notifications by default, but Element X doesn't render them — so they were invisible "ghosts".
|
|
|
|
Fix (after stopping Synapse and backing up `homeserver.db`):
|
|
- `DELETE FROM event_push_actions WHERE stream_ordering <= max(receipt_stream)` per (user, room) — clears receipt-superseded rows.
|
|
- `DELETE FROM event_push_actions WHERE event_id IN (SELECT event_id FROM events WHERE type='m.room.redaction')` — removes redaction-driven notifications.
|
|
- Rebuild `event_push_summary` from the remaining rows (preserve `thread_id='main'` to satisfy the NOT-NULL check constraint).
|
|
|
|
Result: 125 → 4 (legitimate WhatsApp unreads).
|
|
|
|
Then, to stop this recurring, an **override push-rule** was installed for `@admin` via the client API:
|
|
```
|
|
PUT /_matrix/client/v3/pushrules/global/override/suppress_redactions
|
|
{ "actions": [], "conditions": [{ "kind": "event_match", "key": "type", "pattern": "m.room.redaction" }] }
|
|
```
|
|
With empty `actions`, all `m.room.redaction` events are silently ignored by the push system. Inspect/remove via the same endpoint. (Note: rule ids beginning with `.` are reserved for server defaults — custom ids cannot start with a dot.)
|
|
|
|
### 2026-05-15 — disk-full incident, rootfs grown to 16 GiB, bridges restarted
|
|
Element/Synapse was unreliable for ~2 days: messages dropping, sync requests stalling, mautrix bridges (slack/signal/meta/linkedin/whatsapp) all exited around 2026-05-13. Root cause: rootfs reached 100% (19 MB free of 7.8 G); Synapse logs showed `OSError: [Errno 28] No space left on device` during flush. Fix: `pct resize 118 rootfs +8G` (host `pve` VG had plenty of headroom), `journalctl --vacuum-size=50M` to recover ~200 M, `systemctl restart matrix-synapse`, then `docker compose up -d` in each `/root/mautrix-*` directory. Also expanded this wiki page to document the bridges, synapse-admin, and the actual DB backend (Postgres, not the SQLite leftover).
|
|
|
|
### 2026-04-28 — wiki entry created
|
|
Initial documentation.
|