Bootstrap Homelab-Docs wiki

Initial documentation of the hubris Proxmox homelab as a cross-linked
markdown wiki. Per-node pages, cross-cutting infrastructure pages, an
investigation log, and an operations cheatsheet. Each node and topic
ends with a Changelog section so changes can be tracked in-place going
forward.

Refreshed against live state on 2026-04-28 — 14 active LXCs (109
syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test
state of the 2026-04-21 hubris crash-loop investigation.

infrastructure/monitoring.md

@@ -0,0 +1,75 @@
+# Monitoring — `claudio-monitor`
+
+Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
+
+## Why
+
+After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
+
+## Components
+
+- **Repo:** `dtoro/claudio-monitor`
+- **Checkout:** `/opt/claudio-monitor` on the PVE host
+- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
+- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
+- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
+- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
+- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
+- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
+
+## Per-LXC data
+
+A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
+
+## Matrix-side plugin
+
+`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
+- `/monitor status`
+- `/monitor history [N]`
+- `/monitor clear <key>`
+
+Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
+
+Events posted to claudio-bot `/notify`:
+```json
+{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
+```
+`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
+
+## Current thresholds & active alerts
+
+Active warnings at the time monitor was deployed (2026-04-21):
+- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
+- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
+
+## Testing
+
+- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
+- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
+- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
+
+## Not yet deployed (tracked in plan file)
+
+- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
+- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
+
+## Related repos
+
+- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
+- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
+- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
+
+## Related pages
+- [Hubris host](../hosts/hubris.md)
+- [claudio-bot (123)](../containers/123-claudio-bot.md)
+- [HAOS VM (108)](../vms/108-haos.md)
+- [Backups (disabled)](backups.md)
+- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+
+## Changelog
+
+### 2026-04-28 — wiki entry created
+Initial documentation.
+
+### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
+General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.