# Monitoring — `claudio-monitor`

Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.

## Why

After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.

## Components

- **Repo:** `dtoro/claudio-monitor`
- **Checkout:** `/opt/claudio-monitor` on the PVE host
- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).

## Per-LXC data

A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.

## Matrix-side plugin

`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
- `/monitor status`
- `/monitor history [N]`
- `/monitor clear <key>`

Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.

Events posted to claudio-bot `/notify`:
```json
{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
```
`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.

## Current thresholds & active alerts

Active warnings at the time monitor was deployed (2026-04-21):
- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%

## Testing

- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.

## Not yet deployed (tracked in plan file)

- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.

## Related repos

- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.

## Related pages
- [Hubris host](../hosts/hubris.md)
- [claudio-bot (123)](../containers/123-claudio-bot.md)
- [HAOS VM (108)](../vms/108-haos.md)
- [Backups (disabled)](backups.md)
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)

## Changelog

### 2026-04-28 — wiki entry created
Initial documentation.

### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.