claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes

- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core
- Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only)
- Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor
- Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift
- Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options)
- Wire Matrix (matrix:dtoro) as health alert delivery platform
- Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml
- Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md

infrastructure/monitoring.md

@@ -1,75 +1,60 @@
-# Monitoring — `claudio-monitor`
+# Monitoring — Hermes health watchdog
 
-Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
+Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
+`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
 
-## Why
+## Current approach
 
-After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
+Two layers:
 
-## Components
+1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
+   the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
+   service reachability, and apt/docker drift across all hosts.
 
-- **Repo:** `dtoro/claudio-monitor`
-- **Checkout:** `/opt/claudio-monitor` on the PVE host
-- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
-- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
-- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
-- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
-- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
-- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
+2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
+   cron. Silent when healthy. When thresholds breach, sends an actionable alert
+   to Matrix (`@dtoro:avispero`) with options the user can reply to directly
+   (e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
+   the selected option via SSH.
 
-## Per-LXC data
+Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
+apt >10/>50 upgradable, services down.
 
-A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
+Home Assistant pulls PVE metrics independently via its Proxmox VE integration
+(unaffected by this change).
 
-## Matrix-side plugin
+## Legacy: claudio-monitor (deprecated 2026-06-04)
 
-`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
-- `/monitor status`
-- `/monitor history [N]`
-- `/monitor clear <key>`
+The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
+min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
+on LXC 123:9090. All components decommissioned:
 
-Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
+| Component | Fate |
+|-----------|------|
+| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
+| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
+| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
+| `claudio-monitor.timer` | Disabled on hubris |
+| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
+| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
 
-Events posted to claudio-bot `/notify`:
-```json
-{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
-```
-`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
-
-## Current thresholds & active alerts
-
-Active warnings at the time monitor was deployed (2026-04-21):
-- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
-- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
-
-## Testing
-
-- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
-- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
-- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
-
-## Not yet deployed (tracked in plan file)
-
-- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
-- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
-
-## Related repos
-
-- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
-- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
-- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
+For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
 
 ## Related pages
 - [Hubris host](../hosts/hubris.md)
-- [claudio-bot (123)](../containers/123-claudio-bot.md)
 - [HAOS VM (108)](../vms/108-haos.md)
 - [Backups (disabled)](backups.md)
-- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
+- [Homelab context distribution](homelab-context.md)
 
 ## Changelog
 
+### 2026-06-04 — migrated to Hermes health watchdog
+claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
+On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
+Matrix alerts. LXC 123 destroyed, repos archived.
+
 ### 2026-04-28 — wiki entry created
 Initial documentation.
 
 ### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
-General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.
+General health monitor with per-LXC checks. MQTT/REST push paths ripped out.