Files
oikos/archive/knowledge/infrastructure/monitoring.md
dtoro 6b75f7302d db as source of truth: wiki→seeds, archive old artifacts, knowledge ingestion
- Migrations 010 (content_hash) + 011 (search tsvector column)
- new: internal/knowledge/seed.go — knowledge seed ingest engine
- new: internal/httpapi/knowledge.go — SearchKnowledge + GetEntityKnowledge
- wire knowledge ingest into oikos seed pipeline
- convert all 36 wiki docs + 6 investigations + 12 runbooks → seeds/knowledge.yaml
- archive: knowledge/wiki/→archive/, oikos/cards/→archive/, .hermes/plans/→archive/
- delete: 9 superseded Python kernel files, ledger/, mcp/build_host_files.py
- remove empty knowledge/ directory tree
2026-07-07 20:22:30 +02:00

2.4 KiB

Monitoring — Hermes health watchdog

Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy claudio-monitor + claudio-bot IPC pipeline on 2026-06-04.

Current approach

Two layers:

  1. On-demand: ask Hermes "how's the homelab?" or run homelab health — loads the homelab-hardware-health skill, checks hardware temps, LXC resources, service reachability, and apt/docker drift across all hosts.

  2. Cron watchdog: homelab-health-watchdog runs every 15 minutes via Hermes cron. Silent when healthy. When thresholds breach, sends an actionable alert to Matrix (@dtoro:avispero) with options the user can reply to directly (e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on the selected option via SSH.

Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C, apt >10/>50 upgradable, services down.

Home Assistant pulls PVE metrics independently via its Proxmox VE integration (unaffected by this change).

Legacy: claudio-monitor (deprecated 2026-06-04)

The old system was a bash watchdog on hubris (claudio-monitor.timer, every 5 min) that POSTed alerts to a Matrix bot (@claudio:avispero) via an IPC server on LXC 123:9090. All components decommissioned:

Component Fate
LXC 123 (claudio-bot) Destroyed 2026-06-04
dtoro/claudio-bot Archived (read-only) on Gitea
dtoro/claudio-monitor Archived (read-only) on Gitea
claudio-monitor.timer Disabled on hubris
/opt/claudio-monitor/ Still on hubris (cleanup pending)
/etc/claudio-monitor/ Still on hubris (cleanup pending)

For the full deprecation plan, see plans/2026-06-04_130000-deprecate-claudio-bot.md.

Changelog

2026-06-04 — migrated to Hermes health watchdog

claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring. On-demand homelab health via extended skill; 15-min cron watchdog with actionable Matrix alerts. LXC 123 destroyed, repos archived.

2026-04-28 — wiki entry created

Initial documentation.

2026-04-21 — claudio-monitor stood up; thermal-watch removed

General health monitor with per-LXC checks. MQTT/REST push paths ripped out.