Files
oikos/knowledge/wiki/infrastructure/monitoring.md
dtoro 8a6422bd7d docs: move narrative wiki under knowledge/wiki/ (phase 3)
Problem: node and cross-cutting narratives lived at the repo root
(containers/, vms/, infrastructure/, host .md files), interleaved with the
machine-readable substrate.

Change:
- Move containers/ -> knowledge/wiki/containers/, vms/ -> knowledge/wiki/vms/,
  infrastructure/ -> knowledge/wiki/infrastructure/, hosts/{hubris,strong}.md ->
  knowledge/wiki/hosts/, infrastructure/references/ -> knowledge/sources/references/,
  GLOSSARY.md -> knowledge/GLOSSARY.md.
- Add knowledge/{index.md,log.md,sources/index.md} scaffolding.
- Rewrite all relative links repo-wide via a path-resolving mapper (inbound +
  outbound + between-moved-files), including .hermes/, runbooks, operations,
  investigations, plans, README, AGENTS.
- Repoint inventory.yaml doc_page fields and regenerate hosts/*.yaml (which
  embed doc_page); update oikos/gen-topology.py output path, candidate doc
  paths, and footer links; update code-comment doc paths.

Substrate untouched in place: inventory.yaml, hosts/*.yaml (regenerated,
idempotent), oikos/ code, mcp/, secrets/, bin/.

Verification:
- Logical broken-link set identical to pre-move baseline (net 128 -> 127; the
  topology regen fixed one, introduced none). Remaining are pre-existing refs
  to destroyed/archived nodes, out of scope for this move.
- gen-topology.py --check exit 0 (in sync); cards carry knowledge/wiki/ doc paths.
- build_host_files.py idempotent; all inventory doc_page targets resolve.
- MCP contract verified: get_page/search_docs/get_changelog resolve moved pages.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 14:35:23 +02:00

2.4 KiB

Monitoring — Hermes health watchdog

Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy claudio-monitor + claudio-bot IPC pipeline on 2026-06-04.

Current approach

Two layers:

  1. On-demand: ask Hermes "how's the homelab?" or run homelab health — loads the homelab-hardware-health skill, checks hardware temps, LXC resources, service reachability, and apt/docker drift across all hosts.

  2. Cron watchdog: homelab-health-watchdog runs every 15 minutes via Hermes cron. Silent when healthy. When thresholds breach, sends an actionable alert to Matrix (@dtoro:avispero) with options the user can reply to directly (e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on the selected option via SSH.

Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C, apt >10/>50 upgradable, services down.

Home Assistant pulls PVE metrics independently via its Proxmox VE integration (unaffected by this change).

Legacy: claudio-monitor (deprecated 2026-06-04)

The old system was a bash watchdog on hubris (claudio-monitor.timer, every 5 min) that POSTed alerts to a Matrix bot (@claudio:avispero) via an IPC server on LXC 123:9090. All components decommissioned:

Component Fate
LXC 123 (claudio-bot) Destroyed 2026-06-04
dtoro/claudio-bot Archived (read-only) on Gitea
dtoro/claudio-monitor Archived (read-only) on Gitea
claudio-monitor.timer Disabled on hubris
/opt/claudio-monitor/ Still on hubris (cleanup pending)
/etc/claudio-monitor/ Still on hubris (cleanup pending)

For the full deprecation plan, see plans/2026-06-04_130000-deprecate-claudio-bot.md.

Changelog

2026-06-04 — migrated to Hermes health watchdog

claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring. On-demand homelab health via extended skill; 15-min cron watchdog with actionable Matrix alerts. LXC 123 destroyed, repos archived.

2026-04-28 — wiki entry created

Initial documentation.

2026-04-21 — claudio-monitor stood up; thermal-watch removed

General health monitor with per-LXC checks. MQTT/REST push paths ripped out.