Files

dtoro f5cd320433 Bootstrap Homelab-Docs wiki

Initial documentation of the hubris Proxmox homelab as a cross-linked
markdown wiki. Per-node pages, cross-cutting infrastructure pages, an
investigation log, and an operations cheatsheet. Each node and topic
ends with a Changelog section so changes can be tracked in-place going
forward.

Refreshed against live state on 2026-04-28 — 14 active LXCs (109
syncthing currently stopped) + 1 VM (108 haos). Reflects post-A/B-test
state of the 2026-04-21 hubris crash-loop investigation.

2026-04-28 22:53:09 +02:00

4.3 KiB

Raw Blame History

Monitoring — `claudio-monitor`

Per-5-minute host-health watchdog on hubris. Edge-triggered Matrix alerts via claudio-bot (123). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.

Why

After the 2026-04-21 thermal crash, the old single-purpose thermal-watch (which lived in dtoro/backup-library) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit 82f0596 in dtoro/claudio-monitor) once HA started consuming PVE metrics directly.

Components

Repo: dtoro/claudio-monitor
Checkout: /opt/claudio-monitor on the PVE host
No auto-deploy webhook yet — push, then /opt/claudio-monitor/scripts/deploy.sh manually.
Main loop: claudio-monitor.timer (every 5 min) → /opt/claudio-monitor/claudio-monitor.sh → sources lib/check-{thermal,host,lxc}.sh.
Module shape: every check is a check_XXX() function in lib/check-XXX.sh, sourced by the orchestrator. To add: write lib/check-new.sh and add . "$LIB/check-new.sh" + one check_new call in claudio-monitor.sh.
Thresholds: /etc/claudio-monitor/thresholds.env (installed from config/thresholds.env — admin edits there are preserved by deploy.sh).
Tokens: /etc/claudio-monitor/bot.token (same value as /etc/restic/bot.token, copied by hand). Missing → Matrix delivery is silently skipped.
State files: /var/lib/claudio-monitor/<check>-<key>.alert and .counter. notify.sh ships two primitives: maybe_alert (warn/clear hysteresis) and sticky_alert (N-consecutive-ticks).

Per-LXC data

A single pvesh get /cluster/resources --type vm --output-format json per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.

Matrix-side plugin

plugins/monitor.py in dtoro/claudio-bot. Registered in /etc/claudio-bot/config.yaml under plugins:. Slash commands:

/monitor status
/monitor history [N]
/monitor clear <key>

Tools (for NL queries via the LLM): get_alerts, get_monitor_history.

Events posted to claudio-bot /notify:

{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}

event:"info" is the recovery / clear event. The plugin keys active alerts by _alert_key(check, data).

Current thresholds & active alerts

Active warnings at the time monitor was deployed (2026-04-21):

lxc:103 (paperless) — disk 86.9%
lxc:118 (elementsynapse) — disk 86.8%

Testing

Force an alert: NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh (any threshold env override).
Clean state: rm -f /var/lib/claudio-monitor/*.alert.
Verify Matrix delivery: pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor".

Not yet deployed (tracked in plan file)

check-apt.sh + check-docker.sh + claudio-monitor-updates.timer (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on apps (105).
Deploy webhook (scripts/webhook/) mirroring dtoro/backup-library-deploy.

dtoro/backup-library — origin of the now-removed thermal-watch.*.
dtoro/claudio-bot — the monitor plugin lives at plugins/monitor.py.
dtoro/haos-config — empty placeholder created 2026-04-21 for HA /config once bootstrapped from inside HAOS via the SSH addon.

Changelog

2026-04-28 — wiki entry created

Initial documentation.

2026-04-21 — claudio-monitor stood up; thermal-watch removed

General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit 82f0596) because HA gets PVE metrics directly. Matrix-side monitor plugin added in claudio-bot commit e56da25.

4.3 KiB Raw Blame History

Monitoring — claudio-monitor

Why