claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes
- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core - Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only) - Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor - Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift - Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options) - Wire Matrix (matrix:dtoro) as health alert delivery platform - Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml - Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md
This commit is contained in:
@@ -20,7 +20,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
||||
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
|
||||
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
|
||||
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
|
||||
- `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
|
||||
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
|
||||
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
|
||||
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
|
||||
- `192.168.8.190` ([plato (126)](../containers/126-plato.md))
|
||||
@@ -38,7 +38,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
||||
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
|
||||
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
|
||||
| `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
|
||||
| `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
|
||||
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
|
||||
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
||||
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
|
||||
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
|
||||
@@ -58,7 +58,7 @@ Always commit + push. Local-only edits drift. Common ones:
|
||||
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
|
||||
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
|
||||
- `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys)
|
||||
- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys)
|
||||
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
|
||||
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
|
||||
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
|
||||
|
||||
@@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
|
||||
|
||||
## Related
|
||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||
|
||||
@@ -135,6 +135,9 @@ Initial documentation. Six active pipelines.
|
||||
### 2026-04-22 — Artifacto pipeline added
|
||||
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
|
||||
|
||||
### 2026-06-04 — claudio-bot pipeline decommissioned
|
||||
LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
|
||||
|
||||
### 2026-04-21 — mule-image + claudio-bot pipelines added
|
||||
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
|
||||
|
||||
|
||||
@@ -71,7 +71,7 @@ Baked into the systemd units:
|
||||
|
||||
## Notifications
|
||||
|
||||
POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.
|
||||
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
|
||||
|
||||
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
|
||||
|
||||
@@ -95,7 +95,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
|
||||
|
||||
## Related
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- [claudio-bot (123)](../containers/123-claudio-bot.md)
|
||||
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
||||
- [Monitoring](monitoring.md)
|
||||
- [Auto-deploy](auto-deploy.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Homelab context distribution
|
||||
|
||||
The cross-client context-and-secrets system that makes every agent (Claude
|
||||
Code, claudio-bot, future MCP-capable clients) on every machine in the lab
|
||||
Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
|
||||
self-locating and able to read the same source of truth.
|
||||
|
||||
Operational walkthrough for enrolling a new client lives in
|
||||
@@ -13,7 +13,7 @@ page is the architecture reference.
|
||||
| Piece | Host | Path | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
|
||||
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, claudio-bot |
|
||||
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
|
||||
| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab` → `/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
|
||||
| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
|
||||
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
|
||||
@@ -140,4 +140,4 @@ wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
|
||||
`client add --finalize-pubkey` grant flow live so new clients are one
|
||||
ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
|
||||
(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
|
||||
claudio-bot integration so the matrix agent uses inventory at chat-time.
|
||||
Hermes Agent integration so the agent uses inventory at chat-time.
|
||||
|
||||
@@ -1,75 +1,60 @@
|
||||
# Monitoring — `claudio-monitor`
|
||||
# Monitoring — Hermes health watchdog
|
||||
|
||||
Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
|
||||
Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
|
||||
`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
|
||||
|
||||
## Why
|
||||
## Current approach
|
||||
|
||||
After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
|
||||
Two layers:
|
||||
|
||||
## Components
|
||||
1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
|
||||
the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
|
||||
service reachability, and apt/docker drift across all hosts.
|
||||
|
||||
- **Repo:** `dtoro/claudio-monitor`
|
||||
- **Checkout:** `/opt/claudio-monitor` on the PVE host
|
||||
- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
|
||||
- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
|
||||
- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
|
||||
- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
|
||||
- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
|
||||
- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
|
||||
2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
|
||||
cron. Silent when healthy. When thresholds breach, sends an actionable alert
|
||||
to Matrix (`@dtoro:avispero`) with options the user can reply to directly
|
||||
(e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
|
||||
the selected option via SSH.
|
||||
|
||||
## Per-LXC data
|
||||
Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
|
||||
apt >10/>50 upgradable, services down.
|
||||
|
||||
A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
|
||||
Home Assistant pulls PVE metrics independently via its Proxmox VE integration
|
||||
(unaffected by this change).
|
||||
|
||||
## Matrix-side plugin
|
||||
## Legacy: claudio-monitor (deprecated 2026-06-04)
|
||||
|
||||
`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
|
||||
- `/monitor status`
|
||||
- `/monitor history [N]`
|
||||
- `/monitor clear <key>`
|
||||
The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
|
||||
min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
|
||||
on LXC 123:9090. All components decommissioned:
|
||||
|
||||
Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
|
||||
| Component | Fate |
|
||||
|-----------|------|
|
||||
| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
|
||||
| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
|
||||
| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
|
||||
| `claudio-monitor.timer` | Disabled on hubris |
|
||||
| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
|
||||
| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
|
||||
|
||||
Events posted to claudio-bot `/notify`:
|
||||
```json
|
||||
{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
|
||||
```
|
||||
`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
|
||||
|
||||
## Current thresholds & active alerts
|
||||
|
||||
Active warnings at the time monitor was deployed (2026-04-21):
|
||||
- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
|
||||
- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
|
||||
|
||||
## Testing
|
||||
|
||||
- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
|
||||
- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
|
||||
- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
|
||||
|
||||
## Not yet deployed (tracked in plan file)
|
||||
|
||||
- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
|
||||
- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
|
||||
|
||||
## Related repos
|
||||
|
||||
- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
|
||||
- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
|
||||
- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
|
||||
For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
|
||||
|
||||
## Related pages
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- [claudio-bot (123)](../containers/123-claudio-bot.md)
|
||||
- [HAOS VM (108)](../vms/108-haos.md)
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
||||
- [Homelab context distribution](homelab-context.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-04 — migrated to Hermes health watchdog
|
||||
claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
|
||||
On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
|
||||
Matrix alerts. LXC 123 destroyed, repos archived.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
|
||||
General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.
|
||||
General health monitor with per-LXC checks. MQTT/REST push paths ripped out.
|
||||
|
||||
@@ -131,7 +131,6 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|
||||
| 120 | mule-images | `192.168.8.136` | photo-management |
|
||||
| 121 | caddy | `192.168.8.175` | reverse-proxy |
|
||||
| 122 | arriman | `192.168.8.132` | arr-stack |
|
||||
| 123 | claudio-bot | `192.168.8.230` | matrix-agent |
|
||||
| 126 | plato | `192.168.8.190` | app |
|
||||
|
||||
### Workstations
|
||||
|
||||
Reference in New Issue
Block a user