claudio-bot: decommission LXC 123, archive repos, migrate monitoring to Hermes
- Destroy LXC 123 (claudio-bot) — freed 8 GiB rootfs, 512 MiB RAM, 1 core - Archive dtoro/claudio-bot and dtoro/claudio-monitor on Gitea (read-only) - Stop claudio-monitor.timer on hubris, remove /opt/claudio-monitor - Extend homelab-hardware-health skill with LXC resources, service health, apt/docker drift - Create homelab-health-watchdog cron (15 min, Matrix alerts, actionable options) - Wire Matrix (matrix:dtoro) as health alert delivery platform - Update 13 files: inventory, containers/*, infrastructure/*, hosts/*, README, .sops.yaml - Add deprecation plan at plans/2026-06-04_130000-deprecate-claudio-bot.md
This commit is contained in:
@@ -44,7 +44,7 @@ creation_rules:
|
||||
- path_regex: ^secrets/webhook-hmacs\.yaml$
|
||||
# LXCs that run a webhook receiver.
|
||||
age: >-
|
||||
# placeholder — fill with age_pubkey of: apps, caddy, claudio-bot, claudio-monitor host
|
||||
# placeholder — fill with age_pubkey of: apps, caddy
|
||||
|
||||
- path_regex: ^secrets/turn-shared-secret\.yaml$
|
||||
# coturn TURN long-term-credential password. Consumed by hubris (which
|
||||
|
||||
@@ -29,7 +29,6 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links:
|
||||
| 120 | [mule-images](containers/120-mule-images.md) | 192.168.8.136 | Mule-image / mulita photos |
|
||||
| 121 | [caddy](containers/121-caddy.md) | 192.168.8.175 | Reverse proxy |
|
||||
| 122 | [arriman](containers/122-arriman.md) | 192.168.8.132 | Docker host (\*arr stack) |
|
||||
| 123 | [claudio-bot](containers/123-claudio-bot.md) | 192.168.8.230 | Matrix control plane |
|
||||
| 124 | [authentik](containers/124-authentik.md) | 192.168.8.180 | SSO + split-horizon DNS |
|
||||
| 126 | [plato](containers/126-plato.md) | 192.168.8.190 | Plato (notes/discovery workspace) |
|
||||
|
||||
@@ -37,7 +36,7 @@ See the full table in [`containers/index.md`](containers/index.md). Quick links:
|
||||
- [DNS — split-horizon](infrastructure/dns.md)
|
||||
- [Ingress — Caddy + VPS traefik](infrastructure/ingress.md)
|
||||
- [Mesh — Tailscale → Netbird migration](infrastructure/mesh.md)
|
||||
- [Monitoring — claudio-monitor](infrastructure/monitoring.md)
|
||||
- [Monitoring — Hermes health watchdog](infrastructure/monitoring.md)
|
||||
- [Media permissions — `media` GID 10000](infrastructure/media-permissions.md)
|
||||
- [SSH access](infrastructure/ssh-access.md)
|
||||
- [Backups — restic on external drive (disabled)](infrastructure/backups.md)
|
||||
@@ -61,7 +60,7 @@ Time-stamped incident notes / experiments in [`investigations/`](investigations/
|
||||
```
|
||||
- **Cross-linking is mandatory.** If a page references another node or system, link to it. Treat orphans as a bug.
|
||||
- **Live state wins.** When something here disagrees with `pct config` / `docker inspect` / running config, fix the wiki *and* note the change in the relevant changelog.
|
||||
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image, claudio-bot) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
|
||||
- **Tracked configs.** A node whose config lives in a Gitea repo (Caddy, Gitea customizations, Artifacto, mule-image) is auto-deployed via webhook — see [auto-deploy](infrastructure/auto-deploy.md). Edits there must be pushed, not left local.
|
||||
- **No secrets.** This is a private repo on `git.hubris.network`, but still: paths to secret files are fine, secret values are not.
|
||||
|
||||
## Maintaining this wiki
|
||||
|
||||
@@ -24,7 +24,7 @@ Behind [Authentik forward-auth](124-authentik.md). API path `/api/*` bypasses fo
|
||||
- Documents at `/mnt/library/documents` (owner `www-data:www-data`, mode 750 — *not* on the `media` group, by design).
|
||||
|
||||
## Known issues
|
||||
- Disk usage was 86.9% at last claudio-monitor reading on 2026-04-21. Monitor or grow rootfs.
|
||||
- ~~Disk usage was 86.9% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
||||
|
||||
## Related
|
||||
- [Authentik](124-authentik.md)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# 118 — `elementsynapse`
|
||||
|
||||
Matrix homeserver (Synapse). Backs `@dtoro:avispero` and `@claudio:avispero`.
|
||||
Matrix homeserver (Synapse). Backs `@dtoro:avispero`.
|
||||
|
||||
## At a glance
|
||||
- **Hostname:** `elementsynapse`
|
||||
@@ -33,10 +33,10 @@ All five bridges run as plain `docker compose` stacks under `/root/mautrix-<name
|
||||
- `/var/lib/matrix-synapse/media_store` is the dominant grower (~2 GiB at last check). If disk pressure returns, purge remote media via the Synapse admin API before resizing further.
|
||||
|
||||
## Known issues
|
||||
- ~~Disk usage was 86.8% at last claudio-monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
||||
- ~~Disk usage was 86.8% at last legacy monitor reading on 2026-04-21~~ — resolved by growing rootfs to 16 GiB on 2026-05-15.
|
||||
|
||||
## Related
|
||||
- [claudio-bot (123)](123-claudio-bot.md) — connects directly to `192.168.8.239:8008` (avoids hairpin-NAT TLS issue on the public URL)
|
||||
- ~~[claudio-bot (123)](123-claudio-bot.md)~~ — decommissioned 2026-06-04, replaced by Hermes Agent
|
||||
- [Caddy](121-caddy.md)
|
||||
- [DNS](../infrastructure/dns.md)
|
||||
- [Monitoring](../infrastructure/monitoring.md)
|
||||
|
||||
@@ -1,4 +1,9 @@
|
||||
# 123 — `claudio-bot`
|
||||
# 123 — `claudio-bot` (DEPRECATED — destroyed 2026-06-04)
|
||||
|
||||
> **This LXC was destroyed on 2026-06-04.** Replaced by Hermes Agent on mac-mini.
|
||||
> Monitoring migrated to `homelab-hardware-health` skill + 15-min Hermes cronjob.
|
||||
> Repos `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived (read-only) on Gitea.
|
||||
> See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) for full details.
|
||||
|
||||
Matrix-resident control plane. Bot account `@claudio:avispero` joined to a private room; accepts slash commands and natural language; relays infra notifications.
|
||||
|
||||
@@ -66,6 +71,12 @@ Push to `dtoro/claudio-bot` → gitea webhook → `http://192.168.8.230:9797/dep
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-04 — LXC destroyed; replaced by Hermes Agent
|
||||
LXC 123 destroyed via `pct destroy 123 --purge`. Bot service stopped, systemd
|
||||
units disabled. `dtoro/claudio-bot` and `dtoro/claudio-monitor` archived on
|
||||
Gitea. Monitoring replaced by Hermes `homelab-health-watchdog` cron job.
|
||||
`@claudio:avispero` Matrix account decommissioned.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
|
||||
@@ -14,7 +14,6 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
|
||||
| 120 | [mule-images](120-mule-images.md) | 192.168.8.136 | priv | 6 | 12 GiB | 60 GiB | `/mnt/library` + `/dev/dri` (iGPU passthrough) | `photos.hubris.network` | running |
|
||||
| 121 | [caddy](121-caddy.md) | 192.168.8.175 | unpriv | 1 | 512 MiB | 6 GiB | — | (terminates all `*.hubris.network`) | running |
|
||||
| 122 | [arriman](122-arriman.md) | 192.168.8.132 | priv | 4 | 8 GiB | 24 GiB | `/mnt/library` | `jellyseerr` / `qbit` / `sab` | running |
|
||||
| 123 | [claudio-bot](123-claudio-bot.md) | 192.168.8.230 | unpriv | 1 | 512 MiB | 8 GiB | — | — | running |
|
||||
| 124 | [authentik](124-authentik.md) | 192.168.8.180 | priv | 2 | 4 GiB | 20 GiB | — | `auth.hubris.network` | running |
|
||||
| 126 | [plato](126-plato.md) | 192.168.8.190 | priv | 2 | 2 GiB | 8 GiB | `/mnt/library/documents/plato` | `plato.hubris.network` | running |
|
||||
|
||||
@@ -26,6 +25,7 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
|
||||
| 100 | arr (yunohost) | ~2026-04-28 | Migrated to docker stack on [arriman](122-arriman.md); planned retention window expired |
|
||||
| 106 | flaresolverr | ~2026-04-28 | Folded into the arriman docker compose |
|
||||
| 116 | heaper | 2026-05-14 | Decommissioned by user; data subtree at `/mnt/library/heaper` (224 MiB) retained |
|
||||
| 123 | claudio-bot | 2026-06-04 | Replaced by Hermes Agent on mac-mini; monitoring migrated to `homelab-health-watchdog` cron. See [deprecation plan](../plans/2026-06-04_130000-deprecate-claudio-bot.md) |
|
||||
| 109 | syncthing | 2026-05-14 | Decommissioned by user; `/mnt/library/syncthing` was already empty |
|
||||
| 125 | seafile | 2026-05-13 | Seafile Pro evaluation, user disliked the product; teardown also removed `files.hubris.network` from caddy + dnsmasq |
|
||||
| 107 | marimo | between 2026-04-21 and 2026-04-28 | Decommissioned |
|
||||
@@ -41,7 +41,7 @@ All containers live on [`hubris`](../hosts/hubris.md). Each row links to the per
|
||||
- All net0 are `bridge=vmbr0`, `ip=dhcp` except [124 (authentik)](124-authentik.md) which is statically `192.168.8.180/24`. IPs are stable via the LAN router's DHCP reservations.
|
||||
- `onboot=1` on every container — the host brings them up after `pve-guests.service`.
|
||||
- Bind mounts are declared as `mp0: /mnt/library,mp=/mnt/library`. Containers that don't mount `/mnt/library` don't need it.
|
||||
- Most containers are privileged. Unprivileged ones (`101`, `118`, `121`, `123`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
|
||||
- Most containers are privileged. Unprivileged ones (`101`, `118`, `121`) require an idmap block in their conf to participate in the [media GID 10000](../infrastructure/media-permissions.md) standard.
|
||||
|
||||
## Related
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
# DEPRECATED — LXC 123 destroyed 2026-06-04. Replaced by Hermes Agent.
|
||||
# Generated by mcp/build_host_files.py from inventory.yaml.
|
||||
# Do NOT edit by hand — your changes will be overwritten.
|
||||
# Source of truth: ../inventory.yaml
|
||||
|
||||
@@ -20,7 +20,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
||||
- `127.0.0.1` (gitea customizations on [LXC 104](../containers/104-gitea.md))
|
||||
- `192.168.8.175` ([caddy (121)](../containers/121-caddy.md))
|
||||
- `192.168.8.205` ([apps (105)](../containers/105-apps.md) — Artifacto)
|
||||
- `192.168.8.230` ([claudio-bot (123)](../containers/123-claudio-bot.md))
|
||||
- ~~`192.168.8.230` (claudio-bot — destroyed 2026-06-04)~~
|
||||
- `192.168.8.136` ([mule-images (120)](../containers/120-mule-images.md))
|
||||
- `192.168.8.77` ([hubris host](../hosts/hubris.md) — backup-library)
|
||||
- `192.168.8.190` ([plato (126)](../containers/126-plato.md))
|
||||
@@ -38,7 +38,7 @@ The app repo at `/opt/<thing>` is the working tree, but the deploy tooling (`web
|
||||
| `dtoro/mule-image` | [mule-images (120)](../containers/120-mule-images.md) `/opt/mule-image/` | B | `http://192.168.8.136:9797/deploy` | 6 | `docker compose up -d --build` |
|
||||
| `dtoro/Artifacto` | [apps (105)](../containers/105-apps.md) `/opt/artifacto/` | B | `http://192.168.8.205:9798/deploy` | 7 | `docker compose up -d --build` |
|
||||
| `dtoro/Plato` | [plato (126)](../containers/126-plato.md) `/opt/plato/app/` | B | `http://192.168.8.190:9799/deploy` | 8 | `docker compose up -d --build` |
|
||||
| `dtoro/claudio-bot` | [claudio-bot (123)](../containers/123-claudio-bot.md) `/opt/claudio-bot/` | A | `http://192.168.8.230:9797/deploy` | (orig) | `pip install` + `systemctl restart claudio-bot` |
|
||||
| `dtoro/claudio-bot` | ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04) | ⊘ | `http://192.168.8.230:9797/deploy` (dead) | (archived) | Repo archived — LXC destroyed |
|
||||
| `dtoro/backup-library` | [hubris host](../hosts/hubris.md) `/opt/backup-library/` | A | `http://192.168.8.77:9798/deploy` | (orig) | runs `deploy.sh` (preserves admin-edited `/etc/restic/include-*.list`) |
|
||||
| `dtoro/Homelab-Docs` → homelab-mcp | [apps (105)](../containers/105-apps.md) `/opt/homelab-mcp/` | B | `http://192.168.8.205:9811/deploy` | 10 | reinstalls `homelab-mcp.service` + restart |
|
||||
| `dtoro/Homelab-Docs` → secrets-issuance | [apps (105)](../containers/105-apps.md) `/opt/secrets-issuance/` | B | `http://192.168.8.205:9821/deploy` | 11 | reinstalls `secrets-issuance.service` + restart |
|
||||
@@ -58,7 +58,7 @@ Always commit + push. Local-only edits drift. Common ones:
|
||||
- `/opt/artifacto/` ↔ `dtoro/Artifacto` (auto-deploys)
|
||||
- `/opt/mule-image/` ↔ `dtoro/mule-image` (auto-deploys)
|
||||
- `/opt/plato/app/` ↔ `dtoro/Plato` (auto-deploys)
|
||||
- `/opt/claudio-bot/` ↔ `dtoro/claudio-bot` (auto-deploys)
|
||||
- ~~`/opt/claudio-bot/` ↔ `dtoro/claudio-bot`~~ (destroyed 2026-06-04)
|
||||
- `/opt/backup-library/` ↔ `dtoro/backup-library` (auto-deploys)
|
||||
- `/opt/homelab-mcp/` + `/opt/secrets-issuance/` ↔ `dtoro/Homelab-Docs` (auto-deploys both, see [homelab-context](homelab-context.md))
|
||||
|
||||
@@ -117,7 +117,7 @@ If you're not sure what's already lurking, run `homelab apt-audit --fleet` and l
|
||||
|
||||
## Related
|
||||
- [Gitea (104)](../containers/104-gitea.md) — webhook source for all of these
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [claudio-bot (123)](../containers/123-claudio-bot.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Caddy (121)](../containers/121-caddy.md), [apps (105)](../containers/105-apps.md), [mule-images (120)](../containers/120-mule-images.md), [hubris host](../hosts/hubris.md) — webhook targets
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Operations cheatsheet](../operations/commands.md) — `homelab apt-audit` / `homelab apt-upgrade` reference
|
||||
|
||||
@@ -135,6 +135,9 @@ Initial documentation. Six active pipelines.
|
||||
### 2026-04-22 — Artifacto pipeline added
|
||||
Webhook id 7 on `dtoro/Artifacto` (port 9798 on apps). `app.ini` `ALLOWED_HOST_LIST` extended.
|
||||
|
||||
### 2026-06-04 — claudio-bot pipeline decommissioned
|
||||
LXC 123 destroyed, `dtoro/claudio-bot` archived. Webhook port 9797 dead.
|
||||
|
||||
### 2026-04-21 — mule-image + claudio-bot pipelines added
|
||||
Webhook id 6; receiver on apps' sibling `/opt/mule-deploy/`. Same shape used for claudio-bot.
|
||||
|
||||
|
||||
@@ -71,7 +71,7 @@ Baked into the systemd units:
|
||||
|
||||
## Notifications
|
||||
|
||||
POST to [claudio-bot (123)](../containers/123-claudio-bot.md) `http://192.168.8.230:9090/notify` with header `X-Bot-Token: $(cat /etc/restic/bot.token)`. Token in `/etc/restic/bot.token` is the same value as the bot's `ipc.token` — rotate together.
|
||||
~~POST to claudio-bot (123) `http://192.168.8.230:9090/notify`~~ — IPC endpoint dead since 2026-06-04. When backups are re-enabled, wire notifications to Hermes `send_message` via Matrix instead.
|
||||
|
||||
`OnFailure=notify-failure@%n.service` on the backup unit fires a synchronous notify as belt-and-suspenders for cases where the wrapper itself died before reaching its own notify.
|
||||
|
||||
@@ -95,7 +95,7 @@ Moved out of this repo to `dtoro/claudio-monitor` on 2026-04-21 (commit `50dc213
|
||||
|
||||
## Related
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- [claudio-bot (123)](../containers/123-claudio-bot.md)
|
||||
- ~~[claudio-bot (123)](../containers/123-claudio-bot.md)~~ (destroyed 2026-06-04)
|
||||
- [Monitoring](monitoring.md)
|
||||
- [Auto-deploy](auto-deploy.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Homelab context distribution
|
||||
|
||||
The cross-client context-and-secrets system that makes every agent (Claude
|
||||
Code, claudio-bot, future MCP-capable clients) on every machine in the lab
|
||||
Code, Hermes Agent, future MCP-capable clients) on every machine in the lab
|
||||
self-locating and able to read the same source of truth.
|
||||
|
||||
Operational walkthrough for enrolling a new client lives in
|
||||
@@ -13,7 +13,7 @@ page is the architecture reference.
|
||||
| Piece | Host | Path | Role |
|
||||
| --- | --- | --- | --- |
|
||||
| Source of truth | [gitea (104)](../containers/104-gitea.md) | `dtoro/Homelab-Docs.git` | Inventory + wiki + service code |
|
||||
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, claudio-bot |
|
||||
| Per-client clone | every enrolled client | `/opt/homelab-context/` | Read by `homelab` CLI, MCP server, Hermes Agent |
|
||||
| `homelab` CLI | every enrolled client | `/usr/local/bin/homelab` → `/opt/homelab-context/bin/homelab` (symlink) | Operator surface for enroll/secret/ssh/pct |
|
||||
| Per-client age key | every enrolled client | `/etc/age/key.txt` (0600 root) | Decrypts SOPS-encrypted secrets the client is a recipient on |
|
||||
| MCP server | [apps (105)](../containers/105-apps.md) | `homelab-mcp.service` on port 9810 (https://mcp.hubris.network/mcp) | 14 tools: 8 context (get_host, search_docs, …) + 5 read-only management (get_service_status, tail_log, …) + list_my_secrets |
|
||||
@@ -140,4 +140,4 @@ wired (hook ids 10 + 11). `homelab refresh-creds` + atomic
|
||||
`client add --finalize-pubkey` grant flow live so new clients are one
|
||||
ceremony instead of four manual steps. Outstanding: bootstrap mac-mini
|
||||
(macOS, exercises launchd) + ludo-mini + the remaining LXCs;
|
||||
claudio-bot integration so the matrix agent uses inventory at chat-time.
|
||||
Hermes Agent integration so the agent uses inventory at chat-time.
|
||||
|
||||
@@ -1,75 +1,60 @@
|
||||
# Monitoring — `claudio-monitor`
|
||||
# Monitoring — Hermes health watchdog
|
||||
|
||||
Per-5-minute host-health watchdog on [hubris](../hosts/hubris.md). Edge-triggered Matrix alerts via [claudio-bot (123)](../containers/123-claudio-bot.md). Home Assistant pulls the same metrics independently from the official Proxmox VE integration — claudio-monitor focuses on alerting only.
|
||||
Homelab health monitoring via Hermes Agent on mac-mini. Replaced the legacy
|
||||
`claudio-monitor` + `claudio-bot` IPC pipeline on 2026-06-04.
|
||||
|
||||
## Why
|
||||
## Current approach
|
||||
|
||||
After the [2026-04-21 thermal crash](../investigations/2026-04-21-hubris-crash-loop.md), the old single-purpose `thermal-watch` (which lived in `dtoro/backup-library`) was extended into a general health monitor. Single sink: claudio-bot. The MQTT/REST push paths were ripped out 2026-04-21 (commit `82f0596` in `dtoro/claudio-monitor`) once HA started consuming PVE metrics directly.
|
||||
Two layers:
|
||||
|
||||
## Components
|
||||
1. **On-demand:** ask Hermes "how's the homelab?" or run `homelab health` — loads
|
||||
the `homelab-hardware-health` skill, checks hardware temps, LXC resources,
|
||||
service reachability, and apt/docker drift across all hosts.
|
||||
|
||||
- **Repo:** `dtoro/claudio-monitor`
|
||||
- **Checkout:** `/opt/claudio-monitor` on the PVE host
|
||||
- **No auto-deploy webhook yet** — push, then `/opt/claudio-monitor/scripts/deploy.sh` manually.
|
||||
- **Main loop:** `claudio-monitor.timer` (every 5 min) → `/opt/claudio-monitor/claudio-monitor.sh` → sources `lib/check-{thermal,host,lxc}.sh`.
|
||||
- **Module shape:** every check is a `check_XXX()` function in `lib/check-XXX.sh`, sourced by the orchestrator. To add: write `lib/check-new.sh` and add `. "$LIB/check-new.sh"` + one `check_new` call in `claudio-monitor.sh`.
|
||||
- **Thresholds:** `/etc/claudio-monitor/thresholds.env` (installed from `config/thresholds.env` — admin edits there are preserved by `deploy.sh`).
|
||||
- **Tokens:** `/etc/claudio-monitor/bot.token` (same value as `/etc/restic/bot.token`, copied by hand). Missing → Matrix delivery is silently skipped.
|
||||
- **State files:** `/var/lib/claudio-monitor/<check>-<key>.alert` and `.counter`. `notify.sh` ships two primitives: `maybe_alert` (warn/clear hysteresis) and `sticky_alert` (N-consecutive-ticks).
|
||||
2. **Cron watchdog:** `homelab-health-watchdog` runs every 15 minutes via Hermes
|
||||
cron. Silent when healthy. When thresholds breach, sends an actionable alert
|
||||
to Matrix (`@dtoro:avispero`) with options the user can reply to directly
|
||||
(e.g. "resize rootfs", "investigate", "snooze 24h"). Hermes takes action on
|
||||
the selected option via SSH.
|
||||
|
||||
## Per-LXC data
|
||||
Thresholds: LXC disk >80% warn/>90% critical, NVMe >60°C/>70°C, CPU >70°C/>80°C,
|
||||
apt >10/>50 upgradable, services down.
|
||||
|
||||
A single `pvesh get /cluster/resources --type vm --output-format json` per tick gives CPU/mem/disk for every LXC and VM. Same call HA's PVE integration uses — keep them aligned.
|
||||
Home Assistant pulls PVE metrics independently via its Proxmox VE integration
|
||||
(unaffected by this change).
|
||||
|
||||
## Matrix-side plugin
|
||||
## Legacy: claudio-monitor (deprecated 2026-06-04)
|
||||
|
||||
`plugins/monitor.py` in `dtoro/claudio-bot`. Registered in `/etc/claudio-bot/config.yaml` under `plugins:`. Slash commands:
|
||||
- `/monitor status`
|
||||
- `/monitor history [N]`
|
||||
- `/monitor clear <key>`
|
||||
The old system was a bash watchdog on hubris (`claudio-monitor.timer`, every 5
|
||||
min) that POSTed alerts to a Matrix bot (`@claudio:avispero`) via an IPC server
|
||||
on LXC 123:9090. All components decommissioned:
|
||||
|
||||
Tools (for NL queries via the LLM): `get_alerts`, `get_monitor_history`.
|
||||
| Component | Fate |
|
||||
|-----------|------|
|
||||
| LXC 123 (claudio-bot) | Destroyed 2026-06-04 |
|
||||
| `dtoro/claudio-bot` | Archived (read-only) on Gitea |
|
||||
| `dtoro/claudio-monitor` | Archived (read-only) on Gitea |
|
||||
| `claudio-monitor.timer` | Disabled on hubris |
|
||||
| `/opt/claudio-monitor/` | Still on hubris (cleanup pending) |
|
||||
| `/etc/claudio-monitor/` | Still on hubris (cleanup pending) |
|
||||
|
||||
Events posted to claudio-bot `/notify`:
|
||||
```json
|
||||
{"plugin": "monitor", "event": "critical|warning|info", "check": "thermal|host|lxc|apt|docker", "message": "...", "data": {...}}
|
||||
```
|
||||
`event:"info"` is the recovery / clear event. The plugin keys active alerts by `_alert_key(check, data)`.
|
||||
|
||||
## Current thresholds & active alerts
|
||||
|
||||
Active warnings at the time monitor was deployed (2026-04-21):
|
||||
- `lxc:103` ([paperless](../containers/103-paperless.md)) — disk 86.9%
|
||||
- `lxc:118` ([elementsynapse](../containers/118-elementsynapse.md)) — disk 86.8%
|
||||
|
||||
## Testing
|
||||
|
||||
- Force an alert: `NVME_WARN_TEMP_C=1 /opt/claudio-monitor/claudio-monitor.sh` (any threshold env override).
|
||||
- Clean state: `rm -f /var/lib/claudio-monitor/*.alert`.
|
||||
- Verify Matrix delivery: `pct exec 123 -- journalctl -u claudio-bot --since "30s ago" | grep "plugin=monitor"`.
|
||||
|
||||
## Not yet deployed (tracked in plan file)
|
||||
|
||||
- `check-apt.sh` + `check-docker.sh` + `claudio-monitor-updates.timer` (daily at 04:00) — apt upgradable counts per host/LXC, docker image digest drift on [apps (105)](../containers/105-apps.md).
|
||||
- Deploy webhook (`scripts/webhook/`) mirroring `dtoro/backup-library-deploy`.
|
||||
|
||||
## Related repos
|
||||
|
||||
- `dtoro/backup-library` — origin of the now-removed `thermal-watch.*`.
|
||||
- `dtoro/claudio-bot` — the `monitor` plugin lives at `plugins/monitor.py`.
|
||||
- `dtoro/haos-config` — empty placeholder created 2026-04-21 for HA `/config` once bootstrapped from inside HAOS via the SSH addon.
|
||||
For the full deprecation plan, see `plans/2026-06-04_130000-deprecate-claudio-bot.md`.
|
||||
|
||||
## Related pages
|
||||
- [Hubris host](../hosts/hubris.md)
|
||||
- [claudio-bot (123)](../containers/123-claudio-bot.md)
|
||||
- [HAOS VM (108)](../vms/108-haos.md)
|
||||
- [Backups (disabled)](backups.md)
|
||||
- [Investigation: 2026-04-21 crash loop](../investigations/2026-04-21-hubris-crash-loop.md)
|
||||
- [Homelab context distribution](homelab-context.md)
|
||||
|
||||
## Changelog
|
||||
|
||||
### 2026-06-04 — migrated to Hermes health watchdog
|
||||
claudio-monitor + claudio-bot IPC pipeline replaced by Hermes-native monitoring.
|
||||
On-demand `homelab health` via extended skill; 15-min cron watchdog with actionable
|
||||
Matrix alerts. LXC 123 destroyed, repos archived.
|
||||
|
||||
### 2026-04-28 — wiki entry created
|
||||
Initial documentation.
|
||||
|
||||
### 2026-04-21 — claudio-monitor stood up; thermal-watch removed
|
||||
General health monitor with per-LXC checks. MQTT/REST push paths ripped out (commit `82f0596`) because HA gets PVE metrics directly. Matrix-side `monitor` plugin added in claudio-bot commit `e56da25`.
|
||||
General health monitor with per-LXC checks. MQTT/REST push paths ripped out.
|
||||
|
||||
@@ -131,7 +131,6 @@ are managed by `ssh/deploy-keys.sh`. SSH user is `root`.
|
||||
| 120 | mule-images | `192.168.8.136` | photo-management |
|
||||
| 121 | caddy | `192.168.8.175` | reverse-proxy |
|
||||
| 122 | arriman | `192.168.8.132` | arr-stack |
|
||||
| 123 | claudio-bot | `192.168.8.230` | matrix-agent |
|
||||
| 126 | plato | `192.168.8.190` | app |
|
||||
|
||||
### Workstations
|
||||
|
||||
@@ -272,16 +272,7 @@ hosts:
|
||||
mounts:
|
||||
- /mnt/library
|
||||
age_pubkey: ''
|
||||
claudio-bot:
|
||||
kind: lxc
|
||||
pve_id: 123
|
||||
host: hubris
|
||||
os: linux
|
||||
role: matrix-agent
|
||||
lan_ip: 192.168.8.230
|
||||
notes:
|
||||
- Reads /opt/homelab-context/ on startup
|
||||
age_pubkey: age1xmkeq968areza2necqyq0065dpeegngzyr6dhagh0n6pl33lccfqe5mqn9
|
||||
# 123 (claudio-bot) — destroyed 2026-06-04, replaced by Hermes Agent
|
||||
authentik:
|
||||
kind: lxc
|
||||
pve_id: 124
|
||||
|
||||
318
plans/2026-06-04_130000-deprecate-claudio-bot.md
Normal file
318
plans/2026-06-04_130000-deprecate-claudio-bot.md
Normal file
@@ -0,0 +1,318 @@
|
||||
# Deprecate claudio-bot (LXC 123) — Hermes Agent now serves as control plane
|
||||
|
||||
## Goal
|
||||
|
||||
Phase out the **claudio-bot ecosystem** (LXC 123, claudio-monitor, IPC server)
|
||||
now that Hermes Agent is configured and running on mac-mini. Hermes replaces
|
||||
claudio-bot's role as the homelab control plane — accepting commands, relaying
|
||||
alerts, and answering queries about infrastructure state.
|
||||
|
||||
## What is claudio-bot
|
||||
|
||||
A three-component ecosystem deployed 2026-04-20:
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| **claudio-bot** | LXC 123 (192.168.8.230) | Matrix bot `@claudio:avispero` — slash commands, NL queries, infra notifications |
|
||||
| **claudio-monitor** | hubris (PVE host) | 5-min watchdog → sends alerts to claudio-bot's IPC server |
|
||||
| **IPC server** | LXC 123:9090 | `/notify`, `/propose`, `/status` endpoints — used by monitor + backup wrapper |
|
||||
|
||||
Dependencies:
|
||||
- Matrix synapse on LXC 118 (elementsynapse) — claudio-bot connects to `192.168.8.239:8008`
|
||||
- Gitea auto-deploy webhook → LXC 123:9797 (pipeline `dtoro/claudio-bot`)
|
||||
- LLM backend: LM Studio on mac-mini (192.168.8.174:1234) or Anthropic API
|
||||
- Secrets at `/etc/claudio-bot/` (config.yaml, matrix.token, anthropic.key, lmstudio.key, ipc.token)
|
||||
|
||||
## What Hermes replaces
|
||||
|
||||
| claudio-bot function | Hermes replacement |
|
||||
|---------------------|-------------------|
|
||||
| Matrix slash commands | Hermes CLI + `send_message` for outbound notifications |
|
||||
| NL infrastructure queries | Hermes with `homelab` CLI + MCP tools (already working) |
|
||||
| Monitor alert relay | Hermes-native: (a) on-demand `homelab health` via extended skill, (b) cronjob every 15 min with actionable alerts (not just passive notifications) |
|
||||
| IPC server (port 9090) | **Retired** — Hermes doesn't need a separate IPC layer; monitoring runs directly via SSH + `send_message` |
|
||||
| LLM backend | Already handled by Hermes (OpenRouter → deepseek/deepseek-v4-pro) |
|
||||
| Auto-deploy webhook (9797) | **Retired** — repo archived |
|
||||
|
||||
## Full blast radius (files referencing claudio)
|
||||
|
||||
### Homelab-context repo (`/opt/homelab-context/`)
|
||||
|
||||
#### Core topology
|
||||
- `inventory.yaml:275` — `claudio-bot:` host entry (LXC 123, matrix-agent)
|
||||
- `hosts/claudio-bot.yaml` — auto-generated host file (delete)
|
||||
- `.sops.yaml:47` — placeholder comment listing claudio-bot/claudio-monitor
|
||||
|
||||
#### Documentation (containers)
|
||||
- `containers/123-claudio-bot.md` — full container doc
|
||||
- `containers/index.md:17` — LXC table row
|
||||
- `containers/118-elementsynapse.md:3,36,39` — `@claudio:avispero` reference, disk reading, direct connection note
|
||||
- `containers/103-paperless.md:27` — disk reading from claudio-monitor
|
||||
|
||||
#### Documentation (infrastructure)
|
||||
- `infrastructure/monitoring.md` — **ENTIRE FILE** is about claudio-monitor → claudio-bot pipeline
|
||||
- `infrastructure/backups.md:74,94,98` — IPC notify, thermal monitoring moved out, related link
|
||||
- `infrastructure/auto-deploy.md:23,41,50,61,120,138-139` — Gitea webhook target, pipeline table, port 9797
|
||||
- `infrastructure/ssh-access.md:134` — LXC table row
|
||||
- `infrastructure/homelab-context.md:4,16,143` — client list, integration note
|
||||
|
||||
#### Other
|
||||
- `operations/commands.md:17` — `pvesh` bulk query note (passive reference, can stay)
|
||||
- `README.md:32,40,64` — LXC table, monitoring link, tracked configs mention
|
||||
- `mcp/deploy/webhook/webhook.py:5-6` — port comment (9797, 9799)
|
||||
- `hosts/hubris.md:57,96` — `/opt/claudio-monitor` checkout, LXC 109 cleanup note (passive)
|
||||
- `vms/108-haos.md:30` — claudio-monitor MQTT removal note (historical, can stay)
|
||||
- `investigations/2026-04-21-hubris-crash-loop.md:27` — historical reference (keep)
|
||||
|
||||
### External repos (Gitea)
|
||||
- `dtoro/claudio-bot` — the bot itself
|
||||
- `dtoro/claudio-monitor` — the watchdog scripts
|
||||
|
||||
### External state (hubris)
|
||||
- `/opt/claudio-bot/` — bot checkout (LXC 123)
|
||||
- `/opt/claudio-monitor/` — monitor checkout (hubris host)
|
||||
- `/etc/claudio-bot/` — secrets directory
|
||||
- `/etc/claudio-deploy/` — deploy git credentials
|
||||
- `/etc/claudio-monitor/` — thresholds + bot.token
|
||||
- `/var/lib/claudio-monitor/` — state files (.alert, .counter)
|
||||
- `claudio-bot.service` — systemd unit (LXC 123)
|
||||
- `claudio-monitor.timer` + `.service` — systemd timer (hubris)
|
||||
- Gitea webhook id 6 (claudio-bot-deploy) + webhook for claudio-monitor (9799) on apps LXC 105
|
||||
|
||||
### External state (workstations)
|
||||
- Nothing on mac-mini, republic-laptop, or ludo-mini references claudio directly
|
||||
|
||||
## Phased deprecation plan
|
||||
|
||||
### Phase 1 — Stop & verify (low-risk, reversible)
|
||||
|
||||
**Goal**: Stop claudio services, verify nothing breaks, keep LXC 123 paused (not destroyed).
|
||||
|
||||
1. **Stop claudio-monitor on hubris**
|
||||
```bash
|
||||
ssh hubris systemctl disable --now claudio-monitor.timer
|
||||
```
|
||||
The 5-minute watchdog stops. Home Assistant still gets PVE metrics independently via its own integration — no alerting gap there.
|
||||
|
||||
2. **Stop claudio-bot on LXC 123**
|
||||
```bash
|
||||
ssh claudio-bot systemctl stop claudio-bot
|
||||
```
|
||||
The Matrix bot goes offline. Nobody can send slash commands to `@claudio:avispero`.
|
||||
|
||||
3. **Pause LXC 123** (not destroy yet)
|
||||
```bash
|
||||
ssh hubris pct shutdown 123
|
||||
```
|
||||
IPC server (port 9090) and deploy webhook (port 9797) become unreachable.
|
||||
|
||||
4. **Verify no breakage** (run for 24-48h)
|
||||
- Check Gitea webhook logs — the `dtoro/claudio-bot` webhook will 404 on pushes (non-fatal)
|
||||
- Check Caddy logs — no upstream errors for claudio-related paths (there shouldn't be any — no public hostname)
|
||||
- Check DNS — no queries for claudio-bot.hubris.network (doesn't have a public hostname)
|
||||
- Confirm Hermes works normally for all homelab operations
|
||||
|
||||
### Phase 2 — Hermes-native health monitoring (replaces claudio-monitor)
|
||||
|
||||
**Goal**: Two capabilities that together exceed what claudio-monitor did:
|
||||
|
||||
1. **On-demand health check** — user asks "how's the homelab?" → Hermes runs a
|
||||
comprehensive check across all hosts/LXCs and reports results conversationally.
|
||||
2. **Cron-based watchdog** — runs every 15 minutes, checks thresholds, raises
|
||||
alarms with actionable options (not just passive alerts). User can respond
|
||||
and Hermes takes action directly.
|
||||
|
||||
**Key difference from claudio-monitor**: The old system was a bash script that
|
||||
POSTed JSON to an IPC server that a Matrix bot reformatted into a flat message.
|
||||
The Hermes-native approach gives the LLM the raw metrics and lets it reason about
|
||||
what's important before alerting — and crucially, the user can reply to an alert
|
||||
and Hermes can SSH into hubris to resize a rootfs, restart a service, or
|
||||
investigate deeper.
|
||||
|
||||
#### 2a. Extend `homelab-hardware-health` skill
|
||||
|
||||
The existing skill covers CPU/NVMe temps. Expand it to be the **single
|
||||
homelab health check skill** covering:
|
||||
|
||||
- **Hardware**: CPU temps (sysfs), NVMe/SATA SSD temps (smartctl), block device inventory
|
||||
- **LXC resources**: `pvesh get /cluster/resources --type vm --output-format json` → disk %, CPU %, RAM usage per container
|
||||
- **Service health**: Check key services respond (gitea:3000, caddy:80/443, synapse:8008, paperless, nextcloud, jellyfin)
|
||||
- **System drift**: `apt list --upgradable` counts per host, docker image pull digests on apps LXC 105
|
||||
|
||||
Thresholds (from old `/etc/claudio-monitor/thresholds.env`):
|
||||
|
||||
| Metric | Warning | Critical |
|
||||
|--------|---------|----------|
|
||||
| LXC disk usage | > 80% | > 90% |
|
||||
| NVMe temp | > 60°C | > 70°C |
|
||||
| CPU temp | > 70°C | > 80°C |
|
||||
| Apt upgradable | > 10 | > 50 |
|
||||
| Service down | any | 2+ services |
|
||||
|
||||
Update the skill file at `devops/homelab-hardware-health/SKILL.md` with:
|
||||
- Expanded trigger phrases ("check homelab health", "how's the lab", "health report")
|
||||
- LXC resource check commands
|
||||
- Service health check commands
|
||||
- Thresholds table
|
||||
- "If you find issues, present options to the user" instruction
|
||||
|
||||
#### 2b. Create Hermes cronjob `homelab-health-watchdog`
|
||||
|
||||
```bash
|
||||
hermes cron create \
|
||||
--schedule 'every 15m' \
|
||||
--name 'homelab-health-watchdog' \
|
||||
--prompt 'Load skill homelab-hardware-health. Run a full health check across all hosts (hubris + all LXCs). Check: (1) hardware temps, (2) LXC disk/CPU/RAM via pvesh, (3) service reachability for gitea, caddy, synapse, paperless, nextcloud, jellyfin, (4) apt upgradable counts. Compare against thresholds in the skill. If nothing is critical, say nothing (no message = healthy). If anything breaches thresholds, send a concise alert with: what broke, current value vs threshold, and 2-3 actionable options the user can reply to. DO NOT send routine "all clear" messages.'
|
||||
```
|
||||
|
||||
This runs every 15 minutes (Hermes cron is LLM-driven, heavier than a bash
|
||||
script — 15 min is appropriate vs claudio-monitor's 5 min). When thresholds
|
||||
are clean, the agent stays silent. When something breaches, the user gets a
|
||||
message like:
|
||||
|
||||
> ⚠️ LXC 103 (paperless) disk at 91% (> 90% critical)
|
||||
>
|
||||
> Options: [resize rootfs to 16 GiB] [investigate what's using space] [snooze 24h]
|
||||
|
||||
User picks an option → Hermes takes action (SSH into hubris, run `pct resize`,
|
||||
grow the filesystem inside the container).
|
||||
|
||||
#### 2c. Add `homelab-health` alias to `homelab` CLI
|
||||
|
||||
For quick on-demand checks from the terminal:
|
||||
|
||||
```bash
|
||||
homelab health # runs the skill inline via Hermes
|
||||
# or equivalently:
|
||||
hermes --prompt "Load homelab-hardware-health, run full health check"
|
||||
```
|
||||
|
||||
### Phase 3 — Documentation & inventory cleanup
|
||||
|
||||
**Goal**: Update homelab-context to reflect claudio deprecation, without destroying history.
|
||||
|
||||
#### Files to edit
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `inventory.yaml` | Mark `claudio-bot` entry as deprecated (comment + `status: deprecated`). Do NOT remove — keeps historical ID mapping. |
|
||||
| `containers/123-claudio-bot.md` | Add deprecation banner at top, changelog entry |
|
||||
| `containers/index.md` | Change status from "running" to "stopped (deprecated)" |
|
||||
| `infrastructure/monitoring.md` | Rewrite to document new Hermes-based monitoring. Keep claudio-monitor section as "legacy (deprecated 2026-06-04)". |
|
||||
| `infrastructure/auto-deploy.md` | Mark claudio-bot and claudio-monitor webhook entries as deprecated |
|
||||
| `infrastructure/backups.md` | Update notification section (IPC path no longer available). Note: backups themselves are still disabled — this is forward-looking. |
|
||||
| `infrastructure/ssh-access.md` | Mark LXC 123 as "offline (deprecated)" |
|
||||
| `infrastructure/homelab-context.md` | Remove claudio-bot from client list |
|
||||
| `containers/118-elementsynapse.md` | Note `@claudio:avispero` is decommissioned |
|
||||
| `containers/103-paperless.md` | Remove stale disk reading reference (already resolved — rootfs grew to 16 GiB 2026-05-15 per 118 doc) |
|
||||
| `README.md` | Update LXC table, remove monitoring link |
|
||||
| `hosts/claudio-bot.yaml` | Add deprecation banner (or delete if auto-generated — it is, so just update inventory) |
|
||||
| `.sops.yaml` | Remove claudio-bot from placeholder comment |
|
||||
|
||||
#### Files to leave untouched (historical)
|
||||
|
||||
| File | Reason |
|
||||
|------|--------|
|
||||
| `investigations/2026-04-21-hubris-crash-loop.md` | Historical investigation — claudio-monitor is factual context |
|
||||
| `vms/108-haos.md:30` | Historical note about MQTT removal — factual |
|
||||
| `mcp/deploy/webhook/webhook.py:5-6` | Comment only — harmless, keep for port mapping context |
|
||||
| `operations/commands.md:17` | Passive reference — harmless |
|
||||
|
||||
### Phase 4 — Purge (after confirmation window)
|
||||
|
||||
**Goal**: Full removal after confirming Phase 1-3 is stable for ≥2 weeks.
|
||||
|
||||
1. **Destroy LXC 123**
|
||||
```bash
|
||||
ssh hubris pct destroy 123 --purge
|
||||
```
|
||||
Frees 8 GiB rootfs, 512 MiB RAM, 1 core.
|
||||
|
||||
2. **Remove claudio-monitor from hubris**
|
||||
```bash
|
||||
ssh hubris rm -rf /opt/claudio-monitor /etc/claudio-monitor /var/lib/claudio-monitor
|
||||
ssh hubris systemctl disable claudio-monitor.timer # already done in phase 1
|
||||
```
|
||||
|
||||
3. **Archive Gitea repos**
|
||||
- Archive `dtoro/claudio-bot` (mark read-only in Gitea)
|
||||
- Archive `dtoro/claudio-monitor` (mark read-only in Gitea)
|
||||
|
||||
4. **Remove Gitea webhooks**
|
||||
- Remove claudio-bot-deploy webhook (id 6) from Gitea
|
||||
- Remove claudio-monitor webhook (9799) from apps LXC 105
|
||||
- Remove `192.168.8.230` from Gitea `app.ini` `ALLOWED_HOST_LIST`
|
||||
|
||||
5. **Final documentation cleanup**
|
||||
- Remove `containers/123-claudio-bot.md` (or move to `containers/_destroyed/`)
|
||||
- Remove `hosts/claudio-bot.yaml`
|
||||
- Remove claudio-bot row from `containers/index.md`
|
||||
- Update all cross-references (see blast radius above)
|
||||
|
||||
## Files changed by phase
|
||||
|
||||
### Phase 3 (documentation)
|
||||
- `/opt/homelab-context/inventory.yaml`
|
||||
- `/opt/homelab-context/containers/123-claudio-bot.md`
|
||||
- `/opt/homelab-context/containers/index.md`
|
||||
- `/opt/homelab-context/infrastructure/monitoring.md`
|
||||
- `/opt/homelab-context/infrastructure/auto-deploy.md`
|
||||
- `/opt/homelab-context/infrastructure/backups.md`
|
||||
- `/opt/homelab-context/infrastructure/ssh-access.md`
|
||||
- `/opt/homelab-context/infrastructure/homelab-context.md`
|
||||
- `/opt/homelab-context/containers/118-elementsynapse.md`
|
||||
- `/opt/homelab-context/containers/103-paperless.md`
|
||||
- `/opt/homelab-context/README.md`
|
||||
- `/opt/homelab-context/hosts/claudio-bot.yaml`
|
||||
- `/opt/homelab-context/.sops.yaml`
|
||||
|
||||
### Phase 4 (destruction)
|
||||
- Proxmox: LXC 123 destroyed
|
||||
- Gitea: `dtoro/claudio-bot` archived, `dtoro/claudio-monitor` archived
|
||||
- Gitea webhooks removed (apps LXC 105)
|
||||
- hubris: `/opt/claudio-monitor/`, `/etc/claudio-monitor/`, `/var/lib/claudio-monitor/` removed
|
||||
- `containers/123-claudio-bot.md` → moved to `_destroyed/` or deleted
|
||||
|
||||
## Risks & tradeoffs
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|-----------|
|
||||
| **Monitoring gap** — no alerts during Phase 1-2 transition | Phase 1 stops the old timer but Phase 2 builds the Hermes-native replacement immediately. If there's a gap, keep claudio-monitor.timer running until the Hermes cronjob is verified with a test alert. |
|
||||
| **Backup notifications** — the backup wrapper's IPC notify path is dead | Backups are already DISABLED. When re-enabled, wire them to the new Hermes monitoring pipeline instead. |
|
||||
| **Matrix `@claudio:avispero`** — users may still try to interact | The bot just won't respond. Could leave a notice in the Matrix room. |
|
||||
| **Gitea webhook failures** — pushes to `dtoro/claudio-bot` will 404 | Non-fatal. Archive the repo in Phase 4 to stop pushes entirely. |
|
||||
| **Losing monitoring history** — state files on hubris | Keep `/var/lib/claudio-monitor/` until Phase 4. No rush. |
|
||||
| **LM Studio dependency** — claudio-bot was using mac-mini's LM Studio | Hermes already uses OpenRouter. LM Studio on mac-mini can stay for other uses or be reclaimed. |
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Should the Matrix `@claudio:avispero` account be deactivated or left as-is?** It has no public hostname and was invite-only to a private room.
|
||||
|
||||
2. **Are you OK archiving `dtoro/claudio-bot` and `dtoro/claudio-monitor` on Gitea (read-only, preserved), or do you want them fully deleted?**
|
||||
|
||||
3. **Should LXC 123 be paused (reversible) or destroyed immediately?** Recommend pause for at least 2 weeks as a safety net.
|
||||
|
||||
4. **Which messaging platform should the health watchdog deliver alerts to?** Currently Hermes is connected via CLI (mac-mini). To get push alerts, we need a `send_message` target — Telegram, Discord, or Matrix are available. Which do you prefer?
|
||||
|
||||
## Verification
|
||||
|
||||
After Phase 1:
|
||||
- `ssh hubris systemctl is-active claudio-monitor.timer` → `inactive`
|
||||
- `ssh claudio-bot systemctl is-active claudio-bot` → `inactive` (or unreachable if LXC stopped)
|
||||
- `ssh hubris pct status 123` → `stopped`
|
||||
- `curl -s http://192.168.8.230:9090/status` → connection refused (IPC dead)
|
||||
|
||||
After Phase 2:
|
||||
- `hermes cron list` shows `homelab-health-watchdog` running every 15 min
|
||||
- "check homelab health" → Hermes runs comprehensive check, reports temps + LXC resources + service status + apt drift
|
||||
- Force an alert: `NVME_WARN_TEMP_C=1` on hubris → watchdog catches it on next 15-min tick → alert arrives via Hermes messaging with actionable options
|
||||
- Alert format includes 2-3 options user can reply to, not just passive notification
|
||||
|
||||
After Phase 3:
|
||||
- All cross-references in homelab-context point to deprecated/removed status
|
||||
- `grep -r 'claudio' /opt/homelab-context/ --include='*.md' --include='*.yaml' | grep -v '_destroyed\|investigations\|changelog'` → only historical references remain
|
||||
|
||||
After Phase 4:
|
||||
- `ssh hubris pct list | grep 123` → no output
|
||||
- `ssh hubris ls /opt/claudio-monitor` → no such file
|
||||
- Gitea `dtoro/claudio-bot` → archived badge
|
||||
Reference in New Issue
Block a user