oikos/plans/2026-06-04_130000-deprecate-claudio-bot.md

# Deprecate claudio-bot (LXC 123) — Hermes Agent now serves as control plane

## Goal

Phase out the **claudio-bot ecosystem** (LXC 123, claudio-monitor, IPC server)
now that Hermes Agent is configured and running on mac-mini. Hermes replaces
claudio-bot's role as the homelab control plane — accepting commands, relaying
alerts, and answering queries about infrastructure state.

## What is claudio-bot

A three-component ecosystem deployed 2026-04-20:

| Component | Location | Purpose |
|-----------|----------|---------|
| **claudio-bot** | LXC 123 (192.168.8.230) | Matrix bot `@claudio:avispero` — slash commands, NL queries, infra notifications |
| **claudio-monitor** | hubris (PVE host) | 5-min watchdog → sends alerts to claudio-bot's IPC server |
| **IPC server** | LXC 123:9090 | `/notify`, `/propose`, `/status` endpoints — used by monitor + backup wrapper |

Dependencies:
- Matrix synapse on LXC 118 (elementsynapse) — claudio-bot connects to `192.168.8.239:8008`
- Gitea auto-deploy webhook → LXC 123:9797 (pipeline `dtoro/claudio-bot`)
- LLM backend: LM Studio on mac-mini (192.168.8.174:1234) or Anthropic API
- Secrets at `/etc/claudio-bot/` (config.yaml, matrix.token, anthropic.key, lmstudio.key, ipc.token)

## What Hermes replaces

| claudio-bot function | Hermes replacement |
|---------------------|-------------------|
| Matrix slash commands | Hermes CLI + `send_message` for outbound notifications |
| NL infrastructure queries | Hermes with `homelab` CLI + MCP tools (already working) |
| Monitor alert relay | Hermes-native: (a) on-demand `homelab health` via extended skill, (b) cronjob every 15 min with actionable alerts (not just passive notifications) |
| IPC server (port 9090) | **Retired** — Hermes doesn't need a separate IPC layer; monitoring runs directly via SSH + `send_message` |
| LLM backend | Already handled by Hermes (OpenRouter → deepseek/deepseek-v4-pro) |
| Auto-deploy webhook (9797) | **Retired** — repo archived |

## Full blast radius (files referencing claudio)

### Homelab-context repo (`/opt/homelab-context/`)

#### Core topology
- `inventory.yaml:275` — `claudio-bot:` host entry (LXC 123, matrix-agent)
- `hosts/claudio-bot.yaml` — auto-generated host file (delete)
- `.sops.yaml:47` — placeholder comment listing claudio-bot/claudio-monitor

#### Documentation (containers)
- `containers/123-claudio-bot.md` — full container doc
- `containers/index.md:17` — LXC table row
- `containers/118-elementsynapse.md:3,36,39` — `@claudio:avispero` reference, disk reading, direct connection note
- `containers/103-paperless.md:27` — disk reading from claudio-monitor

#### Documentation (infrastructure)
- `infrastructure/monitoring.md` — **ENTIRE FILE** is about claudio-monitor → claudio-bot pipeline
- `infrastructure/backups.md:74,94,98` — IPC notify, thermal monitoring moved out, related link
- `infrastructure/auto-deploy.md:23,41,50,61,120,138-139` — Gitea webhook target, pipeline table, port 9797
- `infrastructure/ssh-access.md:134` — LXC table row
- `infrastructure/homelab-context.md:4,16,143` — client list, integration note

#### Other
- `operations/commands.md:17` — `pvesh` bulk query note (passive reference, can stay)
- `README.md:32,40,64` — LXC table, monitoring link, tracked configs mention
- `mcp/deploy/webhook/webhook.py:5-6` — port comment (9797, 9799)
- `hosts/hubris.md:57,96` — `/opt/claudio-monitor` checkout, LXC 109 cleanup note (passive)
- `vms/108-haos.md:30` — claudio-monitor MQTT removal note (historical, can stay)
- `investigations/2026-04-21-hubris-crash-loop.md:27` — historical reference (keep)

### External repos (Gitea)
- `dtoro/claudio-bot` — the bot itself
- `dtoro/claudio-monitor` — the watchdog scripts

### External state (hubris)
- `/opt/claudio-bot/` — bot checkout (LXC 123)
- `/opt/claudio-monitor/` — monitor checkout (hubris host)
- `/etc/claudio-bot/` — secrets directory
- `/etc/claudio-deploy/` — deploy git credentials
- `/etc/claudio-monitor/` — thresholds + bot.token
- `/var/lib/claudio-monitor/` — state files (.alert, .counter)
- `claudio-bot.service` — systemd unit (LXC 123)
- `claudio-monitor.timer` + `.service` — systemd timer (hubris)
- Gitea webhook id 6 (claudio-bot-deploy) + webhook for claudio-monitor (9799) on apps LXC 105

### External state (workstations)
- Nothing on mac-mini, republic-laptop, or ludo-mini references claudio directly

## Phased deprecation plan

### Phase 1 — Stop & verify (low-risk, reversible)

**Goal**: Stop claudio services, verify nothing breaks, keep LXC 123 paused (not destroyed).

1. **Stop claudio-monitor on hubris**
   ```bash
   ssh hubris systemctl disable --now claudio-monitor.timer
   ```
   The 5-minute watchdog stops. Home Assistant still gets PVE metrics independently via its own integration — no alerting gap there.

2. **Stop claudio-bot on LXC 123**
   ```bash
   ssh claudio-bot systemctl stop claudio-bot
   ```
   The Matrix bot goes offline. Nobody can send slash commands to `@claudio:avispero`.

3. **Pause LXC 123** (not destroy yet)
   ```bash
   ssh hubris pct shutdown 123
   ```
   IPC server (port 9090) and deploy webhook (port 9797) become unreachable.

4. **Verify no breakage** (run for 24-48h)
   - Check Gitea webhook logs — the `dtoro/claudio-bot` webhook will 404 on pushes (non-fatal)
   - Check Caddy logs — no upstream errors for claudio-related paths (there shouldn't be any — no public hostname)
   - Check DNS — no queries for claudio-bot.hubris.network (doesn't have a public hostname)
   - Confirm Hermes works normally for all homelab operations

### Phase 2 — Hermes-native health monitoring (replaces claudio-monitor)

**Goal**: Two capabilities that together exceed what claudio-monitor did:

1. **On-demand health check** — user asks "how's the homelab?" → Hermes runs a
   comprehensive check across all hosts/LXCs and reports results conversationally.
2. **Cron-based watchdog** — runs every 15 minutes, checks thresholds, raises
   alarms with actionable options (not just passive alerts). User can respond
   and Hermes takes action directly.

**Key difference from claudio-monitor**: The old system was a bash script that
POSTed JSON to an IPC server that a Matrix bot reformatted into a flat message.
The Hermes-native approach gives the LLM the raw metrics and lets it reason about
what's important before alerting — and crucially, the user can reply to an alert
and Hermes can SSH into hubris to resize a rootfs, restart a service, or
investigate deeper.

#### 2a. Extend `homelab-hardware-health` skill

The existing skill covers CPU/NVMe temps. Expand it to be the **single
homelab health check skill** covering:

- **Hardware**: CPU temps (sysfs), NVMe/SATA SSD temps (smartctl), block device inventory
- **LXC resources**: `pvesh get /cluster/resources --type vm --output-format json` → disk %, CPU %, RAM usage per container
- **Service health**: Check key services respond (gitea:3000, caddy:80/443, synapse:8008, paperless, nextcloud, jellyfin)
- **System drift**: `apt list --upgradable` counts per host, docker image pull digests on apps LXC 105

Thresholds (from old `/etc/claudio-monitor/thresholds.env`):

| Metric | Warning | Critical |
|--------|---------|----------|
| LXC disk usage | > 80% | > 90% |
| NVMe temp | > 60°C | > 70°C |
| CPU temp | > 70°C | > 80°C |
| Apt upgradable | > 10 | > 50 |
| Service down | any | 2+ services |

Update the skill file at `devops/homelab-hardware-health/SKILL.md` with:
- Expanded trigger phrases ("check homelab health", "how's the lab", "health report")
- LXC resource check commands
- Service health check commands
- Thresholds table
- "If you find issues, present options to the user" instruction

#### 2b. Create Hermes cronjob `homelab-health-watchdog`

```bash
hermes cron create \
  --schedule 'every 15m' \
  --name 'homelab-health-watchdog' \
  --prompt 'Load skill homelab-hardware-health. Run a full health check across all hosts (hubris + all LXCs). Check: (1) hardware temps, (2) LXC disk/CPU/RAM via pvesh, (3) service reachability for gitea, caddy, synapse, paperless, nextcloud, jellyfin, (4) apt upgradable counts. Compare against thresholds in the skill. If nothing is critical, say nothing (no message = healthy). If anything breaches thresholds, send a concise alert with: what broke, current value vs threshold, and 2-3 actionable options the user can reply to. DO NOT send routine "all clear" messages.'
```

This runs every 15 minutes (Hermes cron is LLM-driven, heavier than a bash
script — 15 min is appropriate vs claudio-monitor's 5 min). When thresholds
are clean, the agent stays silent. When something breaches, the user gets a
message like:

> ⚠️ LXC 103 (paperless) disk at 91% (> 90% critical)
>
> Options: [resize rootfs to 16 GiB] [investigate what's using space] [snooze 24h]

User picks an option → Hermes takes action (SSH into hubris, run `pct resize`,
grow the filesystem inside the container).

#### 2c. Add `homelab-health` alias to `homelab` CLI

For quick on-demand checks from the terminal:

```bash
homelab health          # runs the skill inline via Hermes
# or equivalently:
hermes --prompt "Load homelab-hardware-health, run full health check"
```

### Phase 3 — Documentation & inventory cleanup

**Goal**: Update homelab-context to reflect claudio deprecation, without destroying history.

#### Files to edit

| File | Action |
|------|--------|
| `inventory.yaml` | Mark `claudio-bot` entry as deprecated (comment + `status: deprecated`). Do NOT remove — keeps historical ID mapping. |
| `containers/123-claudio-bot.md` | Add deprecation banner at top, changelog entry |
| `containers/index.md` | Change status from "running" to "stopped (deprecated)" |
| `infrastructure/monitoring.md` | Rewrite to document new Hermes-based monitoring. Keep claudio-monitor section as "legacy (deprecated 2026-06-04)". |
| `infrastructure/auto-deploy.md` | Mark claudio-bot and claudio-monitor webhook entries as deprecated |
| `infrastructure/backups.md` | Update notification section (IPC path no longer available). Note: backups themselves are still disabled — this is forward-looking. |
| `infrastructure/ssh-access.md` | Mark LXC 123 as "offline (deprecated)" |
| `infrastructure/homelab-context.md` | Remove claudio-bot from client list |
| `containers/118-elementsynapse.md` | Note `@claudio:avispero` is decommissioned |
| `containers/103-paperless.md` | Remove stale disk reading reference (already resolved — rootfs grew to 16 GiB 2026-05-15 per 118 doc) |
| `README.md` | Update LXC table, remove monitoring link |
| `hosts/claudio-bot.yaml` | Add deprecation banner (or delete if auto-generated — it is, so just update inventory) |
| `.sops.yaml` | Remove claudio-bot from placeholder comment |

#### Files to leave untouched (historical)

| File | Reason |
|------|--------|
| `investigations/2026-04-21-hubris-crash-loop.md` | Historical investigation — claudio-monitor is factual context |
| `vms/108-haos.md:30` | Historical note about MQTT removal — factual |
| `mcp/deploy/webhook/webhook.py:5-6` | Comment only — harmless, keep for port mapping context |
| `operations/commands.md:17` | Passive reference — harmless |

### Phase 4 — Purge (after confirmation window)

**Goal**: Full removal after confirming Phase 1-3 is stable for ≥2 weeks.

1. **Destroy LXC 123**
   ```bash
   ssh hubris pct destroy 123 --purge
   ```
   Frees 8 GiB rootfs, 512 MiB RAM, 1 core.

2. **Remove claudio-monitor from hubris**
   ```bash
   ssh hubris rm -rf /opt/claudio-monitor /etc/claudio-monitor /var/lib/claudio-monitor
   ssh hubris systemctl disable claudio-monitor.timer  # already done in phase 1
   ```

3. **Archive Gitea repos**
   - Archive `dtoro/claudio-bot` (mark read-only in Gitea)
   - Archive `dtoro/claudio-monitor` (mark read-only in Gitea)

4. **Remove Gitea webhooks**
   - Remove claudio-bot-deploy webhook (id 6) from Gitea
   - Remove claudio-monitor webhook (9799) from apps LXC 105
   - Remove `192.168.8.230` from Gitea `app.ini` `ALLOWED_HOST_LIST`

5. **Final documentation cleanup**
   - Remove `containers/123-claudio-bot.md` (or move to `containers/_destroyed/`)
   - Remove `hosts/claudio-bot.yaml`
   - Remove claudio-bot row from `containers/index.md`
   - Update all cross-references (see blast radius above)

## Files changed by phase

### Phase 3 (documentation)
- `/opt/homelab-context/inventory.yaml`
- `/opt/homelab-context/containers/123-claudio-bot.md`
- `/opt/homelab-context/containers/index.md`
- `/opt/homelab-context/infrastructure/monitoring.md`
- `/opt/homelab-context/infrastructure/auto-deploy.md`
- `/opt/homelab-context/infrastructure/backups.md`
- `/opt/homelab-context/infrastructure/ssh-access.md`
- `/opt/homelab-context/infrastructure/homelab-context.md`
- `/opt/homelab-context/containers/118-elementsynapse.md`
- `/opt/homelab-context/containers/103-paperless.md`
- `/opt/homelab-context/README.md`
- `/opt/homelab-context/hosts/claudio-bot.yaml`
- `/opt/homelab-context/.sops.yaml`

### Phase 4 (destruction)
- Proxmox: LXC 123 destroyed
- Gitea: `dtoro/claudio-bot` archived, `dtoro/claudio-monitor` archived
- Gitea webhooks removed (apps LXC 105)
- hubris: `/opt/claudio-monitor/`, `/etc/claudio-monitor/`, `/var/lib/claudio-monitor/` removed
- `containers/123-claudio-bot.md` → moved to `_destroyed/` or deleted

## Risks & tradeoffs

| Risk | Mitigation |
|------|-----------|
| **Monitoring gap** — no alerts during Phase 1-2 transition | Phase 1 stops the old timer but Phase 2 builds the Hermes-native replacement immediately. If there's a gap, keep claudio-monitor.timer running until the Hermes cronjob is verified with a test alert. |
| **Backup notifications** — the backup wrapper's IPC notify path is dead | Backups are already DISABLED. When re-enabled, wire them to the new Hermes monitoring pipeline instead. |
| **Matrix `@claudio:avispero`** — users may still try to interact | The bot just won't respond. Could leave a notice in the Matrix room. |
| **Gitea webhook failures** — pushes to `dtoro/claudio-bot` will 404 | Non-fatal. Archive the repo in Phase 4 to stop pushes entirely. |
| **Losing monitoring history** — state files on hubris | Keep `/var/lib/claudio-monitor/` until Phase 4. No rush. |
| **LM Studio dependency** — claudio-bot was using mac-mini's LM Studio | Hermes already uses OpenRouter. LM Studio on mac-mini can stay for other uses or be reclaimed. |

## Open questions

1. **Should the Matrix `@claudio:avispero` account be deactivated or left as-is?** It has no public hostname and was invite-only to a private room.

2. **Are you OK archiving `dtoro/claudio-bot` and `dtoro/claudio-monitor` on Gitea (read-only, preserved), or do you want them fully deleted?**

3. **Should LXC 123 be paused (reversible) or destroyed immediately?** Recommend pause for at least 2 weeks as a safety net.

4. **Which messaging platform should the health watchdog deliver alerts to?** Currently Hermes is connected via CLI (mac-mini). To get push alerts, we need a `send_message` target — Telegram, Discord, or Matrix are available. Which do you prefer?

## Verification

After Phase 1:
- `ssh hubris systemctl is-active claudio-monitor.timer` → `inactive`
- `ssh claudio-bot systemctl is-active claudio-bot` → `inactive` (or unreachable if LXC stopped)
- `ssh hubris pct status 123` → `stopped`
- `curl -s http://192.168.8.230:9090/status` → connection refused (IPC dead)

After Phase 2:
- `hermes cron list` shows `homelab-health-watchdog` running every 15 min
- "check homelab health" → Hermes runs comprehensive check, reports temps + LXC resources + service status + apt drift
- Force an alert: `NVME_WARN_TEMP_C=1` on hubris → watchdog catches it on next 15-min tick → alert arrives via Hermes messaging with actionable options
- Alert format includes 2-3 options user can reply to, not just passive notification

After Phase 3:
- All cross-references in homelab-context point to deprecated/removed status
- `grep -r 'claudio' /opt/homelab-context/ --include='*.md' --include='*.yaml' | grep -v '_destroyed\|investigations\|changelog'` → only historical references remain

After Phase 4:
- `ssh hubris pct list | grep 123` → no output
- `ssh hubris ls /opt/claudio-monitor` → no such file
- Gitea `dtoro/claudio-bot` → archived badge