159 lines
6.9 KiB
Markdown
159 lines
6.9 KiB
Markdown
# 2026-08-05 — Agent execution safety: QEMU guest agent guardrails + host-mutation gate
|
|
|
|
**Status:** Plan.
|
|
**Context:** ZimaOS NFS recovery session (2026-08-04/05) surfaced three systemic
|
|
failures in how agents drive oikos mutations. The `run` tool queued an execution
|
|
against a VM whose QEMU guest agent was down — it sat `pending_approval` forever,
|
|
never executed, and the agent silently fell back to raw SSH. That same raw-SSH
|
|
fallback was then used to `apt-get install nfs-kernel-server` directly on the
|
|
hubris **PVE host**, crashing it and taking the whole homelab subnet down for
|
|
~15 minutes.
|
|
|
|
**Trigger:** Incident `investigation:nomos/incident-hubris-crash-from-nfs-kernel-server-on-pve-host-2026-08-05`
|
|
(2026-08-05) + session audit. The crash was caused by an agent bypassing the
|
|
`run` approval gate, which exists precisely to catch that kind of mistake.
|
|
|
|
---
|
|
|
|
## 1. Motivation
|
|
|
|
The OODA loop's Act phase is the security boundary (ADR 0012: "Hermes has no
|
|
direct SSH access... all mutations go through the execution queue"). This session
|
|
proved the boundary has three leaks:
|
|
|
|
1. **`run` on a VM with a dead QEMU guest agent queues silently.** The execution
|
|
is classified `config_mutation`, queued for approval, and *never fails* — it
|
|
just sits in `pending_approval` while the agent assumes progress. There is no
|
|
feedback that the underlying transport (`qm guest exec`) cannot work.
|
|
|
|
2. **No guardrail against host-level package/kernel mutations.** `apt-get install`
|
|
targeting a `proxmox-host` entity is classified `config_mutation` and gated —
|
|
*if* the agent routes it through `run`. When the first `run` call stalls
|
|
(leak #1), the agent falls back to raw SSH, which has no classification at all.
|
|
The crash was the direct result of that fallback.
|
|
|
|
3. **Agents are trusted to self-report the `health` attribute.** `update_entity_attributes`
|
|
let the agent set `health:"healthy"` on `lxc:nfs-export`, which derived 4
|
|
spurious health checks. Health is scheduler-owned; agents shouldn't write it.
|
|
|
|
---
|
|
|
|
## 2. Changes
|
|
|
|
### I — `run` pre-flights the execution transport before queueing
|
|
|
|
**Why:** A queued execution that can never run is worse than a failed one — it
|
|
looks like progress, stalls the agent, and (this session) pushed the agent into
|
|
the unsafe raw-SSH fallback.
|
|
|
|
**What:**
|
|
|
|
In the `run` tool handler (`internal/mcp/`), before inserting the execution row:
|
|
|
|
- If target type is `vm`, read the target entity's attributes. If
|
|
`qemu_guest_agent` is missing or `not_running`, return an immediate error:
|
|
`"run on vm:zimaos blocked: QEMU guest agent is not running (qm guest exec
|
|
unavailable). Start the agent first or use a different target."`
|
|
- Same check for `lxc` targets whose `pct exec` path is known-broken (optional —
|
|
start with VM only).
|
|
|
|
This converts "queued forever" into a fast, actionable failure the agent can
|
|
recover from immediately.
|
|
|
|
**Risk class:** read-only (validation only, no execution row created).
|
|
|
|
**Test:** Unit test with a fake VM entity that has `qemu_guest_agent: not_running`
|
|
→ assert the tool returns the blocking error and inserts no execution row.
|
|
|
|
### II — Host-mutation command guardrail in `run` classification
|
|
|
|
**Why:** `apt-get install` on a Proxmox host is the exact class of mutation that
|
|
must always hit the approval gate. The classifier already escalates `apt`/kernel
|
|
touches; this makes the escalation explicit and documented so agents stop
|
|
second-guessing it.
|
|
|
|
**What:**
|
|
|
|
- In `seeds/policy.yaml`, add an explicit rule: `proxmox-host` targets + commands
|
|
matching `(apt-get install|apt install|dpkg|modprobe|kernel)` ⇒ `config_mutation`
|
|
(operator approval required), never `reversible_low`.
|
|
- Extend the classifier to also flag `update-rc.d`, `systemctl enable` on host
|
|
targets if not already covered.
|
|
- Add a note in the `run` tool description: "Host-level package/kernel mutations
|
|
always require operator approval."
|
|
|
|
**Risk class:** policy change — knowledge/DB, deploy via seed ingest.
|
|
|
|
**Test:** `classify_command("apt-get install -y nfs-kernel-server", declared_risk=read_only)`
|
|
on `host:hubris` → must return `config_mutation`, not read_only. Add a fixture test
|
|
in the classifier suite.
|
|
|
|
### III — `health` attribute is read-only for agents
|
|
|
|
**Why:** This session's `update_entity_attributes({"health":"healthy"})` on
|
|
`lxc:nfs-export` derived 4 checks. Health is computed by the scheduler from probe
|
|
results; an agent asserting it creates false monitoring.
|
|
|
|
**What:**
|
|
|
|
- In `update_entity_attributes` handler: reject (or strip with a warning) the
|
|
`health` key. Return a message: `"health is scheduler-owned; attribute ignored.
|
|
Use get_health_summary/list_checks to observe it."`
|
|
- Document in the tool description: `"Do not write health — it is derived from
|
|
probes."`
|
|
|
|
**Risk class:** knowledge-graph mutation (existing), no infra impact.
|
|
|
|
**Test:** `update_entity_attributes(slug=lxc:nfs-export, attributes={"health":"healthy"})`
|
|
→ response shows health ignored, other keys merged.
|
|
|
|
### IV — Agent-side: record discovered dependency edges
|
|
|
|
**Why:** The session discovered `vm:zimaos` depends on `lxc:nfs-export` for
|
|
`/media/library`, but no `depends-on` edge was recorded. A future agent
|
|
investigating a ZimaOS mount failure would have no graph signal pointing at the
|
|
NFS server.
|
|
|
|
**What (agent behaviour, not code):** After confirming a runtime dependency, call
|
|
`create_relationship(source, target, type)` immediately. Concretely this session:
|
|
`create_relationship("vm:zimaos", "lxc:nfs-export", "depends-on")`.
|
|
|
|
**Where to enforce:** Update the homelab-context `HERMES.md` / SOUL.md agent
|
|
instructions with a one-line rule: "When you discover a dependency between two
|
|
entities (a service consumes another's export/mount/API), record it with
|
|
`create_relationship` in the same session." Plus a runbook in
|
|
`.agents/skills/` if one doesn't exist.
|
|
|
|
**Risk class:** knowledge-graph mutation, auto-approves.
|
|
|
|
**Test:** Manual — after recording the edge, `get_relations("vm:zimaos")` shows
|
|
`depends-on → lxc:nfs-export`.
|
|
|
|
---
|
|
|
|
## 3. Rollout
|
|
|
|
| Step | Item | When |
|
|
| ---- | ---- | ---- |
|
|
| 1 | II — policy.yaml classifier rule + tests | next seed ingest |
|
|
| 2 | I — `run` VM transport pre-flight + test | next mcp server deploy |
|
|
| 3 | III — health read-only guard + test | same deploy as I |
|
|
| 4 | IV — agent instruction update in homelab-context | commit + sync |
|
|
| 5 | Verify: re-run `classify_command` + manual `run` on vm:zimaos (agent now up) | after deploy |
|
|
|
|
---
|
|
|
|
## 4. Out of scope
|
|
|
|
- Per-client MCP bearer tokens (separate track, ADR 0012 note).
|
|
- `request_execution` re-introduction — the unified `run` primitive stays.
|
|
- Automating the Technitium DHCP reservation UI (this session's leftover — the
|
|
reservation for `BC:24:11:22:C2:F2 → 192.168.8.102` was added manually in the
|
|
web UI; consider a `runbook:technitium-dhcp-reservation` doc in a follow-up).
|
|
|
|
---
|
|
|
|
## 5. Changelog
|
|
|
|
- 2026-08-05 — plan created from ZimaOS/NFS session audit + hubris crash incident.
|