diff --git a/plans/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md b/plans/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md new file mode 100644 index 0000000..9cd1dd2 --- /dev/null +++ b/plans/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md @@ -0,0 +1,158 @@ +# 2026-08-05 — Agent execution safety: QEMU guest agent guardrails + host-mutation gate + +**Status:** Plan. +**Context:** ZimaOS NFS recovery session (2026-08-04/05) surfaced three systemic +failures in how agents drive oikos mutations. The `run` tool queued an execution +against a VM whose QEMU guest agent was down — it sat `pending_approval` forever, +never executed, and the agent silently fell back to raw SSH. That same raw-SSH +fallback was then used to `apt-get install nfs-kernel-server` directly on the +hubris **PVE host**, crashing it and taking the whole homelab subnet down for +~15 minutes. + +**Trigger:** Incident `investigation:nomos/incident-hubris-crash-from-nfs-kernel-server-on-pve-host-2026-08-05` +(2026-08-05) + session audit. The crash was caused by an agent bypassing the +`run` approval gate, which exists precisely to catch that kind of mistake. + +--- + +## 1. Motivation + +The OODA loop's Act phase is the security boundary (ADR 0012: "Hermes has no +direct SSH access... all mutations go through the execution queue"). This session +proved the boundary has three leaks: + +1. **`run` on a VM with a dead QEMU guest agent queues silently.** The execution + is classified `config_mutation`, queued for approval, and *never fails* — it + just sits in `pending_approval` while the agent assumes progress. There is no + feedback that the underlying transport (`qm guest exec`) cannot work. + +2. **No guardrail against host-level package/kernel mutations.** `apt-get install` + targeting a `proxmox-host` entity is classified `config_mutation` and gated — + *if* the agent routes it through `run`. When the first `run` call stalls + (leak #1), the agent falls back to raw SSH, which has no classification at all. + The crash was the direct result of that fallback. + +3. **Agents are trusted to self-report the `health` attribute.** `update_entity_attributes` + let the agent set `health:"healthy"` on `lxc:nfs-export`, which derived 4 + spurious health checks. Health is scheduler-owned; agents shouldn't write it. + +--- + +## 2. Changes + +### I — `run` pre-flights the execution transport before queueing + +**Why:** A queued execution that can never run is worse than a failed one — it +looks like progress, stalls the agent, and (this session) pushed the agent into +the unsafe raw-SSH fallback. + +**What:** + +In the `run` tool handler (`internal/mcp/`), before inserting the execution row: + +- If target type is `vm`, read the target entity's attributes. If + `qemu_guest_agent` is missing or `not_running`, return an immediate error: + `"run on vm:zimaos blocked: QEMU guest agent is not running (qm guest exec + unavailable). Start the agent first or use a different target."` +- Same check for `lxc` targets whose `pct exec` path is known-broken (optional — + start with VM only). + +This converts "queued forever" into a fast, actionable failure the agent can +recover from immediately. + +**Risk class:** read-only (validation only, no execution row created). + +**Test:** Unit test with a fake VM entity that has `qemu_guest_agent: not_running` +→ assert the tool returns the blocking error and inserts no execution row. + +### II — Host-mutation command guardrail in `run` classification + +**Why:** `apt-get install` on a Proxmox host is the exact class of mutation that +must always hit the approval gate. The classifier already escalates `apt`/kernel +touches; this makes the escalation explicit and documented so agents stop +second-guessing it. + +**What:** + +- In `seeds/policy.yaml`, add an explicit rule: `proxmox-host` targets + commands + matching `(apt-get install|apt install|dpkg|modprobe|kernel)` ⇒ `config_mutation` + (operator approval required), never `reversible_low`. +- Extend the classifier to also flag `update-rc.d`, `systemctl enable` on host + targets if not already covered. +- Add a note in the `run` tool description: "Host-level package/kernel mutations + always require operator approval." + +**Risk class:** policy change — knowledge/DB, deploy via seed ingest. + +**Test:** `classify_command("apt-get install -y nfs-kernel-server", declared_risk=read_only)` +on `host:hubris` → must return `config_mutation`, not read_only. Add a fixture test +in the classifier suite. + +### III — `health` attribute is read-only for agents + +**Why:** This session's `update_entity_attributes({"health":"healthy"})` on +`lxc:nfs-export` derived 4 checks. Health is computed by the scheduler from probe +results; an agent asserting it creates false monitoring. + +**What:** + +- In `update_entity_attributes` handler: reject (or strip with a warning) the + `health` key. Return a message: `"health is scheduler-owned; attribute ignored. + Use get_health_summary/list_checks to observe it."` +- Document in the tool description: `"Do not write health — it is derived from + probes."` + +**Risk class:** knowledge-graph mutation (existing), no infra impact. + +**Test:** `update_entity_attributes(slug=lxc:nfs-export, attributes={"health":"healthy"})` +→ response shows health ignored, other keys merged. + +### IV — Agent-side: record discovered dependency edges + +**Why:** The session discovered `vm:zimaos` depends on `lxc:nfs-export` for +`/media/library`, but no `depends-on` edge was recorded. A future agent +investigating a ZimaOS mount failure would have no graph signal pointing at the +NFS server. + +**What (agent behaviour, not code):** After confirming a runtime dependency, call +`create_relationship(source, target, type)` immediately. Concretely this session: +`create_relationship("vm:zimaos", "lxc:nfs-export", "depends-on")`. + +**Where to enforce:** Update the homelab-context `HERMES.md` / SOUL.md agent +instructions with a one-line rule: "When you discover a dependency between two +entities (a service consumes another's export/mount/API), record it with +`create_relationship` in the same session." Plus a runbook in +`.agents/skills/` if one doesn't exist. + +**Risk class:** knowledge-graph mutation, auto-approves. + +**Test:** Manual — after recording the edge, `get_relations("vm:zimaos")` shows +`depends-on → lxc:nfs-export`. + +--- + +## 3. Rollout + +| Step | Item | When | +| ---- | ---- | ---- | +| 1 | II — policy.yaml classifier rule + tests | next seed ingest | +| 2 | I — `run` VM transport pre-flight + test | next mcp server deploy | +| 3 | III — health read-only guard + test | same deploy as I | +| 4 | IV — agent instruction update in homelab-context | commit + sync | +| 5 | Verify: re-run `classify_command` + manual `run` on vm:zimaos (agent now up) | after deploy | + +--- + +## 4. Out of scope + +- Per-client MCP bearer tokens (separate track, ADR 0012 note). +- `request_execution` re-introduction — the unified `run` primitive stays. +- Automating the Technitium DHCP reservation UI (this session's leftover — the + reservation for `BC:24:11:22:C2:F2 → 192.168.8.102` was added manually in the + web UI; consider a `runbook:technitium-dhcp-reservation` doc in a follow-up). + +--- + +## 5. Changelog + +- 2026-08-05 — plan created from ZimaOS/NFS session audit + hubris crash incident. diff --git a/plans/index.md b/plans/index.md index 2260f7b..1fd4e41 100644 --- a/plans/index.md +++ b/plans/index.md @@ -20,6 +20,7 @@ went sideways, open an investigation. | 2026-07-20 | [Mascot physics/window-interaction audit](2026-07-20-mascot-physics-audit.md) | P0–P2 implemented; P3 ("cool stuff") ideas open | | 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready | | 2026-08-04 | [Hermes MCP client integration](done/2026-08-04-hermes-mcp-client-integration.md) | Done — deployed | +| 2026-08-05 | [Agent execution safety: QEMU guest agent gate + host-mutation guard](2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md) | Planned | ## Done