oikos/plans/done/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md

2026-08-05 — Agent execution safety: QEMU guest agent guardrails + host-mutation gate

Status: Plan. Context: ZimaOS NFS recovery session (2026-08-04/05) surfaced three systemic failures in how agents drive oikos mutations. The run tool queued an execution against a VM whose QEMU guest agent was down — it sat pending_approval forever, never executed, and the agent silently fell back to raw SSH. That same raw-SSH fallback was then used to apt-get install nfs-kernel-server directly on the hubris PVE host, crashing it and taking the whole homelab subnet down for ~15 minutes.

Trigger: Incident investigation:nomos/incident-hubris-crash-from-nfs-kernel-server-on-pve-host-2026-08-05 (2026-08-05) + session audit. The crash was caused by an agent bypassing the run approval gate, which exists precisely to catch that kind of mistake.

---

1. Motivation

The OODA loop's Act phase is the security boundary (ADR 0012: "Hermes has no direct SSH access... all mutations go through the execution queue"). This session proved the boundary has three leaks:

1. run on a VM with a dead QEMU guest agent queues silently. The execution is classified config_mutation, queued for approval, and never fails — it just sits in pending_approval while the agent assumes progress. There is no feedback that the underlying transport (qm guest exec) cannot work.

2. No guardrail against host-level package/kernel mutations. apt-get install targeting a proxmox-host entity is classified config_mutation and gated — if the agent routes it through run. When the first run call stalls (leak #1), the agent falls back to raw SSH, which has no classification at all. The crash was the direct result of that fallback.

3. Agents are trusted to self-report the health attribute. update_entity_attributes let the agent set health:"healthy" on lxc:nfs-export, which derived 4 spurious health checks. Health is scheduler-owned; agents shouldn't write it.

---

2. Changes

I — run pre-flights the execution transport before queueing

Why: A queued execution that can never run is worse than a failed one — it looks like progress, stalls the agent, and (this session) pushed the agent into the unsafe raw-SSH fallback.

What:

In the run tool handler (internal/mcp/), before inserting the execution row:

• If target type is vm, read the target entity's attributes. If qemu_guest_agent is missing or not_running, return an immediate error: "run on vm:zimaos blocked: QEMU guest agent is not running (qm guest exec unavailable). Start the agent first or use a different target."
• Same check for lxc targets whose pct exec path is known-broken (optional — start with VM only).

This converts "queued forever" into a fast, actionable failure the agent can recover from immediately.

Risk class: read-only (validation only, no execution row created).

Test: Unit test with a fake VM entity that has qemu_guest_agent: not_running → assert the tool returns the blocking error and inserts no execution row.

II — Host-mutation command guardrail in run classification

Why: apt-get install on a Proxmox host is the exact class of mutation that must always hit the approval gate. The classifier already escalates apt/kernel touches; this makes the escalation explicit and documented so agents stop second-guessing it.

What:

• In seeds/policy.yaml, add an explicit rule: proxmox-host targets + commands matching (apt-get install|apt install|dpkg|modprobe|kernel) ⇒ config_mutation (operator approval required), never reversible_low.
• Extend the classifier to also flag update-rc.d, systemctl enable on host targets if not already covered.
• Add a note in the run tool description: "Host-level package/kernel mutations always require operator approval."

Risk class: policy change — knowledge/DB, deploy via seed ingest.

Test: classify_command("apt-get install -y nfs-kernel-server", declared_risk=read_only) on host:hubris → must return config_mutation, not read_only. Add a fixture test in the classifier suite.

III — health attribute is read-only for agents

Why: This session's update_entity_attributes({"health":"healthy"}) on lxc:nfs-export derived 4 checks. Health is computed by the scheduler from probe results; an agent asserting it creates false monitoring.

What:

• In update_entity_attributes handler: reject (or strip with a warning) the health key. Return a message: "health is scheduler-owned; attribute ignored. Use get_health_summary/list_checks to observe it."
• Document in the tool description: "Do not write health — it is derived from probes."

Risk class: knowledge-graph mutation (existing), no infra impact.

Test: update_entity_attributes(slug=lxc:nfs-export, attributes={"health":"healthy"}) → response shows health ignored, other keys merged.

IV — Agent-side: record discovered dependency edges

Why: The session discovered vm:zimaos depends on lxc:nfs-export for /media/library, but no depends-on edge was recorded. A future agent investigating a ZimaOS mount failure would have no graph signal pointing at the NFS server.

What (agent behaviour, not code): After confirming a runtime dependency, call create_relationship(source, target, type) immediately. Concretely this session: create_relationship("vm:zimaos", "lxc:nfs-export", "depends-on").

Where to enforce: Update the homelab-context HERMES.md / SOUL.md agent instructions with a one-line rule: "When you discover a dependency between two entities (a service consumes another's export/mount/API), record it with create_relationship in the same session." Plus a runbook in .agents/skills/ if one doesn't exist.

Risk class: knowledge-graph mutation, auto-approves.

Test: Manual — after recording the edge, get_relations("vm:zimaos") shows depends-on → lxc:nfs-export.

---

3. Rollout

Step	Item	When
1	II — policy.yaml classifier rule + tests	next seed ingest
2	I — run VM transport pre-flight + test	next mcp server deploy
3	III — health read-only guard + test	same deploy as I
4	IV — agent instruction update in homelab-context	commit + sync
5	Verify: re-run classify_command + manual run on vm:zimaos (agent now up)	after deploy

---

4. Out of scope

• Per-client MCP bearer tokens (separate track, ADR 0012 note).
• request_execution re-introduction — the unified run primitive stays.
• Automating the Technitium DHCP reservation UI (this session's leftover — the reservation for BC:24:11:22:C2:F2 → 192.168.8.102 was added manually in the web UI; consider a runbook:technitium-dhcp-reservation doc in a follow-up).

---

5. Changelog

• 2026-08-05 — plan created from ZimaOS/NFS session audit + hubris crash incident.