oikos/plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md

2026-08-03 — Session review: service:haos monitoring + agent capability gaps

Status: Plan (audit complete; ready to implement). Reviewed session: 23da10db-46a9-444c-bbde-ca9457bd9087 — "Work out what monitoring checks service:haos should have and configure them." Method: Direct Postgres read of agent_sessions/agent_messages/ agent_activity/session_plan_steps on the prod mac-mini (oikos prod runs here in docker compose project oikos; gateway :8092), cross-referenced with the code paths in internal/mcp, internal/httpapi, internal/policy, internal/checkdefaults, internal/db/seed.go.

---

1. Session audit (objective vs outcome)

Dimension	Finding
Objective	Determine + configure monitoring checks for service:haos (HAOS VM 108, home.hubris.network, 192.168.8.101:8123).
Outcome	❌ Failed/stuck. status=executing, outcome=null ~4 min after last activity (UTC); never reached a terminal state. Only the existing check:vm-status:vm:haos:0 stub got populated; the three new checks (http:service, http:ingress, cert-expiry) and their ingress:/cert: entities were never created.
Tool calls	116 (vs the >30 N+1 failure signature). ~45 redundant list_entities/get_entity/get_relations, then a ~15-min storm of run doing find/grep/cat on prod source.
Plan	8 steps proposed; steps 1–4 genuinely done; step 5 falsely marked "done" after both its tool calls errored entity not found; steps 6–8 never started.
Operator friction	3 manual interventions: status, proceed, "why dony you use the mcp?"; plus a 44-minute approval stall (19:52→20:36) on two trivial reachability curls.
Severity	blocker (capability gap) + friction (classifier, plan-state, reaping).

Timeline (UTC)

• 19:45–19:50 — read-only exploration; run correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
• 19:50 — propose_plan (8 steps).
• 19:52 — two curl … -o /dev/null -w '%{http_code}' reachability probes → both classified config_mutation → one queued for approval (019fc92e…), second blocked ("approval already pending").
• 19:52 → 20:36 (44 min) — idle, waiting on operator approval.
• 20:36 — approval granted ("auto via assent window"); both curls → 200/200.
• 20:37 — step 4 ✅: populated check:vm-status:vm:haos:0 + checks edge.
• 20:37:58 — step 5 ❌: update_entity_attributes("check:http:service:haos:0") → entity not found; create_relationship → source entity not found. (There is no create tool.)
• 20:38–20:47 — spiral: search_knowledge (empty), then run find/grep/cat across /opt/homelab-context/**/*.go to reverse-engineer check creation. Reads checkdefaults.go, monitoring.go, default_checks.go, checks.go, coverage.go.
• 20:42 — sets service:haos monitoring: ["http"] via update_entity_attributes, hoping checkdefaults.Ensure() auto-generates. It does not (see A2).
• 20:42–20:50 — tries to reach the REST API directly: psql on hubris (cmd 127), docker exec on hubris (docker absent), curl http://192.168.178.25:8090 (wrong subnet; real net is 192.168.8.x; exit 7), curl http://oikos-api:8090 (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), ssh root@192.168.178.25 (no route). Final update_entity_attributes on ingress:/cert: → not found.
• 20:50:38 — last activity: a failed 30s run. Session goes silent, never terminates.

---

2. Root-cause findings (with code evidence)

A1 — No entity-creation capability in the MCP toolset (the blocker)

internal/mcp/tools.go registers 37 tools; the only entity-mutation surface is update_entity_attributes (merge into an existing entity) and create_relationship (needs existing source+target). Neither can create a new entity. The capability does exist at the HTTP layer — CreateEntity (internal/httpapi/impl.go:865, POST /api/v1/entities) — it is simply not exposed to the agent. Every "set up / onboard / configure entity X" task that needs a new check/ingress/cert/service hits this wall.

A2 — MCP update_entity_attributes bypasses ensureDefaultChecks

ensureDefaultChecks (internal/httpapi/default_checks.go:20) is invoked only from the HTTP handlers: CreateEntity (impl.go:1012) and PatchEntity (impl.go:1280). grep ensureDefaultChecks internal/mcp/ → no matches: the MCP tool writes attributes straight to the store, so flipping service:haos monitoring:["http"] never regenerated its checks. The agent's fallback strategy was structurally doomed via MCP.

A3 — -o /dev/null curl idiom misclassified as config_mutation

internal/policy/command.go:106 curlMutateRe matches (?:^|\s)-(?:d|F|T|o)\b — so -o (output-file) is treated as mutation. The canonical read-only reachability probe curl -sS -o /dev/null -w '%{http_code}' … therefore escalates to approval. This is the entire 44-minute stall. (curlIsReadOnly at command.go:307 only passes for GET with no -o/-d/-X/>.) A pure GET that discards the body is the single most common health probe and shouldn't need approval.

A4 — No platform self-knowledge doc for the check lifecycle

search_knowledge("create check entity how to add new check monitoring") → empty. The agent re-derived the whole mechanism from source on prod (~15 min, dozens of run). There is no agent/operator runbook explaining: check slugs are check:<kind>:<target>:<n>; check_defs are derived from the type's monitoring spec by checkdefaults.Ensure; Ensure runs at seed/deploy and on HTTP create/patch, not via MCP.

A5 — False plan progress (step marked done on failure)

At 20:37:58 both tool calls for step 5 returned error: entity not found, yet the agent advanced step 5→done. Plan-state integrity hole: a step whose actions error should not transition to done. (session_plan_steps confirms seq 5 = done.)

A6 — No "missing-capability" escalation; self-grounding failures

On detecting the dead-end (no create tool) the agent never told the operator "I lack a tool to create entities — please create them"; instead it tried to bypass its own platform. Grounding errors: invented IP 192.168.178.25 (real LAN is 192.168.8.x), ran run against ws:mac-mini ("unsupported target — must be host:/lxc:/vm:"), assumed docker exists on hubris, assumed the docker-alias oikos-api resolves from hubris. The agent didn't query get_entity("service:oikos") for the real address.

A7 — Sessions never reap from executing

Last activity 20:50; status still executing with no turn running. There is no idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide: 176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly approval-stalls and entity-not-found — same families.)

A8 — N+1 tool fan-out (116 calls)

Dozens of redundant list_entities/get_entity/get_relations before proposing a plan, plus the source-reading run storm. Above the >30-per-turn signature; indicates weak bulk-tool use and under-constrained exploration before planning.

---

3. Improvement plan (ordered)

Scope decision (confirmed with operator): general create_entity MCP tool + wire regen — solves this case and the 67-entity blast radius (§4).

Task 1 — create_entity MCP tool (fixes A1; the centerpiece)

• Register a new tool create_entity(slug, type, name, attributes?) in internal/mcp/tools.go that reuses httpapi.CreateEntity (impl.go:865) / the same store path — do not hand-roll. It must run ensureDefaultChecks (free, since it goes through the create path).
• Approval policy: no approval required for the entity itself — it mutates the knowledge graph, matching the existing no-approval stance of update_entity_attributes/create_relationship/upsert_knowledge. (Derived checks are safe/read-side; if a check kind is ever deemed mutating, gate that in the scheduler, not here.)
• Validate type against entity_types; reject unknown slugs/types with a clear error. Idempotent on existing slug (return the existing entity, mirroring the HTTP ETag/conflict behavior).
• Expose to the agent via the tool-list build path used by cmd/nomos/agent.go.

Task 2 — MCP update_entity_attributes triggers ensureDefaultChecks (fixes A2)

• After the attribute merge in the MCP handler, call ensureDefaultChecks with the post-merge entity (same args as impl.go:1280). This makes "set monitoring → checks regenerate" work via MCP, matching HTTP semantics.
• Mind the default_checks.go:14-19 caveat: a service whose address comes from its host edge may still produce no checks until the hosting edge exists — log/return that as an explicit result so the agent knows to create the edge next.

Task 3 — Classifier: read-only curl with -o /dev/null (fixes A3)

• In internal/policy/command.go curlIsReadOnly, treat -o /dev/null (and --output /dev/null) as read-only — it's a no-op sink. Keep -o <realpath> as mutation. Add TestClassifyCommand_CurlDevNull_ReadOnly next to the existing TestClassifyCommand_CurlPipeSh_ConfigMutation.
• Coach complement: in nomos/SOUL.md, note that reachability probes should use curl -I or -o /dev/null GETs (now read-only) rather than POSTs.

Task 4 — Plan-state integrity: don't mark done on errored actions (fixes A5)

• In cmd/nomos (agent.go/tasks.go where update_plan_step is emitted), a step whose turn ended with only error/not-found tool results must not auto-advance to done; leave it running/blocked and surface the failure to the operator. Minimal: if every tool call in the step returned an error:* result, hold the step.

Task 5 — Stuck-session reaping (fixes A7)

• Add an idle sweep (extend the existing continuation/idle worker in cmd/nomos) that transitions a session from executing→failed (or a new stuck) when no turn has run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.

Task 6 — Missing-capability escalation + grounding (fixes A6)

• nomos/SOUL.md: when a mutation tool returns entity … not found on a create intent, the agent must stop and ask the operator (or now use create_entity) rather than pivot to run/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead get_entity("service:oikos") for the real API address. run targets must be host:/lxc:/vm: slugs (state the contract explicitly).

Task 7 — Runbook: "how checks work / how to add monitoring" (fixes A4)

• Upsert a knowledge doc (via upsert_knowledge, linked to the agent:nomos and document:infrastructure/monitoring entities) covering: check slug grammar, checkdefaults.Ensure triggers (seed + HTTP create/patch, now also MCP), the monitoring per-entity override, the host-edge caveat, and the canonical way to add monitoring to an entity (create/patch entity → checks derive).

Task 8 — (Lower priority) exploration budget / bulk-tool use (A8)

• nomos/SOUL.md: prefer list_entities(limit) + get_entity_knowledge bulk calls over N+1 get_entity/get_relations fans; cap pre-plan exploration. Optional guardrail in agent.go (warn at >N same-tool calls per turn).

Recommended sequence

1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona hardening.)

---

4. Uncovered cases — the capability-gap blast radius

The existing F1–F8 plans (2026-08-03-nomos-chat-reliability-and-ux-audit.md, shipped v0.15.0) and the turn-scheduler review cover only UI / streaming / turn serialization / connection UX. None addresses agent capability or MCP↔HTTP integration. This session exposes the uncovered class:

• 67 entities currently have no check: relationship (DB query): 40 lxc, 26 service, 1 vm. Any "add monitoring to X" task fails identically until Tasks 1+2.
• Whole task families blocked by the no-create gap: onboarding a new host/LXC/VM, declaring a new service/ingress/cert/dns, adding any check that doesn't already exist, registering a relationship target that doesn't exist yet. All currently require an operator to hand-edit seeds/inventory.yaml and re-seed.
• MCP↔HTTP semantic drift (generalize A2): audit other MCP mutation tools for side-effects that the HTTP handlers perform but the MCP path skips (check regen, drift-flagging, audit fields, idempotency). Each is a latent "agent did the right thing but nothing happened" bug.
• Classifier read-only false-positives (generalize A3): beyond -o /dev/null, review other common read-only idioms that escalate (curl with benign flags, compound read-only commands) — friction compounds into approval stalls and stuck sessions.
• No terminal/stuck reaping (generalize A7): any turn that ends unresolved leaves the session executing forever; the UI never shows "done/failed".

---

5. Validation

• Task 1/2: go test ./internal/mcp/... ./internal/httpapi/... — new test creates check:http:service:haos:0 via create_entity, asserts the entity exists and that a check_def row was derived; then update_entity_attributes(service:haos, monitoring:["http"]) via MCP and assert checks regenerate (currently absent).
• Task 3: go test ./internal/policy/ — curl -sS -o /dev/null -w '%{http_code}' URL ⇒ read_only; curl -o /tmp/x URL ⇒ config_mutation.
• Task 4: cmd/nomos test — a step whose only tool result is error:* stays non-done.
• Task 5: idle-sweep test — session with no turn for N min and no pending approval ⇒ failed (+ event emitted).
• End-to-end re-run: replay the haos goal against a local nomos; expect the three checks + ingress:/cert: entities created in <15 tool calls with zero approvals and a done outcome.

6. Out of scope / open questions

• Whether create_entity for sensitive types (e.g. secret, key) should require approval even though it's graph-only — recommend: same no-approval stance now, add type-specific gating later if abused.
• The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct stuck status vs reuse failed.
• Whether to also expose a delete_entity/retire_entity MCP tool (not needed for this case; lifecycle retirement is a separate flow).