oikos/plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md

# 2026-08-03 — Session review: `service:haos` monitoring + agent capability gaps

**Status:** Plan (audit complete; ready to implement).
**Reviewed session:** `23da10db-46a9-444c-bbde-ca9457bd9087` — *"Work out what
monitoring checks service:haos should have and configure them."*
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
`agent_activity`/`session_plan_steps` on the prod mac-mini (oikos prod runs here
in docker compose project `oikos`; gateway `:8092`), cross-referenced with the
code paths in `internal/mcp`, `internal/httpapi`, `internal/policy`,
`internal/checkdefaults`, `internal/db/seed.go`.

---

## 1. Session audit (objective vs outcome)

| Dimension | Finding |
|---|---|
| Objective | Determine + configure monitoring checks for `service:haos` (HAOS VM 108, `home.hubris.network`, `192.168.8.101:8123`). |
| Outcome | ❌ **Failed/stuck.** `status=executing`, `outcome=null` ~4 min after last activity (UTC); never reached a terminal state. Only the *existing* `check:vm-status:vm:haos:0` stub got populated; the three **new** checks (`http:service`, `http:ingress`, `cert-expiry`) and their `ingress:`/`cert:` entities were never created. |
| Tool calls | **116** (vs the >30 N+1 failure signature). ~45 redundant `list_entities`/`get_entity`/`get_relations`, then a ~15-min storm of `run` doing `find`/`grep`/`cat` on prod source. |
| Plan | 8 steps proposed; steps 1–4 genuinely done; **step 5 falsely marked "done"** after both its tool calls errored `entity not found`; steps 6–8 never started. |
| Operator friction | 3 manual interventions: `status`, `proceed`, *"why dony you use the mcp?"*; plus a **44-minute approval stall** (19:52→20:36) on two trivial reachability curls. |
| Severity | **blocker** (capability gap) + **friction** (classifier, plan-state, reaping). |

### Timeline (UTC)
- **19:45–19:50** — read-only exploration; `run` correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
- **19:50** — `propose_plan` (8 steps).
- **19:52** — two `curl … -o /dev/null -w '%{http_code}'` reachability probes → both classified `config_mutation` → one queued for approval (`019fc92e…`), second blocked ("approval already pending").
- **19:52 → 20:36 (44 min)** — idle, waiting on operator approval.
- **20:36** — approval granted ("auto via assent window"); both curls → 200/200.
- **20:37** — step 4 ✅: populated `check:vm-status:vm:haos:0` + `checks` edge.
- **20:37:58** — step 5 ❌: `update_entity_attributes("check:http:service:haos:0")` → **`entity not found`**; `create_relationship` → **`source entity not found`**. *(There is no create tool.)*
- **20:38–20:47** — spiral: `search_knowledge` (empty), then `run find/grep/cat` across `/opt/homelab-context/**/*.go` to reverse-engineer check creation. Reads `checkdefaults.go`, `monitoring.go`, `default_checks.go`, `checks.go`, `coverage.go`.
- **20:42** — sets `service:haos` `monitoring: ["http"]` via `update_entity_attributes`, hoping `checkdefaults.Ensure()` auto-generates. **It does not** (see A2).
- **20:42–20:50** — tries to reach the REST API directly: `psql` on hubris (cmd 127), `docker exec` on hubris (docker absent), `curl http://192.168.178.25:8090` (wrong subnet; real net is `192.168.8.x`; exit 7), `curl http://oikos-api:8090` (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), `ssh root@192.168.178.25` (no route). Final `update_entity_attributes` on `ingress:`/`cert:` → `not found`.
- **20:50:38** — last activity: a failed 30s `run`. Session goes silent, never terminates.

---

## 2. Root-cause findings (with code evidence)

### A1 — No entity-creation capability in the MCP toolset  *(the blocker)*
`internal/mcp/tools.go` registers **37 tools**; the only entity-mutation surface is
`update_entity_attributes` (merge into an **existing** entity) and
`create_relationship` (needs **existing** source+target). Neither can create a new
entity. The capability **does** exist at the HTTP layer — `CreateEntity`
(`internal/httpapi/impl.go:865`, `POST /api/v1/entities`) — it is simply not exposed
to the agent. Every "set up / onboard / configure entity X" task that needs a new
check/ingress/cert/service hits this wall.

### A2 — MCP `update_entity_attributes` bypasses `ensureDefaultChecks`
`ensureDefaultChecks` (`internal/httpapi/default_checks.go:20`) is invoked **only**
from the HTTP handlers: `CreateEntity` (`impl.go:1012`) and `PatchEntity`
(`impl.go:1280`). `grep ensureDefaultChecks internal/mcp/` → **no matches**: the MCP
tool writes attributes straight to the store, so flipping `service:haos`
`monitoring:["http"]` never regenerated its checks. The agent's fallback strategy
was structurally doomed via MCP.

### A3 — `-o /dev/null` curl idiom misclassified as `config_mutation`
`internal/policy/command.go:106` `curlMutateRe` matches `(?:^|\s)-(?:d|F|T|o)\b` — so
`-o` (output-file) is treated as mutation. The canonical read-only reachability probe
`curl -sS -o /dev/null -w '%{http_code}' …` therefore escalates to approval. This is
the entire 44-minute stall. (`curlIsReadOnly` at `command.go:307` only passes for GET
with no `-o`/`-d`/`-X`/`>`.) A pure GET that discards the body is the single most
common health probe and shouldn't need approval.

### A4 — No platform self-knowledge doc for the check lifecycle
`search_knowledge("create check entity how to add new check monitoring")` → empty.
The agent re-derived the whole mechanism from source on prod (~15 min, dozens of
`run`). There is no agent/operator runbook explaining: check slugs are
`check:<kind>:<target>:<n>`; `check_defs` are derived from the type's `monitoring`
spec by `checkdefaults.Ensure`; Ensure runs at **seed/deploy** and on **HTTP
create/patch**, not via MCP.

### A5 — False plan progress (step marked done on failure)
At 20:37:58 both tool calls for step 5 returned `error: entity not found`, yet the
agent advanced step 5→`done`. Plan-state integrity hole: a step whose actions error
should not transition to `done`. (`session_plan_steps` confirms seq 5 = `done`.)

### A6 — No "missing-capability" escalation; self-grounding failures
On detecting the dead-end (no create tool) the agent never told the operator *"I lack
a tool to create entities — please create them"*; instead it tried to bypass its own
platform. Grounding errors: invented IP `192.168.178.25` (real LAN is `192.168.8.x`),
ran `run` against `ws:mac-mini` ("unsupported target — must be host:/lxc:/vm:"),
assumed `docker` exists on hubris, assumed the docker-alias `oikos-api` resolves from
hubris. The agent didn't query `get_entity("service:oikos")` for the real address.

### A7 — Sessions never reap from `executing`
Last activity 20:50; status still `executing` with no turn running. There is no
idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide:
176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly
approval-stalls and entity-not-found — same families.)

### A8 — N+1 tool fan-out (116 calls)
Dozens of redundant `list_entities`/`get_entity`/`get_relations` before proposing a
plan, plus the source-reading `run` storm. Above the >30-per-turn signature; indicates
weak bulk-tool use and under-constrained exploration before planning.

---

## 3. Improvement plan (ordered)

**Scope decision (confirmed with operator):** general `create_entity` MCP tool **+
wire regen** — solves this case and the 67-entity blast radius (§4).

### Task 1 — `create_entity` MCP tool  *(fixes A1; the centerpiece)*
- Register a new tool `create_entity(slug, type, name, attributes?)` in
  `internal/mcp/tools.go` that **reuses** `httpapi.CreateEntity`
  (`impl.go:865`) / the same store path — do not hand-roll. It must run
  `ensureDefaultChecks` (free, since it goes through the create path).
- **Approval policy:** no approval required for the entity itself — it mutates the
  knowledge graph, matching the existing no-approval stance of
  `update_entity_attributes`/`create_relationship`/`upsert_knowledge`. (Derived checks
  are safe/read-side; if a check kind is ever deemed mutating, gate *that* in the
  scheduler, not here.)
- Validate `type` against `entity_types`; reject unknown slugs/types with a clear
  error. Idempotent on existing slug (return the existing entity, mirroring the HTTP
  `ETag`/conflict behavior).
- Expose to the agent via the tool-list build path used by `cmd/nomos/agent.go`.

### Task 2 — MCP `update_entity_attributes` triggers `ensureDefaultChecks`  *(fixes A2)*
- After the attribute merge in the MCP handler, call `ensureDefaultChecks` with the
  post-merge entity (same args as `impl.go:1280`). This makes "set monitoring → checks
  regenerate" work via MCP, matching HTTP semantics.
- Mind the `default_checks.go:14-19` caveat: a service whose address comes from its
  host edge may still produce no checks until the hosting edge exists — log/return
  that as an explicit result so the agent knows to create the edge next.

### Task 3 — Classifier: read-only `curl` with `-o /dev/null`  *(fixes A3)*
- In `internal/policy/command.go` `curlIsReadOnly`, treat `-o /dev/null` (and
  `--output /dev/null`) as read-only — it's a no-op sink. Keep `-o <realpath>` as
  mutation. Add `TestClassifyCommand_CurlDevNull_ReadOnly` next to the existing
  `TestClassifyCommand_CurlPipeSh_ConfigMutation`.
- Coach complement: in `nomos/SOUL.md`, note that reachability probes should use
  `curl -I` or `-o /dev/null` GETs (now read-only) rather than POSTs.

### Task 4 — Plan-state integrity: don't mark `done` on errored actions  *(fixes A5)*
- In `cmd/nomos` (`agent.go`/`tasks.go` where `update_plan_step` is emitted), a step
  whose turn ended with only error/`not-found` tool results must **not** auto-advance
  to `done`; leave it `running`/`blocked` and surface the failure to the operator.
  Minimal: if every tool call in the step returned an `error:*` result, hold the step.

### Task 5 — Stuck-session reaping  *(fixes A7)*
- Add an idle sweep (extend the existing continuation/idle worker in `cmd/nomos`) that
  transitions a session from `executing`→`failed` (or a new `stuck`) when no turn has
  run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal
  handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.

### Task 6 — Missing-capability escalation + grounding  *(fixes A6)*
- `nomos/SOUL.md`: when a mutation tool returns `entity … not found` on a create
  intent, the agent must **stop and ask the operator** (or now use `create_entity`)
  rather than pivot to `run`/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead
  `get_entity("service:oikos")` for the real API address. `run` targets must be
  `host:/lxc:/vm:` slugs (state the contract explicitly).

### Task 7 — Runbook: "how checks work / how to add monitoring"  *(fixes A4)*
- Upsert a knowledge doc (via `upsert_knowledge`, linked to the `agent:nomos` and
  `document:infrastructure/monitoring` entities) covering: check slug grammar,
  `checkdefaults.Ensure` triggers (seed + HTTP create/patch, now also MCP), the
  `monitoring` per-entity override, the host-edge caveat, and the canonical way to add
  monitoring to an entity (create/patch entity → checks derive).

### Task 8 — (Lower priority) exploration budget / bulk-tool use  *(A8)*
- `nomos/SOUL.md`: prefer `list_entities(limit)` + `get_entity_knowledge` bulk calls
  over N+1 `get_entity`/`get_relations` fans; cap pre-plan exploration. Optional
  guardrail in `agent.go` (warn at >N same-tool calls per turn).

### Recommended sequence
1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the
approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona
hardening.)

---

## 4. Uncovered cases — the capability-gap blast radius

The existing F1–F8 plans (`2026-08-03-nomos-chat-reliability-and-ux-audit.md`,
shipped v0.15.0) and the turn-scheduler review cover **only** UI / streaming / turn
serialization / connection UX. **None** addresses agent *capability* or
MCP↔HTTP integration. This session exposes the uncovered class:

- **67 entities currently have no `check:` relationship** (DB query): 40 `lxc`, 26
  `service`, 1 `vm`. Any "add monitoring to X" task fails identically until Tasks 1+2.
- **Whole task families blocked by the no-create gap:** onboarding a new host/LXC/VM,
  declaring a new service/ingress/cert/dns, adding any check that doesn't already
  exist, registering a relationship target that doesn't exist yet. All currently
  require an operator to hand-edit `seeds/inventory.yaml` and re-seed.
- **MCP↔HTTP semantic drift (generalize A2):** audit other MCP mutation tools for
  side-effects that the HTTP handlers perform but the MCP path skips (check regen,
  drift-flagging, audit fields, idempotency). Each is a latent "agent did the right
  thing but nothing happened" bug.
- **Classifier read-only false-positives (generalize A3):** beyond `-o /dev/null`,
  review other common read-only idioms that escalate (`curl` with benign flags,
  compound read-only commands) — friction compounds into approval stalls and stuck
  sessions.
- **No terminal/`stuck` reaping (generalize A7):** any turn that ends unresolved
  leaves the session `executing` forever; the UI never shows "done/failed".

---

## 5. Validation

- **Task 1/2:** `go test ./internal/mcp/... ./internal/httpapi/...` — new test creates
  `check:http:service:haos:0` via `create_entity`, asserts the entity exists **and**
  that a `check_def` row was derived; then `update_entity_attributes(service:haos,
  monitoring:["http"])` via MCP and assert checks regenerate (currently absent).
- **Task 3:** `go test ./internal/policy/` — `curl -sS -o /dev/null -w '%{http_code}'
  URL` ⇒ `read_only`; `curl -o /tmp/x URL` ⇒ `config_mutation`.
- **Task 4:** `cmd/nomos` test — a step whose only tool result is `error:*` stays
  non-`done`.
- **Task 5:** idle-sweep test — session with no turn for N min and no pending approval
  ⇒ `failed` (+ event emitted).
- **End-to-end re-run:** replay the haos goal against a local nomos; expect the three
  checks + `ingress:`/`cert:` entities created in <15 tool calls with **zero**
  approvals and a `done` outcome.

## 6. Out of scope / open questions
- Whether `create_entity` for sensitive types (e.g. `secret`, `key`) should require
  approval even though it's graph-only — recommend: same no-approval stance now, add
  type-specific gating later if abused.
- The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct
  `stuck` status vs reuse `failed`.
- Whether to also expose a `delete_entity`/`retire_entity` MCP tool (not needed for
  this case; lifecycle retirement is a separate flow).