oikos/plans/done/2026-08-04-session-audit-agent-reliability.md

2026-08-04 — Session audit: agent reliability, plan system, and learning loop gaps

Status: Done — all 12 tasks implemented, tested, deployed in v0.21.0. Verified in production with live session tests (classifications, feedback, token tracking, execution linkage all confirmed). Reviewed sessions: Past 5 completed plus ZimaOS continuation (268895a5) Method: Direct Postgres read of agent_sessions/agent_messages/ agent_activity/session_plan_steps/executions/classifications/feedback/ patterns/skills/approvals/nomos_plan_executions/audit_log on the prod mac-mini. Cross-referenced with internal/audit/, internal/mcp/, internal/httpapi/, internal/policy/.

---

1. Sessions audited

#	Session	Turns	Run calls	Failures	Plan steps (done/total)	Duration	Outcome
S1	Pocket-pascal deploy (a433b386)	8	111	1	10/11	1.5h	Success (truncated by turn limit)
S2	SSH re-investigation (6f0ade08)	4	42	0	0/4	11m	Success (all steps replaced)
S3	Webhook HMAC (0f509508)	2	40	0	5/5	2m	Success (clean; best session)
S4	Check scripts (d458a5f8)	23	96	4	0/14	2h	Success (3 plan gens; 0 steps done)
S5	ZimaOS outage (268895a5)	18+10	209	8	0/5	2h30m	Marked success; dashboard still broken

Headline: 85% session success rate, but plan adherence is ~38% (15/39 steps ever reached done). The ZimaOS session was the worst: marked success while the dashboard was still down, then burned 81 more calls and 30 minutes chasing irrelevant DHCP reservations.

---

2. Cross-cutting findings

F1 — MCP 30s client timeout kills every long-running command (P0)

All 9 run failures across the audit are Post "http://api:8090/mcp": context deadline exceeded at exactly 30s. Commands with sleep 30, qm shutdown+wait, or async poll loops always hit this. The agent retries with longer sleeps and hits the same wall.

Root cause: cmd/nomos MCP client uses a 30s timeout; the run tool blocks synchronously waiting for command completion. No async path exists for long-running commands.

F2 — Premature success: complete_task fires before the goal is actually met (P0)

Session 268895a5 called complete_task(success) at 18:29 with summary "zimaos returns 200." The endpoint was serving ttyd (terminal), not the ZimaOS dashboard. The agent conflated "HTTPS 200" with "dashboard works." The user had to resume the session.

Root cause: No pre-completion validation. The agent can mark success with a summary that doesn't match reality. complete_task is a write-and-forget operation with no state check.

F3 — Wrong target: run doesn't validate the target can execute the command (P0)

In the ZimaOS continuation, qm stop 100 --skiplock was executed on lxc:dns. The command failed (qm: command not found) because the agent copied the command from a previous run but forgot to change the target. Similarly, cmd/nomos tried docker exec on host:hubris (no docker).

Root cause: run(target, command) in internal/mcp/server.go doesn't validate that the target type can execute the given command. A simple allowlist would catch qm/pct/pvesh on non-host targets.

F4 — Plan system is decorative: 62% of steps never reach done (P1)

Across 5 sessions: 39 plan steps. 24 (62%) were replaced, 15 (38%) reached done. Session d458a5f8 had 3 complete plan regenerations with zero completed steps. The agent replaces plans instead of completing or explicitly skipping steps.

Root cause: Plan steps carry status (pending/running/done/failed/ skipped/blocked/replaced) but replaced has no replaced_reason field. The model can silently replace every step and the system doesn't flag it. No constraint ties propose_plan to existing plan state.

F5 — No plan on session resume: continuation sessions run ad-hoc (P1)

The ZimaOS continuation (18:29→19:01) had 81 calls with zero propose_plan calls. The agent ran ad-hoc tool calls with no structure.

Root cause: When a session resumes, the must_have_plan guard is already satisfied by the old (completed) plan. The agent doesn't re-plan on resume.

F6 — Scope expansion / rabbit holes: agent chases irrelevant sub-goals (P2)

In the ZimaOS continuation, the agent spent ~40 calls trying to fix Technitium DHCP reservations — a completely different subsystem from the goal ("make the dashboard reachable"). The ZimaOS dashboard hadn't started since July 19 (3-week-old issue), making the DHCP reservation effort moot. The agent never surfaced a question like "This is pre-existing — should I still fix DHCP?"

Root cause: No scope gate. When the agent pivots to a subsystem unrelated to the stated goal, nothing stops it. The session_questions mechanism exists (2 calls in 193 sessions) but the model never uses it.

F7 — Command generation errors: malformed bash from the LLM (P2)

In the ZimaOS continuation, the agent generated:

• head - n instead of head -n → bash syntax error
• echo "---" && \n curl ... → literal \n in command → ambiguous redirect

Root cause: The LLM generates bash commands inline in content blocks. No syntax validation, no escape-character handling. The run tool should reject malformed commands before execution.

F8 — Learning pipeline completely empty (P4)

Table	Rows
classifications	0
feedback	0
patterns	0
skills	0

Despite 1,884 executions and 263 approvals, the system learns nothing from outcomes. The ZimaOS session discovered that the dashboard hadn't started since July 19 — this was never persisted. The HMAC trailing-newline discovery was persisted manually; if the agent forgot upsert_knowledge, it would be lost.

F9 — Observability gaps (P3)

Gap	Detail
Token tracking	agent_activity.token_count is NULL for every row
Execution linkage	nomos_plan_executions is empty; audit_log.session_id is null
Plan quality	No metric for step completion rate (currently 38%)
MCP timeout rate	No counter for run calls that hit the client timeout

F10 — Execution success rate is 74% (P2)

1,884 executions: 1,400 completed (74%), 295 failed (15.6%), 152 cancelled (8%), 37 denied (2%). One in four execution attempts doesn't complete.

---

3. Improvement plan (ordered by impact/effort)

Task 1 — Prevent premature complete_task(success) (fixes F2)

internal/mcp/tools.go — In the complete_task handler, when outcome=success: require the summary field to contain a verifiable state assertion. Minimum: if the goal mentions a URL, check that the summary doesn't contradict known state. Lightweight: log a warning if the summary says "returns 200" but the last ping_service or run result says otherwise.

Coach: nomos/SOUL.md — explicit rule: "Before calling complete_task(success), restate the user's original goal in your own words and verify each condition. If any condition is 'probably works' rather than 'verified,' ask the operator or set outcome=partial."

Task 2 — Target validation in run (fixes F3)

internal/mcp/server.go — In the run handler, before dispatching: validate that the command prefix matches the target type.

pct/qm/pvesh/iptables → only host:* targets
systemctl/docker      → host:* or lxc:* targets
curl/nmap/ss          → any target

If mismatched, return a clear error: "Cannot run qm on lxc:dns — qm is a Proxmox host command. Use target host:hubris or host:strong." Do not classify or execute.

Test: TestClassifyCommand_WrongTarget → commands with host-only prefixes on LXC targets return error without execution.

Task 3 — Raise MCP client timeout; add async path for long-running commands (fixes F1)

cmd/nomos — Raise the MCP client timeout from 30s to 120s.

internal/mcp/server.go — For run commands that the classifier determines will exceed the client timeout (presence of sleep, wait, timeout in the command), return immediately with an execution_id and status running. The agent already has get_execution_status — use it:

1. Classify the command; if it contains sleep, wait, or shell constructs that imply polling, flag it as async_potential.
2. Start the command, return the execution_id immediately.
3. Agent polls with get_execution_status(execution_id).
4. If the client timeout is hit mid-poll, the execution continues on the server — it's not lost.

Test: run with sleep 60; echo done on host:hubris → returns immediately (not 30s timeout), get_execution_status eventually returns completed.

Task 4 — Plan step integrity: require replaced_reason on replacement (fixes F4)

session_plan_steps migration — Add replaced_reason TEXT column.

cmd/nomos — When the agent emits update_plan_step with status=replaced, require a non-empty replaced_reason. Valid reasons: wrong_diagnosis, scope_change, blocked, superseded, operator_override.

Coach: nomos/SOUL.md — explicit rule: "Complete (status=done) or explicitly skip (status=skipped) steps. Use status=replaced only when the entire plan generation is wrong; include the reason. Replacing all steps with no reason is a session-quality violation."

Task 5 — Force propose_plan on session resume (fixes F5)

cmd/nomos — When a session with status done or failed receives a new user message, reset the plan state: clear step status, require a new propose_plan call before any run calls. The "must have plan" guard should consider the resumed session as plan-less until a fresh propose_plan is called.

Guard: set_goal + propose_plan must be called before any run in a resumed session. Reuse the existing "No plan — call set_goal then propose_plan" error from internal/mcp/server.go.

Task 6 — Scope gate: surface session_questions on context switch (fixes F6)

Coach: nomos/SOUL.md — explicit rule: "Before pivoting to a subsystem not mentioned in the user's goal, ask via session_questions. Example: 'The dashboard logs show it hasn't started since July 19. Do you want me to debug the dashboard service itself [A], skip it and just stabilize the IP [B], or stop here [C]?'"

cmd/nomos prompt — Add to the system prompt: "When the investigation leads to a subsystem or root cause unrelated to the expressed goal, surface a session_question before taking action."

Task 7 — Auto-upsert knowledge on session close (fixes F8)

cmd/nomos — On complete_task (any outcome: success, partial, failure), auto-generate a knowledge entry:

title: "<date>: <goal summary>"
content: "## Outcome\n<outcome>\n## Root cause\n<extracted>\n## What was done\n<summary>\n## What was left\n<unresolved>"
tags: [session:<id>]
about: [entities involved]

This ensures every session leaves a trace regardless of whether the agent remembered to call upsert_knowledge.

Task 8 — Token tracking (fixes F9)

cmd/nomos — After each LLM call, extract usage.prompt_tokens, usage.completion_tokens, usage.total_tokens from the response and write to agent_activity.token_count. Currently the field exists but is never populated (NULL for all rows).

Task 9 — Execution linkage (fixes F9)

internal/mcp/server.go — When run creates an execution, write a row into nomos_plan_executions linking session_id, plan_step_seq, and execution_id.

internal/mcp/server.go — Pass session_id (from MCP request headers) into audit_log writes. Currently audit_log.session_id is NULL — the createAuditLog function in internal/httpapi/impl.go receives the correlation_id but not the session_id from the MCP path.

Task 10 — Plan quality metric (fixes F9)

cmd/nomos — At session close, compute: completed_steps / total_steps_per_plan (currently ~38%). Log as a metric or write as a session attribute. Track over time to measure plan-adherence improvements from Tasks 4+5.

Task 11 — Auto-classify every run call (fixes F8)

internal/mcp/server.go — The run handler already calls the classifier (classifyCommand in internal/policy/command.go) to determine risk_class and approval route. Write the result to the classifications table. Currently the table is empty (0 rows) despite 1,884 executions being classified.

Task 12 — Auto-feedback on session close (fixes F8)

cmd/nomos — On complete_task, generate a feedback entry:

session_id: <id>
outcome: <outcome>
observation: <summary>
lesson: <extracted from complete_task.summary>
side_effects: <entities created/modified during session>

cmd/oikos — Add a daily cron or scheduler job that reads recent feedback entries and extracts patterns (recurring root causes, same-fix applied multiple times, known-broken services). Seed the pattern table.

Task 13 — Command syntax validation in run (fixes F7)

internal/mcp/server.go — Before executing a run command, do lightweight bash syntax validation:

- Reject literal \n in commands (should be ; or &&)
- Reject commands where the last line ends with \ (backslash-continuation) 
  but no next line
- Warn on common typos: "head - n", "grep - i", spaces before flags
- Reject `&& \n` patterns (the LLM sometimes inserts literal \n between && chains)

Task 14 — Stuck-session reaping (from prior plan; re-confirmed)

This session exhibited the same idle zombie pattern (23da10db — no closed_at, status failed but outcome failure). Task 5 from the 2026-08-03 plan is still open. Copying here for completeness.

---

4. Recommended sequence

P0 (blocks operational waste):
  1 → 2 → 3

P1 (fixes plan architecture):
  4 → 5

P2 (cognitive guardrails):
  6 → 7 → 13

P3 (observability):
  8 → 9 → 10

P4 (learning loop):
  11 → 12

Sequence rationale: Tasks 1-3 stop the worst outcomes (premature success, wrong-target execution, MCP timeouts). Tasks 4-5 make the plan system actually useful instead of decorative. Tasks 6-7 add guardrails that prevent the ZimaOS rabbit-hole class of failure. Tasks 8-10 give us visibility into whether any of the previous tasks are working. Tasks 11-12 close the learning loop.

---

5. Validation

Task	Test
1	Session with goal "make X reachable" where last ping shows 502 → complete_task(success) is rejected or warns
2	run("lxc:dns", "qm stop 100") → error: "qm is a Proxmox host command"
3	run with sleep 45; echo done → returns execution_id immediately, get_execution_status shows final result
4	update_plan_step(status=replaced) with no reason → rejected; with reason → accepted
5	Resumed session calls run before propose_plan → blocked: "No plan — call propose_plan"
6	Agent pivots to unrelated subsystem → session_questions is called before action
7	complete_task → knowledge entry created automatically with session link
8	agent_activity.token_count is non-NULL after any LLM call
9	nomos_plan_executions has rows linking session + step + execution
10	Session close writes plan_adherence attribute (step-completion %)
11	classifications table has 1 row per run call with risk_class + route
12	complete_task → auto feedback entry; daily pattern job finds recurring issues
13	run with head - n /etc/hosts → rejected with clear error about malformed command

---

6. Out of scope / open questions

• Whether to raise auto_act from off for reversible_low actions (separate policy decision; would reduce approval pileup without code changes).
• Whether to add a delete_entity MCP tool for lifecycle management (separate from this reliability plan).
• The exact TTL for stuck-session reaping (30 min recommended, confirmed in 2026-08-03 plan).
• Whether run async mode should be opt-in (command contains sleep/wait) or universal (every run returns immediately, agent always polls). Recommend opt-in for now — most commands complete in <5s.