oikos/plans/done/2026-07-10-autonomous-plan-execution.md

2026-07-10 — Autonomous plan execution: close the observation gap

Status: Done — 2026-07-11. Full scope shipped, including the Option B stretch goal: atomic pct_create decomposition, robust assent-window open/extend, SOUL persist-through-errors + maxIterations=40, and event-driven auto-continuation (cmd/nomos/continue.go). Commits 233b5e4, d2f749d, 657e1a8, d529688, 84ecb6b.

The real problem (not the one we kept fixing)

Operator, verbatim: "the agent seems to stop when it encounters the first error, it does not recover from it… my goal is that the agent can do anything once a plan has been approved."

We have patched ~10 individual failure modes (DNS, gateway, sshExec timeout, substring bug, slug collisions, docker CLI, assent window…). Every one was real. None fixed the thing the operator keeps hitting, because they all fixed individual commands — and the problem is the loop, not the commands.

Root cause: the agent never sees the result of the thing it started

The agent runs in discrete request→response turns. Provisioning executions are asynchronous: request_execution(pct_create) queues an execution, fires the real SSH work in a goroutine (go executeApprovedViaAPI(...), internal/mcp/server.go), and returns "provisioning now" immediately. The multi-minute result lands in the DB after the agent's turn has already ended.

So the agent literally is not running when the error happens. It cannot react to a failure it never observes. The only way the result re-enters the agent's reasoning is if a human types "continue" to start a new turn — the human is the event loop. Read the failing session (7c25edaa, 18 messages): the operator typed "continue" / "continue?" / "??" / "proceed" eight times, each one just ticking the agent forward one async step. The agent was recovering (it correctly diagnosed the docker-CLI issue and proposed fixes) — it simply could not proceed one step without a human tick.

Two concrete asymmetries prove the diagnosis:

1. run is synchronous, pct_create is async. Inside an assent window, the general run tool executes the command inline and returns stdout/exit-status to the agent (server.go ~L514) — the agent sees the result and can continue. pct_create in the same window auto-approves and then go-routines the work — the agent sees nothing. The failure-prone path is the unobservable one.
2. pct_create is monolithic and all-or-nothing. It does create + apt + docker + post_install + verify in one SSH call. Even if it were synchronous, the agent could only see "the whole thing failed at some point," not step 3 of 6 — so it can't surgically fix step 3 and resume. Recovery requires intermediate observation.

Secondary (real but downstream): "continue" is not an assent word (cmd/nomos/assent.go), so in that session the assent window never even opened — every step stayed gated, compounding the ticking.

The reframe: Nomos should work like a coding agent

A coding agent (Claude Code) runs a command, sees the output, runs the next, fixes errors inline, all in one continuous session — it does not stop and ask a human to forward it after each command. That is exactly "do anything once the plan is approved." The homelab agent needs the same loop:

approve the plan → agent runs step → observes result → runs next step / on failure diagnoses + adapts + retries → … → verifies goal met → reports.

The machinery for this already exists in the run tool (synchronous, observable, auto-executing within an assent window). Provisioning just doesn't use it — it uses a black box. The fix is to make the whole system consistent with the model run already embodies.

Target architecture

1. One observable primitive; retire the async black box

• Everything the agent does — including provisioning — is a sequence of synchronous run calls whose real output (stdout, stderr, exit code) returns inline. No goroutine hand-off for agent-initiated work.
• Decompose pct_create. Keep a thin pct_create that only does the fast, atomic container creation (create + start + register), returning synchronously. Move package install / service setup / post_install / verify out into agent-driven run steps. Now the agent observes each step and can fix a failed one without redoing the container.
• Net: the agent orchestrates create → apt → install → configure → up → verify, seeing each result, exactly like a human operator at a shell.

2. Approve the plan = an autonomy grant the agent executes to completion

• The assent/autonomy window already exists. Make it robust:
  • Opening it must not depend on a magic word list. "continue", "go", "do it", "proceed", clicking Approve, or approving the first queued step should all open/extend it. Safer: when the operator approves ANY step of a plan, treat that as opening the window for the rest of that plan.
  • Within the window: read-only + config_mutation run steps execute inline, no re-prompt. Destructive still stops for typed confirmation — but a destructive step described in the approved plan can be pre-authorized so the agent isn't blocked mid-flow on something already shown and approved.
• The window is the scope boundary: "you may do what the plan needs on this target; you may not wander outside it."

3. The agent persists through errors (prompt + loop)

• SOUL: "You are the executor of the approved plan. Run it step by step, observing each result. On failure, do not stop and hand back — diagnose (read logs / inspect state), form a hypothesis, fix it, and retry or take an alternative path. Continue until the goal is verified working or you are genuinely blocked (you need information only the operator has, or a step exceeds the approved scope). Never end a turn with a half-finished plan just because one command failed."
• maxIterations sized for a full provision-with-recovery (raise 25 → ~40) and count observation/read-only steps cheaply so recovery attempts aren't starved.

4. Long-running steps: keep the turn alive, or auto-continue

A synchronous apt install is ~1–2 min; a full stack up is longer. Options, in preference order:

• A (simplest, ship first): synchronous run with the existing 10-min cap; the streaming turn stays open (the chat UI already holds the SSE). Emit progress events so the operator sees liveness (already built — elapsed timer).
• B (for very long ops): event-driven auto-continuation — when an async execution tied to an active plan completes, a worker re-invokes Nomos automatically with the result (the system becomes the event loop, not the human). More plumbing; do only if A's long turns prove problematic.

Why this is the root fix, not another patch

Every prior fix made an individual command more likely to succeed. This makes the agent able to notice and respond when one doesn't — which is the only thing that generalizes to "do anything," because "anything" always includes "the first thing didn't work." You cannot enumerate every failure mode of an unbounded action space; you can give the agent a loop that observes and adapts.

Implementation order

1. Make provisioning observable: decompose pct_create into a fast atomic create + agent-orchestrated run steps for install/config/verify. (Biggest single win — removes the async black box from the failure-prone path.)
2. Robust window open: any approval / any forward-assent opens/extends it; pre-authorize plan-described destructive steps.
3. SOUL persist-through-errors framing + maxIterations bump.
4. Verify end-to-end (below). Only then consider B (auto-continuation).

Verification

• Re-run the exact TypeType deploy. Expected: operator approves the plan once; the agent then creates the container, installs docker (recovering from the Debian docker.io-CLI gap on its own by falling back to get.docker.com), brings up the stack, hits a transient error (e.g. Docker Hub 500), retries on its own, verifies :8082 responds, and reports success — with zero additional "continue" ticks from the operator.
• Failure injection: point a step at a wrong path; confirm the agent reads the error, adapts, and continues rather than ending the turn.

Open questions

• Scope of an autonomy window: per-plan, per-target, time-boxed (30 min now)? What exactly may the agent do inside it without asking again?
• Pre-authorized destructive steps: allow a plan to include a named destructive step (e.g. "destroy the half-provisioned CT and redo") that the agent may execute during recovery without a fresh typed confirmation, since the plan approval covered it? Or always re-confirm destructive, accepting the interruption?
• A vs B: is a single 5–10 min streaming turn acceptable, or do we need event-driven auto-continuation from the start?