oikos/plans/done/2026-07-10-autonomous-plan-execution.md

# 2026-07-10 — Autonomous plan execution: close the observation gap

**Status:** Done — 2026-07-11. Full scope shipped, including the Option B
stretch goal: atomic `pct_create` decomposition, robust assent-window
open/extend, SOUL persist-through-errors + `maxIterations=40`, and
event-driven auto-continuation (`cmd/nomos/continue.go`). Commits `233b5e4`,
`d2f749d`, `657e1a8`, `d529688`, `84ecb6b`.

## The real problem (not the one we kept fixing)

Operator, verbatim: *"the agent seems to stop when it encounters the first error,
it does not recover from it… my goal is that the agent can do anything once a
plan has been approved."*

We have patched ~10 individual failure modes (DNS, gateway, sshExec timeout,
substring bug, slug collisions, docker CLI, assent window…). Every one was real.
None fixed the thing the operator keeps hitting, because they all fixed
**individual commands** — and the problem is the **loop**, not the commands.

## Root cause: the agent never sees the result of the thing it started

The agent runs in discrete request→response turns. Provisioning executions are
**asynchronous**: `request_execution(pct_create)` queues an execution, fires the
real SSH work in a **goroutine** (`go executeApprovedViaAPI(...)`,
[internal/mcp/server.go](../internal/mcp/server.go)), and returns
*"provisioning now"* immediately. The multi-minute result lands in the DB
**after the agent's turn has already ended.**

So the agent literally is not running when the error happens. It cannot react to
a failure it never observes. The only way the result re-enters the agent's
reasoning is if a human types "continue" to start a new turn — **the human is the
event loop.** Read the failing session
(`7c25edaa`, 18 messages): the operator typed "continue" / "continue?" / "??" /
"proceed" **eight times**, each one just ticking the agent forward one async step.
The agent *was* recovering (it correctly diagnosed the docker-CLI issue and
proposed fixes) — it simply could not proceed one step without a human tick.

Two concrete asymmetries prove the diagnosis:

1. **`run` is synchronous, `pct_create` is async.** Inside an assent window, the
   general `run` tool executes the command inline and returns stdout/exit-status
   to the agent ([server.go](../internal/mcp/server.go) ~L514) — the agent *sees*
   the result and can continue. `pct_create` in the same window auto-approves and
   then `go`-routines the work — the agent sees nothing. The failure-prone path
   is the unobservable one.
2. **`pct_create` is monolithic and all-or-nothing.** It does create + apt +
   docker + post_install + verify in one SSH call. Even if it were synchronous,
   the agent could only see "the whole thing failed at some point," not step 3 of
   6 — so it can't surgically fix step 3 and resume. Recovery *requires*
   intermediate observation.

Secondary (real but downstream): "continue" is **not** an assent word
([cmd/nomos/assent.go](../cmd/nomos/assent.go)), so in that session the assent
window never even opened — every step stayed gated, compounding the ticking.

## The reframe: Nomos should work like a coding agent

A coding agent (Claude Code) runs a command, **sees the output**, runs the next,
fixes errors inline, all in one continuous session — it does not stop and ask a
human to forward it after each command. That is exactly "do anything once the
plan is approved." The homelab agent needs the same loop:

> approve the plan → agent runs step → **observes result** → runs next step / on
> failure diagnoses + adapts + retries → … → verifies goal met → reports.

The machinery for this **already exists** in the `run` tool (synchronous,
observable, auto-executing within an assent window). Provisioning just doesn't
use it — it uses a black box. The fix is to make the whole system consistent
with the model `run` already embodies.

## Target architecture

### 1. One observable primitive; retire the async black box

- Everything the agent does — including provisioning — is a sequence of
  **synchronous `run` calls** whose real output (stdout, stderr, exit code)
  returns inline. No goroutine hand-off for agent-initiated work.
- **Decompose `pct_create`.** Keep a thin `pct_create` that only does the fast,
  atomic container creation (create + start + register), returning synchronously.
  Move package install / service setup / post_install / verify **out** into
  agent-driven `run` steps. Now the agent observes each step and can fix a
  failed one without redoing the container.
- Net: the agent orchestrates `create → apt → install → configure → up → verify`,
  seeing each result, exactly like a human operator at a shell.

### 2. Approve the plan = an autonomy grant the agent executes to completion

- The assent/autonomy window already exists. Make it robust:
  - Opening it must not depend on a magic word list. "continue", "go", "do it",
    "proceed", clicking Approve, or approving the first queued step should all
    open/extend it. Safer: when the operator approves ANY step of a plan, treat
    that as opening the window for the rest of that plan.
  - Within the window: read-only + config_mutation `run` steps execute inline,
    no re-prompt. **Destructive still stops** for typed confirmation — but a
    destructive step *described in the approved plan* can be pre-authorized so
    the agent isn't blocked mid-flow on something already shown and approved.
- The window is the scope boundary: "you may do what the plan needs on this
  target; you may not wander outside it."

### 3. The agent persists through errors (prompt + loop)

- SOUL: "You are the executor of the approved plan. Run it step by step,
  observing each result. **On failure, do not stop and hand back — diagnose
  (read logs / inspect state), form a hypothesis, fix it, and retry or take an
  alternative path.** Continue until the goal is verified working or you are
  genuinely blocked (you need information only the operator has, or a step
  exceeds the approved scope). Never end a turn with a half-finished plan just
  because one command failed."
- `maxIterations` sized for a full provision-with-recovery (raise 25 → ~40) and
  count observation/read-only steps cheaply so recovery attempts aren't starved.

### 4. Long-running steps: keep the turn alive, or auto-continue

A synchronous `apt install` is ~1–2 min; a full stack up is longer. Options,
in preference order:
- **A (simplest, ship first):** synchronous `run` with the existing 10-min cap;
  the streaming turn stays open (the chat UI already holds the SSE). Emit
  progress events so the operator sees liveness (already built — elapsed timer).
- **B (for very long ops):** event-driven auto-continuation — when an async
  execution tied to an active plan completes, a worker **re-invokes Nomos**
  automatically with the result (the system becomes the event loop, not the
  human). More plumbing; do only if A's long turns prove problematic.

## Why this is the root fix, not another patch

Every prior fix made an individual command more likely to succeed. This makes
the agent able to **notice and respond when one doesn't** — which is the only
thing that generalizes to "do anything," because "anything" always includes
"the first thing didn't work." You cannot enumerate every failure mode of an
unbounded action space; you can give the agent a loop that observes and adapts.

## Implementation order

1. **Make provisioning observable**: decompose `pct_create` into a fast atomic
   create + agent-orchestrated `run` steps for install/config/verify. (Biggest
   single win — removes the async black box from the failure-prone path.)
2. **Robust window open**: any approval / any forward-assent opens/extends it;
   pre-authorize plan-described destructive steps.
3. **SOUL persist-through-errors** framing + `maxIterations` bump.
4. Verify end-to-end (below). Only then consider **B** (auto-continuation).

## Verification

- Re-run the exact TypeType deploy. Expected: operator approves the plan **once**;
  the agent then creates the container, installs docker (recovering from the
  Debian docker.io-CLI gap on its own by falling back to get.docker.com), brings
  up the stack, hits a transient error (e.g. Docker Hub 500), **retries on its
  own**, verifies `:8082` responds, and reports success — **with zero additional
  "continue" ticks from the operator.**
- Failure injection: point a step at a wrong path; confirm the agent reads the
  error, adapts, and continues rather than ending the turn.

## Open questions

- **Scope of an autonomy window**: per-plan, per-target, time-boxed (30 min now)?
  What exactly may the agent do inside it without asking again?
- **Pre-authorized destructive steps**: allow a plan to include a named
  destructive step (e.g. "destroy the half-provisioned CT and redo") that the
  agent may execute during recovery without a fresh typed confirmation, since
  the plan approval covered it? Or always re-confirm destructive, accepting the
  interruption?
- **A vs B**: is a single 5–10 min streaming turn acceptable, or do we need
  event-driven auto-continuation from the start?