dtoro
d2f749d33d
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
160 lines
8.7 KiB
Markdown
160 lines
8.7 KiB
Markdown
# 2026-07-10 — Autonomous plan execution: close the observation gap
|
||
|
||
**Status:** Planned
|
||
|
||
## The real problem (not the one we kept fixing)
|
||
|
||
Operator, verbatim: *"the agent seems to stop when it encounters the first error,
|
||
it does not recover from it… my goal is that the agent can do anything once a
|
||
plan has been approved."*
|
||
|
||
We have patched ~10 individual failure modes (DNS, gateway, sshExec timeout,
|
||
substring bug, slug collisions, docker CLI, assent window…). Every one was real.
|
||
None fixed the thing the operator keeps hitting, because they all fixed
|
||
**individual commands** — and the problem is the **loop**, not the commands.
|
||
|
||
## Root cause: the agent never sees the result of the thing it started
|
||
|
||
The agent runs in discrete request→response turns. Provisioning executions are
|
||
**asynchronous**: `request_execution(pct_create)` queues an execution, fires the
|
||
real SSH work in a **goroutine** (`go executeApprovedViaAPI(...)`,
|
||
[internal/mcp/server.go](../internal/mcp/server.go)), and returns
|
||
*"provisioning now"* immediately. The multi-minute result lands in the DB
|
||
**after the agent's turn has already ended.**
|
||
|
||
So the agent literally is not running when the error happens. It cannot react to
|
||
a failure it never observes. The only way the result re-enters the agent's
|
||
reasoning is if a human types "continue" to start a new turn — **the human is the
|
||
event loop.** Read the failing session
|
||
(`7c25edaa`, 18 messages): the operator typed "continue" / "continue?" / "??" /
|
||
"proceed" **eight times**, each one just ticking the agent forward one async step.
|
||
The agent *was* recovering (it correctly diagnosed the docker-CLI issue and
|
||
proposed fixes) — it simply could not proceed one step without a human tick.
|
||
|
||
Two concrete asymmetries prove the diagnosis:
|
||
|
||
1. **`run` is synchronous, `pct_create` is async.** Inside an assent window, the
|
||
general `run` tool executes the command inline and returns stdout/exit-status
|
||
to the agent ([server.go](../internal/mcp/server.go) ~L514) — the agent *sees*
|
||
the result and can continue. `pct_create` in the same window auto-approves and
|
||
then `go`-routines the work — the agent sees nothing. The failure-prone path
|
||
is the unobservable one.
|
||
2. **`pct_create` is monolithic and all-or-nothing.** It does create + apt +
|
||
docker + post_install + verify in one SSH call. Even if it were synchronous,
|
||
the agent could only see "the whole thing failed at some point," not step 3 of
|
||
6 — so it can't surgically fix step 3 and resume. Recovery *requires*
|
||
intermediate observation.
|
||
|
||
Secondary (real but downstream): "continue" is **not** an assent word
|
||
([cmd/nomos/assent.go](../cmd/nomos/assent.go)), so in that session the assent
|
||
window never even opened — every step stayed gated, compounding the ticking.
|
||
|
||
## The reframe: Nomos should work like a coding agent
|
||
|
||
A coding agent (Claude Code) runs a command, **sees the output**, runs the next,
|
||
fixes errors inline, all in one continuous session — it does not stop and ask a
|
||
human to forward it after each command. That is exactly "do anything once the
|
||
plan is approved." The homelab agent needs the same loop:
|
||
|
||
> approve the plan → agent runs step → **observes result** → runs next step / on
|
||
> failure diagnoses + adapts + retries → … → verifies goal met → reports.
|
||
|
||
The machinery for this **already exists** in the `run` tool (synchronous,
|
||
observable, auto-executing within an assent window). Provisioning just doesn't
|
||
use it — it uses a black box. The fix is to make the whole system consistent
|
||
with the model `run` already embodies.
|
||
|
||
## Target architecture
|
||
|
||
### 1. One observable primitive; retire the async black box
|
||
|
||
- Everything the agent does — including provisioning — is a sequence of
|
||
**synchronous `run` calls** whose real output (stdout, stderr, exit code)
|
||
returns inline. No goroutine hand-off for agent-initiated work.
|
||
- **Decompose `pct_create`.** Keep a thin `pct_create` that only does the fast,
|
||
atomic container creation (create + start + register), returning synchronously.
|
||
Move package install / service setup / post_install / verify **out** into
|
||
agent-driven `run` steps. Now the agent observes each step and can fix a
|
||
failed one without redoing the container.
|
||
- Net: the agent orchestrates `create → apt → install → configure → up → verify`,
|
||
seeing each result, exactly like a human operator at a shell.
|
||
|
||
### 2. Approve the plan = an autonomy grant the agent executes to completion
|
||
|
||
- The assent/autonomy window already exists. Make it robust:
|
||
- Opening it must not depend on a magic word list. "continue", "go", "do it",
|
||
"proceed", clicking Approve, or approving the first queued step should all
|
||
open/extend it. Safer: when the operator approves ANY step of a plan, treat
|
||
that as opening the window for the rest of that plan.
|
||
- Within the window: read-only + config_mutation `run` steps execute inline,
|
||
no re-prompt. **Destructive still stops** for typed confirmation — but a
|
||
destructive step *described in the approved plan* can be pre-authorized so
|
||
the agent isn't blocked mid-flow on something already shown and approved.
|
||
- The window is the scope boundary: "you may do what the plan needs on this
|
||
target; you may not wander outside it."
|
||
|
||
### 3. The agent persists through errors (prompt + loop)
|
||
|
||
- SOUL: "You are the executor of the approved plan. Run it step by step,
|
||
observing each result. **On failure, do not stop and hand back — diagnose
|
||
(read logs / inspect state), form a hypothesis, fix it, and retry or take an
|
||
alternative path.** Continue until the goal is verified working or you are
|
||
genuinely blocked (you need information only the operator has, or a step
|
||
exceeds the approved scope). Never end a turn with a half-finished plan just
|
||
because one command failed."
|
||
- `maxIterations` sized for a full provision-with-recovery (raise 25 → ~40) and
|
||
count observation/read-only steps cheaply so recovery attempts aren't starved.
|
||
|
||
### 4. Long-running steps: keep the turn alive, or auto-continue
|
||
|
||
A synchronous `apt install` is ~1–2 min; a full stack up is longer. Options,
|
||
in preference order:
|
||
- **A (simplest, ship first):** synchronous `run` with the existing 10-min cap;
|
||
the streaming turn stays open (the chat UI already holds the SSE). Emit
|
||
progress events so the operator sees liveness (already built — elapsed timer).
|
||
- **B (for very long ops):** event-driven auto-continuation — when an async
|
||
execution tied to an active plan completes, a worker **re-invokes Nomos**
|
||
automatically with the result (the system becomes the event loop, not the
|
||
human). More plumbing; do only if A's long turns prove problematic.
|
||
|
||
## Why this is the root fix, not another patch
|
||
|
||
Every prior fix made an individual command more likely to succeed. This makes
|
||
the agent able to **notice and respond when one doesn't** — which is the only
|
||
thing that generalizes to "do anything," because "anything" always includes
|
||
"the first thing didn't work." You cannot enumerate every failure mode of an
|
||
unbounded action space; you can give the agent a loop that observes and adapts.
|
||
|
||
## Implementation order
|
||
|
||
1. **Make provisioning observable**: decompose `pct_create` into a fast atomic
|
||
create + agent-orchestrated `run` steps for install/config/verify. (Biggest
|
||
single win — removes the async black box from the failure-prone path.)
|
||
2. **Robust window open**: any approval / any forward-assent opens/extends it;
|
||
pre-authorize plan-described destructive steps.
|
||
3. **SOUL persist-through-errors** framing + `maxIterations` bump.
|
||
4. Verify end-to-end (below). Only then consider **B** (auto-continuation).
|
||
|
||
## Verification
|
||
|
||
- Re-run the exact TypeType deploy. Expected: operator approves the plan **once**;
|
||
the agent then creates the container, installs docker (recovering from the
|
||
Debian docker.io-CLI gap on its own by falling back to get.docker.com), brings
|
||
up the stack, hits a transient error (e.g. Docker Hub 500), **retries on its
|
||
own**, verifies `:8082` responds, and reports success — **with zero additional
|
||
"continue" ticks from the operator.**
|
||
- Failure injection: point a step at a wrong path; confirm the agent reads the
|
||
error, adapts, and continues rather than ending the turn.
|
||
|
||
## Open questions
|
||
|
||
- **Scope of an autonomy window**: per-plan, per-target, time-boxed (30 min now)?
|
||
What exactly may the agent do inside it without asking again?
|
||
- **Pre-authorized destructive steps**: allow a plan to include a named
|
||
destructive step (e.g. "destroy the half-provisioned CT and redo") that the
|
||
agent may execute during recovery without a fresh typed confirmation, since
|
||
the plan approval covered it? Or always re-confirm destructive, accepting the
|
||
interruption?
|
||
- **A vs B**: is a single 5–10 min streaming turn acceptable, or do we need
|
||
event-driven auto-continuation from the start?
|