feat: event-driven auto-continuation — agent runs an approved plan to completion
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -154,6 +154,16 @@ type agentEvent struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
||||||
|
a.chatWith(ctx, sessionID, message, "", emit)
|
||||||
|
}
|
||||||
|
|
||||||
|
// chatWith is chat() with an optional system-injected note appended after the
|
||||||
|
// replayed history. The auto-continuation worker uses it to resume a session
|
||||||
|
// with a finished execution's result ("execution X completed: … — continue the
|
||||||
|
// plan") without persisting a fake user turn. message is normally the new user
|
||||||
|
// message; for a worker continuation it is empty and systemInject carries the
|
||||||
|
// note.
|
||||||
|
func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject string, emit func(agentEvent)) {
|
||||||
correlationID := uuid.New().String()
|
correlationID := uuid.New().String()
|
||||||
|
|
||||||
tools, err := a.buildTools()
|
tools, err := a.buildTools()
|
||||||
@@ -244,6 +254,12 @@ func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(a
|
|||||||
a.openAssentWindow(ctx)
|
a.openAssentWindow(ctx)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Worker continuation: append the finished-execution note so the model
|
||||||
|
// sees the result and decides the next step (proceed / recover / done).
|
||||||
|
if systemInject != "" {
|
||||||
|
messages = append(messages, openai.SystemMessage(systemInject))
|
||||||
|
}
|
||||||
|
|
||||||
for i := 0; i < maxIterations; i++ {
|
for i := 0; i < maxIterations; i++ {
|
||||||
params := openai.ChatCompletionNewParams{
|
params := openai.ChatCompletionNewParams{
|
||||||
Model: openai.ChatModel(a.model),
|
Model: openai.ChatModel(a.model),
|
||||||
@@ -352,6 +368,15 @@ func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(a
|
|||||||
resultJSON, _ := json.Marshal(result)
|
resultJSON, _ := json.Marshal(result)
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, inputStr, string(resultJSON), elapsed, true, correlationID)
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, inputStr, string(resultJSON), elapsed, true, correlationID)
|
||||||
|
|
||||||
|
// Link any execution this tool queued/started back to this
|
||||||
|
// session, so the auto-continuation worker can feed its result
|
||||||
|
// back here when it finishes (see cmd/nomos/continue.go). Async
|
||||||
|
// executions (pct_create, apt_upgrade) are the ones that matter —
|
||||||
|
// their result lands after this turn ends.
|
||||||
|
for _, execID := range extractExecutionIDs(string(resultJSON)) {
|
||||||
|
a.store.linkExecution(ctx, execID, sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
emit(agentEvent{
|
emit(agentEvent{
|
||||||
Type: "tool_result",
|
Type: "tool_result",
|
||||||
Data: map[string]any{"name": tc.Function.Name, "result": result, "id": tc.ID},
|
Data: map[string]any{"name": tc.Function.Name, "result": result, "id": tc.ID},
|
||||||
|
|||||||
135
cmd/nomos/continue.go
Normal file
135
cmd/nomos/continue.go
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// execIDRe matches "execution <uuid>" in a tool result — the phrasing shared
|
||||||
|
// by request_execution / run when they queue or start a gated execution.
|
||||||
|
// Only these async executions need continuation; the synchronous auto-run
|
||||||
|
// path returns its output inline and is already observed in-turn.
|
||||||
|
var execIDRe = regexp.MustCompile(`(?i)execution\s+([0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})`)
|
||||||
|
|
||||||
|
func extractExecutionIDs(toolResult string) []uuid.UUID {
|
||||||
|
matches := execIDRe.FindAllStringSubmatch(toolResult, -1)
|
||||||
|
seen := map[uuid.UUID]bool{}
|
||||||
|
var out []uuid.UUID
|
||||||
|
for _, m := range matches {
|
||||||
|
if id, err := uuid.Parse(m[1]); err == nil && !seen[id] {
|
||||||
|
seen[id] = true
|
||||||
|
out = append(out, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// runContinuationWorker is the event loop that replaces the human typing
|
||||||
|
// "continue". It polls for gated executions that (a) were initiated by a chat
|
||||||
|
// session and (b) have just finished, and — while that agent has an open assent
|
||||||
|
// window (an approved plan is in flight) — feeds each result back into the
|
||||||
|
// agent so it proceeds to the next step or recovers from the failure, all
|
||||||
|
// without an operator tick. Blocks until ctx is cancelled.
|
||||||
|
func (a *agent) runContinuationWorker(ctx context.Context) {
|
||||||
|
if a.store == nil {
|
||||||
|
slog.Warn("nomos: continuation worker disabled (no store)")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
slog.Info("nomos: continuation worker started")
|
||||||
|
ticker := time.NewTicker(4 * time.Second)
|
||||||
|
defer ticker.Stop()
|
||||||
|
for {
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return
|
||||||
|
case <-ticker.C:
|
||||||
|
a.processContinuations(ctx)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (a *agent) processContinuations(ctx context.Context) {
|
||||||
|
pending := a.store.pendingContinuations(ctx, 5)
|
||||||
|
windowOpen := a.store.assentWindowActive(ctx, a.agentID)
|
||||||
|
for _, p := range pending {
|
||||||
|
// Scope gate: only auto-continue while an approved plan is active.
|
||||||
|
// A finished one-off execution with no window is left as-is (marked
|
||||||
|
// continued so we don't re-check it forever) — the operator decides
|
||||||
|
// what happens next, as today.
|
||||||
|
if !windowOpen {
|
||||||
|
a.store.markContinued(ctx, p.ExecID)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
|
||||||
|
a.continueSession(ctx, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// continueSession re-invokes the agent for one finished execution, persisting
|
||||||
|
// the resulting assistant turn just like handleChat does. The operator sees it
|
||||||
|
// on their next load of the session (live push is a follow-up).
|
||||||
|
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||||
|
note := buildContinuationNote(p)
|
||||||
|
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||||
|
|
||||||
|
var toolCalls []map[string]any
|
||||||
|
var finalText string
|
||||||
|
emit := func(ev agentEvent) {
|
||||||
|
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||||
|
if m, ok := ev.Data.(map[string]any); ok {
|
||||||
|
m["type"] = ev.Type
|
||||||
|
toolCalls = append(toolCalls, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ev.Type == "text" {
|
||||||
|
finalText, _ = ev.Data.(string)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Use a generous timeout: a continuation may itself launch further steps.
|
||||||
|
cctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
|
||||||
|
defer cancel()
|
||||||
|
a.chatWith(cctx, p.SessionID, "", note, emit)
|
||||||
|
|
||||||
|
assistantMsg, _ := json.Marshal(map[string]any{
|
||||||
|
"role": "assistant",
|
||||||
|
"text": finalText,
|
||||||
|
"tool_calls": toolCalls,
|
||||||
|
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
||||||
|
})
|
||||||
|
a.store.saveMessage(ctx, p.SessionID, "assistant", assistantMsg)
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildContinuationNote frames the finished execution for the model: what
|
||||||
|
// happened, and what to do about it. The persist-through-errors instruction
|
||||||
|
// lives here (and in SOUL) so the agent recovers instead of stopping.
|
||||||
|
func buildContinuationNote(p pendingContinuation) string {
|
||||||
|
action := p.Action
|
||||||
|
if i := strings.IndexByte(action, ':'); i > 0 && len(action) > 40 {
|
||||||
|
action = action[:i] // keep just the action verb for brevity; params are in the DB
|
||||||
|
}
|
||||||
|
result := p.Result
|
||||||
|
if len(result) > 3000 {
|
||||||
|
result = result[:3000] + "…[truncated]"
|
||||||
|
}
|
||||||
|
var b strings.Builder
|
||||||
|
fmt.Fprintf(&b, "[System: execution %s (%s) finished with status=%s.\nResult: %s\n\n",
|
||||||
|
p.ExecID, action, p.Status, result)
|
||||||
|
switch p.Status {
|
||||||
|
case "completed":
|
||||||
|
b.WriteString("It SUCCEEDED. Continue the approved plan: run the next step. If this was the final step, verify the end goal actually works (e.g. curl the service) and then report success to the operator. Do NOT stop and wait for the operator to say 'continue'.")
|
||||||
|
case "failed", "cancelled":
|
||||||
|
b.WriteString("It FAILED. Do NOT give up or hand back to the operator. Diagnose the cause from the result above (and by running read-only inspection commands if needed), form a hypothesis, fix it, and retry or take an alternative approach. You have an active assent window, so config_mutation steps run without re-approval. Only stop and ask the operator if you are genuinely blocked (need information only they have) or the fix would require a destructive action they haven't approved.")
|
||||||
|
default: // denied / revoked
|
||||||
|
b.WriteString("The operator denied or revoked this step. Stop executing this plan and briefly acknowledge.")
|
||||||
|
}
|
||||||
|
b.WriteString("]")
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
39
cmd/nomos/continue_test.go
Normal file
39
cmd/nomos/continue_test.go
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestExtractExecutionIDs(t *testing.T) {
|
||||||
|
// Real tool-result phrasings that should yield an execution id.
|
||||||
|
pos := map[string]string{
|
||||||
|
`"pct_create on host:strong auto-approved via assent window — execution 019f4b19-eafd-74ed-baa6-d24a27b3f52c running."`: "019f4b19-eafd-74ed-baa6-d24a27b3f52c",
|
||||||
|
`"run on lxc:caddy requires approval (risk: config_mutation) — execution 019f4af7-7eff-7723-b38c-b540b267f407 queued."`: "019f4af7-7eff-7723-b38c-b540b267f407",
|
||||||
|
`"apt_upgrade on host:hubris auto-approved via assent window — execution 019f4b58-c88c-7767-87dd-044608ced913 running."`: "019f4b58-c88c-7767-87dd-044608ced913",
|
||||||
|
}
|
||||||
|
for in, want := range pos {
|
||||||
|
ids := extractExecutionIDs(in)
|
||||||
|
if len(ids) != 1 || ids[0].String() != want {
|
||||||
|
t.Errorf("extractExecutionIDs(%q) = %v, want [%s]", in, ids, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Synchronous auto-run and read-only results carry no "execution <uuid>"
|
||||||
|
// phrasing — they've already completed inline and must NOT be linked for
|
||||||
|
// continuation.
|
||||||
|
neg := []string{
|
||||||
|
`"run on host:strong (read_only, auto): 09:30 up 8 days"`,
|
||||||
|
`"run on lxc:caddy (config_mutation, auto via assent window): done"`,
|
||||||
|
`[{"slug":"lxc:caddy","health":"healthy"}]`,
|
||||||
|
`"target not found: lxc:nope"`,
|
||||||
|
}
|
||||||
|
for _, in := range neg {
|
||||||
|
if ids := extractExecutionIDs(in); len(ids) != 0 {
|
||||||
|
t.Errorf("extractExecutionIDs(%q) = %v, want none", in, ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// De-dupes repeated ids in one result.
|
||||||
|
dup := `execution 019f4b19-eafd-74ed-baa6-d24a27b3f52c queued ... execution 019f4b19-eafd-74ed-baa6-d24a27b3f52c running`
|
||||||
|
if ids := extractExecutionIDs(dup); len(ids) != 1 {
|
||||||
|
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -63,6 +63,11 @@ func main() {
|
|||||||
os.Exit(1)
|
os.Exit(1)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Event-driven auto-continuation: feed finished async executions back
|
||||||
|
// into the agent so an approved plan runs to completion (and recovers
|
||||||
|
// from failures) without the operator ticking it forward each step.
|
||||||
|
go nAgent.runContinuationWorker(ctx)
|
||||||
|
|
||||||
mux := http.NewServeMux()
|
mux := http.NewServeMux()
|
||||||
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
|
mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
|
||||||
w.WriteHeader(200)
|
w.WriteHeader(200)
|
||||||
|
|||||||
@@ -196,6 +196,83 @@ func (s *store) resolveAgentID(ctx context.Context, slug string) uuid.UUID {
|
|||||||
return id
|
return id
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// linkExecution records that a gated execution was initiated by a chat
|
||||||
|
// session, so the auto-continuation worker can feed its result back to that
|
||||||
|
// session when it finishes. Idempotent — the same execution may appear in
|
||||||
|
// several tool results across a turn.
|
||||||
|
func (s *store) linkExecution(ctx context.Context, execID uuid.UUID, sessionID string) {
|
||||||
|
if s == nil || execID == uuid.Nil || sessionID == "" || sessionID == "ephemeral" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO nomos_plan_executions (execution_id, session_id)
|
||||||
|
VALUES ($1, $2) ON CONFLICT (execution_id) DO NOTHING`, execID, sessionID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// pendingContinuation is one finished execution whose result hasn't yet been
|
||||||
|
// fed back to its originating session.
|
||||||
|
type pendingContinuation struct {
|
||||||
|
ExecID uuid.UUID
|
||||||
|
SessionID string
|
||||||
|
Status string
|
||||||
|
Result string
|
||||||
|
Action string
|
||||||
|
}
|
||||||
|
|
||||||
|
// pendingContinuations returns executions that have reached a terminal state
|
||||||
|
// but haven't been continued yet — the worker's work list. Bounded so one
|
||||||
|
// tick can't fan out unboundedly.
|
||||||
|
func (s *store) pendingContinuations(ctx context.Context, limit int) []pendingContinuation {
|
||||||
|
if s == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
rows, err := s.pool.Query(ctx, `
|
||||||
|
SELECT l.execution_id, l.session_id, e.status,
|
||||||
|
COALESCE(e.result::text, ''), COALESCE(e.action, '')
|
||||||
|
FROM nomos_plan_executions l
|
||||||
|
JOIN executions e ON e.entity_id = l.execution_id
|
||||||
|
WHERE l.continued_at IS NULL
|
||||||
|
AND e.status IN ('completed', 'failed', 'cancelled', 'denied', 'revoked')
|
||||||
|
ORDER BY l.created_at
|
||||||
|
LIMIT $1`, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []pendingContinuation
|
||||||
|
for rows.Next() {
|
||||||
|
var p pendingContinuation
|
||||||
|
if err := rows.Scan(&p.ExecID, &p.SessionID, &p.Status, &p.Result, &p.Action); err == nil {
|
||||||
|
out = append(out, p)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// markContinued stamps an execution as fed-back so the worker won't process it
|
||||||
|
// again (prevents an auto-continuation loop).
|
||||||
|
func (s *store) markContinued(ctx context.Context, execID uuid.UUID) {
|
||||||
|
if s == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
s.pool.Exec(ctx, `UPDATE nomos_plan_executions SET continued_at = now() WHERE execution_id = $1`, execID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// assentWindowActive reports whether this agent currently has an open assent
|
||||||
|
// window — the scope gate for auto-continuation. We only auto-continue
|
||||||
|
// executions that are part of an approved plan, never stray one-off actions.
|
||||||
|
func (s *store) assentWindowActive(ctx context.Context, agentID uuid.UUID) bool {
|
||||||
|
if s == nil || agentID == uuid.Nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
var expires time.Time
|
||||||
|
key := "assent_window.agent:" + agentID.String()
|
||||||
|
if err := s.pool.QueryRow(ctx, `SELECT value::timestamptz FROM autonomy_settings WHERE key = $1`, key).Scan(&expires); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return time.Now().Before(expires)
|
||||||
|
}
|
||||||
|
|
||||||
// logActivity records a tool call. agent_id is the agent entity UUID and is
|
// logActivity records a tool call. agent_id is the agent entity UUID and is
|
||||||
// NOT NULL in the schema, so we skip logging when it can't be resolved.
|
// NOT NULL in the schema, so we skip logging when it can't be resolved.
|
||||||
// The (nullable) session_id column carries the conversation id.
|
// The (nullable) session_id column carries the conversation id.
|
||||||
|
|||||||
@@ -1491,6 +1491,21 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
|||||||
// for every other risk class, including destructive.
|
// for every other risk class, including destructive.
|
||||||
_, _ = tx.Exec(ctx, `UPDATE executions SET status = 'approved' WHERE entity_id = $1`, execID)
|
_, _ = tx.Exec(ctx, `UPDATE executions SET status = 'approved' WHERE entity_id = $1`, execID)
|
||||||
|
|
||||||
|
// Approving a plan step — by ANY route (this endpoint backs both
|
||||||
|
// the chat Approve button and chat-assent) — opens/extends the
|
||||||
|
// agent's assent window. This is the scope gate the Nomos
|
||||||
|
// auto-continuation worker checks: with the window open, the
|
||||||
|
// finished execution's result is fed back to the agent so it runs
|
||||||
|
// the plan to completion. Without opening it here, approving via
|
||||||
|
// the button (instead of typing "go ahead") would silently not
|
||||||
|
// auto-continue.
|
||||||
|
var agentID *uuid.UUID
|
||||||
|
if qerr := tx.QueryRow(ctx, "SELECT agent_id FROM executions WHERE entity_id = $1", execID).Scan(&agentID); qerr == nil && agentID != nil {
|
||||||
|
expires := time.Now().Add(30 * time.Minute).UTC().Format(time.RFC3339)
|
||||||
|
_, _ = tx.Exec(ctx, `INSERT INTO autonomy_settings (key, value) VALUES ($1, $2)
|
||||||
|
ON CONFLICT (key) DO UPDATE SET value = $2`, "assent_window.agent:"+agentID.String(), expires)
|
||||||
|
}
|
||||||
|
|
||||||
slog.Info("httpapi: approved execution queued",
|
slog.Info("httpapi: approved execution queued",
|
||||||
"execution_id", execID, "target", targetSlug, "action", actionStr)
|
"execution_id", execID, "target", targetSlug, "action", actionStr)
|
||||||
} else {
|
} else {
|
||||||
|
|||||||
25
migrations/017_nomos_plan_executions.up.sql
Normal file
25
migrations/017_nomos_plan_executions.up.sql
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
-- 017_nomos_plan_executions.up.sql
|
||||||
|
-- Links a gated execution back to the chat session that initiated it, so the
|
||||||
|
-- Nomos auto-continuation worker can re-invoke the agent for that session when
|
||||||
|
-- the (asynchronous) execution finishes. This is the "the system is the event
|
||||||
|
-- loop, not the human" foundation: the human no longer types "continue" after
|
||||||
|
-- every async step — the worker feeds each execution's result back into the
|
||||||
|
-- agent automatically.
|
||||||
|
--
|
||||||
|
-- Owned by the nomos process. execution_id references the execution entity by
|
||||||
|
-- UUID but intentionally without a hard FK — nomos records the link from the
|
||||||
|
-- tool-result text it gets back, and we don't want a race between the API
|
||||||
|
-- creating the execution entity and nomos linking it to break the insert.
|
||||||
|
CREATE TABLE IF NOT EXISTS nomos_plan_executions (
|
||||||
|
execution_id UUID PRIMARY KEY,
|
||||||
|
session_id UUID NOT NULL,
|
||||||
|
-- when the worker fed this execution's result back to the agent (NULL = not yet)
|
||||||
|
continued_at TIMESTAMPTZ,
|
||||||
|
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Worker query: find terminal executions not yet fed back. Partial index on the
|
||||||
|
-- not-yet-continued rows keeps the poll cheap as history accumulates.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_nomos_plan_exec_pending
|
||||||
|
ON nomos_plan_executions (created_at)
|
||||||
|
WHERE continued_at IS NULL;
|
||||||
@@ -193,6 +193,19 @@ operator if:
|
|||||||
Do NOT stop after every step waiting for "continue". The operator approved
|
Do NOT stop after every step waiting for "continue". The operator approved
|
||||||
the plan — execute it end to end.
|
the plan — execute it end to end.
|
||||||
|
|
||||||
|
**Automatic continuation — you are re-invoked when async steps finish.** Some
|
||||||
|
steps (`pct_create`, `apt_upgrade`) run asynchronously: the tool returns
|
||||||
|
"execution <id> running" immediately, and the actual work (which can take
|
||||||
|
minutes) finishes later. **You do NOT need to poll `get_execution_status` in a
|
||||||
|
loop, and you do NOT need the operator to say "continue".** When such a step
|
||||||
|
finishes, the system automatically re-invokes you with a
|
||||||
|
`[System: execution <id> finished with status=…]` note carrying the result.
|
||||||
|
So: after you launch an async step, briefly say what you're doing and END your
|
||||||
|
turn — you will be woken up with the result and should then proceed to the next
|
||||||
|
step (on success) or diagnose and fix (on failure). Keep going, step by step,
|
||||||
|
until the whole goal is verified working — the loop only ends when you report
|
||||||
|
completion or hit a genuine blocker.
|
||||||
|
|
||||||
**When a step fails:** diagnose the error, try an alternative approach, and
|
**When a step fails:** diagnose the error, try an alternative approach, and
|
||||||
continue. For example, if `docker: command not found` appears, install Docker
|
continue. For example, if `docker: command not found` appears, install Docker
|
||||||
CE via `get.docker.com` and retry. If a package is missing, install it. If a
|
CE via `get.docker.com` and retry. If a package is missing, install it. If a
|
||||||
|
|||||||
159
plans/2026-07-10-autonomous-plan-execution.md
Normal file
159
plans/2026-07-10-autonomous-plan-execution.md
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
# 2026-07-10 — Autonomous plan execution: close the observation gap
|
||||||
|
|
||||||
|
**Status:** Planned
|
||||||
|
|
||||||
|
## The real problem (not the one we kept fixing)
|
||||||
|
|
||||||
|
Operator, verbatim: *"the agent seems to stop when it encounters the first error,
|
||||||
|
it does not recover from it… my goal is that the agent can do anything once a
|
||||||
|
plan has been approved."*
|
||||||
|
|
||||||
|
We have patched ~10 individual failure modes (DNS, gateway, sshExec timeout,
|
||||||
|
substring bug, slug collisions, docker CLI, assent window…). Every one was real.
|
||||||
|
None fixed the thing the operator keeps hitting, because they all fixed
|
||||||
|
**individual commands** — and the problem is the **loop**, not the commands.
|
||||||
|
|
||||||
|
## Root cause: the agent never sees the result of the thing it started
|
||||||
|
|
||||||
|
The agent runs in discrete request→response turns. Provisioning executions are
|
||||||
|
**asynchronous**: `request_execution(pct_create)` queues an execution, fires the
|
||||||
|
real SSH work in a **goroutine** (`go executeApprovedViaAPI(...)`,
|
||||||
|
[internal/mcp/server.go](../internal/mcp/server.go)), and returns
|
||||||
|
*"provisioning now"* immediately. The multi-minute result lands in the DB
|
||||||
|
**after the agent's turn has already ended.**
|
||||||
|
|
||||||
|
So the agent literally is not running when the error happens. It cannot react to
|
||||||
|
a failure it never observes. The only way the result re-enters the agent's
|
||||||
|
reasoning is if a human types "continue" to start a new turn — **the human is the
|
||||||
|
event loop.** Read the failing session
|
||||||
|
(`7c25edaa`, 18 messages): the operator typed "continue" / "continue?" / "??" /
|
||||||
|
"proceed" **eight times**, each one just ticking the agent forward one async step.
|
||||||
|
The agent *was* recovering (it correctly diagnosed the docker-CLI issue and
|
||||||
|
proposed fixes) — it simply could not proceed one step without a human tick.
|
||||||
|
|
||||||
|
Two concrete asymmetries prove the diagnosis:
|
||||||
|
|
||||||
|
1. **`run` is synchronous, `pct_create` is async.** Inside an assent window, the
|
||||||
|
general `run` tool executes the command inline and returns stdout/exit-status
|
||||||
|
to the agent ([server.go](../internal/mcp/server.go) ~L514) — the agent *sees*
|
||||||
|
the result and can continue. `pct_create` in the same window auto-approves and
|
||||||
|
then `go`-routines the work — the agent sees nothing. The failure-prone path
|
||||||
|
is the unobservable one.
|
||||||
|
2. **`pct_create` is monolithic and all-or-nothing.** It does create + apt +
|
||||||
|
docker + post_install + verify in one SSH call. Even if it were synchronous,
|
||||||
|
the agent could only see "the whole thing failed at some point," not step 3 of
|
||||||
|
6 — so it can't surgically fix step 3 and resume. Recovery *requires*
|
||||||
|
intermediate observation.
|
||||||
|
|
||||||
|
Secondary (real but downstream): "continue" is **not** an assent word
|
||||||
|
([cmd/nomos/assent.go](../cmd/nomos/assent.go)), so in that session the assent
|
||||||
|
window never even opened — every step stayed gated, compounding the ticking.
|
||||||
|
|
||||||
|
## The reframe: Nomos should work like a coding agent
|
||||||
|
|
||||||
|
A coding agent (Claude Code) runs a command, **sees the output**, runs the next,
|
||||||
|
fixes errors inline, all in one continuous session — it does not stop and ask a
|
||||||
|
human to forward it after each command. That is exactly "do anything once the
|
||||||
|
plan is approved." The homelab agent needs the same loop:
|
||||||
|
|
||||||
|
> approve the plan → agent runs step → **observes result** → runs next step / on
|
||||||
|
> failure diagnoses + adapts + retries → … → verifies goal met → reports.
|
||||||
|
|
||||||
|
The machinery for this **already exists** in the `run` tool (synchronous,
|
||||||
|
observable, auto-executing within an assent window). Provisioning just doesn't
|
||||||
|
use it — it uses a black box. The fix is to make the whole system consistent
|
||||||
|
with the model `run` already embodies.
|
||||||
|
|
||||||
|
## Target architecture
|
||||||
|
|
||||||
|
### 1. One observable primitive; retire the async black box
|
||||||
|
|
||||||
|
- Everything the agent does — including provisioning — is a sequence of
|
||||||
|
**synchronous `run` calls** whose real output (stdout, stderr, exit code)
|
||||||
|
returns inline. No goroutine hand-off for agent-initiated work.
|
||||||
|
- **Decompose `pct_create`.** Keep a thin `pct_create` that only does the fast,
|
||||||
|
atomic container creation (create + start + register), returning synchronously.
|
||||||
|
Move package install / service setup / post_install / verify **out** into
|
||||||
|
agent-driven `run` steps. Now the agent observes each step and can fix a
|
||||||
|
failed one without redoing the container.
|
||||||
|
- Net: the agent orchestrates `create → apt → install → configure → up → verify`,
|
||||||
|
seeing each result, exactly like a human operator at a shell.
|
||||||
|
|
||||||
|
### 2. Approve the plan = an autonomy grant the agent executes to completion
|
||||||
|
|
||||||
|
- The assent/autonomy window already exists. Make it robust:
|
||||||
|
- Opening it must not depend on a magic word list. "continue", "go", "do it",
|
||||||
|
"proceed", clicking Approve, or approving the first queued step should all
|
||||||
|
open/extend it. Safer: when the operator approves ANY step of a plan, treat
|
||||||
|
that as opening the window for the rest of that plan.
|
||||||
|
- Within the window: read-only + config_mutation `run` steps execute inline,
|
||||||
|
no re-prompt. **Destructive still stops** for typed confirmation — but a
|
||||||
|
destructive step *described in the approved plan* can be pre-authorized so
|
||||||
|
the agent isn't blocked mid-flow on something already shown and approved.
|
||||||
|
- The window is the scope boundary: "you may do what the plan needs on this
|
||||||
|
target; you may not wander outside it."
|
||||||
|
|
||||||
|
### 3. The agent persists through errors (prompt + loop)
|
||||||
|
|
||||||
|
- SOUL: "You are the executor of the approved plan. Run it step by step,
|
||||||
|
observing each result. **On failure, do not stop and hand back — diagnose
|
||||||
|
(read logs / inspect state), form a hypothesis, fix it, and retry or take an
|
||||||
|
alternative path.** Continue until the goal is verified working or you are
|
||||||
|
genuinely blocked (you need information only the operator has, or a step
|
||||||
|
exceeds the approved scope). Never end a turn with a half-finished plan just
|
||||||
|
because one command failed."
|
||||||
|
- `maxIterations` sized for a full provision-with-recovery (raise 25 → ~40) and
|
||||||
|
count observation/read-only steps cheaply so recovery attempts aren't starved.
|
||||||
|
|
||||||
|
### 4. Long-running steps: keep the turn alive, or auto-continue
|
||||||
|
|
||||||
|
A synchronous `apt install` is ~1–2 min; a full stack up is longer. Options,
|
||||||
|
in preference order:
|
||||||
|
- **A (simplest, ship first):** synchronous `run` with the existing 10-min cap;
|
||||||
|
the streaming turn stays open (the chat UI already holds the SSE). Emit
|
||||||
|
progress events so the operator sees liveness (already built — elapsed timer).
|
||||||
|
- **B (for very long ops):** event-driven auto-continuation — when an async
|
||||||
|
execution tied to an active plan completes, a worker **re-invokes Nomos**
|
||||||
|
automatically with the result (the system becomes the event loop, not the
|
||||||
|
human). More plumbing; do only if A's long turns prove problematic.
|
||||||
|
|
||||||
|
## Why this is the root fix, not another patch
|
||||||
|
|
||||||
|
Every prior fix made an individual command more likely to succeed. This makes
|
||||||
|
the agent able to **notice and respond when one doesn't** — which is the only
|
||||||
|
thing that generalizes to "do anything," because "anything" always includes
|
||||||
|
"the first thing didn't work." You cannot enumerate every failure mode of an
|
||||||
|
unbounded action space; you can give the agent a loop that observes and adapts.
|
||||||
|
|
||||||
|
## Implementation order
|
||||||
|
|
||||||
|
1. **Make provisioning observable**: decompose `pct_create` into a fast atomic
|
||||||
|
create + agent-orchestrated `run` steps for install/config/verify. (Biggest
|
||||||
|
single win — removes the async black box from the failure-prone path.)
|
||||||
|
2. **Robust window open**: any approval / any forward-assent opens/extends it;
|
||||||
|
pre-authorize plan-described destructive steps.
|
||||||
|
3. **SOUL persist-through-errors** framing + `maxIterations` bump.
|
||||||
|
4. Verify end-to-end (below). Only then consider **B** (auto-continuation).
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- Re-run the exact TypeType deploy. Expected: operator approves the plan **once**;
|
||||||
|
the agent then creates the container, installs docker (recovering from the
|
||||||
|
Debian docker.io-CLI gap on its own by falling back to get.docker.com), brings
|
||||||
|
up the stack, hits a transient error (e.g. Docker Hub 500), **retries on its
|
||||||
|
own**, verifies `:8082` responds, and reports success — **with zero additional
|
||||||
|
"continue" ticks from the operator.**
|
||||||
|
- Failure injection: point a step at a wrong path; confirm the agent reads the
|
||||||
|
error, adapts, and continues rather than ending the turn.
|
||||||
|
|
||||||
|
## Open questions
|
||||||
|
|
||||||
|
- **Scope of an autonomy window**: per-plan, per-target, time-boxed (30 min now)?
|
||||||
|
What exactly may the agent do inside it without asking again?
|
||||||
|
- **Pre-authorized destructive steps**: allow a plan to include a named
|
||||||
|
destructive step (e.g. "destroy the half-provisioned CT and redo") that the
|
||||||
|
agent may execute during recovery without a fresh typed confirmation, since
|
||||||
|
the plan approval covered it? Or always re-confirm destructive, accepting the
|
||||||
|
interruption?
|
||||||
|
- **A vs B**: is a single 5–10 min streaming turn acceptable, or do we need
|
||||||
|
event-driven auto-continuation from the start?
|
||||||
Reference in New Issue
Block a user