337d577f0096ef17c17350d4904cdf8cd0f792bd
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 337d577f00 |
fix(agent): refuse plan re-proposal + emit done on error (close divergence chain)
Operator-reported bug: on 'proceed with the rest' the agent re-proposed the
plan, duplicating it in the sidebar. Root cause was a three-bug chain, not
one bug:
1. Trigger — model empty-response on 'proceed' (approval vocabulary didn't
list 'proceed', so the agent wasn't sure it was approved and no-op'd).
2. Amplifier — chatWith emitted 'error' without 'done' on empty response
(agent.go:370). The frontend's onComplete saw !receivedDone and
misclassified the model failure as a network disconnect, calling
handleDisconnect -> resumeSession.
3. Divergence — the reconnect note was generic ('report your state'), so
the agent re-proposed + re-executed instead of advancing the plan.
Fixes (shipped, e2e-validated against the live agent on oikos-nomos-1):
- A.2: proposePlan refuses re-proposal once a step has started (returns
errPlanInFlight). Drops the append-mode safety net (commit
|
|||
| 60effcb2fe |
session reliability: reconnect, knowledge loop, retire request_execution
Phase 1 — crash recovery: SSE auto-reconnect + backoff, polling gate during disconnect, connection banner with retry button, empty-response retry 3x, non-terminal resume on empty response, persistent error cards. Phase 2/4 — visibility + continuation: custom ExecutionStatus renderer, approvals extracted on every tool_result (not just done), activity bar with status/goal, SessionDigest live polling, Continue button. Phase 3 — cleanup: complete_task auto-cancels orphaned approvals, deletes assent/destructive window keys, propose_plan marks pending steps as replaced, plan step seq-order enforcement. Phase 5 — knowledge loop: list_lxcs state filter (active/destroyed), SOUL.md unmissable writeback section, propose_plan validation nudge, complete_task writeback check, upsert_knowledge about array support, plan generation grouping in frontend, session approval count badge. Retire request_execution — all mutations now route through run. Updated SOUL.md, AGENTS.md, CLIENTS.md, skills, and agent system notes. Migration 020: plan step generation column, audit_log session_id index, nomos_plan_executions pending-approval index. |
|||
| 3b9c75fa3f |
fix(agent): task completion safety net — stop tasks sticking at Running
Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md. Confirmed live that 50/50 production sessions never reached a terminal status because the model almost never calls complete_task, even for trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it. - Inline safety net (agent.go): a session that never called set_goal never framed itself as a structured task, so its first plain-text turn-end IS the task ending — auto-complete it there instead of leaving status stuck at its creation default forever. - Idle sweep (continue.go, new completion_nudges column): goal-bearing sessions that stall get one nudge, then auto-close with outcome=partial if the nudge goes unanswered, mirroring the pattern resumeSession already uses for a different stuck-session failure mode. Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately separate — deferred until this is deployed and verified live, per the plan's implementation order. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
|||
| 6d4f6de676 |
fix(agent): mark a task failed when its resume permanently gives up (B3)
Fix B3 of plans/2026-07-11-nomos-agent-code-review.md. resumeSession's retry loop (used by both auto-continuation and panel-answered questions) already retried once on a transient LLM failure, but if BOTH attempts came back empty/erroring, the code just logged and returned — the task was left at whatever status it already had (typically 'executing' or 'awaiting_input') with no outcome, no operator-visible signal beyond an inert error line buried in the transcript, and no way to tell a genuinely stuck task apart from one quietly still working. On permanent failure, now calls store.completeTask(outcome='failure', a summary built from the error) so the task board reflects reality instead of showing a task that looks perpetually in-progress. Uses context.Background() for that write, matching resumeSession's own persistence pattern, since the context that led to the failure may itself be in a bad state. This doesn't prevent the operator from continuing to work the task via a fresh chat message afterward — it only replaces silent hanging with a real status. A full live induction of a permanent LLM outage would require breaking the model/API-key config for the whole nomos container — too invasive for this fix's priority. Verified instead that the new branch stays correctly dormant on the happy path: ran a real ask_operator → panel-answer → resume cycle end-to-end and confirmed the task landed at status='executing' with no outcome set, proving the failure-handling code doesn't false-positive on a normal successful resume. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|||
| c5ffaec85b |
fix(agent): panic recovery on every background goroutine (B1+B2)
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together, since the right granularity for B1 in the auto-continuation worker turned out to require B2's restructuring anyway (see below). B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned nothing before this — every explicitly-spawned goroutine (continuation worker, resumed chat turns, async execution dispatch, the SSE listener, two duplicate sshExec implementations' output-collector goroutines) crashed the whole process on an unhandled panic, not just that one goroutine. More consequential post-concurrency: more simultaneous unattended background work means more surface area for one bad input to end every running task. New internal/safego package: Go(label, fn) launches fn in a goroutine with a recover-and-log wrapper. Applied at every bare `go` spawn site across the three packages. Two sites needed bespoke handling instead of the generic helper because their callers block on a channel and a silent recover would just make them hang until timeout: sshExec's output-collector goroutine (two near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go) and httpapi's ListenAndServe goroutine — both now recover AND send a synthetic error result so the waiting select unblocks immediately instead of waiting out the full timeout. httpapi's sseListener got extra treatment: its per-notification handling was extracted into handleNotification with its own recover, so a panic decoding ONE malformed pg_notify payload can't kill the listener goroutine for every connected SSE client — the outer goroutine spawn only needs to guard the connection setup/reconnect code around it. B2: cmd/nomos/continue.go's processContinuations used to run every pending continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the ticker — meaning (a) task B's continuation waited for task A's full (up to 10-minute) resumed turn to finish first, undercutting this session's earlier concurrency work on exactly the path autonomous tasks depend on most, and (b) an unrecovered panic anywhere in that call chain didn't just crash the process (B1) — even WITH B1's recovery wrapped only at the top-level worker spawn, the panic would still unwind the ENTIRE ticker-loop goroutine, silently ending auto-continuation for every task until nomos restarted. Fixed by spawning each pending item via safego.Go individually: real parallelism, and a bad item can now only ever take down its own goroutine. Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete proof — a deliberate panic inside Go() that would otherwise crash the whole test binary; reaching the assertion after it IS the evidence recovery works. Verified live against the rebuilt containers: full chat turn round-tripped correctly (hostname lookup, 2 iterations, normal completion) — no regression from threading safego.Go through the tool-dispatch/continuation paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|||
| 9ef1ba3702 |
fix(concurrency): scope assent/destructive windows to session, not just agent
Fix 1 of plans/2026-07-11-concurrent-task-execution.md — the safety-critical
one. The assent window (and destructive window) were keyed purely by agent
id ("assent_window.agent:<uuid>"). With one agent:nomos entity serving every
concurrent task, this meant approving Task A's plan opened a window that ANY
concurrently-running task's config-mutation/destructive actions could also
ride, auto-executing without their own approval.
- store.go / agent.go: assentWindowActive/openAssentWindow and
destructiveWindowActive/openDestructiveWindow/destructiveWindowKey all gain
a sessionID parameter; keys become
"assent_window.agent:<id>.session:<sessionID>" and
"destructive_window.agent:<id>.target:<slug>.session:<sessionID>". Missing
session id fails closed (no window) rather than falling back to the old
agent-wide key.
- continue.go: the auto-continuation worker's window check moved from once-
per-batch to once-per-pending-item, scoped to that item's own session —
it was previously checking ONE agent-wide window for a batch that can span
multiple tasks.
- agent.go tool-dispatch: injects `_session_id` into a COPY of the wire args
sent to the MCP server (never into the args used for the emitted/logged/
persisted tool call, and never part of any tool's declared InputSchema —
invisible to the model) so the gating checks on the OTHER side of the
process boundary know which task is asking.
- internal/mcp/server.go: assentWindowActive/destructiveWindowActive/
classifyAndGate gain the same sessionID parameter, read from
args["_session_id"] at the three call sites (request_execution's
apt_upgrade/pct_create branches, and the shared classifyAndGate used by
restart/pct_exec/systemctl/run).
Verified against the live stack with the exact scenario from the plan: opened
an assent window for session A only, then called `run` with an identical
config-mutation command for session A (window open) and session B (same
agent, no window). A auto-ran (execution status completed); B correctly
queued for approval (pending_approval) instead of bleeding through — proven
at both the MCP response text and the executions table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|||
| 014e5c74e0 |
feat(tasks): phase 5 — ask_operator (structured question, pause, resume)
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.
- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
that records a session_questions row, moves the task to awaiting_input, emits
question.raised, and ENDS the turn (the agent loop returns after it, so the
agent can't barrel past its own question). The prompt becomes the assistant's
visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
the task to executing:
- Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
background with the answer injected (reusing the continuation machinery,
refactored continueSession → resumeSession). Returns 202; the reply lands via
message polling.
- Chat reply: the next chat message on a task with an open question IS the
answer — auto-closed in handleChat; the turn itself is the resume.
Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|||
| 233b5e4519 |
feat: live visibility into what the agent is running (no more silent waiting)
Operator: "I'd like to be able to see in the chat what the agent is actually running, right now I just wait while nothing happens." Two compounding gaps: 1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push — its result only appeared on a manual page reload, so approving a plan and watching the chat looked completely dead even while the agent was actively working. 2. Even with polling, continueSession only persisted ONE message at the very end of a continuation — a continuation that runs several tool calls before concluding would still show total silence for however long that took. Fixed both: - web/src/lib/stores/chat.ts: polls the current session's messages every 3s between turns (never while a live stream owns the message list) and merges in anything new. Started after a live turn ends and when a session loads; stopped on new-chat/session-switch. - cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message be created as a placeholder and updated in place. - cmd/nomos/continue.go: continueSession now inserts a placeholder the instant it starts (renders as the existing "thinking" dots — immediate feedback that something is happening) and updates that SAME row after EVERY tool call, not just at the end. A poll within ~3s of any tool call landing shows it — individual `run` commands appear as the agent issues them, not just the final rolled-up summary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
|||
| 6f9998fa29 |
fix: auto-continuation silently dropped LLM errors + added outer retry
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.
- Capture "error" events and, if the turn produced no text/tool_calls at all,
persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
produces nothing — the principle behind this whole feature ("don't give up
on the first error") should apply to the continuation mechanism itself, not
just the homelab commands it's continuing.
Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|||
| d2f749d33d |
feat: event-driven auto-continuation — agent runs an approved plan to completion
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
