Implements fixes 1-3 of plans/2026-07-11-task-completion-safety-net.md.
Confirmed live that 50/50 production sessions never reached a terminal
status because the model almost never calls complete_task, even for
trivial single-tool Q&A turns SOUL.md explicitly calls out as needing it.
- Inline safety net (agent.go): a session that never called set_goal never
framed itself as a structured task, so its first plain-text turn-end IS
the task ending — auto-complete it there instead of leaving status stuck
at its creation default forever.
- Idle sweep (continue.go, new completion_nudges column): goal-bearing
sessions that stall get one nudge, then auto-close with outcome=partial
if the nudge goes unanswered, mirroring the pattern resumeSession already
uses for a different stuck-session failure mode.
Fix 4 (backfill of the 50 already-stuck live sessions) is deliberately
separate — deferred until this is deployed and verified live, per the
plan's implementation order.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix F1 of plans/2026-07-11-nomos-agent-code-review.md, the last item.
buildTools called listToolsFull (a tools/list MCP round-trip) at the start of
EVERY chat turn, including every auto-continuation resume — the tool list is
static for the lifetime of one MCP connection, changing only when the api
process re-registers tools (a restart, which this client already detects and
reacts to via reconnectLocked). Re-fetching it every single turn was
avoidable network+parsing work on the hot path.
mcpClient now caches the parsed tool list after its first fetch, guarded by
its own mutex (kept separate from the request-serializing mu so a cache
check never contends with an in-flight doRequest call). reconnectLocked
clears the cache — an api restart may have changed what's registered, so a
stale cache would be wrong, not just slow. fleetSnapshot's get_health_summary
call is deliberately left uncached — it's meant to be "as of now."
Since each session gets its own client (the per-session pool from the
concurrency work), this caches per-task-conversation rather than globally: a
task's FIRST turn still pays the round-trip, every turn after reuses the
cached list — which is exactly the case that mattered (long-running,
heavily-autonomous tasks with many auto-continuation resumes).
Verified live via the api's request log: a brand-new session's first turn
made 3 MCP calls (initialize, tools/list, get_health_summary); a second turn
on the SAME session made exactly 1 (only get_health_summary) — tools/list
correctly skipped.
This completes the implementation order in
plans/2026-07-11-nomos-agent-code-review.md — every A/B/D/E/F finding from
the review (excluding C1, explicitly deferred per operator instruction) is
now fixed, tested, and verified live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A3 of plans/2026-07-11-nomos-agent-code-review.md. handleChat only ever
saved the assistant message ONCE, after a.chat(...) returned, using
ctx := r.Context() for that write — the same context that cancels the instant
the client disconnects (Stop button, tab close, network blip). A disconnect
mid-turn meant the final save ran with an already-cancelled context and its
error was never checked: the entire turn's tool-call history was silently
lost from the persisted transcript, even though real work (executions
launched, knowledge written) had already happened server-side.
Brought handleChat in line with resumeSession's existing pattern
(continue.go): insert a placeholder assistant row immediately, update the
SAME row after every tool call. The key fix is WHICH context the writes use —
a new pctx := context.Background() for every DB write in this handler
(session creation/touch, the user message, question auto-close, the
placeholder + incremental updates, the title update), while ctx/r.Context()
still gates the agent's own work (a.chat) and the SSE writes exactly as
before — a disconnect still correctly stops the agent from doing further
work, it just no longer also erases what it already did.
Verified live: sent a message requiring 6 tool calls (get_entity/
get_relations/get_blast_radius on two targets) and force-killed the client
connection mid-stream with curl -m 12 (confirmed via exit code 28). Before
this fix the persisted transcript would show 0 tool-call entries; after,
all 12 raw tool_use/tool_result entries (6 calls × 2) were present and
correctly attributed by tool name — proving both that progress survives an
abort and that the incremental writes aren't corrupting the data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together,
since the right granularity for B1 in the auto-continuation worker turned
out to require B2's restructuring anyway (see below).
B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned
nothing before this — every explicitly-spawned goroutine (continuation
worker, resumed chat turns, async execution dispatch, the SSE listener, two
duplicate sshExec implementations' output-collector goroutines) crashed the
whole process on an unhandled panic, not just that one goroutine. More
consequential post-concurrency: more simultaneous unattended background work
means more surface area for one bad input to end every running task.
New internal/safego package: Go(label, fn) launches fn in a goroutine with a
recover-and-log wrapper. Applied at every bare `go` spawn site across the
three packages. Two sites needed bespoke handling instead of the generic
helper because their callers block on a channel and a silent recover would
just make them hang until timeout: sshExec's output-collector goroutine (two
near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go)
and httpapi's ListenAndServe goroutine — both now recover AND send a
synthetic error result so the waiting select unblocks immediately instead of
waiting out the full timeout.
httpapi's sseListener got extra treatment: its per-notification handling was
extracted into handleNotification with its own recover, so a panic decoding
ONE malformed pg_notify payload can't kill the listener goroutine for every
connected SSE client — the outer goroutine spawn only needs to guard the
connection setup/reconnect code around it.
B2: cmd/nomos/continue.go's processContinuations used to run every pending
continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the
ticker — meaning (a) task B's continuation waited for task A's full (up to
10-minute) resumed turn to finish first, undercutting this session's earlier
concurrency work on exactly the path autonomous tasks depend on most, and
(b) an unrecovered panic anywhere in that call chain didn't just crash the
process (B1) — even WITH B1's recovery wrapped only at the top-level worker
spawn, the panic would still unwind the ENTIRE ticker-loop goroutine,
silently ending auto-continuation for every task until nomos restarted.
Fixed by spawning each pending item via safego.Go individually: real
parallelism, and a bad item can now only ever take down its own goroutine.
Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete
proof — a deliberate panic inside Go() that would otherwise crash the whole
test binary; reaching the assertion after it IS the evidence recovery works.
Verified live against the rebuilt containers: full chat turn round-tripped
correctly (hostname lookup, 2 iterations, normal completion) — no regression
from threading safego.Go through the tool-dispatch/continuation paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 3 of plans/2026-07-11-concurrent-task-execution.md, the throughput one.
nomos held exactly one *mcpClient for the whole process, shared by every
/chat goroutine. Its mutex was held for the full duration of each tool
round-trip, and `run` executes its SSH command SYNCHRONOUSLY inside that
round-trip (capped at up to 10 minutes) — so while Task A was mid-`run`,
every other task's tool calls, even a trivial get_entity, queued behind that
single lock. Tasks could think (LLM calls) in parallel but never act in
parallel.
The MCP server has no per-connection state to protect (newServer returns one
shared *mcp.Server instance whose handlers close only over the DB
connection pool, already safe for concurrent use) — the mutex existed purely
because the client reused one stateful transport session. So the fix doesn't
touch the server at all:
- New mcpClientPool (cmd/nomos/main.go): one *mcpClient per session id,
created lazily (a real MCP initialize handshake) on first use and cached;
session-less traffic (the ephemeral no-DB-store path, the structured
/query endpoint) gets its own fixed, reused key instead of a fresh
connection per request. Idle clients (20 min past last use — long enough
to outlive a single slow `run`) are evicted on a 5-minute sweep ticker.
- agent.go: `client *mcpClient` → `clients *mcpClientPool`; every call site
(buildTools, fleetSnapshot, the tool-dispatch loop) now resolves its own
session's client via clients.get(sessionID) instead of reaching for one
shared field. A task's own tool calls stay sequential (already true — the
agent loop calls tools one at a time within a turn) but no longer block
anyone else's.
- main.go: handleQuery takes the pool instead of a client (keyed "query", a
fixed non-session slot); shutdown calls pool.closeAll().
Verified live against the deployed stack: fired a slow-but-ungated command
(`ping -c 15 127.0.0.1`, read-only per policy's allowlist, no approval
needed) as Task A, then — 2s into A's run — a trivial hostname lookup as
Task B, both through the real /chat endpoint. Task A's ping genuinely ran
~14.3s (confirmed via its own execution record and the agent's reported
output). Task B returned in 6s total, well before A finished — proving it
was never queued behind A's connection. Before this fix, B would have been
forced to wait out A's entire ~14.3s hold on the single shared client.
This completes plans/2026-07-11-concurrent-task-execution.md's required
scope — only the explicitly optional/deferred Fix 4 (a concurrency/cost cap,
pending real usage data) remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the chat right rail's ad-hoc Digest+Graph stack with a single
TaskContextPanel that renders the task's live working state, driven by the
always-on events stream (not the per-turn chat SSE) so it keeps updating
during server-side auto-continuation/resume:
- GoalHeader: goal + status pill (planning/executing/awaiting_input/done/
failed), sourced from the sessions list.
- PlanProgress: ordered steps with live status icons + progress bar, hydrated
via new GET /sessions/{id}/plan; clicking a step with a target opens its
EntitySheet (no fake "jump to transcript" — bits-ui Collapsible content
isn't force-mounted, so a DOM-scroll jump would silently no-op for
collapsed tool groups).
- OperatorQuestion: the pinned structured question card (prompt/why/entity
chips/option buttons/free-text), hydrated via new GET /sessions/{id}/
questions; answering POSTs to the existing answer endpoint.
- SessionGraph upgraded to a live entity panel: entity.touched pulses the
node (animated ring) and shows "Now touching <slug>"; health.changed shows
a transient diff badge for touched entities.
- SessionDigest gains a success/failure/partial outcome banner and now also
refetches when the task's status changes, not just on session switch.
Two bugs found and fixed while wiring this up:
- workspace.ts's status-refresh trigger only covered goal.set/task.status;
question.raised/answered didn't refresh the sessions list, so GoalHeader's
pill went stale after answering via the panel (resumeSession runs entirely
server-side — no client 'done' event to piggyback a refresh on). Now every
status-affecting event triggers the (debounced) refetch.
- Forgot to rebuild the nomos container after adding the /plan and
/questions endpoints, so they silently fell through to the old default GET
handler — caught via a live curl diff against the running container,
not a code read.
Verified end-to-end against the live stack: goal/plan/question all update
without a reload as the agent works; answering a question via the panel
resumes the agent and the header pill correctly flips to Executing;
entity.touched pulses the live graph.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.
- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
that records a session_questions row, moves the task to awaiting_input, emits
question.raised, and ENDS the turn (the agent loop returns after it, so the
agent can't barrel past its own question). The prompt becomes the assistant's
visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
the task to executing:
- Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
background with the answer injected (reusing the continuation machinery,
refactored continueSession → resumeSession). Returns 202; the reply lands via
message polling.
- Chat reply: the next chat message on a task with an open question IS the
answer — auto-closed in handleChat; the turn itself is the resume.
Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
