Fix A3 of plans/2026-07-11-nomos-agent-code-review.md. handleChat only ever
saved the assistant message ONCE, after a.chat(...) returned, using
ctx := r.Context() for that write — the same context that cancels the instant
the client disconnects (Stop button, tab close, network blip). A disconnect
mid-turn meant the final save ran with an already-cancelled context and its
error was never checked: the entire turn's tool-call history was silently
lost from the persisted transcript, even though real work (executions
launched, knowledge written) had already happened server-side.
Brought handleChat in line with resumeSession's existing pattern
(continue.go): insert a placeholder assistant row immediately, update the
SAME row after every tool call. The key fix is WHICH context the writes use —
a new pctx := context.Background() for every DB write in this handler
(session creation/touch, the user message, question auto-close, the
placeholder + incremental updates, the title update), while ctx/r.Context()
still gates the agent's own work (a.chat) and the SSE writes exactly as
before — a disconnect still correctly stops the agent from doing further
work, it just no longer also erases what it already did.
Verified live: sent a message requiring 6 tool calls (get_entity/
get_relations/get_blast_radius on two targets) and force-killed the client
connection mid-stream with curl -m 12 (confirmed via exit code 28). Before
this fix the persisted transcript would show 0 tool-call entries; after,
all 12 raw tool_use/tool_result entries (6 calls × 2) were present and
correctly attributed by tool name — proving both that progress survives an
abort and that the incremental writes aren't corrupting the data.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together,
since the right granularity for B1 in the auto-continuation worker turned
out to require B2's restructuring anyway (see below).
B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned
nothing before this — every explicitly-spawned goroutine (continuation
worker, resumed chat turns, async execution dispatch, the SSE listener, two
duplicate sshExec implementations' output-collector goroutines) crashed the
whole process on an unhandled panic, not just that one goroutine. More
consequential post-concurrency: more simultaneous unattended background work
means more surface area for one bad input to end every running task.
New internal/safego package: Go(label, fn) launches fn in a goroutine with a
recover-and-log wrapper. Applied at every bare `go` spawn site across the
three packages. Two sites needed bespoke handling instead of the generic
helper because their callers block on a channel and a silent recover would
just make them hang until timeout: sshExec's output-collector goroutine (two
near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go)
and httpapi's ListenAndServe goroutine — both now recover AND send a
synthetic error result so the waiting select unblocks immediately instead of
waiting out the full timeout.
httpapi's sseListener got extra treatment: its per-notification handling was
extracted into handleNotification with its own recover, so a panic decoding
ONE malformed pg_notify payload can't kill the listener goroutine for every
connected SSE client — the outer goroutine spawn only needs to guard the
connection setup/reconnect code around it.
B2: cmd/nomos/continue.go's processContinuations used to run every pending
continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the
ticker — meaning (a) task B's continuation waited for task A's full (up to
10-minute) resumed turn to finish first, undercutting this session's earlier
concurrency work on exactly the path autonomous tasks depend on most, and
(b) an unrecovered panic anywhere in that call chain didn't just crash the
process (B1) — even WITH B1's recovery wrapped only at the top-level worker
spawn, the panic would still unwind the ENTIRE ticker-loop goroutine,
silently ending auto-continuation for every task until nomos restarted.
Fixed by spawning each pending item via safego.Go individually: real
parallelism, and a bad item can now only ever take down its own goroutine.
Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete
proof — a deliberate panic inside Go() that would otherwise crash the whole
test binary; reaching the assertion after it IS the evidence recovery works.
Verified live against the rebuilt containers: full chat turn round-tripped
correctly (hostname lookup, 2 iterations, normal completion) — no regression
from threading safego.Go through the tool-dispatch/continuation paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix A1 of plans/2026-07-11-nomos-agent-code-review.md. isAssent and
isTypedConfirmation used a space-padded word-boundary check for negation
words but a bare strings.Contains for assent/confirm words — confirmed live
via test probes: isAssent("...maybe yesterday's logs...") returned true
("yes" matched inside "yesterday"), and isTypedConfirmation("I haven't
confirmed anything yet") returned true ("confirm" matched inside "confirmed",
and "haven't" wasn't in negationWords — only "don't"/"do not" were).
isTypedConfirmation is the sole gate for DESTRUCTIVE actions, so the second
case meant a message merely stating something hadn't been confirmed could
read as an explicit confirmation.
- Replaced the ad-hoc space-padding/prefix-check negation logic with proper
tokenization (wordTokenRe) + containsPhrase, matching WHOLE tokens/phrases
only — never a mid-word substring. Handles curly apostrophes too (a
pre-existing gap: the old straight-quote-only check would have missed
"don't" typed with a smart quote).
- Added contracted negatives (haven't, hasn't, isn't, wasn't, aren't, can't,
cannot, won't, wouldn't, shouldn't, didn't, doesn't) to negationWords.
Deliberately did NOT add a bare "not" — too broad, would false-negative
ordinary assent like "go ahead, this is not risky".
- Added regression tests for both confirmed cases plus a couple of adjacent
ones (eyesight/isn't, can't confirm) so a future change can't silently
reintroduce either bug.
All existing assent/confirmation tests pass unchanged — this is a pure
robustness fix, not a behavior change for any previously-correct case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full read-through of cmd/nomos/ (agent.go, store.go, main.go, continue.go,
assent.go, tasks.go). Findings, ranked:
- A1 (confirmed via runnable probe): isAssent/isTypedConfirmation use
unpadded substring matching for assent/confirm words while negation uses
word-boundary checks — "yes" matches inside "yesterday", "confirm" matches
inside "confirmed" with no negation word covering contracted negatives
("haven't"). isTypedConfirmation gates DESTRUCTIVE actions specifically.
- A2: chatWith replays a session's ENTIRE message history every turn, no
windowing/token budget — confirmed unbounded against a documented
production case (70 tool calls, 106KB messages).
- A3: a live turn's tool-call history is lost entirely if the client
disconnects mid-stream (single end-of-turn save using the same
connection-tied, possibly-cancelled context) — resumeSession already has
the fix pattern (incremental placeholder+update), handleChat doesn't use it.
- B1: zero recover() anywhere in cmd/nomos/internal/mcp/internal/httpapi —
every explicitly-spawned goroutine (continuation worker, resumeSession,
executeApprovedViaAPI, sse listeners) crashes the whole process on panic.
- B2: auto-continuation processes its batch sequentially, one full LLM turn
at a time, undercutting this session's own concurrency work on exactly the
path autonomous tasks depend on most.
- B3: no terminal state for a permanently-failed auto-continuation.
- C1: nomos's own gateway (port 8092, directly published + mesh-reachable)
has ZERO authentication on any endpoint — chat, session read/delete,
chat-assent approval of gated executions, all open to anyone on the LAN.
- D1-D3: dead code (isTaskTool unused), N+1 query in recordTouched, no
validation on complete_task's outcome enum.
- E: zero automated tests for agent.go/store.go/main.go/tasks.go — including
today's new safety-critical logic (session-scoped windows, mcpClientPool,
proposePlan's append-vs-replace), verified only by live manual testing.
- F1: tool list + fleet snapshot re-fetched every turn (minor).
Prioritized implementation order in the doc: A1 → C1 → B1 → B2 → A3 → D1-3 →
A2 → B3/F1, tests landing alongside each fix rather than as a deferred pass.
Also archives the now-fully-shipped concurrent-task-execution plan to done/
(all 3 required fixes deployed this session; fix 4 explicitly deferred per
its own recommendation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 3 of plans/2026-07-11-concurrent-task-execution.md, the throughput one.
nomos held exactly one *mcpClient for the whole process, shared by every
/chat goroutine. Its mutex was held for the full duration of each tool
round-trip, and `run` executes its SSH command SYNCHRONOUSLY inside that
round-trip (capped at up to 10 minutes) — so while Task A was mid-`run`,
every other task's tool calls, even a trivial get_entity, queued behind that
single lock. Tasks could think (LLM calls) in parallel but never act in
parallel.
The MCP server has no per-connection state to protect (newServer returns one
shared *mcp.Server instance whose handlers close only over the DB
connection pool, already safe for concurrent use) — the mutex existed purely
because the client reused one stateful transport session. So the fix doesn't
touch the server at all:
- New mcpClientPool (cmd/nomos/main.go): one *mcpClient per session id,
created lazily (a real MCP initialize handshake) on first use and cached;
session-less traffic (the ephemeral no-DB-store path, the structured
/query endpoint) gets its own fixed, reused key instead of a fresh
connection per request. Idle clients (20 min past last use — long enough
to outlive a single slow `run`) are evicted on a 5-minute sweep ticker.
- agent.go: `client *mcpClient` → `clients *mcpClientPool`; every call site
(buildTools, fleetSnapshot, the tool-dispatch loop) now resolves its own
session's client via clients.get(sessionID) instead of reaching for one
shared field. A task's own tool calls stay sequential (already true — the
agent loop calls tools one at a time within a turn) but no longer block
anyone else's.
- main.go: handleQuery takes the pool instead of a client (keyed "query", a
fixed non-session slot); shutdown calls pool.closeAll().
Verified live against the deployed stack: fired a slow-but-ungated command
(`ping -c 15 127.0.0.1`, read-only per policy's allowlist, no approval
needed) as Task A, then — 2s into A's run — a trivial hostname lookup as
Task B, both through the real /chat endpoint. Task A's ping genuinely ran
~14.3s (confirmed via its own execution record and the agent's reported
output). Task B returned in 6s total, well before A finished — proving it
was never queued behind A's connection. Before this fix, B would have been
forced to wait out A's entire ~14.3s hold on the single shared client.
This completes plans/2026-07-11-concurrent-task-execution.md's required
scope — only the explicitly optional/deferred Fix 4 (a concurrency/cost cap,
pending real usage data) remains.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the known gap flagged in the previous commit (9131559). A single
module-level `activeController` meant cancelStream()/newChat() always
aborted whichever stream was MOST RECENTLY STARTED, regardless of what the
operator was currently viewing: start Task A, switch to an already-loaded
Task B, click "New task" — the click's cancelStream() would silently abort
Task A's still-running turn, even though the operator was never looking at
it and never asked to cancel it.
- Replaced the single controller with activeControllers (Map<sessionID,
AbortController>) plus pendingController for the brief pre-'session'-event
window of a brand-new task. Registered immediately in sendMessage (keyed by
the continuing session id right away, or held pending until the 'session'
event assigns a new one) and cleaned up on completion.
- cancelStream() now looks up by $currentSession (falling back to
pendingController when no session is assigned yet) — it can only ever
touch the stream belonging to the view being left, never an unrelated
background task's.
- newChat() unchanged in behavior (still calls cancelStream()), now correctly
scoped through the above.
Verified live, reproducing the exact bug: started Task A (slow, 5 tool
calls), switched to an existing Task B, clicked "New task" while viewing
B — Task A was NOT aborted, ran to completion server-side with a full,
correct final summary (previously this exact sequence would have killed it).
Confirmed the positive path is unaffected: started a task, clicked Stop
while actively viewing it — input re-enabled, stream genuinely aborted
("BodyStreamBuffer was aborted"), turn stopped mid-flight as expected.
This closes out Fix 2's scope from
plans/2026-07-11-concurrent-task-execution.md; only Fix 3 (per-session MCP
client pool, throughput) and the optional Fix 4 (concurrency cap) remain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 2 of plans/2026-07-11-concurrent-task-execution.md. sendMessage's SSE
callback mutated the global messages/currentSession stores unconditionally,
assuming only one task's turn is ever in flight. It isn't — the backend runs
every /chat request as its own goroutine with no serialization. Switching to
a different task while a previous one was still streaming let that
background stream's later events (tool_use, text_delta, ..., and worst of
all 'done''s currentSession.set) get applied to whatever the operator is now
looking at: corrupting another task's transcript, or yanking the view back
to the one they left.
- Captures the session a stream belongs to (openedFor at call time, updated
to the real id once the 'session' event assigns one) and checks
$currentSession still matches before every messages/error/streaming
mutation. The task keeps running server-side regardless — dropped events
just mean the live view isn't watching it; navigating back re-hydrates via
REST, same as already happens for auto-continuation.
- The 'session' event itself only claims currentSession if the operator
hasn't already navigated elsewhere since the call started (comparing
against openedFor, which is null for a brand-new task).
- loadSessionMessages/newChat now reset `streaming` to false unconditionally
on navigation — needed so the new guard can't leave a DIFFERENT task's view
stuck showing streaming=true (which would also silently stop startPolling's
loop from ever applying updates, since it bails while $streaming is true).
Known residual gap, not fixed here (matches the plan's "contained fix, not a
rearchitecture" scope): activeController is still a single global slot, so
starting a new task while another is mid-stream, then clicking "New task"
again, aborts whichever stream that slot last pointed at rather than only
the one being left. A genuine multi-session controller/store is the
plan's deferred "stretch" fix, not required for correctness here.
Verified live: started Task A with a deliberately slow 4-tool-call turn,
switched to an existing Task B mid-stream — Task B's transcript stayed
correct with zero A-originated entries and the input was NOT stuck disabled.
Task A kept running and completed normally server-side (status=done, full
6-tool transcript, 5-entity graph); navigating back loaded its complete,
uncorrupted result via REST.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fix 1 of plans/2026-07-11-concurrent-task-execution.md — the safety-critical
one. The assent window (and destructive window) were keyed purely by agent
id ("assent_window.agent:<uuid>"). With one agent:nomos entity serving every
concurrent task, this meant approving Task A's plan opened a window that ANY
concurrently-running task's config-mutation/destructive actions could also
ride, auto-executing without their own approval.
- store.go / agent.go: assentWindowActive/openAssentWindow and
destructiveWindowActive/openDestructiveWindow/destructiveWindowKey all gain
a sessionID parameter; keys become
"assent_window.agent:<id>.session:<sessionID>" and
"destructive_window.agent:<id>.target:<slug>.session:<sessionID>". Missing
session id fails closed (no window) rather than falling back to the old
agent-wide key.
- continue.go: the auto-continuation worker's window check moved from once-
per-batch to once-per-pending-item, scoped to that item's own session —
it was previously checking ONE agent-wide window for a batch that can span
multiple tasks.
- agent.go tool-dispatch: injects `_session_id` into a COPY of the wire args
sent to the MCP server (never into the args used for the emitted/logged/
persisted tool call, and never part of any tool's declared InputSchema —
invisible to the model) so the gating checks on the OTHER side of the
process boundary know which task is asking.
- internal/mcp/server.go: assentWindowActive/destructiveWindowActive/
classifyAndGate gain the same sessionID parameter, read from
args["_session_id"] at the three call sites (request_execution's
apt_upgrade/pct_create branches, and the shared classifyAndGate used by
restart/pct_exec/systemctl/run).
Verified against the live stack with the exact scenario from the plan: opened
an assent window for session A only, then called `run` with an identical
config-mutation command for session A (window open) and session B (same
agent, no window). A auto-ran (execution status completed); B correctly
queued for approval (pending_approval) instead of bleeding through — proven
at both the MCP response text and the executions table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New plan grounds three concurrency issues found by tracing the actual code
(not assumed): the assent/destructive windows are keyed by agent id only
(no session dimension), so an approved plan in one task can auto-run
unapproved actions in a concurrently-running task; nomos shares one
mutex-guarded MCP client across all sessions, so a single slow `run` call
serializes every other task's tool calls behind it; and chat.ts's SSE
callback has no session guard, so switching tasks mid-stream lets the
backgrounded task's events corrupt whatever's now displayed. Proposes
session-scoping the windows (critical/first), a frontend stream guard
(contained/second), a per-session MCP client pool (throughput/third), and
an optional concurrency cap (deferred pending real usage data).
Also archives the goal-oriented-chat-control-panel plan to done/ — all 7
phases shipped and are live in production (SHA e30813a) — fixing its
internal relative links for the new depth and pointing forward to the new
concurrency plan as follow-up hardening.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the gap that made the knowledge loop optional/implicit: every
non-trivial task now has an EXPLICIT first plan step (research) and last
plan step (write back), not just background behavior the model might skip.
New MCP tools (the agent had no way to do these before — only REST endpoints
existed, unexposed to it):
- update_entity_attributes(slug, attributes): shallow-merge new/changed facts
into an entity (an IP, a version, a discovered port) so a future task
doesn't have to rediscover them from scratch. No approval required — this
updates the knowledge graph, not live infra.
- create_relationship(source, target, type): record a discovered edge
(depends-on, hosts, provides, ...). Idempotent, FK-validated against the
ontology's relationship_types, no approval required.
SOUL.md: restructured the task loop so step 1 is explicitly "gather
knowledge, not just status" (get_entity_knowledge, search_knowledge,
get_relations, get_blast_radius, http_get) and the last step before
complete_task is explicitly "write back" (update_entity_attributes,
create_relationship, upsert_knowledge) — both called out as real plan
entries the operator should see in propose_plan, not silent side-work. This
is what prevents the graph drifting from reality and is the concrete
mechanism behind "tasks compound."
propose_plan's tool description reinforces the same first-step/last-step
convention at the call site.
Verified against the live stack: both tools registered and callable via MCP;
update_entity_attributes merged an attribute correctly; create_relationship
rejected an invalid type (FK violation, clear error) and succeeded with a
valid type+direction, confirmed idempotent (2 calls, 1 row).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause: proposePlan unconditionally deleted and replaced the whole
session_plan_steps list on every call. The model isn't strictly held to
"call propose_plan once with the full list" — nothing stopped it (and
production evidence + live testing showed it happening) from calling
propose_plan once per step as it worked. Each such call wiped every
already-completed step, so the operator only ever saw the model's latest
single step ("1/1") instead of the real, growing plan.
Fix, two layers:
- store.go: proposePlan now only does a destructive replace when no step
has left 'pending' yet (a genuine pre-execution revision). Once any step
has started, a new call APPENDS after the current max seq instead of
wiping — so the panel accumulates the full history regardless of how the
model chooses to call the tool. plan.proposed now carries `appended` so
the frontend knows whether to replace or append.
- workspace.ts: plan.proposed handler respects `appended` (update vs set).
- tasks.go / SOUL.md: strengthened the propose_plan description and task-
loop guidance to call it ONCE with the complete step list end-to-end,
using update_plan_step (not re-calling propose_plan) to advance — fixing
the root behavioral cause, with the store-side append as a safety net
that holds even if the model still calls it incrementally.
Verified: forced the exact incremental-call pattern (propose_plan with 1
step, mark it running, propose_plan again with 1 more step) — the second
call appended at seq 2 instead of erasing seq 1, and its plan.proposed
event carried appended=true.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the chat right rail's ad-hoc Digest+Graph stack with a single
TaskContextPanel that renders the task's live working state, driven by the
always-on events stream (not the per-turn chat SSE) so it keeps updating
during server-side auto-continuation/resume:
- GoalHeader: goal + status pill (planning/executing/awaiting_input/done/
failed), sourced from the sessions list.
- PlanProgress: ordered steps with live status icons + progress bar, hydrated
via new GET /sessions/{id}/plan; clicking a step with a target opens its
EntitySheet (no fake "jump to transcript" — bits-ui Collapsible content
isn't force-mounted, so a DOM-scroll jump would silently no-op for
collapsed tool groups).
- OperatorQuestion: the pinned structured question card (prompt/why/entity
chips/option buttons/free-text), hydrated via new GET /sessions/{id}/
questions; answering POSTs to the existing answer endpoint.
- SessionGraph upgraded to a live entity panel: entity.touched pulses the
node (animated ring) and shows "Now touching <slug>"; health.changed shows
a transient diff badge for touched entities.
- SessionDigest gains a success/failure/partial outcome banner and now also
refetches when the task's status changes, not just on session switch.
Two bugs found and fixed while wiring this up:
- workspace.ts's status-refresh trigger only covered goal.set/task.status;
question.raised/answered didn't refresh the sessions list, so GoalHeader's
pill went stale after answering via the panel (resumeSession runs entirely
server-side — no client 'done' event to piggyback a refresh on). Now every
status-affecting event triggers the (debounced) refetch.
- Forgot to rebuild the nomos container after adding the /plan and
/questions endpoints, so they silently fell through to the old default GET
handler — caught via a live curl diff against the running container,
not a code read.
Verified end-to-end against the live stack: goal/plan/question all update
without a reload as the agent works; answering a question via the panel
resumes the agent and the header pill correctly flips to Executing;
entity.touched pulses the live graph.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the chat surface as tasks:
- New Tasks.svelte: a card grid of tasks, each showing status (Running /
Needs input / Done / Failed), the goal as title, the outcome summary, and
relative time; filterable by status with live counts; delete on hover;
"New task" and per-card click open the conversation.
- Board updates LIVE off the events stream (goal.set / task.status /
question.*) via an explicit liveEvents.subscribe with a debounced refetch —
scanning all events newer than the last seen, since entity.touched bursts
bury task events below index 0.
- App shell: primary nav "Chat" → "Tasks" (board is now the home route),
"New chat" → "New task", conversation header gets a Tasks / Conversation
breadcrumb. Removed the superseded Sessions page.
- api.ts Session type carries the task fields (goal/status/outcome/summary).
Also fixes a pre-existing SSE bug that blocked ALL live updates app-wide:
writeSSE emitted `event: <type>`, which EventSource only delivers to
addEventListener(type) handlers — but stores/events.ts (and every page reading
liveEvents) consumes via onmessage, which never fires for named events. So the
live stream delivered nothing to the UI. Dropped the event-name line; the type
is already in the JSON payload, and new event types now need zero client
changes. SSE test still green (it parses data: lines).
Verified in the browser against the live stack: the board renders 50 tasks
with correct status buckets; a goal-driven task appears and flips to a Done
card with its summary in real time without a reload; Events page confirms the
stream now delivers to onmessage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The last backend piece: when the agent hits a decision only the operator can
make, it surfaces a structured question instead of guessing or stalling.
- ask_operator(prompt, why?, options?, context_entities?): nomos-local tool
that records a session_questions row, moves the task to awaiting_input, emits
question.raised, and ENDS the turn (the agent loop returns after it, so the
agent can't barrel past its own question). The prompt becomes the assistant's
visible message so the question also shows inline in the transcript.
- Two resume paths, both close the question + emit question.answered + return
the task to executing:
- Panel: POST /sessions/{id}/questions/{qid}/answer → resumes the agent in the
background with the answer injected (reusing the continuation machinery,
refactored continueSession → resumeSession). Returns 202; the reply lands via
message polling.
- Chat reply: the next chat message on a task with an open question IS the
answer — auto-closed in handleChat; the turn itself is the resume.
Verified end-to-end: forcing a decision paused the task at awaiting_input with
the structured question (prompt/why/options/entities); a panel answer resumed
the agent (it acknowledged host:strong and continued); a plain chat reply
auto-closed a second question. Cleanup + tests green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gives a task a legible, live-advancing plan via three more nomos-local tools:
- set_goal(goal): records the task goal, status → planning, emits goal.set.
- propose_plan(steps[]): persists ordered steps (clean replace for v1 — a
revision starts a new list), status → executing, emits plan.proposed with
the persisted steps (id+seq) so the panel can address them.
- update_plan_step(seq, status, execution_id?): advances a step, stamping
started_at/finished_at, emits plan.step.started/finished. Anchors the event
to the step's target entity when it has one.
Belt-and-suspenders: when an execution linked to a step reaches a terminal
state, the api auto-closes the step (closePlanStepForExecution in
emitExecutionEvent) and emits plan.step.finished — so the board stays honest
even if the agent forgets to close a step it started.
Verified end-to-end: a goal-driven task fired goal.set → plan.proposed →
2× step.started/finished → task.status on the SSE stream; both steps persisted
done with start/finish timestamps; status progressed planning→executing→done.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds the compounding knowledge loop the task model is built around:
- complete_task(outcome, summary): a nomos-LOCAL, session-scoped tool (the
shared MCP server has no session id). Introduces the local-tool mechanism —
buildTools appends task tools, the agent loop routes them to handleTaskTool
instead of the MCP client. Sets the task's terminal status/outcome/summary,
mirrors it onto the task entity, and emits task.status.
- Knowledge → task linkage: after a successful upsert_knowledge in a task,
nomos links the note to the task entity (documents) and emits
knowledge.recorded, so the task's outcome view shows what it learned. The
note's about-link to the involved entity (written by upsert_knowledge) is the
retrieval path future tasks use.
- SOUL: every chat is a task loop — retrieve prior knowledge FIRST
(get_entity_knowledge on the target), plan, execute, record learnings, then
complete_task. Scales down for trivial read-only tasks.
- deleteSession now cleans up the task entity, its relationships, and its
task-scoped events (was orphaning them); the knowledge doc itself and its
about-links survive, as knowledge should outlive the task.
Verified end-to-end: a task recorded a note and completed; task.status +
knowledge.recorded hit the SSE stream; status=done/outcome=success persisted;
the note linked to both lxc:caddy (retrieval) and the task; a future
get_entity_knowledge(lxc:caddy) surfaces it; delete cleaned edges+events (0/0/0)
while the knowledge survived.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
As the agent runs a task, record which entities each tool call references:
write an idempotent task —involves→ entity relationship and publish one
entity.touched event per entity (correlation_id = session, data {slug,tool}).
Extracted from tool ARGS only — never results — so a bulk fleet query can't
drag every entity into the task graph; bulk/no-slug tools stay silent.
Emitted from the nomos agent loop rather than the shared MCP wrapper, which
has no session id. The involves edges make a task's graph neighborhood its
involved-entity set (queryable via get_relations) — the substrate for the
knowledge loop; the events are the live pulse the context panel consumes in
phase 6.
Verified end-to-end on the local stack: a chat referencing lxc:caddy/lxc:gitea
produced entity.touched on the browser SSE stream with slug+tool+correlation,
and exactly one involves edge per entity despite repeated touches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.
nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.
No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reframes the chat surface as a board of tasks: each task carries a goal,
a plan approved once, a lifecycle status, an outcome, and a knowledge
loop that links learnings to the involved entities (and the task entity
itself) via relationships so future tasks compound. Supersedes the
sidebar-only framing and the free-form chat portion of the control-room
web UI plan.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audited all 10 active plan docs against the codebase (not just commit
titles). 5 were fully shipped and stale-tagged "Planned"/"In Progress" —
moved to done/ with verification notes. The other 4 got corrected
Planned→In Progress status plus concrete remaining-gap notes so the next
pass doesn't re-derive what's already done.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The plan's "learning view" (runbook success-rate trends, promoted
skills, capability timeline) assumes the patterns/skills/feedback
pipeline is populated. It isn't: all three tables are empty in
production and nothing in the codebase ever writes to feedback, so
building the UI against them today would ship a permanently-empty
page. Scoped instead around data that's real and growing —
executions — while still wiring up /patterns and /skills so the page
needs no rework once that pipeline exists.
New /api/v1/learning/timeline (per-verb first-success date + success
rate, parsed via the existing splitAction helper) and
/api/v1/learning/trend (30-day daily success/fail counts), both
read-only queries against executions. Patterns and skills sections
call the existing (untouched) ListPatterns/ListSkills endpoints and
render an explanatory empty state instead of nothing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found live: a chat request to restart caddy (the reverse proxy for the
whole fleet) executed instantly over SSH with zero approval. Root
cause was in request_execution's legacy handler — restart, pct_exec,
and systemctl (outside enable/disable) executed immediately with a
hardcoded risk_class='reversible_low' that was never actually checked
against anything, bypassing the classifier entirely. Only the `run`
tool's commands were ever gated.
Extracted the run tool's classify -> execute-or-queue logic into a
shared classifyAndGate() and route restart/pct_exec/systemctl through
it too, so every mutating path — regardless of which tool the model
reaches for — gets the same read-only/config-mutation/destructive
classification and approval gate. systemctl restart is already covered
by an existing classifier test (config_mutation), so no new test
needed; the gap was that request_execution never called the
classifier at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pending-approval cards showed target and risk but not what else the
action would affect — the operator approved config_mutation/destructive
commands blind to downstream impact, even though the graph-walk
(blast_radius() SQL, GetBlastRadius endpoint) already existed and was
just never wired into the approval path.
Fetch it once per pending approval and render "Affects N downstream: …"
on both the normal and destructive approval cards, reusing the existing
fetchBlastRadius() API client function which was already written but
unused anywhere.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ops "Executions" tab showed raw target UUIDs, alphabetical (not
recency) order, and a stale status vocabulary from an earlier schema
iteration — never actually usable as a live "what's happening" view.
Replaced with a new recency-ordered /api/v1/activity/recent endpoint
and matching table (human-readable action summaries, risk/status
badges, duration, inline error preview).
Also added /api/v1/activity/session/{id} + a collapsible SessionDigest
panel in the chat rail, answering "what did this session actually do"
(executions by status, entities touched, knowledge written) — the
missing piece for proactive outcome reporting to be visible in the UI,
not just in the chat transcript.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Verified live immediately after deploying: the endpoint returned 200 with
correct-looking stats (total=56, agent_authored=2) but items=[] always,
regardless of limit/source. Root cause: pgx v5 can't scan a timestamptz
column directly into a Go string — Scan() errored on every single row, and
that error was silently swallowed by a bare `continue`, so every row was
dropped with no trace in the logs. Fixed by casting updated_at::text in the
SQL (matching how every other handler in this codebase already returns
timestamps) and logging scan failures instead of swallowing them, so this
class of bug can't hide silently again.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First slice of the observability/learning UI (the "see the system come alive
and learn" ask). The Knowledge page was search-only — blank until you typed —
so the knowledge Nomos now writes via upsert_knowledge was invisible unless
you knew to search for it. Now the page LEADS with what the system knows and
is learning:
- internal/httpapi/knowledge.go: GET /api/v1/knowledge/recent — recency-ordered
knowledge + a stats header (total, agent-authored, learned-this-week,
by-kind). Custom route (not OpenAPI-generated), same auth as the rest.
- web Knowledge page rewrite: stat cards up top (Total / Written by Nomos /
Learned this week / runbooks-investigations), then a "Recently learned" feed
with agent-authored notes highlighted and badged "learned by Nomos", tags,
and relative timestamps. A toggle filters to Nomos-only. Search still works,
now as a mode you enter/clear rather than the whole page.
This turns "the system is getting smarter" from a claim into something you
watch fill up: every gotcha the agent records shows here within seconds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
From the last (successful) TypeType deploy session, two gaps the operator hit:
1. Knowledge write-back — the missing half of the loop.
The agent could read the knowledge base (search_knowledge/get_entity_knowledge)
but had no way to WRITE it, so everything it learned (the Dragonfly memlock
rlimit gotcha, the NAT-hairpin DNS issue, etc.) lived only in an ephemeral
chat message and was lost — the system could never actually "get better."
This is the `upsert_knowledge` MCP tool the 2026-07-08 gaps plan called for.
- internal/mcp/server.go: upsert_knowledge(title, content, about?, tags?,
kind?) writes a document/investigation/runbook entity + knowledge_entities
row (search column is generated), upserts by slug so re-titling updates in
place, and optionally links it to the entity it's about so
get_entity_knowledge surfaces it there.
- SOUL.md: capture non-obvious findings/deploys/gotchas as part of finishing
work, not only when asked "what did we learn".
2. "I had to ask for status multiple times."
The clearest cause: a long working turn (64 tool calls) that exhausted the
iteration cap ended with a bare "max iterations reached without final
answer" — a dead end that forced the operator to ask what happened.
- cmd/nomos/agent.go: on exhaustion, make one final no-tools LLM call
(finalSummary) asking for a status report — what was accomplished, current
state, what remains — so the turn always ends with a real outcome.
- maxIterations 25 -> 40 (the decomposed per-step pct_create flow legitimately
needs more steps).
- SOUL.md: always end a turn with a clear outcome; never end silently or on a
bare tool call — the operator can't see the tools working and reads silence
as "nothing happened".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator: "I'd like to be able to see in the chat what the agent is actually
running, right now I just wait while nothing happens." Two compounding gaps:
1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push —
its result only appeared on a manual page reload, so approving a plan and
watching the chat looked completely dead even while the agent was actively
working.
2. Even with polling, continueSession only persisted ONE message at the very
end of a continuation — a continuation that runs several tool calls before
concluding would still show total silence for however long that took.
Fixed both:
- web/src/lib/stores/chat.ts: polls the current session's messages every 3s
between turns (never while a live stream owns the message list) and merges
in anything new. Started after a live turn ends and when a session loads;
stopped on new-chat/session-switch.
- cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message
be created as a placeholder and updated in place.
- cmd/nomos/continue.go: continueSession now inserts a placeholder the
instant it starts (renders as the existing "thinking" dots — immediate
feedback that something is happening) and updates that SAME row after
EVERY tool call, not just at the end. A poll within ~3s of any tool call
landing shows it — individual `run` commands appear as the agent issues
them, not just the final rolled-up summary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit fixed a context-cancellation bug in the auto-approve path
and appeared to fix things, but re-testing end-to-end after deploy showed the
execution STILL never completed — just via a different symptom
("no pending execution found for approval" in the logs). Dug further and
found the real, deeper bug underneath: this whole mechanism has never
actually worked.
autoApprove() directly flipped BOTH approvals.status and executions.status to
'approved' via raw SQL, then called executeApprovedViaAPI to POST to the
decision endpoint. But DecideApproval's own logic specifically looks for the
execution still at status='pending_approval' to find and dispatch the real
SSH work (executeApprovedAction) — autoApprove's premature flip meant that
lookup always found zero rows. DecideApproval's UpdateApprovalStatus call
also silently no-ops the same way (sqlc :exec doesn't surface "0 rows
affected" as an error). Every assent-window auto-approved pct_create/
apt_upgrade has been sitting at 'approved' forever with the real work never
triggered — indistinguishable from "still running" until you check.
Fix: remove autoApprove() entirely. Call executeApprovedViaAPI directly
against the untouched pending_approval row from createApproval — identical
to the manual Approve-button path, just without the human click. DecideApproval
is now the single place that transitions status and dispatches, for both the
manual and auto-approved paths, closing the class of bug where two code paths
raced to do the same state transition.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live testing the new atomic pct_create: an assent-window
auto-approved pct_create appeared to "run" (logged "auto-approved... running
now") but the execution stayed stuck at 'approved' forever. Root cause:
`go executeApprovedViaAPI(ctx, ...)` passed the MCP tool-call's own context —
which is cancelled the instant the triggering /chat request's HTTP response
completes, i.e. on every normal turn. The spawned goroutine's POST to the
approval-decision endpoint died with "context canceled" before it could even
start the real work, and nothing surfaced this to the operator or the agent —
the execution just sat at 'approved' with no error, indistinguishable from
"still running."
This is exactly the context-lifetime bug class httpapi's own approval
goroutine (executeApprovedAction) already avoided by using
context.Background() — it had just been missed in these two call sites
(apt_upgrade and pct_create auto-approve). Fixed both to use
context.Background(), matching the correct pattern already in place
elsewhere. Audited for other goroutines spawned with a request-scoped ctx —
none found; the sshExec internal goroutines are synchronous/waited-on via
select and correctly scoped to the call.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two remaining open points from the auto-continuation work.
1. Atomic pct_create (observability, the bigger of the two):
pct_create used to bundle create + apt install + post_install script into
one black-box multi-minute SSH call — the agent got back a single opaque
success/fail with no way to see (or fix) which step actually broke.
Removed the whole post-create provisioning block (and the now-dead
provisionScript/sanitizePkgs helpers + their tests). pct_create is now
create + start + register ONLY — fast, and its result is fed back to the
agent via auto-continuation almost immediately. The agent installs
packages and runs setup as its OWN sequence of `run` calls against the new
lxc:<hostname>, observing each command's real output and able to diagnose
and retry exactly the step that failed — the same recovery loop already
proven for the general case, now applied to installs too, instead of
requiring a separate black-box mechanism.
- services/post_install removed from the pct_create params struct and
from the MCP tool schema/SOUL.md docs.
- SOUL.md: explains the new flow, moves the Docker CLI gotcha and DNS
troubleshooting guidance to be steps the agent runs itself.
2. Scoped destructive window (targeted autonomy for recovery):
Verified live in the previous session that a destructive recovery (a
failed destroy needing stop-then-destroy on the same container) required
TWO separate typed confirmations for what was clearly one recovery
action. Added a narrow, TARGET-scoped 15-minute grant
(destructive_window.agent:<id>.target:<slug> in autonomy_settings,
shared key format across cmd/nomos and internal/mcp) that opens only
after an EXPLICIT typed confirmation (never loose assent) or an explicit
button-approval of a destructive step, and only ever covers further
destructive commands against that SAME target. A different target always
needs its own fresh confirmation — this narrows risk instead of loosening
it globally, unlike broadening the general assent window to cover
destructive actions would have.
- cmd/nomos/store.go: openDestructiveWindow/destructiveWindowActive/
executionTarget.
- cmd/nomos/agent.go: opens the window when a typed confirmation grants a
destructive chat-assent execution.
- internal/mcp/server.go: `run` tool checks the window before gating a
destructive command; auto-runs if active.
- internal/httpapi/phase3.go: DecideApproval opens the same window when a
destructive execution is approved via the button/API, for parity with
the chat-assent path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live: the new auto-continuation worker (previous commit) worked end
to end for the happy path (provision -> auto-verify -> report success, zero
operator ticks). But testing a failure-recovery case (a destroy that failed
because the container was still running) surfaced a real bug: continueSession's
emit closure only captured "text" events, so when chatWith ended the turn on
an "error" event (LLM returned an empty/refusal response, internal retry also
empty), the worker persisted a completely blank, uninformative "auto" message
— no sign anything had gone wrong, undermining observability of the very
mechanism just built.
- Capture "error" events and, if the turn produced no text/tool_calls at all,
persist an explanatory placeholder instead of blank.
- Add one outer retry of the whole chatWith call when the first attempt
produces nothing — the principle behind this whole feature ("don't give up
on the first error") should apply to the continuation mechanism itself, not
just the homelab commands it's continuing.
Also verified live: recovery-from-failure works via the normal chat path once
prompted, and cleaned up the test container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:
1. Chat-assent fallback: when the operator says 'proceed' but the
preceding turn had NO pending approvals (agent proposed plan in text
without calling request_execution), inject a system note telling the
agent to execute the plan now. Opens the assent window so subsequent
config_mutation commands auto-run.
2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
proposing a plan, not wait for 'proceed' first. This ensures a
pending approval exists for chat-assent to grant.
3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
installs the daemon but NOT the docker CLI binary. Must use
get.docker.com in post_install. Added 'handling errors' section:
diagnose, try alternatives, continue — don't stop after one failure.
When the operator has approved a plan via chat assent (assent window
active), pct_create and apt_upgrade now auto-approve and execute
instead of queuing for a separate approval round. The auto-approve
path updates the approval+execution status in the DB, then calls the
HTTP API's decision endpoint to trigger executeApprovedAction — same
code path as a manual Approve button, consistent audit trail.
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:
1. Compound read-only commands (e.g. 'systemctl status; journalctl')
defaulted to config_mutation — now splits on ;/&&/||/| and classifies
as read_only if all segments are inspection verbs. Added grep, wc,
sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.
2. curl|sh was classified destructive, forcing typed confirmation for
legitimate installs (get.docker.com). Demoted to config_mutation —
loose assent grants it, no typed phrase needed.
3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
on non-blocked steps'. Added assent window section instructing agent
to carry out the full plan after approval.
4. Assent window: when operator approves a plan via chat assent, a
30-minute window opens where config_mutation commands auto-run
without re-approval. Agent writes expiry to autonomy_settings; MCP
run tool checks it before gating. Destructive never auto-runs.
5. System note after approval now says 'CONTINUE executing the full
plan — do not stop and wait for continue.'
Verified live that after deploying the "fixed" bridge-bound pre-flight, it
still let a known-bad vmbr0+192.168.8.2 config straight through to a full
pct_create with no error. Root cause: the check used
`strings.Contains(pingOut, "REACHABLE")` against markers "REACHABLE" /
"UNREACHABLE" — but "UNREACHABLE" contains "REACHABLE" as a substring, so the
containment check was true for BOTH outcomes. The pre-flight was structurally
incapable of ever failing, regardless of the actual ping result.
Fixed with distinct, non-overlapping markers (PREFLIGHT_OK/PREFLIGHT_FAIL)
and exact-match comparison, pulled into a small gatewayPreflightPassed()
helper with a unit test asserting the exact historical bug case
("UNREACHABLE" must be false) so this bug class can't silently recur.
Re-verified live end-to-end: manually re-tested the exact ping command
(confirmed UNREACHABLE via vmbr0), and this was caught only by actually
running the check against production, not by reading the code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified live that the pre-flight check added in the previous commit had a
real gap: a plain `ping <gateway>` from the Proxmox host succeeds via the
HOST's own routing table (which can have routes to a subnet through paths
the host alone knows about), even when the CONTAINER — attached via a plain
bridge with only a naive on-link default route — can never actually ARP that
gateway. Confirmed by creating a real test container on vmbr0 with
gw=192.168.8.2: the host-wide ping had said "reachable," but pinging from
inside the container showed 100% packet loss. Fixed by binding the pre-flight
ping to the specific requested bridge (`ping -I <bridge>`), which correctly
rejects vmbr0 for that gateway instead of false-positiving via the host's
broader routing table.
Also confirmed live: vmbr1 does exist and is up on strong (contrary to the
possibly-stale host doc), matching what romm/seanime's docs already said.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Investigated why the operator couldn't get past "no DNS/connectivity" across
multiple retries even after Nomos correctly diagnosed and fixed the gateway
(192.168.8.1 -> 192.168.8.2). It still failed. Root cause, confirmed from
strong's own documented network topology: on `strong`, vmbr0 physically
bridges only to 192.168.178.0/24 — the 192.168.8.0/24 service network is
reached via a Fritz!Box static route, not a local bridge. A container
attached to vmbr0 can never reach a 192.168.8.x gateway no matter which
address in that range is picked; ARP for it just gets silently dropped
(matching the earlier hang symptom). The gateway was never the problem — the
bridge was. 192.168.8.0/24 is also segmented into /28 blocks each with their
own gateway (192.168.8.2 is only the .0-.15 block's gateway), so even a
correct bridge with a copy-pasted gateway from a different block would still
fail.
No amount of retrying with a different gateway guess could have fixed this —
the missing fact (which bridge reaches which subnet, and the per-/28 gateway)
isn't inferable from the subnet alone.
- pct_create gets a `bridge` param (was hardcoded to vmbr0) so a correct
bridge can actually be requested once known.
- Fast pre-flight: for any static IP, ping the gateway from the target HOST
before creating anything. Was: a bad config took a multi-minute hang (or,
after last commit's timeout fix, ~2min) before failing. Now: ~2 seconds,
with a message that explicitly says not to guess a different gateway in
the same subnet — find a real neighbor's config or use DHCP.
- SOUL.md: DHCP is now framed as the default, not a fallback; static IP
requires finding an existing LXC on the same host in the same /28 and
copying its bridge+gateway verbatim — inventing one is explicitly called
out as the failure mode that caused this exact incident.
- MCP tool schema: pct_create's params description now documents `bridge`
and the neighbor-copy rule directly in what the model reads at call time,
not just in SOUL.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).
Two compounding bugs made this unrecoverable without manual intervention:
1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
NO execution timeout — `session.CombinedOutput()` blocks until the remote
command exits, with no deadline. A hung remote process blocks the Go
goroutine forever; the execution can never leave 'running', and the
operator has no way to make it stop. Fixed: both now race the SSH call
against a 10-minute hard timeout, closing the session/client and
returning a clear "timed out after 10m0s" error if exceeded. (The
mcp/server.go copy also still had the original "swallowed non-zero exit"
bug from before that fix was applied to httpapi's copy only — fixed here
too.)
2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
connectivity — it doesn't; a black-holed network can make each call hang
far past the resolver's nominal timeout, so the documented "~90s" budget
was never real. Wrapped every attempt in `timeout 3` so the wall-clock
budget is now actually enforced (~2min worst case), and the failure
message now suggests checking the net0 gateway.
Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):
- InlineApproval's track() polling loop had its own ~6min ceiling and simply
STOPPED polling after that — silently going stale before the backend (now
correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
with margin, and added a distinct 'stalled' state if that's ever exceeded
(explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
execution's created_at), the actual command being run, and the execution
ID — previously just a static "this can take a minute" with zero
information. Also added command display to the destructive pending-
approval card for full transparency before confirming.
Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.
1. InlineApproval.svelte — all of last session's live-status/self-heal work —
was never imported or rendered anywhere. Chat.svelte had its own separate,
much dumber approval bar (no status tracking, no destructive handling, just
silently disappears after clicking) that WAS the one users actually saw.
Deleted the dead bar and its state; InlineApproval now renders per-message.
2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
so any approval raised by the newer `run` tool was invisible — no card, no
feedback, nothing to self-heal, forcing the operator to the Ops page with
zero acknowledgement in the conversation. This was the actual proximate
cause of last night's destroy-135 session. Fixed to match on response
shape, not tool name, so it doesn't silently break again for the next new
gated tool.
3. Nomos was telling operators "type something like 'I confirm destroy 135'"
for destructive actions (SOUL.md) but no backend path ever consumed that
phrase — chat-assent explicitly (and correctly) excludes destructive from
loose assent, but I never built the alternative. Added
isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
requires an explicit "confirm" statement, only applies to destructive-
flagged pending approvals.
4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
wrong/confusing for a destroy or arbitrary `run` command. Now says
"Completed on <target>" and shows the actual command output, verified live
against the real destroy-135 execution.
Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live testing hit `entities_slug_key` violations: exec slugs used an 8-char
prefix of a UUIDv7, whose leading bytes encode a millisecond timestamp — two
executions created seconds apart can share a prefix. Use the full UUID
(guaranteed unique) for the exec entity's slug/name in request_execution, the
new `run` tool, and the REST RequestExecution handler — all three had the
same pattern.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Found live: `run` against lxc:caddy failed with "missing pve_id" even though
pve_id=121 was present — caddy is an inventory-seeded LXC with no `host`
attribute at all (only pct_create-provisioned LXCs set one). The combined
query scanned attributes->>'host' (SQL NULL) into a plain Go string, which
errors the whole Scan — including the pve_id column that scanned fine.
COALESCE the host column to '' so a missing host attribute degrades to the
documented default instead of failing the whole resolution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.
- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
read-only allowlist + destructive denylist, default-escalate to
config_mutation for anything else. Classification can only ESCALATE the
caller's declared risk, never de-escalate it (destructive always wins even
if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
purpose, optional declared_risk. Read-only commands execute immediately;
everything else queues an approval exactly like pct_create today, executed
via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
helpers (mcp + httpapi) fix this for both the new `run` action and existing
actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
'config_mutation' on every approve, silently corrupting the audit ledger for
every other risk class; (2) denying/revoking an approval never updated the
linked execution's status, so it stayed 'pending_approval' forever instead
of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
Scoped to the immediately-preceding assistant turn's pending approvals only
— an old "yes" can't retroactively approve something new. Destructive-risk
actions are excluded from loose assent. Approves via the same HTTP decision
endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
just its own button. Previously the banner stayed stuck showing
Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
risk gate"); documents chat-assent behavior and the destructive exception.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator directives: (1) the UI must show what's executing, its risk
classification, live status, and what knowledge the session created — the
system's growth should be visible, not just trusted. (2) approval should be
granted by chat assent ("go ahead"), not a separate button; destructive
actions still require a typed confirmation phrase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Evaluate the agent's action path against the OIKOS.md design. Finding: the
intended model (unlimited runbook-driven actions gated by a risk classifier)
already exists on paper and in scaffolding, but the live agent path regressed
to a hard-coded 5-action enum that bypasses the classifier. Plan a layered
realignment: (0) one general gated `run` primitive, (1) runbooks-as-data as the
reliable fast-path, (2) learning. Chosen v1 posture: approve-most (read-only
auto-runs, all state changes gate). Incremental, each step shippable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fresh Debian LXCs have no locale configured, spamming "apt-listchanges:
Can't set locale" / perl warnings across every install and breaking some
packages' post-install scripts. Pin LANG/LC_ALL=C.UTF-8 (and hoist
DEBIAN_FRONTEND) at the top of the in-container bootstrap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Production session provisioned the container but the service never installed:
apt failed with "Temporary failure resolving deb.debian.org" — a static-IP LXC
whose assigned nameserver couldn't resolve. The operator also got zero feedback:
the approval banner just sat there with no running/complete/failed status.
Backend robustness (provisionScript):
- Wait for real DNS/connectivity inside the container before apt, and self-heal
/etc/resolv.conf to a public resolver (1.1.1.1/8.8.8.8) if the assigned one
is dead. `set -e` after the gate so apt/post_install failures surface.
- apt-get update/install with Acquire::Retries=3.
Frontend feedback (InlineApproval):
- After approve, poll GET /executions/{id} and show live phase: submitting →
provisioning… → provisioned successfully / execution failed (with the error).
- add getExecution() to api.ts.
Agent guidance (SOUL.md):
- omit vmid (auto-assigned), prefer dhcp, docker-compose-plugin is not in Debian
(use docker.io + get.docker.com), end post_install with a health check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The final "UPDATE executions SET result=$::jsonb" built its payload with
fmt.Sprintf and only escaped newlines. apt/pct output contains quotes,
backslashes and control chars, so the payload was invalid JSON, the jsonb
cast failed, and the (unchecked) UPDATE was silently discarded — the
execution stayed 'approved' with a NULL result even though the LXC was fully
provisioned (verified live: vmid auto-assigned, container running, service
installed, post_install ran).
- executeApprovedAction: marshal result via json.Marshal; log UPDATE errors
- add jsonErr() helper; route all pct_create failure-path results through it
- mcp/server.go: add jsonOut() for restart/systemctl/pct_exec inline results
- regression test for JSON validity on quote/backslash/control-char output
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-ups found while verifying the approve→provision path end to end:
- vmid is now optional: the early required-field check rejected vmid:0
before the cluster VMID guard could auto-assign a free id. Only hostname
is required now; 0 (or a collision) resolves to `pvesh get /cluster/nextid`.
- net0: use ip=dhcp with no gateway when no static IP is given (Proxmox
rejects gw alongside dhcp); only attach gw for a static CIDR.
- bump post-create settle to 10s so a DHCP lease is up before apt runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
