The root cause behind "the agent stops at the first error and doesn't recover":
provisioning executions run ASYNCHRONOUSLY (pct_create fires the SSH work in a
goroutine and returns "running" immediately), so the agent's turn ENDS before
the result exists. The agent literally isn't running when the step fails — it
can't react to a failure it never observes. The only thing that fed results
back was the operator typing "continue" after every async step: the human was
the event loop. (In the flagged 18-message session the operator typed
continue/proceed/?? eight times while the agent correctly diagnosed each failure
but couldn't advance a step on its own.)
This makes the system the event loop instead:
- migrations/017: nomos_plan_executions links each gated execution to the chat
session that started it.
- cmd/nomos: after a tool result, any "execution <uuid>" it started is linked
to the session. A background worker (continue.go) polls for those executions
reaching a terminal state and — while the agent has an open assent window (an
approved plan is in flight) — re-invokes the agent with the result
("execution X completed/failed: <result>"), so it proceeds to the next step
or diagnoses+fixes the failure, with no operator tick. Guarded against loops
(mark-continued before running) and bounded by the 30-min window.
- chatWith(): chat() variant that injects the finished-execution note after
replayed history without persisting a fake user turn.
- DecideApproval: approving a step by ANY route (button or chat-assent) now
opens the assent window, so auto-continuation works regardless of how the
operator approved — previously only typing "go ahead" opened it.
- SOUL: the agent is told it will be auto-re-invoked when async steps finish —
don't poll get_execution_status, don't wait for "continue"; end the turn and
keep going step by step until the goal is verified or a genuine blocker.
This is the root fix, not another per-command patch: you can't enumerate every
failure of an unbounded action space, but you can give the agent a loop that
observes each result and adapts — because "do anything" always includes "the
first attempt failed."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three fixes for the session where the agent proposed a plan, waited
for 'proceed', then re-queued instead of being auto-approved:
1. Chat-assent fallback: when the operator says 'proceed' but the
preceding turn had NO pending approvals (agent proposed plan in text
without calling request_execution), inject a system note telling the
agent to execute the plan now. Opens the assent window so subsequent
config_mutation commands auto-run.
2. SOUL.md: instruct agent to ALWAYS call request_execution/run when
proposing a plan, not wait for 'proceed' first. This ensures a
pending approval exists for chat-assent to grant.
3. SOUL.md: stronger Docker instructions — Debian 13's docker.io package
installs the daemon but NOT the docker CLI binary. Must use
get.docker.com in post_install. Added 'handling errors' section:
diagnose, try alternatives, continue — don't stop after one failure.
Agent stopped after every approval step, forcing operator to type
'continue' 7× per deploy session. Root causes and fixes:
1. Compound read-only commands (e.g. 'systemctl status; journalctl')
defaulted to config_mutation — now splits on ;/&&/||/| and classifies
as read_only if all segments are inspection verbs. Added grep, wc,
sort, uniq, cut, tr, dpkg -l, apt list, docker stats to allowlist.
2. curl|sh was classified destructive, forcing typed confirmation for
legitimate installs (get.docker.com). Demoted to config_mutation —
loose assent grants it, no typed phrase needed.
3. SOUL.md said 'STOP after queuing' — replaced with 'continue working
on non-blocked steps'. Added assent window section instructing agent
to carry out the full plan after approval.
4. Assent window: when operator approves a plan via chat assent, a
30-minute window opens where config_mutation commands auto-run
without re-approval. Agent writes expiry to autonomy_settings; MCP
run tool checks it before gating. Destructive never auto-runs.
5. System note after approval now says 'CONTINUE executing the full
plan — do not stop and wait for continue.'
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.
1. InlineApproval.svelte — all of last session's live-status/self-heal work —
was never imported or rendered anywhere. Chat.svelte had its own separate,
much dumber approval bar (no status tracking, no destructive handling, just
silently disappears after clicking) that WAS the one users actually saw.
Deleted the dead bar and its state; InlineApproval now renders per-message.
2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
so any approval raised by the newer `run` tool was invisible — no card, no
feedback, nothing to self-heal, forcing the operator to the Ops page with
zero acknowledgement in the conversation. This was the actual proximate
cause of last night's destroy-135 session. Fixed to match on response
shape, not tool name, so it doesn't silently break again for the next new
gated tool.
3. Nomos was telling operators "type something like 'I confirm destroy 135'"
for destructive actions (SOUL.md) but no backend path ever consumed that
phrase — chat-assent explicitly (and correctly) excludes destructive from
loose assent, but I never built the alternative. Added
isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
requires an explicit "confirm" statement, only applies to destructive-
flagged pending approvals.
4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
wrong/confusing for a destroy or arbitrary `run` command. Now says
"Completed on <target>" and shows the actual command output, verified live
against the real destroy-135 execution.
Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the first slice of plans/2026-07-10-general-gated-execution.md:
Nomos gets one general execution tool instead of only a fixed action enum,
gated by an automatic risk classifier, and approval can be granted by the
operator just replying in chat instead of clicking a button.
- internal/policy/command.go: ClassifyCommand(cmd, declaredRisk) — rule-based
read-only allowlist + destructive denylist, default-escalate to
config_mutation for anything else. Classification can only ESCALATE the
caller's declared risk, never de-escalate it (destructive always wins even
if declared read_only). Compound commands (&&, ;, |, $()) never qualify for
the read-only fast path. Full test corpus.
- internal/mcp/server.go: new `run` MCP tool — target (host:/lxc:), command,
purpose, optional declared_risk. Read-only commands execute immediately;
everything else queues an approval exactly like pct_create today, executed
via httpapi's existing executeApprovedAction. Also fixes a real latent bug:
pct_exec resolved an LXC's host attribute without the "host:" prefix, so it
could never find the Proxmox host — new resolveExecTarget/resolveRunTarget
helpers (mcp + httpapi) fix this for both the new `run` action and existing
actions that route through the same execution path.
- internal/httpapi/phase3.go: "run" case in executeApprovedAction; fixes two
bugs found while wiring this up — (1) DecideApproval hardcoded risk_class to
'config_mutation' on every approve, silently corrupting the audit ledger for
every other risk class; (2) denying/revoking an approval never updated the
linked execution's status, so it stayed 'pending_approval' forever instead
of reflecting the decision.
- cmd/nomos/assent.go: deterministic (not LLM-judged) chat-assent detection.
Scoped to the immediately-preceding assistant turn's pending approvals only
— an old "yes" can't retroactively approve something new. Destructive-risk
actions are excluded from loose assent. Approves via the same HTTP decision
endpoint the UI button calls, so both paths share one audit trail.
- web/.../InlineApproval.svelte: self-healing poll — a pending approval card
now picks up being decided via ANY path (chat assent, Ops page, Matrix), not
just its own button. Previously the banner stayed stuck showing
Approve/Deny even after the action had already run elsewhere.
- nomos/SOUL.md: `run` is now the general capability ("no fixed menu, only a
risk gate"); documents chat-assent behavior and the destructive exception.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "asks permission but never acts": the approved pct_create
execution failed to parse because the LLM emitted `"privileged":0` /
`"nesting":1` (numbers) into strict `bool` fields, so the container was
never created. Compounded by a hardcoded template name (debian-13.0-1)
that no longer exists on the host, and no way for the agent to read the web.
- flexBool: accept 0/1, "true", bool for privileged/nesting (the exact prod failure)
- pct_create template pre-flight: list host cache, validate/auto-pick newest debian
- pct_create services[] + post_install: one approval provisions a working service
- new http_get MCP tool (sanitized, size-capped, SSRF-guarded) — agent can read repos/sites
- request_execution description: target=host, full JSON schema + example
- SOUL.md: agent CAN fetch the web; prefer one-step provisioning
- default model deepseek-v4-flash -> v4-pro; maxIterations 15 -> 25
- unit tests for flexBool, template resolve, pkg sanitize, HTML sanitize + SSRF block
Verified live on host:strong with a throwaway VMID 999: template auto-resolved,
container created + booted, services installed, post_install ran, then destroyed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
Problem: every host/service/lxc/etc. entity_status row was permanently
stuck at 'unknown' since creation. Verified against the live DB:
metric_samples had 17,559 rows, 100% attached to type='check' probe
entities and 0% to any real monitored entity; only 25 check entities
ever had real health written. check_defs.entity_id (the probe's own
bookkeeping entity) and check_defs.target_id (the host/service actually
being observed) were both real fields, but the scheduler wrote
UpsertEntityStatus/InsertMetricSample/emitSchedulerEvent keyed by
entity_id instead of target_id — so every check ran and every result
was real, it just landed on the wrong row. This is the mechanism behind
observed drift: the agent's dashboard/health tools reported the
internal probes' state, never the actual fleet.
Change:
- scheduler.go: runCheck/resolveSignal now resolve targetID from
cd.TargetID (falling back to the check's own id if unset) and write
status/metrics/events there. Signals stay keyed by the check entity,
unchanged, matching their existing resolution logic.
- Added a staleness sweep to housekeeping(): an entity whose last
observation is older than 3x its fastest enabled check's interval
(floor 5m) is marked 'stale' and emits health.stale, so a stalled
scheduler or disabled check_def can no longer look like current data
forever.
- migrations/016: deletes the now-orphaned check-entity entity_status
rows so dashboard/fleet-health rollups stop double-counting probes as
monitored entities. Historical metric_samples on check entities are
left as-is (time-series data, not safe to reattribute).
- openapi.yaml + regenerated gen code: Entity gains health/last_check_at;
'stale' added to the health enum everywhere it's used.
- dashboard.go / GetFleetHealth / nomos's get_health_summary MCP tool:
exclude type='check' entities from rollups.
- nomos/agent.go: replay prior turns' tool_use/tool_result pairs into
the conversation instead of dropping them (previously only final text
was replayed, forcing the agent to re-derive fleet state every turn),
and inject a compact live fleet-health snapshot into the system prompt
each turn so it starts oriented instead of spending an iteration on
discovery.
Risk: config_mutation (schema-adjacent — new migration, no destructive
DDL, additive DELETE only on orphaned rows). No behavior change until
oikos-api/oikos-scheduler/nomos are rebuilt and redeployed.
Verification: go build/vet clean across the repo. Ran this worktree's
own API binary against the live dev Postgres on an alternate port
(read-only from the live containers' perspective) and confirmed
/api/v1/entities now returns health/last_check_at, and the dashboard
health rollup dropped from double-counting to an honest 168 unmonitored
entities (matches reality pre-deploy — the live scheduler hasn't run
the fixed code yet). Confirmed check_defs.target_id correctly maps
multiple checks to host:hubris via direct psql query.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
After an api (MCP server) restart, nomos held a dead session id and every
tool call failed with "unexpected end of JSON input" until nomos was manually
restarted — which happens on every deploy. The MCP client now detects a
rejected session (4xx or empty body) and transparently re-initializes and
retries once. Also raise the SSE scanner buffer to 4MB so large tool results
don't exceed the 64KB default token limit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
http.FileServer canonicalizes /index.html -> "./", which for /ui/ produced a
301 redirect loop and made the control room unreachable. Serve embedded files
directly with http.ServeContent instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
