Operator: "I'd like to be able to see in the chat what the agent is actually
running, right now I just wait while nothing happens." Two compounding gaps:
1. The auto-continuation worker (cmd/nomos/continue.go) had zero live push —
its result only appeared on a manual page reload, so approving a plan and
watching the chat looked completely dead even while the agent was actively
working.
2. Even with polling, continueSession only persisted ONE message at the very
end of a continuation — a continuation that runs several tool calls before
concluding would still show total silence for however long that took.
Fixed both:
- web/src/lib/stores/chat.ts: polls the current session's messages every 3s
between turns (never while a live stream owns the message list) and merges
in anything new. Started after a live turn ends and when a session loads;
stopped on new-chat/session-switch.
- cmd/nomos/store.go: insertMessageReturningID/updateMessage — lets a message
be created as a placeholder and updated in place.
- cmd/nomos/continue.go: continueSession now inserts a placeholder the
instant it starts (renders as the existing "thinking" dots — immediate
feedback that something is happening) and updates that SAME row after
EVERY tool call, not just at the end. A poll within ~3s of any tool call
landing shows it — individual `run` commands appear as the agent issues
them, not just the final rolled-up summary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).
Two compounding bugs made this unrecoverable without manual intervention:
1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
NO execution timeout — `session.CombinedOutput()` blocks until the remote
command exits, with no deadline. A hung remote process blocks the Go
goroutine forever; the execution can never leave 'running', and the
operator has no way to make it stop. Fixed: both now race the SSH call
against a 10-minute hard timeout, closing the session/client and
returning a clear "timed out after 10m0s" error if exceeded. (The
mcp/server.go copy also still had the original "swallowed non-zero exit"
bug from before that fix was applied to httpapi's copy only — fixed here
too.)
2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
connectivity — it doesn't; a black-holed network can make each call hang
far past the resolver's nominal timeout, so the documented "~90s" budget
was never real. Wrapped every attempt in `timeout 3` so the wall-clock
budget is now actually enforced (~2min worst case), and the failure
message now suggests checking the net0 gateway.
Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):
- InlineApproval's track() polling loop had its own ~6min ceiling and simply
STOPPED polling after that — silently going stale before the backend (now
correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
with margin, and added a distinct 'stalled' state if that's ever exceeded
(explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
execution's created_at), the actual command being run, and the execution
ID — previously just a static "this can take a minute" with zero
information. Also added command display to the destructive pending-
approval card for full transparency before confirming.
Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Root cause of "chat gave me no further feedback — had to go to Ops": two
compounding bugs, found by reading the actual production session transcript.
1. InlineApproval.svelte — all of last session's live-status/self-heal work —
was never imported or rendered anywhere. Chat.svelte had its own separate,
much dumber approval bar (no status tracking, no destructive handling, just
silently disappears after clicking) that WAS the one users actually saw.
Deleted the dead bar and its state; InlineApproval now renders per-message.
2. chat.ts's extractApprovals hardcoded `tool.name === 'request_execution'`,
so any approval raised by the newer `run` tool was invisible — no card, no
feedback, nothing to self-heal, forcing the operator to the Ops page with
zero acknowledgement in the conversation. This was the actual proximate
cause of last night's destroy-135 session. Fixed to match on response
shape, not tool name, so it doesn't silently break again for the next new
gated tool.
3. Nomos was telling operators "type something like 'I confirm destroy 135'"
for destructive actions (SOUL.md) but no backend path ever consumed that
phrase — chat-assent explicitly (and correctly) excludes destructive from
loose assent, but I never built the alternative. Added
isTypedConfirmation() (cmd/nomos/assent.go): stricter than loose assent,
requires an explicit "confirm" statement, only applies to destructive-
flagged pending approvals.
4. InlineApproval's completed-state hardcoded "Provisioned successfully" —
wrong/confusing for a destroy or arbitrary `run` command. Now says
"Completed on <target>" and shows the actual command output, verified live
against the real destroy-135 execution.
Verified live in a real browser against the production API/DB (dev server
proxying to :8090): the historical stuck session now retroactively renders
both executions as resolved with correct wording and real output.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace text-based regex parsing in InlineApproval with structured
pendingApprovals extracted from request_execution tool results. The tool
result text is deterministic (not LLM-generated), making UUID extraction
reliable regardless of how the LLM rephrases the response.
- Fix ToolCallGroup reactivity: wasActive = active captured initial
value. Now uses (active) so re-runs on prop changes.
- Extract approvals in both live streaming (done event) and history loading
for consistent behavior on resumed sessions.
- Empty/refusal responses retried once, then surfaced as errors instead of silent blanks
- Chinese refusal boilerplate detected via denylist + non-ASCII heuristic
- Bulk-tool preference added to SOUL.md (list_lxcs over per-entity get_lxc_state)
- Tool results truncated to 4KB on persist; get_state_snapshot filters null-state entities
- Session delete (DELETE /sessions/{id} + confirm-on-second-click UI)
- Session titles auto-generated from assistant answer instead of raw user message
Problem: the web UI felt dead and hard to navigate — the Entities table
had no health/freshness signal (just a meaningless row-mutation
timestamp), no way to see what was actually monitoring an entity,
sessions couldn't be reopened, and every drill-down was a full page
navigation that lost the list.
Change:
- Entities table: Updated column replaced with a health dot + relative
"checked Xm ago", sourced from the backend's new health/last_check_at
fields.
- New EntityDetailContent.svelte extracted from EntityDetail.svelte and
shared between the full #/entity/:slug page and a new EntitySheet.svelte
opened from the Entities table (master-detail, row click opens a panel
instead of navigating away). Adds a Monitoring card listing the
entity's check_defs (kind, interval, enabled/disabled with
click-to-toggle via the existing PatchCheck endpoint) and renders
attributes as key/value pairs instead of raw JSON.
- Sessions: fixed a bug where clicking a session loaded it into the
chat store but never navigated to the chat page, so nothing appeared
to happen. Added a SessionRail inside Chat so switching sessions
never leaves the chat surface.
- Fixed the local dev proxy (vite.config.ts): production Caddy strips
the /agent prefix before forwarding to nomos; the dev proxy didn't,
so every session/chat fetch 404'd locally while working in prod.
- Found and fixed a real latent bug while testing the session fix:
chat.ts's loadSessionMessages passed the persisted tool_calls array
straight through, but nomos stores the tool_use and tool_result as
two entries sharing one id. Chat.svelte's keyed {#each tool (tool.id)}
throws on the duplicate key, which silently blanked the entire
message list — invisible until sessions were actually clickable.
Fixed by merging tool_calls by id before rendering, matching the
shape the live-streaming path already produces.
- UI polish: sidebar logo is now just the omicron mark in white (was
icon+text in the accent color); removed the sheet overlay's
backdrop-blur (distracting per feedback); the Attributes/Relations/
Signals grids used viewport-based lg:/3xl: breakpoints, which forced
multi-column layouts based on browser width regardless of the sheet's
actual rendered width — switched to Tailwind v4 container queries
(@lg:/@2xl:/@3xl:) so layout responds to the real available width in
both the full page and the narrower sheet.
Risk: reversible_low (UI-only; no destructive operations; the tool_calls
merge and dev-proxy fix are corrections to broken paths, not behavior
changes to working ones).
Verification: npx tsc --noEmit clean (excluding pre-existing unrelated
.svelte type-resolution warnings). Manually verified in the browser
preview against the live dev API: Entities table health column renders
correctly; clicking a row opens the EntitySheet with a populated
Monitoring card (16 checks for host:hubris, verified via psql that
check_defs.target_id links them correctly); clicking a session now
loads its full transcript inline (was blank before the tool_calls fix);
sheet has no blur and lays out single/multi-column correctly at the
sheet's actual width.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
