Full review (plans/2026-07-17-codebase-review-and-cleanup.md) covering Go, web SPA, and docs. Applied low-risk doc/tooling fixes; code refactors and dead-code deletions are listed as actionable recommendations pending approval. Doc fixes: - AGENTS.md: remove ghost of retired request_execution (contradicted the retire notice above it); fix knowledge/wiki/ -> archive/knowledge/; replace brittle counts (33 tools, 36 docs, 20 checks) with pointers to source; drop point-in-time dates. - OIKOS.md: fix broken plan link (now in done/); 001-011 -> 001-020; 15 MCP tools -> pointer; replace hardcoded knowledge counts. - README.md: 15 tools -> pointer; fix wails plan link (now in done/); complete internal/ package list (add checkdefaults, observability, safego); add cmd/desktop/ to repo layout. - commands.md, page-templates.md: fix broken links; HERMES.md -> NOMOS.md. Plans housekeeping: - Move 4 done 2026-07-14 plans from plans/ to plans/done/. - Reconcile plans/index.md: add the 2 missing 2026-07-14 entries and the 2 missing 2026-07-15 done entries; add this review. - Fix stale plan path in migrations/020 comment. New docs: - docs/index.md and docs/operations/README.md (folder READMEs per writing-style.md). Tooling: - web/package.json: add check/typecheck/lint scripts + svelte-check devDep. - Makefile: desktop-package version now reads from VERSION file instead of hardcoded 0.1.0. VERSION 0.7.6 -> 0.7.7 (patch: docs + tooling only).
1175 lines
55 KiB
Markdown
1175 lines
55 KiB
Markdown
# 2026-07-14 — Session reliability & UX audit: "empty responses, disconnects, and silent failures"
|
||
|
||
**Status:** Planned — 2026-07-14. Audit from a Nomos production session that
|
||
exhibited multiple reliability gaps. Findings are grounded in code paths and
|
||
empirical session data from the live DB (`00a344e4` + `42424b92`).
|
||
|
||
## Session under audit
|
||
|
||
| Session | Goal | Messages | Outcome | Top tools |
|
||
|---|---|---|---|---|
|
||
| `00a344e4` (most recent) | Upgrade 4 low-risk LXCs | 22 (11 turns) | success | run (76), request_execution (86), update_plan_step (42), get_execution_status (24) |
|
||
| `42424b92` (prior failure) | Fleet health check | 3 messages | **failure** | — empty response |
|
||
|
||
---
|
||
|
||
## Findings — Part A: UX reliability (the session-breaking class)
|
||
|
||
### 1. Empty/unusable responses — "Nomos returned an empty or unusable response — please retry."
|
||
|
||
**Where:** `cmd/nomos/agent.go:362-371` (`isRefusalOrEmpty`) + `cmd/nomos/continue.go:207-259` (`resumeSession` retry).
|
||
|
||
**What happened:** The LLM (deepseek/deepseek-v4-pro over OpenRouter) returned
|
||
a blank or non-English response. `maxLLMRetries = 1` (agent.go:33) gives exactly
|
||
one retry inside `chatWith`. `resumeSession` adds a second outer retry
|
||
(continue.go:216). After both fail, the auto-continuation marks the task as
|
||
`failure` with `"Auto-resume failed after retrying: Nomos returned an empty
|
||
or unusable response — please retry."`.
|
||
|
||
**Root cause:** No state-specific recovery. The user sees "please retry" but
|
||
(a) the task is marked terminal (`failure`), so retrying in the same session
|
||
won't work — the agent won't auto-resume a completed task; (b) there's no
|
||
suggestion about *how* to retry (new chat message? resend the same prompt?);
|
||
(c) the error surface in the UI is a plain text `error` store update that
|
||
disappears on the next send.
|
||
|
||
**Severity:** Blocker — this is the session-ending failure class.
|
||
|
||
### 2. Client loses connection and does not recover
|
||
|
||
**Where:** `cmd/nomos/main.go:153-272` (SSE `handleChat`) + `web/src/lib/stores/chat.ts:218-331` (`sendMessage`).
|
||
|
||
**What happened:** SSE relies on a persistent HTTP connection. When it drops
|
||
(network blip, server restart, 10-min timeout), the frontend's stream error
|
||
callback sets `error` but:
|
||
- No auto-reconnect. The user has to logout/log in to get a fresh SSE stream.
|
||
- No fallback to polling during a stream outage. The polling loop
|
||
(`startPolling`, chat.ts:152) deliberately bails when `$streaming` is true.
|
||
- The error message is not persistent — it clears on next send.
|
||
|
||
**Root cause:** SSE streams have no reconnect logic. The connection check is
|
||
binary: streaming or not. There's no "connection lost but task still running"
|
||
state.
|
||
|
||
**Severity:** Blocker — forces logout/login cycle for the user.
|
||
|
||
### 3. Agent stops mid-plan — needs "status" nudge to wake up
|
||
|
||
**Where:** `cmd/nomos/continue.go:100-155` (`runContinuationWorker` + `processContinuations`).
|
||
|
||
**What happened:** The auto-continuation worker polls every 4 seconds for
|
||
finished executions and checks `assentWindowActive` (continue.go:145). If the
|
||
assent window expired or was never opened, the execution is marked `continued`
|
||
without actually resuming — the agent silently drops the ball. The idle sweep
|
||
(continue.go:49-65) only fires every **5 minutes**, and its first pass only
|
||
nudges the agent with a system note. If that nudge gets lost (empty response,
|
||
or the model ignores it), the next sweep auto-closes the task.
|
||
|
||
**Root causes:**
|
||
- `assentWindowActive` check is too aggressive — if the window expired while
|
||
an execution was running, the result is silently discarded.
|
||
- No user-visible indication that the agent is "waiting for something."
|
||
- The idle sweep's 5-minute interval + 2-pass nudge-then-close is too slow
|
||
for a "the agent just stopped talking" situation.
|
||
|
||
**Severity:** Blocker — operator has to manually ask "status?" to resume work.
|
||
|
||
### 4. No feedback when background work is running
|
||
|
||
**Where:** `web/src/lib/stores/chat.ts:149-170` (polling) + `web/src/lib/components/ToolCallGroup.svelte` (tool display).
|
||
|
||
**What happened:** When the agent queues an execution (e.g. `apt upgrade -y`),
|
||
the tool result says "execution <uuid> queued — requires approval." After
|
||
approval, the execution runs in the background. The chat shows:
|
||
- ToolCallGroup: the `run` tool call with its result (static, no updates).
|
||
- InlineApproval: the approval card polls execution status and shows it.
|
||
|
||
But: when the auto-continuation worker picks up the finished execution and
|
||
resumes the agent, the agent's next turn (polling for its messages) is
|
||
invisible unless the user happens to be watching the session at that exact
|
||
moment. There's no "agent is working on step 4 of 6" indicator.
|
||
|
||
**Root causes:**
|
||
- Polling only updates the message list — there's no activity indicator.
|
||
- `get_execution_status` results are rendered as generic JSON in
|
||
ToolCallGroup, not as custom execution-status cards.
|
||
- No "last activity" timestamp in the chat header.
|
||
- SessionDigest is computed at session fetch time, not live-updated.
|
||
|
||
**Severity:** Friction — the agent IS working, the user just can't tell.
|
||
|
||
### 5. Approvals sometimes not shown/asked in chat
|
||
|
||
**Where:** `web/src/lib/stores/chat.ts:33-52` (`extractApprovals`) + `web/src/lib/components/InlineApproval.svelte`.
|
||
|
||
**What happened:** `extractApprovals` scans tool results for `"requires approval"` +
|
||
execution UUID. But it only runs on the SSE `done` event (chat.ts:296-302).
|
||
If the stream disconnects before `done` (finding #2), approvals are never
|
||
extracted in the live view — they'd appear on reload but not in real time.
|
||
|
||
Additional gap: the extraction uses `t.args?.action ?? t.args?.purpose ?? t.name`
|
||
(chat.ts:43). For `run` tools, `t.name` is `"run"` — not informative. The
|
||
purpose text is in `t.args.purpose` but it's a full sentence, not a short label.
|
||
|
||
**Root causes:**
|
||
- Approvals only extracted on `done`, not on each `tool_result`.
|
||
- No fallback extraction on message reload/poll.
|
||
- The card label doesn't clearly show what's being approved.
|
||
|
||
**Severity:** Friction — approvals ARE in the system (Ops page shows them),
|
||
but the chat doesn't always surface them.
|
||
|
||
### 6. No custom renderer for `get_execution_status` — 24 calls rendered as raw JSON
|
||
|
||
**Where:** `web/src/lib/components/ToolCallGroup.svelte` (generic tool display).
|
||
|
||
**What happened:** The agent called `get_execution_status` **24 times** in the
|
||
session. Each result is a JSON blob with execution ID, status, target, output,
|
||
error. All 24 are rendered as generic collapsed JSON in ToolCallGroup — the
|
||
user has to expand each one manually and parse the JSON to understand what's
|
||
happening.
|
||
|
||
**Fix needed:** Custom renderer via the tool-renderers system (`web/src/lib/tool-renderers/`)
|
||
that shows: execution ID (truncated), status badge (running/completed/failed),
|
||
target slug, elapsed time, collapsible output. Optionally auto-poll while
|
||
the execution is `running`.
|
||
|
||
**Severity:** Friction — 24 calls in one session makes the chat cluttered and
|
||
hard to follow.
|
||
|
||
### 7. Plan changed mid-flight, stale steps left behind
|
||
|
||
**Where:** `cmd/nomos/tasks.go:182-205` (`propose_plan`) + `cmd/nomos/store.go:367-423` (plan storage).
|
||
|
||
**What happened:** The agent called `propose_plan` 6 times in the session.
|
||
Each call appends new steps (store.go:418-423: `appended=true`). But:
|
||
- Five `propose_plan` calls mean five generations of plan steps.
|
||
- Old steps that were pending when a new plan was proposed remain in the list
|
||
with their old status (some `running`, some never started).
|
||
- `update_plan_step` only updates by `seq` number — if the new plan reuses
|
||
the same `seq` values, old steps get overwritten. If it adds new `seq`
|
||
values, old steps remain orphaned.
|
||
|
||
**Root causes:**
|
||
- No auto-cleanup of pending steps when a new plan is proposed.
|
||
- The plan panel shows ALL steps across all plan generations.
|
||
- No "plan revision" tracking — just a flat append.
|
||
|
||
**Fix needed:** When `propose_plan` is called again, auto-set any still-pending
|
||
steps (status ≠ done/failed/skipped/blocked) from the previous generation to
|
||
`replaced`. Track a `generation` column on plan steps so the frontend can
|
||
collapse or dim old generations.
|
||
|
||
**Severity:** Friction — the operator's plan view shows progress that doesn't
|
||
reflect reality, eroding trust.
|
||
|
||
### 8. 4 orphaned approvals after session completed
|
||
|
||
**Where:** `cmd/nomos/store.go:480-509` (`completeTask`) — no approval cleanup.
|
||
|
||
**What happened:** When the agent calls `complete_task`, it updates
|
||
`agent_sessions.status` and `outcome`. But it does NOT:
|
||
- Cancel any `pending_approval` executions linked to this session.
|
||
- Close any `approved` but unfinished executions.
|
||
- Clean up the `autonomy_settings` assent window rows.
|
||
|
||
**Root causes:**
|
||
- `completeTask` is a simple status update with no cascading cleanup.
|
||
- No lifecycle link between `agent_sessions.id` and `executions.session_id`
|
||
for approval cleanup.
|
||
- The executions table has `correlation_id` but `completeTask` doesn't use it
|
||
to find and cancel open executions.
|
||
|
||
**Fix needed:** `completeTask` should auto-cancel any executions in
|
||
`pending_approval` or `approved` state for this session. Also clean up the
|
||
assent window key from `autonomy_settings`.
|
||
|
||
**Severity:** Blocker — leaves the operator with a stale approval count,
|
||
confusion about what's pending, and no automatic cleanup path.
|
||
|
||
---
|
||
|
||
## Findings — Part B: Agent knowledge loop (what the agent learned vs. what it wrote back)
|
||
|
||
### 9. ZERO `update_entity_attributes` or `create_relationship` calls in the entire session
|
||
|
||
**Where:** Session `00a344e4` — 22 messages, 11 turns, 348 total tool calls.
|
||
|
||
**What happened:** The agent discovered **massive drift** between the DB and
|
||
reality but wrote **none** of it back to entity attributes or relationships:
|
||
|
||
| Discovery | Tool used | Written back? |
|
||
|---|---|---|
|
||
| 13 LXCs are `state: destroyed` | `get_entity` × 14 | **No** |
|
||
| 5 LXCs reachable via pct exec (SSH auth fails) | `run` × 5 + `get_entity` | **No** |
|
||
| teddycloud runs on `host:hubris` not `host:strong` | `get_relations` | **No** |
|
||
| rclone was false-negative (resolver glitch), actually alive | `get_entity` | **No** |
|
||
| romm has broken `pct config` (nodes/h error) | `run` | **No** |
|
||
| Updated upgrade counts for 4 LXCs | `run` (apt-get upgrade) | **No** |
|
||
|
||
Every future fleet audit will re-discover these same facts from scratch,
|
||
issuing the same 44 wasted tool calls (14 `request_execution` against
|
||
destroyed LXCs + 14 `get_entity` to confirm they're destroyed).
|
||
|
||
**Root cause:** The plan's final step descriptions were too vague:
|
||
- Plan 1 step 4: "Record findings" → agent called `upsert_knowledge` only.
|
||
- Plan 2 step 4: "Write back knowledge" → agent called `upsert_knowledge` only.
|
||
- Plan 3 step 5: "Write back results" → agent called `upsert_knowledge` only.
|
||
|
||
The SOUL.md (lines 86-103) is **explicit** about the three-step writeback:
|
||
`update_entity_attributes` → `create_relationship` → `upsert_knowledge` →
|
||
`complete_task`. But the plan step descriptions the agent wrote for itself
|
||
said "Record findings" — too ambiguous. The agent satisfied this with
|
||
narrative knowledge alone and skipped structured entity updates entirely.
|
||
|
||
**Severity:** Blocker — the knowledge loop is broken. The graph keeps drifting
|
||
further from reality each session because nothing is written back.
|
||
|
||
### 10. `list_lxcs` returns all LXCs including destroyed ones — no state filter
|
||
|
||
**Where:** `internal/mcp/server.go:664-675`
|
||
|
||
**What happened:** The `list_lxcs` MCP tool queries `WHERE e.type = 'lxc'` with
|
||
no state filter. The result included 13 destroyed LXCs mixed with 19 live ones.
|
||
The agent then:
|
||
1. Issued 14 `request_execution` calls against no-IP LXCs (all failed because
|
||
the LXCs are destroyed → 14 wasted tool calls).
|
||
2. Issued 14 `get_entity` calls to discover they're `state: destroyed`.
|
||
3. Had to manually classify which are live and which are destroyed.
|
||
|
||
If `list_lxcs` excluded destroyed entities (or accepted a `state` filter),
|
||
these 28 tool calls would have been eliminated from the first turn alone —
|
||
reducing turn 1 from 92 tool calls to 64 (and making the plan simpler).
|
||
|
||
**Root cause:** `list_lxcs` is a fixed SQL query. The `list_entities` API
|
||
already supports `state` filtering (entities.sql.go:118:
|
||
`$1::text IS NULL OR e.state = $1`), but `list_lxcs` hardcodes its query
|
||
without exposing the parameter.
|
||
|
||
**Severity:** Friction — wastes ~30% of tool calls per fleet-wide audit.
|
||
|
||
### 11. Knowledge recorded was good quality but linked too broadly
|
||
|
||
**Where:** Session `00a344e4` — 4 knowledge entries via `upsert_knowledge`.
|
||
|
||
**What was recorded:**
|
||
| Entry | Kind | Linked to | Quality |
|
||
|---|---|---|---|
|
||
| "Fleet-wide apt audit — 2026-07-10" | investigation | `cluster:homelab` | Good — full table + caveats |
|
||
| "Fleet-wide apt audit — complete — 2026-07-10" | investigation | `cluster:homelab` | Duplicate of v1 — should have updated v1 instead |
|
||
| "Low-risk tier upgrade results — 2026-07-10" | investigation | `cluster:homelab` | Good — per-LXC breakdown |
|
||
| "How to run fleet apt upgrades — lessons" | document | `agent:nomos` | Good — actionable runbook |
|
||
|
||
**What should have been different:**
|
||
1. The two audit investigations are near-duplicates — v2 supersedes v1 but v1
|
||
wasn't marked as superseded. `upsert_knowledge` with the same title should
|
||
update the existing entry.
|
||
2. All entries linked to `cluster:homelab` — none to individual LXCs. The
|
||
upgraded LXCs (`lxc:nfs-export`, `lxc:gitea`, `lxc:dns`,
|
||
`lxc:auth-outpost`) should each have the upgrade result linked via `about`.
|
||
Future `get_entity_knowledge("lxc:nfs-export")` returns nothing for this
|
||
session's work.
|
||
3. The 13 destroyed LXCs aren't linked to any knowledge entry documenting
|
||
*why* they were destroyed. A future agent looking at `lxc:arr-yunohost`
|
||
won't find the knowledge entry saying "migrated to arriman, 2026-04-28."
|
||
|
||
**Root cause:** The agent used `about: cluster:homelab` for everything.
|
||
The `upsert_knowledge` tool accepts a single `about` slug, so the agent
|
||
cannot link one entry to multiple entities. Multi-entity linking would
|
||
require either multiple `upsert_knowledge` calls (one per LXC) or extending
|
||
the tool to accept an `about` array.
|
||
|
||
**Severity:** Friction — knowledge exists but is hard to discover per-entity.
|
||
|
||
### 12. Plan step descriptions are too vague — agent interprets them weakly
|
||
|
||
**Where:** The plan steps the agent proposed for itself across 3 plans.
|
||
|
||
**What happened:** The agent wrote plan steps like:
|
||
- "Record findings" / "Update knowledge base if anything notable"
|
||
- "Write back knowledge" / "Record deprecated list and SSH strategy"
|
||
- "Write back results" / "Record what was upgraded and any issues."
|
||
|
||
Every time, the agent satisfied these with `upsert_knowledge` alone and
|
||
skipped `update_entity_attributes` + `create_relationship`. The SOUL.md
|
||
instructions are explicit about the three-step writeback, but the plan
|
||
step descriptions the agent generated for itself didn't reinforce this.
|
||
|
||
**Root cause:** The agent proposes its own plan steps. The tool description
|
||
for `propose_plan` (tasks.go:36-70) says the last step should include
|
||
`update_entity_attributes / create_relationship / upsert_knowledge`. But the
|
||
agent still wrote vague step descriptions. The instruction is present but
|
||
not being followed.
|
||
|
||
**Severity:** Blocker — this is the direct cause of finding #9.
|
||
|
||
### 13. Plan step completion order was wrong — step 5 marked done before step 4
|
||
|
||
**Where:** Turn 11 (the "lets stop here" final turn) in session `00a344e4`.
|
||
|
||
**What happened:** The agent called `update_plan_step` in this order:
|
||
1. `seq=5, status=done` — "Write back results"
|
||
2. `seq=4, status=done` — "Upgrade auth-outpost"
|
||
3. `seq=1, status=done` — "Upgrade nfs-export"
|
||
4. `seq=2, status=done` — "Upgrade gitea"
|
||
5. `seq=3, status=done` — "Upgrade dns"
|
||
|
||
The plan panel would have shown step 5 completing before step 4, then
|
||
steps 1-3 completing in reverse order. This is the "plan changed mid-air
|
||
and the agent forgot to keep its progress up to date" issue — the agent
|
||
rushed to close all remaining steps in the final turn, in no particular
|
||
order, without verifying each one was actually done.
|
||
|
||
**Root cause:** No ordering validation on `update_plan_step`. The agent can
|
||
mark any step as done in any order. At minimum, steps should complete in
|
||
seq order. The frontend should also handle out-of-order completions
|
||
gracefully (don't reorder the list just because completions arrived out of
|
||
sequence).
|
||
|
||
**Severity:** Friction — the progress view looks wrong, eroding trust.
|
||
|
||
### 14. The 1st turn had 92 tool calls — 30% were wasted on destroyed LXCs
|
||
|
||
**Where:** Turn 1 of session `00a344e4`.
|
||
|
||
**Breakdown:**
|
||
| Category | Count | Tool calls |
|
||
|---|---|---|
|
||
| Plan/task management | 12 | set_goal × 2, propose_plan × 2, update_plan_step × 8 |
|
||
| Research | 2 | list_lxcs × 1, search_knowledge × 1 |
|
||
| Legitimate audits | 50 | request_execution against reachable LXCs |
|
||
| **Wasted — destroyed LXCs** | **28** | request_execution × 14 (failed), get_entity × 14 (confirm destroyed) |
|
||
|
||
92 tool calls in one turn. The agent was efficient (it batched them), but 28
|
||
of them were entirely avoidable if `list_lxcs` had filtered out destroyed
|
||
entities or if the DB had up-to-date entity attributes from a prior session.
|
||
|
||
**Root cause:** Combination of #9 (no entity attributes written back) + #10
|
||
(no state filter on list_lxcs). Each problem compounds the other.
|
||
|
||
**Severity:** Friction — costs latency, OpenRouter credits, and model context
|
||
window. A single fleet audit shouldn't need 92 tool calls.
|
||
|
||
---
|
||
|
||
## Findings — Part C: The "wins" (things that worked well)
|
||
|
||
Despite the issues above, the session had several things working correctly
|
||
that should be preserved:
|
||
|
||
- **Chat assent → `run` upgrade path works.** Once the agent learned to use
|
||
`run` instead of `request_execution`, upgrades auto-ran under the assent
|
||
window without re-approval. This is the correct pattern.
|
||
- **`pct exec` fallback discovered autonomously.** The agent realized 5 LXCs
|
||
fail SSH but `pct exec` from their Proxmox host works. No operator input
|
||
needed — the agent investigated and found the alternative.
|
||
- **The agent self-corrected `request_execution → run`.** When `request_execution`
|
||
calls got stuck at `pending_approval` despite chat assent, the agent
|
||
diagnosed the problem ("assent window only covers `run` commands") and
|
||
switched tools. Good resilience.
|
||
- **Knowledge content quality was high.** All 4 knowledge entries had structured
|
||
tables, relevant caveats, and actionable instructions. Content-wise, the
|
||
knowledge loop is producing good output — just not linking it to the
|
||
right entities.
|
||
- **The agent documented a process lesson as a reusable document** ("How to run
|
||
fleet apt upgrades"). This is exactly the kind of knowledge that prevents
|
||
future sessions from repeating mistakes. The format (runbook with examples +
|
||
verification commands) is correct.
|
||
- **Multi-step plan execution worked end-to-end.** 3 plans, 21 `update_plan_step`
|
||
calls, all steps eventually completed. The plan mechanism itself is solid —
|
||
the issues are in step descriptions and writeback completeness.
|
||
|
||
---
|
||
|
||
## Improvement plan — with concrete fix descriptions
|
||
|
||
### Phase 1: Crash recovery (addresses #1, #2, #3 — the "session dies" class)
|
||
|
||
#### 1.1 — SSE auto-reconnect + fallback to polling
|
||
|
||
**Files:** `web/src/lib/stores/chat.ts`, `web/src/lib/api.ts`
|
||
|
||
**What:** When the SSE stream drops (network blip, server restart), the frontend
|
||
should automatically reconnect instead of requiring a logout/login cycle.
|
||
|
||
**How:**
|
||
1. Add a `reconnectCount` state to `sendMessage()` in chat.ts. When the stream
|
||
errors or completes without a `done` event (chat.ts:316-330), set a
|
||
`disconnected = true` flag on the current session instead of calling
|
||
`streaming.set(false)`.
|
||
2. When `disconnected` is true, start a backoff reconnect loop: wait 1s, 2s,
|
||
4s (capped at 8s), then re-post to `/chat` with the *same* `session_id`
|
||
and an empty message string + `resume: true` flag. The backend's `handleChat`
|
||
routes this into `resumeSession` with a system note like `"[System: the
|
||
stream reconnected — continue from where you left off.]"`. On reconnect
|
||
success, stop the loop and clear `disconnected`.
|
||
3. In `streamChat()` (`api.ts:105-156`), the `catch` and `finally` blocks
|
||
call the same callbacks but need to distinguish "aborted by user"
|
||
(`AbortError`) from "connection dropped": don't call `onDone()` on
|
||
network errors — let the new `onDisconnect` callback handle it instead.
|
||
Add a third callback param: `onDisconnect: (reason: string) => void`.
|
||
4. Maximum 3 reconnect attempts. After exhausting retries, fall back to
|
||
polling: set `streaming` to false, clear `disconnected`, and call
|
||
`startPolling(sessionId)` (the existing poll loop in chat.ts:152-170).
|
||
The task continues server-side — the poller will catch whatever happened
|
||
during the outage.
|
||
|
||
#### 1.2 — Connection-lost banner with retry button
|
||
|
||
**Files:** `web/src/pages/Chat.svelte`
|
||
|
||
**What:** A persistent banner above the input area that shows when the SSE
|
||
connection is lost, with a "Reconnect" button and a countdown timer for the
|
||
next auto-retry.
|
||
|
||
**How:**
|
||
1. Add a `connectionState` store to `chat.ts`: `'connected' | 'disconnected' | 'reconnecting'`.
|
||
Expose it via a `connection` export.
|
||
2. In `Chat.svelte`, add a banner between the message area and the input bar
|
||
(around line 170, replacing the current `{#if $error}` block):
|
||
```svelte
|
||
{#if $connection === 'disconnected'}
|
||
<div class="connection-banner">Agent connection lost. <Button size="sm" onclick={reconnect}>Reconnect</Button></div>
|
||
{:else if $connection === 'reconnecting'}
|
||
<div class="connection-banner muted">Reconnecting in {countdown}s… <Button size="sm" variant="ghost" onclick={cancelReconnect}>Cancel</Button></div>
|
||
{/if}
|
||
```
|
||
3. `reconnect()` triggers an immediate reconnect attempt (reset the backoff
|
||
timer, call `sendMessage("", { resume: true })`).
|
||
4. The current `$error` banner (Chat.svelte:164-170) becomes the non-connection
|
||
error path — displayed for LLM errors, tool errors, etc. This is a sibling
|
||
banner, not a replacement.
|
||
|
||
#### 1.3 — On stream drop, immediately poll for messages
|
||
|
||
**Files:** `web/src/lib/stores/chat.ts`
|
||
|
||
**What:** When the SSE stream drops, don't wait for the user to log out/in.
|
||
Start the poller immediately so the chat shows whatever the agent did
|
||
server-side during the outage.
|
||
|
||
**How:**
|
||
1. In `sendMessage()`'s error/complete callbacks (chat.ts:316-330), after
|
||
setting `disconnected` (from 1.1), call `startPolling(sessionId)` immediately
|
||
instead of waiting for the reconnect loop or manual reload.
|
||
2. `startPolling` already bails when `$streaming` is true (chat.ts:156).
|
||
After the stream drops, `streaming` stays true because of `disconnected`.
|
||
Change the gate: allow polling when `disconnected` is true even if
|
||
`streaming` is true. The polled messages are from the persisted DB, so
|
||
they won't conflict with the dead SSE stream.
|
||
3. When reconnection succeeds (the SSE stream is live again), stop polling
|
||
to avoid double-rendering.
|
||
|
||
#### 1.4 — Empty-response: retry 3 times instead of 2
|
||
|
||
**Files:** `cmd/nomos/agent.go:33`
|
||
|
||
**What:** `maxLLMRetries = 1` means the inner loop (agent.go:331-375) retries
|
||
once, and `resumeSession` (continue.go:216) retries the whole call once — 2
|
||
total chances. DeepSeek sometimes needs 3.
|
||
|
||
**How:**
|
||
1. Change `const maxLLMRetries = 1` to `const maxLLMRetries = 2` at
|
||
`agent.go:33`. This gives 3 total attempts in the inner loop.
|
||
2. In `resumeSession` (continue.go:216), change `for attempt := 0; attempt < 2`
|
||
to `for attempt := 0; attempt < 3` for 3 outer-loop attempts.
|
||
3. On each retry inside `resumeSession`, inject a stronger system note:
|
||
`"[System: your previous response was empty or invalid — the operator is
|
||
waiting. Produce a real response this time.]"` instead of just re-running
|
||
the same prompt.
|
||
|
||
#### 1.5 — Empty-response: don't auto-complete the task on failure
|
||
|
||
**Files:** `cmd/nomos/continue.go:252-258`
|
||
|
||
**What:** When `resumeSession` exhausts retries, it calls `completeTask` with
|
||
`outcome='failure'` (continue.go:256). This marks the session as terminal,
|
||
so the next user message in chat can't resume it — the user has to know to
|
||
start a new task.
|
||
|
||
**How:**
|
||
1. Remove the `completeTask` call at continue.go:256. Instead, persist a
|
||
`"resume_failed"` system note as a regular assistant message in the
|
||
transcript so the user sees what happened:
|
||
```go
|
||
resumeFailedNote := fmt.Sprintf("[System: auto-resume failed after retrying: %s. The task is paused — send another message to try again.]", errText)
|
||
```
|
||
2. Leave `agent_sessions.status` at `'executing'` (or whatever it was before
|
||
the failed resume). The user's next chat message will re-enter `handleChat`,
|
||
which replays the full history and picks up from the last tool calls.
|
||
3. Only auto-close if the task was already in a genuinely terminal state
|
||
(check `status` before deciding).
|
||
|
||
#### 1.6 — Persistent, dismissible error card in chat
|
||
|
||
**Files:** `web/src/lib/stores/chat.ts` (new store: `chatErrors`), `web/src/pages/Chat.svelte`
|
||
|
||
**What:** The current `$error` store (chat.ts:72) is a single string that
|
||
disappears on the next `sendMessage()`. For empty-response errors, the user
|
||
needs a card that stays visible until dismissed, explaining what went wrong
|
||
and suggesting a recovery action.
|
||
|
||
**How:**
|
||
1. Add `chatErrors` as a writable store of error objects:
|
||
```ts
|
||
interface ChatError { id: string; message: string; dismissible: boolean; action?: string }
|
||
```
|
||
2. In `sendMessage()`'s error handler, push a `ChatError` instead of setting
|
||
`error.set(err)`. The `action` field suggests recovery (e.g. "type 'status'
|
||
to check what happened" or "click Retry to resend").
|
||
3. In `Chat.svelte`, render `chatErrors` as dismissible cards above the input
|
||
bar (replacing or alongside the current `$error` banner):
|
||
```svelte
|
||
{#each $chatErrors as err (err.id)}
|
||
<div class="error-card">
|
||
<span>{err.message}</span>
|
||
{#if err.action}<Button size="xs" variant="outline" onclick={() => dismissError(err.id)}>{err.action}</Button>{/if}
|
||
<button onclick={() => dismissError(err.id)}>×</button>
|
||
</div>
|
||
{/each}
|
||
```
|
||
4. Clear `chatErrors` on `newChat()` but NOT on `sendMessage()` — errors
|
||
survive across messages until explicitly dismissed.
|
||
|
||
---
|
||
|
||
### Phase 2: Visibility (addresses #4, #5, #6 — "what is the agent doing?")
|
||
|
||
#### 2.1 — Custom renderer for `get_execution_status`
|
||
|
||
**Files:** New: `web/src/lib/tool-renderers/ExecutionStatus.svelte`, `web/src/lib/tool-renderers/index.ts`
|
||
|
||
**What:** `get_execution_status` results are large JSON blobs rendered in
|
||
ToolCallGroup's generic `<pre>` blocks. A custom renderer shows execution
|
||
state inline with a status badge, live polling, and collapsible output.
|
||
|
||
**How:**
|
||
1. Create `web/src/lib/tool-renderers/ExecutionStatus.svelte`:
|
||
- Props: `tool: ToolCallResult`
|
||
- Extract from `tool.result`: `execution_id`, `status`, `target`, `action`,
|
||
`result`, `error`, `duration_ms`
|
||
- Render a compact card:
|
||
```
|
||
[spinner/check/X] execution 019f5f.. | apt_upgrade on lxc:nfs-export | completed (12.3s)
|
||
▶ output: 0 upgraded, 0 newly installed...
|
||
```
|
||
- If status is `running` or `approved`, auto-poll `getExecution(id)` every
|
||
3 seconds (via `api.ts`) and update the card in place. Stop when terminal.
|
||
- Show elapsed wall time (live counter while running).
|
||
- Collapsible output section (default collapsed for completed, expanded
|
||
for failed with error text in red).
|
||
|
||
2. Register in `tool-renderers.ts`:
|
||
```ts
|
||
import ExecutionStatus from './tool-renderers/ExecutionStatus.svelte'
|
||
registerToolRenderer({
|
||
match: (t) => t.name === 'get_execution_status' && t.type === 'tool_result',
|
||
component: ExecutionStatus
|
||
})
|
||
```
|
||
|
||
3. In `ToolCallGroup.svelte`, the tool will now be matched by `getToolRenderer()`
|
||
and rendered inline by `Chat.svelte`'s `getInlineTools` — no changes needed
|
||
to the existing ToolCallGroup unless the card should also appear inside the
|
||
group (it should — add the custom card to ToolCallGroup's body too, or
|
||
simply ensure `getInlineTools` claims it and ToolCallGroup's `unmatched`
|
||
prop excludes it).
|
||
|
||
#### 2.2 — Extract approvals on `tool_result` events, not just `done`
|
||
|
||
**Files:** `web/src/lib/stores/chat.ts:243-277`
|
||
|
||
**What:** `extractApprovals()` only runs on the SSE `done` event. If the stream
|
||
drops before `done`, the approvals never appear in chat (they exist on the Ops
|
||
page but the chat shows nothing).
|
||
|
||
**How:**
|
||
1. In the `tool_result` branch of `sendMessage()` (chat.ts:258-277), after
|
||
updating the tool in `activeTools` and the message's `tools` array, run
|
||
`extractApprovals` on the full `tools` array of the current message and
|
||
set `msg.pendingApprovals` immediately:
|
||
```ts
|
||
// inside tool_result handler, after updating the tool:
|
||
messages.update((ms) => {
|
||
const last = ms[ms.length - 1]
|
||
if (last && last.role === 'assistant') {
|
||
last.tools = last.tools.map((t) => t.id === ev.data.id ? updated : t)
|
||
last.pendingApprovals = extractApprovals(last.tools) // <-- add this
|
||
}
|
||
return [...ms]
|
||
})
|
||
```
|
||
2. Keep the `done`-event extraction as a final sanity pass (it catches any
|
||
edge case where `tool_result` arrived before the tool was registered in
|
||
`activeTools`).
|
||
|
||
#### 2.3 — Approval card label: use `purpose` text
|
||
|
||
**Files:** `web/src/lib/stores/chat.ts:43`
|
||
|
||
**What:** `extractApprovals()` picks `t.args?.action ?? t.args?.purpose ?? t.name`.
|
||
For `run` tools, `t.name` is `"run"` — not informative. The `purpose` text is
|
||
a full sentence like "Upgrade nfs-export (21 packages)". Better to truncate it.
|
||
|
||
**How:**
|
||
1. Change the label logic in `extractApprovals`:
|
||
```ts
|
||
action: t.args?.purpose?.slice(0, 60) ?? t.args?.action ?? t.name ?? 'unknown',
|
||
```
|
||
`purpose` is always present and meaningful for `run` calls; falling back to
|
||
`action` (for `request_execution`) and then `name` (last resort). Truncate
|
||
to 60 chars to fit the card.
|
||
|
||
#### 2.4 — "Agent is working" indicator in chat header
|
||
|
||
**Files:** `web/src/pages/Chat.svelte`
|
||
|
||
**What:** The chat page has no indicator that the agent is doing autonomous work
|
||
(executions running, auto-continuation polling, plan steps advancing). The user
|
||
stares at a static chat and wonders if anything is happening.
|
||
|
||
**How:**
|
||
1. Add a `sessionActivity` store in `chat.ts`:
|
||
```ts
|
||
interface SessionActivity {
|
||
lastMessageAt: Date | null
|
||
executionsRunning: number
|
||
planStep: string | null // "3/6 — Upgrade nfs-export"
|
||
}
|
||
```
|
||
2. Update `sessionActivity` from:
|
||
- Poll fetches: when a new message arrives via `startPolling`, set `lastMessageAt`.
|
||
- SSE events: `tool_result` and `text_delta` events reset `lastMessageAt`.
|
||
- API calls to fetch active executions for the session (new minimal endpoint
|
||
or derived from the session digest).
|
||
3. In `Chat.svelte`, render a thin header bar above the messages area:
|
||
```svelte
|
||
{#if $currentSession}
|
||
<div class="activity-bar">
|
||
{#if $activity.executionsRunning > 0}
|
||
<LoaderCircleIcon class="animate-spin" /> {$activity.executionsRunning} running
|
||
{/if}
|
||
{#if $activity.planStep}
|
||
<span>{$activity.planStep}</span>
|
||
{/if}
|
||
<span class="ml-auto text-muted-foreground">Last active: {timeago($activity.lastMessageAt)}</span>
|
||
</div>
|
||
{/if}
|
||
```
|
||
|
||
#### 2.5 — SessionDigest: poll live when session is active
|
||
|
||
**Files:** `web/src/lib/components/SessionDigest.svelte`
|
||
|
||
**What:** SessionDigest fetches data once when the session loads. For an active
|
||
session with background work, it should refresh periodically.
|
||
|
||
**How:**
|
||
1. In `SessionDigest.svelte`, add a `$effect` that watches `$currentSession`
|
||
and the session's status:
|
||
```svelte
|
||
$effect(() => {
|
||
if (!sessionId || status === 'done' || status === 'failed') return
|
||
const timer = setInterval(() => fetchDigest(sessionId), 10000)
|
||
return () => clearInterval(timer)
|
||
})
|
||
```
|
||
2. Only re-render changed parts — Svelte's reactivity handles this since
|
||
`digest` is a reactive declaration. The fetch updates the store; the
|
||
template re-renders only the changed values (e.g. `executions.running` goes
|
||
from 4→3).
|
||
|
||
#### 2.6 — Live execution count in the task header
|
||
|
||
**Files:** `web/src/pages/Chat.svelte`, `web/src/lib/api.ts`
|
||
|
||
**What:** The task context panel shows plan progress, but the chat itself has
|
||
no execution counter. "3 executions running" in the chat header tells the user
|
||
at a glance that work is happening.
|
||
|
||
**How:**
|
||
1. Add a `fetchSessionExecutions(sessionId)` to `api.ts` that calls
|
||
`GET /api/v1/executions?session_id={id}&status=running,approved`.
|
||
(This endpoint may need adding — or use the existing executions endpoint
|
||
with a new `session_id` filter.)
|
||
2. Expose as a derived store in `chat.ts`; poll it alongside the message
|
||
poller.
|
||
3. Integrate with the activity bar from 2.4 (they share the same data).
|
||
|
||
---
|
||
|
||
### Phase 3: Cleanup (addresses #7, #8 — "leftover state")
|
||
|
||
#### 3.1 — `complete_task`: auto-cancel pending executions
|
||
|
||
**Files:** `cmd/nomos/store.go:480-509`
|
||
|
||
**What:** When the agent calls `complete_task`, any executions in
|
||
`pending_approval` or `approved` state for this session stay that way forever.
|
||
The approvals list on the Ops page shows stale items, and the operator
|
||
has to manually cancel them.
|
||
|
||
**How:**
|
||
1. In `completeTask`, BEFORE updating `agent_sessions`, run two cleanup queries:
|
||
```sql
|
||
-- 1. Cancel pending-approval executions linked to this session
|
||
UPDATE executions SET status = 'cancelled', result = '{"message": "task completed — auto-cancelled"}'::jsonb
|
||
WHERE entity_id IN (
|
||
SELECT execution_id FROM nomos_plan_executions WHERE session_id = $1
|
||
) AND status IN ('pending_approval', 'approved', 'queued');
|
||
|
||
-- 2. Mark them as continued so the worker won't try to auto-continue them
|
||
UPDATE nomos_plan_executions SET continued_at = now()
|
||
WHERE session_id = $1 AND continued_at IS NULL;
|
||
```
|
||
2. For each cancelled execution, emit an `execution.cancelled` event so the
|
||
live approvals list refreshes (same pattern as `observability.Event` calls
|
||
elsewhere in store.go).
|
||
3. The sqlcgen observability event from `completeTask` already fires
|
||
`task.status` — add `cancelled_count: N` to the event data so the frontend
|
||
can show "task completed (3 pending approvals auto-cancelled)."
|
||
|
||
#### 3.2 — `complete_task`: delete assent window + destructive window
|
||
|
||
**Files:** `cmd/nomos/store.go:480-509`
|
||
|
||
**What:** The `autonomy_settings` table holds the assent window key
|
||
(`nomos:assent_window:<agent>:<session>`) and destructive window key
|
||
(`nomos:destructive:<agent>:<target>:<session>`). On task completion,
|
||
these are stale and should be cleaned up.
|
||
|
||
**How:**
|
||
1. Add a cleanup query to `completeTask`:
|
||
```sql
|
||
DELETE FROM autonomy_settings
|
||
WHERE key = $1 OR key LIKE $2
|
||
```
|
||
Where `$1` = the session's assent window key and `$2` = the destructive
|
||
window pattern for this session.
|
||
2. Compute the keys: `assentWindowKey(a.agentID, sessionID)` and
|
||
`nomos:destructive:<agent>:*:<session>` pattern.
|
||
3. This is a single DELETE before the return — no transaction needed, fire
|
||
and forget (failure is logged, not blocking).
|
||
|
||
#### 3.3 — `propose_plan`: auto-replace pending steps with `replaced` status
|
||
|
||
**Files:** `cmd/nomos/store.go:367-423`
|
||
|
||
**What:** When the agent calls `propose_plan` again mid-flight, new steps are
|
||
appended after the current max seq (store.go:379-389). Pending steps from the
|
||
old plan (status = `'pending'`) remain in the list forever — they were never
|
||
started and never will be.
|
||
|
||
**How:**
|
||
1. In `proposePlan`, after the `DELETE FROM session_plan_steps WHERE session_id = $1`
|
||
for the fresh-start path (store.go:386), add ELSE logic for the append path:
|
||
```sql
|
||
-- Before appending new steps, mark any still-pending steps as 'replaced'
|
||
UPDATE session_plan_steps
|
||
SET status = 'replaced', finished_at = now()
|
||
WHERE session_id = $1 AND status = 'pending';
|
||
```
|
||
2. The `replaced` status is already a valid terminal state — the plan panel
|
||
should treat it the same as `skipped` (dimmed, no progress contribution).
|
||
3. Add `'replaced'` to the stamp switch in `updatePlanStep` (store.go:436-441)
|
||
so no one can accidentally un-replace a step.
|
||
|
||
#### 3.4 — Plan steps: `generation` column + frontend collapse
|
||
|
||
**Files:** `cmd/nomos/store.go` (schema migration + proposePlan), `web/src/lib/components/PlanPanel.svelte`
|
||
|
||
**What:** With 3.3, old steps get marked `replaced`, but the flat list still
|
||
shows every step ever created. The operator sees a confusing mix of old and
|
||
new plan steps. A `generation` column lets the frontend group and collapse.
|
||
|
||
**How:**
|
||
1. Migration: `ALTER TABLE session_plan_steps ADD COLUMN generation INTEGER NOT NULL DEFAULT 1;`
|
||
2. In `proposePlan`, resolve the new generation number:
|
||
```sql
|
||
SELECT COALESCE(MAX(generation), 0) + 1 FROM session_plan_steps WHERE session_id = $1
|
||
```
|
||
If this is a fresh-start (no steps started, delete + re-insert), reset to 1.
|
||
If appending, use next generation number (typically 2, 3, ...).
|
||
3. Each new step inserted gets this `generation` value.
|
||
4. In the `planStep` struct (store.go:567), add `Generation int \`json:"generation"\``.
|
||
5. In `getPlanSteps`, include `generation` in the SELECT.
|
||
6. In `PlanPanel.svelte` (or wherever plan steps are rendered), group by
|
||
`generation`. Show the current generation (highest number) expanded; collapse
|
||
older ones with a label like "Plan v1 (replaced)" and a count of steps.
|
||
|
||
#### 3.5 — Sessions list: show open-approval count per session
|
||
|
||
**Files:** `cmd/nomos/store.go` (new query), `cmd/nomos/main.go` (API response),
|
||
`web/src/lib/components/SessionRail.svelte`
|
||
|
||
**What:** The sessions list shows title, status, time. Adding "2 approvals
|
||
pending" tells the operator at a glance which sessions have open actions.
|
||
|
||
**How:**
|
||
1. Add a `pendingApprovalCount` method on store:
|
||
```go
|
||
func (s *store) pendingApprovalCounts(ctx context.Context, sessionIDs []string) map[string]int {
|
||
// SELECT l.session_id, COUNT(*) FROM nomos_plan_executions l
|
||
// JOIN executions e ON e.entity_id = l.execution_id
|
||
// WHERE l.session_id = ANY($1) AND e.status = 'pending_approval'
|
||
// GROUP BY l.session_id
|
||
}
|
||
```
|
||
2. In `handleSessionsList` (main.go:276), call this for the returned sessions
|
||
and add `pending_approvals` to each session JSON object.
|
||
3. In `SessionRail.svelte`, show a yellow badge next to sessions with
|
||
`pending_approvals > 0`. Clicking navigates to that session AND opens
|
||
the approvals section.
|
||
|
||
---
|
||
|
||
### Phase 4: Continuation hardening (addresses #3 — "agent stops")
|
||
|
||
#### 4.1 — When assent window is missing, inject a visible note
|
||
|
||
**Files:** `cmd/nomos/continue.go:145-151`
|
||
|
||
**What:** When `assentWindowActive` returns false, the execution is marked
|
||
`continued` without resuming. The user sees nothing. Instead, persist a
|
||
visible note in the transcript explaining WHY the agent didn't continue.
|
||
|
||
**How:**
|
||
1. In `processContinuations` (continue.go:145-151), before `markContinued`, insert
|
||
a system note as an assistant message in the transcript:
|
||
```go
|
||
note := fmt.Sprintf("[System: execution %s finished, but the assent window for this session is not active (may have expired). The agent will not auto-continue. Reply 'continue' or re-approve the plan to resume.]", p.ExecID)
|
||
a.store.saveMessage(context.Background(), p.SessionID, "assistant", jsonNote)
|
||
```
|
||
2. This makes the auto-continue failure visible to the operator in the chat
|
||
history — they can see WHY the agent stopped and what to do about it.
|
||
|
||
#### 4.2 — Re-open expired assent window for running executions
|
||
|
||
**Files:** `cmd/nomos/continue.go:145-151`
|
||
|
||
**What:** The plan was approved. The execution ran. The window expired during
|
||
execution. Penalizing timing is wrong — the agent should still get the result
|
||
and continue.
|
||
|
||
**How:**
|
||
1. In `processContinuations`, when `assentWindowActive` is false, don't just
|
||
mark it — check if the session's goal is set and the task status is
|
||
`'executing'` (meaning a plan was approved and is being worked):
|
||
```go
|
||
if !a.store.assentWindowActive(ctx, a.agentID, p.SessionID) {
|
||
sesh, _ := a.store.getSession(ctx, p.SessionID)
|
||
if sesh.Goal != "" && sesh.Status == "executing" {
|
||
// Plan was approved, work is in progress — re-open the window
|
||
a.openAssentWindow(ctx, p.SessionID)
|
||
// THEN continue (don't markContinued — execute the normal path)
|
||
} else {
|
||
// No plan, no goal — genuinely shouldn't continue (4.1's note)
|
||
a.store.markContinued(ctx, p.ExecID)
|
||
continue
|
||
}
|
||
}
|
||
```
|
||
2. The `a.store.markContinued(...)` line at continue.go:152 must move inside
|
||
the `if !active` block. Currently it fires unconditionally before the
|
||
`safego.Go` — that's a bug: it marks executions as continued even when
|
||
the continuation IS dispatched, which is safe but unnecessary. With this
|
||
fix, it becomes important that `markContinued` ONLY fires when we're
|
||
NOT continuing.
|
||
|
||
#### 4.3 — Reduce idle sweep interval to 2 min (first pass)
|
||
|
||
**Files:** `cmd/nomos/continue.go:55`
|
||
|
||
**What:** The idle sweep fires every 5 minutes. A stopped agent feels dead
|
||
long before 5 minutes pass.
|
||
|
||
**How:**
|
||
1. Change `time.NewTicker(5 * time.Minute)` to `time.NewTicker(2 * time.Minute)`
|
||
at continue.go:55.
|
||
2. Keep the two-pass logic (nudge → auto-close) but at 2-minute spacing
|
||
instead of 5-minute. Net: an agent that stops responding due to a stuck
|
||
continuation is nudged at 2 min and auto-closed at 4 min instead of 5
|
||
and 10.
|
||
|
||
#### 4.4 — "Continue" button in chat
|
||
|
||
**Files:** `web/src/lib/stores/chat.ts`, `web/src/pages/Chat.svelte`,
|
||
`cmd/nomos/main.go`
|
||
|
||
**What:** When the agent seems stuck, the operator should be able to click
|
||
"Continue" instead of typing "continue" or "status?".
|
||
|
||
**How:**
|
||
1. Add a `resumeSession` API call to `api.ts`:
|
||
```ts
|
||
export function resumeSession(sessionId: string): Promise<boolean> {
|
||
return fetchWithAuth(`${BASE}/sessions/${sessionId}/resume`, { method: 'POST' }).then(r => r.ok)
|
||
}
|
||
```
|
||
2. Add a handler in `main.go`:
|
||
```go
|
||
case "/resume":
|
||
note := "[System: the operator wants you to continue. Pick up where you left off — execute the next step of the plan, diagnose and fix any failures, or report progress if everything is done.]"
|
||
safego.Go("nomos:resume-session", func() { a.resumeSession(context.Background(), sessionID, note) })
|
||
w.WriteHeader(202)
|
||
```
|
||
3. In `Chat.svelte`, add a "Continue" button in the activity bar from 2.4.
|
||
Show it when:
|
||
- The session is active (`status === 'executing'` or `'awaiting_input'`).
|
||
- The most recent message is from the agent (not the user).
|
||
- No stream is active.
|
||
- More than 30 seconds have passed since the last message (agent might
|
||
be stuck).
|
||
On click, calls `resumeSession($currentSession)`, starts polling, and
|
||
shows a brief "Agent resumed" toast.
|
||
4. Debounce: disable the button for 30 seconds after clicking to prevent
|
||
spam.
|
||
|
||
---
|
||
|
||
### Phase 5: Knowledge loop (addresses #9, #10, #11, #12, #13, #14 — "the graph keeps drifting")
|
||
|
||
#### 5.1 — `list_lxcs`: add `state` filter parameter
|
||
|
||
**Files:** `internal/mcp/server.go:664-675`
|
||
|
||
**What:** `list_lxcs` returns all 33 LXCs including 13 destroyed ones. The SQL
|
||
for `list_entities` already supports `state` filtering — port the same pattern.
|
||
|
||
**How:**
|
||
1. Add an optional `state` parameter to the tool's `InputSchema`:
|
||
```go
|
||
InputSchema: objSchema(
|
||
prop{"state", "string", "Optional: filter by entity state (active, destroyed, …)"},
|
||
),
|
||
```
|
||
2. Change the SQL to use the parameter:
|
||
```sql
|
||
WHERE e.type = 'lxc'
|
||
AND ($1::text IS NULL OR e.state = $1)
|
||
ORDER BY (e.attributes->>'pve_id')::int
|
||
```
|
||
3. Default (no filter) returns all LXCs — backward compatible. When `state`
|
||
is passed, only matching entities are returned.
|
||
4. Update SOUL.md to recommend: `list_lxcs(state="active")` for fleet audits,
|
||
`list_lxcs(state="destroyed")` for cleanup lists. This alone eliminates
|
||
~30% of tool calls from the 92-call first turn.
|
||
|
||
#### 5.2 — `upsert_knowledge`: accept `about` as an array of entity slugs
|
||
|
||
**Files:** `internal/mcp/server.go` (the upsert_knowledge handler)
|
||
|
||
**What:** `upsert_knowledge` accepts a single `about` slug (e.g. `cluster:homelab`).
|
||
The agent can't link one knowledge entry to multiple entities without calling
|
||
the tool once per entity. The upgrade results session should have linked to
|
||
each upgraded LXC individually.
|
||
|
||
**How:**
|
||
1. Add `about` as an **array** in the tool schema (accept both single string
|
||
and array for backward compat):
|
||
```go
|
||
InputSchema: objSchema(
|
||
prop{"title", "string", "…"},
|
||
prop{"content", "string", "…"},
|
||
prop{"kind", "string", "…"},
|
||
prop{"tags", "string", "…"},
|
||
prop{"about", "array", "Entity slugs this knowledge is about (e.g. ['lxc:nfs-export', 'lxc:gitea'])"},
|
||
),
|
||
```
|
||
2. In the handler, normalize single string → single-element array. Create
|
||
`documented` relationships for each slug: `knowledge_entity → about → entity`.
|
||
3. This lets the agent record: "Low-risk tier upgrades" about
|
||
`[lxc:nfs-export, lxc:gitea, lxc:dns, lxc:auth-outpost]` in one call.
|
||
|
||
#### 5.3 — SOUL.md: make the writeback instruction unmissable
|
||
|
||
**Files:** `nomos/SOUL.md` (lines 86-103)
|
||
|
||
**What:** The instructions exist but the agent skipped them. The plan step
|
||
descriptions the agent wrote for itself were too vague.
|
||
|
||
**How:**
|
||
1. Add a **bold, standalone section** near the top of SOUL.md (before the
|
||
"Every chat is a task" section):
|
||
```
|
||
## ⚠️ AFTER EVERY TASK: WRITE BACK OR LOSE IT
|
||
|
||
What you discovered but didn't write back is **lost** — the next session
|
||
starts from scratch. Before calling complete_task, you MUST:
|
||
|
||
1. `update_entity_attributes` — ANY concrete fact (IP, version, host, port,
|
||
state) for ANY entity you learned about. Every LXC you queried, every
|
||
target you ran against.
|
||
2. `create_relationship` — ANY edge you discovered (hosts, depends-on,
|
||
provides). Every "X runs on Y" fact.
|
||
3. `upsert_knowledge` — narrative: what you did, what broke, the fix.
|
||
Link to ALL affected entities via `about`.
|
||
|
||
**The plan's LAST step must list these by name.** Not "record findings" —
|
||
"1. update_entity_attributes for each audited LXC, 2. create_relationship
|
||
for any discovered host/container edges, 3. upsert_knowledge." Future you
|
||
depends on this.
|
||
```
|
||
2. Update the `propose_plan` tool description (tasks.go:42-45) to add
|
||
stronger language: "Your LAST step MUST have a detail line that literally
|
||
says '1. update_entity_attributes for X, 2. create_relationship for Y,
|
||
3. upsert_knowledge about Z' so you don't skip entity writeback."
|
||
3. This is a cheap, high-impact change — it doesn't require code, just
|
||
clearer instructions.
|
||
|
||
#### 5.4 — `propose_plan`: validate that the last step mentions entity writeback tools
|
||
|
||
**Files:** `cmd/nomos/tasks.go:196-205` (propose_plan handler)
|
||
|
||
**What:** The agent can propose a plan whose last step is "Record findings" and
|
||
no one checks whether it includes entity attribute updates. Add a validation
|
||
nudge.
|
||
|
||
**How:**
|
||
1. In `propose_plan`, after persisting steps, check if the final step's
|
||
`title` or `detail` contains `"update_entity_attributes"` or
|
||
`"create_relationship"`:
|
||
```go
|
||
lastStep := steps[len(steps)-1]
|
||
hasWriteback := strings.Contains(lastStep.Title+lastStep.Detail, "update_entity_attributes") ||
|
||
strings.Contains(lastStep.Title+lastStep.Detail, "create_relationship")
|
||
if !hasWriteback {
|
||
return fmt.Sprintf("Plan set: %d step(s). ⚠️ The final step doesn't mention update_entity_attributes or create_relationship. Without those, any facts you discovered about entities will be lost. Consider adding them to the last step.", len(persisted)), true
|
||
}
|
||
```
|
||
2. This doesn't *enforce* — it's a nudge in the tool result. The agent sees it
|
||
and can self-correct in the same turn (call `update_plan_step` to fix the
|
||
last step's detail, or call `propose_plan` again with a corrected plan).
|
||
|
||
#### 5.5 — `complete_task`: validate that entity attributes were written back
|
||
|
||
**Files:** `cmd/nomos/store.go:480-509`
|
||
|
||
**What:** If the agent calls `complete_task` without having called
|
||
`update_entity_attributes` or `create_relationship` in this session, the
|
||
completion message should warn about it.
|
||
|
||
**How:**
|
||
1. Add a check in `completeTask`: query `audit_log` for this session's use of
|
||
`update_entity_attributes` and `create_relationship` within the last N
|
||
minutes (the session's active window):
|
||
```sql
|
||
SELECT EXISTS(
|
||
SELECT 1 FROM audit_log
|
||
WHERE details->>'session_id' = $1
|
||
AND action IN ('update_entity_attributes', 'create_relationship')
|
||
AND created_at > now() - interval '60 minutes'
|
||
) AS has_writeback
|
||
```
|
||
2. If no writeback was done, return `"Task marked success: … ⚠️ No entity
|
||
attributes or relationships were updated in this session. Call
|
||
update_entity_attributes and create_relationship before completing
|
||
to keep the graph current."`.
|
||
3. The existing `complete_task` return value (tasks.go:264) includes this
|
||
text, which becomes the assistant's visible message — the operator and
|
||
the agent both see the warning.
|
||
|
||
#### 5.6 — Plan step ordering: enforce seq-order on completion
|
||
|
||
**Files:** `cmd/nomos/store.go:431-473` (updatePlanStep)
|
||
|
||
**What:** The agent marked step 5 as done before step 4 (finding #13). The
|
||
frontend sees steps completing in nonsensical order.
|
||
|
||
**How:**
|
||
1. In `updatePlanStep`, when status is `done/failed/skipped/blocked`, check
|
||
that all lower seq numbers are also in a terminal state:
|
||
```sql
|
||
UPDATE session_plan_steps
|
||
SET status = $3, …
|
||
WHERE session_id = $1 AND seq = $2
|
||
AND NOT EXISTS (
|
||
SELECT 1 FROM session_plan_steps ps
|
||
WHERE ps.session_id = $1
|
||
AND ps.seq < $2
|
||
AND ps.status = 'pending'
|
||
)
|
||
RETURNING id, target_slug
|
||
```
|
||
If no rows are affected (because a prior step is still pending), return
|
||
an error: `"Cannot complete step %d — step %d is still pending."`
|
||
2. This prevents the out-of-order completion. The agent must mark steps in
|
||
order. If the design genuinely allows out-of-order (parallel steps), add
|
||
a `depends_on_seq` column — but for now, strict ordering is simpler and
|
||
correct for the agent's sequential plan style.
|
||
|
||
#### 5.7 — Audit log: persist tool results (not just args) for analysis
|
||
|
||
**Files:** `cmd/nomos/agent.go:446-461` (logActivity call)
|
||
|
||
**What:** The audit log records `tool`, `args`, `result`, `success` per tool
|
||
call. To audit a session like this one programmatically, we need to know
|
||
which entities were touched, what their state was before/after, and whether
|
||
entity attributes were updated.
|
||
|
||
**How:**
|
||
1. Add a `session_id` column to the `audit_log` entries written by
|
||
`logActivity` (agent.go:447 and 461). Currently the log has no session
|
||
linkage — you can't query "what did session X do?"
|
||
2. Add structured fields: `entity_slugs` (array of slugs found in args) and
|
||
`tool_name` (already present as `action`). This lets queries like:
|
||
```sql
|
||
SELECT COUNT(*) FROM audit_log
|
||
WHERE session_id = $1 AND action = 'update_entity_attributes'
|
||
```
|
||
which 5.5 needs.
|
||
3. This is a low-priority improvement that makes 5.5's check fast (a log
|
||
query vs. a full text scan of `result_json`).
|
||
|
||
---
|
||
|
||
### Database migration (updated — covers 3.4, 3.5, and 5.7)
|
||
|
||
```sql
|
||
-- Add generation tracking to plan steps
|
||
ALTER TABLE session_plan_steps ADD COLUMN IF NOT EXISTS generation INTEGER NOT NULL DEFAULT 1;
|
||
|
||
-- Add index for pending-approval lookup by session
|
||
CREATE INDEX IF NOT EXISTS idx_nomos_plan_executions_session
|
||
ON nomos_plan_executions (session_id) WHERE continued_at IS NULL;
|
||
|
||
-- Add session_id to audit_log for per-session analysis
|
||
ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS session_id UUID;
|
||
CREATE INDEX IF NOT EXISTS idx_audit_log_session ON audit_log (session_id);
|
||
```
|
||
|
||
---
|
||
|
||
## Implementation order
|
||
|
||
1. **Phase 1 first** — the session-ending failures (#1 empty responses, #2 disconnects, #3 agent stops) are stop-ship issues that ruin the user experience.
|
||
2. **Phase 5 second** — the knowledge loop (#9, #10, #11, #12, #13, #14) is the most impactful set of improvements per token spent. Making `list_lxcs` filterable (5.1) and updating SOUL.md (5.3) are 30-minute changes that eliminate 30% of wasted tool calls and close the writeback gap. The `about` array (5.2) and completion validation (5.5) make the knowledge graph actually learn from sessions.
|
||
3. **Phase 2 third** — visibility improvements. Custom renderers and activity indicators are polish compared to the knowledge loop actually working.
|
||
4. **Phase 3 fourth** — cleanup of leftover state. The `complete_task` cancellation (3.1, 3.2) directly addresses the 4 orphaned approvals.
|
||
5. **Phase 4 last** — continuation hardening builds on the other phases.
|
||
|
||
## What this plan does NOT address
|
||
|
||
- C1 (unauthenticated nomos gateway) — already in `plans/2026-07-11-nomos-agent-code-review.md`.
|
||
- `request_execution` enum retirement — already in `plans/2026-07-10-general-gated-execution.md` Layer 1.
|
||
- Token-aware history windowing — already in the code review plan (A2), deferred.
|
||
- Auto-act revival — already in the general-gated-execution plan, deferred.
|
||
- Full audit of the `list_lxcs` output format — the `__renderer: lxc_list` annotation works for the frontend but the JSON structure is still flat (no distinction between active/destroyed entities). A richer shape (grouped by state) is future work.
|
||
- Deduplication of `upsert_knowledge` entries with the same title — the tool currently creates a new entry each time. Same-title updates should upsert instead of duplicate (future work).
|
||
|
||
## Verification
|
||
|
||
- **1.1-1.3**: Kill the nomos process mid-turn, confirm the frontend shows a reconnect banner and resumes polling within 3 seconds.
|
||
- **1.4-1.6**: Send a prompt known to produce empty responses, confirm retries fire 3x and the error card stays visible until dismissed.
|
||
- **2.1**: Open a session with `get_execution_status` calls, confirm each renders as a status card with live polling, not raw JSON.
|
||
- **2.2**: Queue a run that requires approval, confirm the approval card appears immediately (not just on `done`).
|
||
- **3.1-3.2**: Complete a task that has pending approvals, confirm they're auto-cancelled and the assent window is cleaned up.
|
||
- **3.3-3.4**: Call `propose_plan` twice in one session, confirm old pending steps are marked `replaced` and the panel shows only the current plan.
|
||
- **4.1-4.2**: Run an execution that finishes after the assent window expires, confirm the agent still picks it up and resumes.
|
||
- **5.1**: Call `list_lxcs(state="active")` — confirm only active LXCs are returned, not destroyed ones. Re-run the apt audit scenario — confirm the first turn has ~64 tool calls instead of 92.
|
||
- **5.2**: Call `upsert_knowledge` with `about: ["lxc:nfs-export", "lxc:gitea"]` — confirm knowledge is linked to both entities. Query `get_entity_knowledge("lxc:nfs-export")` — confirm the entry appears.
|
||
- **5.3**: Send a new fleet audit request. Confirm the agent's proposed plan has a last step that explicitly lists `update_entity_attributes`, `create_relationship`, and `upsert_knowledge` by name.
|
||
- **5.4**: Propose a plan whose last step says "Record findings" without mentioning entity writeback. Confirm the tool result warns about it.
|
||
- **5.5**: Complete a task that never called `update_entity_attributes`. Confirm the completion message warns "No entity attributes or relationships were updated."
|
||
- **5.6**: Call `update_plan_step(seq=5, status="done")` while step 4 is `pending`. Confirm the call returns an error: "Cannot complete step 5 — step 4 is still pending."
|