oikos/plans/done/2026-07-11-concurrent-task-execution.md

2026-07-11 — Concurrent task execution: safety + throughput + frontend correctness

Status: Done — 2026-07-11. All three required fixes shipped and deployed: session-scoped assent/destructive windows (commit 9ef1ba3), the frontend stream-corruption guard + per-session controllers (9131559, 6a8fb43), and the per-session MCP client pool (a4ea542). Fix 4 (concurrency/cost cap) remains explicitly deferred pending real usage data, per this doc's own recommendation.

Goal

Multiple tasks already run "at the same time" at the HTTP/goroutine level — nothing in nomos serializes whole turns. But tracing the actual code (not assuming) surfaces three real gaps that make concurrent tasks unsafe or broken today, in decreasing severity: a cross-task authorization bleed, a throughput bottleneck that makes concurrency mostly illusory, and a frontend state-corruption bug. This plan fixes all three.

Findings (grounded)

1. CRITICAL — the assent window is scoped to the agent, not the task

store.go:816, agent.go:129-143: openAssentWindow/assentWindowActive key on "assent_window.agent:" + a.agentID.String() — there is exactly one agent:nomos entity, so this key is global across every task. It's checked at three MCP call sites (internal/mcp/server.go:454, :488, :1363) purely as "does the agent currently have an open window," with no way to know which task's tool call is asking.

Concrete failure: operator approves Task A's plan → 30-minute window opens → operator starts Task B while that window is still open → Task B's run/request_execution config-mutation calls also auto-execute, because the check has no session dimension. The operator never approved Task B's plan.

store.go:313-345 destructiveWindowKey/ openDestructiveWindow/destructiveWindowActive have the same shape (keyed agent:<id>.target:<slug>, no session) — narrower blast radius (needs a second task hitting the same target within 15 minutes of an explicit typed confirmation elsewhere) but the same class of bug.

2. Tool calls across ALL tasks funnel through one mutex — concurrency is mostly illusory

main.go:45: nomos creates exactly one *mcpClient at startup, shared by every handleChat goroutine. Its mu sync.Mutex (main.go:419) is held for the full duration of each doRequest round-trip. run's MCP handler executes the SSH command synchronously inside that round-trip and is capped at up to 10 minutes. So while Task A is mid-run, every other task's tool calls — even a trivial get_entity — queue behind that single mutex until it returns. Tasks can think (LLM calls) in parallel, but cannot act in parallel; one slow task stalls all others' progress.

The MCP server side has no session-scoped in-memory state to protect — newServer(pool, agentID) returns one shared *mcp.Server instance whose tool handlers close only over pool (safe for concurrent use — pgxpool is a connection pool) and agentID (internal/mcp/server.go:51-74). The mutex exists purely because nomos's client reuses one stateful transport session, not because the server needs it. This is fixable without touching the server.

3. Frontend: the chat store is a global singleton — switching tasks mid-stream corrupts the view

chat.ts:160-269 sendMessage's SSE callback mutates messages/currentSession by reaching for ms[ms.length - 1] — i.e. it assumes the array it's mutating still belongs to the task it was opened for. Nothing in the callback checks that. chat.ts:111-117 loadSessionMessages (fired when you click a different task in the sidebar or the board) does not cancel or otherwise account for a still-open stream from the task you're leaving — it just calls messages.set(...) and currentSession.set(sessionId).

Concrete failure: start Task A, while it's still streaming click into Task B from the Tasks board → messages/currentSession now reflect Task B → Task A's still-open SSE stream delivers its next tool_use/text_delta → the callback appends it onto what is now Task B's last message, and on done calls currentSession.set(taskAId), flipping the app back to Task A underneath the operator. This is a real bug independent of anything else in this plan — it's why "switch away from a running task to start another" currently looks broken even though the backend handles it fine.

(By contrast, workspace.ts's live events are already correctly session-scoped — applyEvent checks ev.correlation_id !== sid before doing anything — because that mechanism was built for this from phase 6. The bug is confined to the older, per-turn chat.ts streaming path.)

Already fine, no change needed

• DB access: pgxpool.Pool is a connection pool; concurrent queries from multiple task goroutines are its normal use case.
• Auto-continuation worker (continue.go): already scoped per session (pendingContinuation.SessionID) — processes its poll batch sequentially (5/tick) but never mixes state across sessions. Sequential processing is a throughput nit, not a correctness bug; not in scope here.
• Task board (Tasks.svelte): event-driven refresh already handles any number of concurrently-changing tasks correctly — it re-lists, it doesn't hold per-task live state.

Design

Fix 1 — session-scope the assent and destructive windows

Thread sessionID through to the MCP call sites. nomos already knows the session id when it calls a tool (agent.go:363); the MCP wire protocol doesn't restrict tool-call args to the declared schema (argsMap just unmarshals whatever JSON object arrives), so nomos can inject an internal _session_id into the args it sends over the wire — invisible to the model (never in the tool's InputSchema, so it never appears in what the LLM sees or is asked to supply) but readable server-side.

• agent.go: build a wire-args copy with _session_id added just before a.client.callTool(...) (leave the args used for history/logging unmodified — the model's own tool-call record shouldn't show an internal field it never set).
• internal/mcp/server.go: assentWindowActive/openAssentWindow/ destructiveWindow* gain a sessionID parameter; the key becomes assent_window.agent:<id>.session:<sessionID> (and similarly for the destructive window). Every call site (run, request_execution, the apt_upgrade/pct_create sub-cases) reads _session_id from argsMap and passes it through.
• agent.go openAssentWindow gains the same sessionID param, called from its two existing call sites (agent.go:253, :269), which are already inside chatWith and have sessionID in scope.
• Fallback: if _session_id is missing (defensive — shouldn't happen since nomos always sets it), treat as "no window" (fail closed, require approval) rather than falling back to the old agent-wide key.

Fix 2 — per-session MCP client (remove the throughput bottleneck)

Replace the single global *mcpClient with a small map of clients keyed by session id, created lazily on first tool call for that session and evicted after a period of inactivity (e.g. 10 minutes past the session's last activity — long enough to outlive a slow run, short enough not to leak connections for abandoned tasks). Guard the map itself with a mutex (cheap — only held for map lookup/insert, not for the duration of a call); each individual client keeps its own mu scoped to its own session's calls, so Task A's slow run only serializes Task A's own tool calls (which are already inherently sequential within one turn — the agent loop calls tools one at a time) and never blocks Task B.

• New mcpClientPool type in cmd/nomos: get(sessionID) *mcpClient (creates+initializes on miss), sweep() (evicts idle clients, called on a ticker alongside the existing continuation-worker ticker). "ephemeral"/"" session ids (no persisted session) get their own dedicated client, not pooled per-request, to avoid a connection-per-message churn for the no-DB-store path.
• agent holds the pool instead of one client; handleQuery (the structured /query endpoint, main.go:330) picks a short-lived or dedicated client the same way.
• No server-side change needed (per finding 2's analysis — the server has no per-connection state to protect).

Fix 3 — frontend: don't let a background stream corrupt the active view

Minimal, contained fix (not a rearchitecture): capture the session id a sendMessage stream belongs to, and have its callback check that currentSession still matches before mutating messages/streaming. If the operator has navigated away, the stream's events are silently dropped from the UI (the task keeps running server-side regardless — the events are also flowing on the global stream, and if the operator navigates back, loadSessionMessages's poll + REST hydration picks up whatever landed while they were away, same as it already does for auto-continuation).

• chat.ts sendMessage: capture const streamSessionID = ... once the 'session' event assigns it; every subsequent branch of the callback (tool_use, tool_result, text_delta, text, done, error) first checks get(currentSession) === streamSessionID (or the pre-assignment optimistic session) before touching messages.
• loadSessionMessages: no change needed once the above guard exists — it already correctly sets messages/currentSession for the task being opened; the guard just stops the other task's stream from clobbering it afterward.
• Out of scope for this pass: a genuine multi-pane "watch two tasks stream live side by side" UI. Not needed for correctness — the Tasks board already shows live status for every task via workspace.ts's correctly-scoped events; only the single-focus Chat transcript view needs this guard.

Fix 4 (optional) — a concurrency/cost guardrail

Nothing currently stops an operator from starting many tasks in a tight loop, each spending real LLM API budget in parallel. Consider a simple semaphore in nomos (NOMOS_MAX_CONCURRENT_TASKS, default e.g. 5) that handleChat acquires before starting a turn and releases on completion; over the cap, queue or reject with a clear "N tasks already running, try again shortly" rather than letting an unbounded burst hit OpenRouter. This is an operational safeguard, not a correctness fix — flagged as optional/lower priority.

Implementation order

1. Fix 1 (assent/destructive window session-scoping) — the only one that's a genuine safety bug (auto-running unapproved actions in another task); ship first regardless of anything else.
2. Fix 3 (frontend stream guard) — small, contained, fixes a visibly broken UX (switching tasks looks corrupted) independent of Fix 2.
3. Fix 2 (per-session MCP client pool) — the throughput fix; more moving parts (lifecycle/eviction), ship after the safety fix lands and is verified, since both touch the same call sites (agent.go tool dispatch).
4. Fix 4 (concurrency cap) — optional, only if real usage shows a need.

Verification

• Fix 1: approve Task A's plan (open its window); concurrently start Task B and have it attempt a config-mutation run command without approving Task B's plan — confirm Task B's action is queued for approval (not auto-run), while Task A's own subsequent steps keep auto-running. SELECT key FROM autonomy_settings WHERE key LIKE 'assent_window%' should show session-scoped keys.
• Fix 2: start Task A with a run step that sleeps ~60s; concurrently start Task B with a trivial get_entity call; confirm Task B's tool result returns immediately rather than waiting on Task A. Confirm the client map evicts idle entries (sweep() logged, connection count doesn't grow unbounded across many sequential tasks).
• Fix 3: start Task A, before it finishes click into Task B on the board, confirm Task B's transcript stays correct (no Task-A tool calls appended) and currentSession doesn't flip back to Task A when its stream eventually completes in the background. Navigate back to Task A afterward and confirm its full transcript (including what happened while unwatched) loads correctly via REST.

Open questions

• Idle eviction window for Fix 2: 10 minutes was a guess balancing "outlive a slow run" against "don't leak connections." Worth checking actual run durations in production (executions.duration_ms) before picking a final number.
• Fix 4's cap and behavior on overflow: queue vs. reject vs. no cap at all — depends on real usage patterns once concurrent tasks are actually safe (Fix 1) and performant (Fix 2). Defer the decision until there's data.
• Destructive-window session-scoping: bundle into Fix 1 (same shape, same PR) or treat as a follow-up given its narrower blast radius? Leaning bundle — it's the same three-line change pattern applied to one more function.