fix(agent): panic recovery on every background goroutine (B1+B2)
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together, since the right granularity for B1 in the auto-continuation worker turned out to require B2's restructuring anyway (see below). B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned nothing before this — every explicitly-spawned goroutine (continuation worker, resumed chat turns, async execution dispatch, the SSE listener, two duplicate sshExec implementations' output-collector goroutines) crashed the whole process on an unhandled panic, not just that one goroutine. More consequential post-concurrency: more simultaneous unattended background work means more surface area for one bad input to end every running task. New internal/safego package: Go(label, fn) launches fn in a goroutine with a recover-and-log wrapper. Applied at every bare `go` spawn site across the three packages. Two sites needed bespoke handling instead of the generic helper because their callers block on a channel and a silent recover would just make them hang until timeout: sshExec's output-collector goroutine (two near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go) and httpapi's ListenAndServe goroutine — both now recover AND send a synthetic error result so the waiting select unblocks immediately instead of waiting out the full timeout. httpapi's sseListener got extra treatment: its per-notification handling was extracted into handleNotification with its own recover, so a panic decoding ONE malformed pg_notify payload can't kill the listener goroutine for every connected SSE client — the outer goroutine spawn only needs to guard the connection setup/reconnect code around it. B2: cmd/nomos/continue.go's processContinuations used to run every pending continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the ticker — meaning (a) task B's continuation waited for task A's full (up to 10-minute) resumed turn to finish first, undercutting this session's earlier concurrency work on exactly the path autonomous tasks depend on most, and (b) an unrecovered panic anywhere in that call chain didn't just crash the process (B1) — even WITH B1's recovery wrapped only at the top-level worker spawn, the panic would still unwind the ENTIRE ticker-loop goroutine, silently ending auto-continuation for every task until nomos restarted. Fixed by spawning each pending item via safego.Go individually: real parallelism, and a bad item can now only ever take down its own goroutine. Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete proof — a deliberate panic inside Go() that would otherwise crash the whole test binary; reaching the assertion after it IS the evidence recovery works. Verified live against the rebuilt containers: full chat turn round-tripped correctly (hostname lookup, 2 iterations, normal completion) — no regression from threading safego.Go through the tool-dispatch/continuation paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -24,6 +24,7 @@ import (
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/policy"
|
||||
"github.com/dtoro/oikos/internal/safego"
|
||||
"github.com/google/jsonschema-go/jsonschema"
|
||||
"github.com/google/uuid"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
@@ -474,7 +475,9 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
|
||||
// tool call, cancelled the instant the chat turn's HTTP
|
||||
// response completes (every normal turn) — a goroutine
|
||||
// meant to outlive the request must not inherit its context.
|
||||
go executeApprovedViaAPI(context.Background(), id, targetSlug, "apt_upgrade:"+params)
|
||||
safego.Go("mcp:executeApprovedViaAPI:apt_upgrade", func() {
|
||||
executeApprovedViaAPI(context.Background(), id, targetSlug, "apt_upgrade:"+params)
|
||||
})
|
||||
slog.Info("mcp: apt_upgrade auto-approved via assent window", "execution_id", id)
|
||||
return textResult(fmt.Sprintf("apt_upgrade on %s auto-approved via assent window — execution %s running.", targetSlug, id)), nil
|
||||
}
|
||||
@@ -492,7 +495,9 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
|
||||
// See the apt_upgrade case above for why there's no
|
||||
// pre-flip-status "autoApprove" step here anymore, and why
|
||||
// this uses context.Background().
|
||||
go executeApprovedViaAPI(context.Background(), id, targetSlug, "pct_create:"+params)
|
||||
safego.Go("mcp:executeApprovedViaAPI:pct_create", func() {
|
||||
executeApprovedViaAPI(context.Background(), id, targetSlug, "pct_create:"+params)
|
||||
})
|
||||
slog.Info("mcp: pct_create auto-approved via assent window", "execution_id", id)
|
||||
return textResult(fmt.Sprintf("pct_create on %s auto-approved via assent window — execution %s running. The LXC is being provisioned now.", targetSlug, id)), nil
|
||||
}
|
||||
@@ -1132,6 +1137,21 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
go func() {
|
||||
// Recovers a panic in CombinedOutput (SSH library internals, rare but
|
||||
// not impossible) and reports it as a failed command instead of
|
||||
// crashing the whole api process — every gated action runs through
|
||||
// this function, so an unrecovered panic here would take down every
|
||||
// concurrently-running task's execution, not just this one. Without
|
||||
// this, a panic would ALSO silently degrade to "wait out the full
|
||||
// timeout" (done never receives, the select below falls through to
|
||||
// its time.After case) rather than crashing outright — recovering
|
||||
// and sending an immediate result is strictly better: the caller
|
||||
// finds out now, not after sshExecTimeout.
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
||||
}
|
||||
}()
|
||||
out, err := session.CombinedOutput(command)
|
||||
done <- result{out, err}
|
||||
}()
|
||||
|
||||
Reference in New Issue
Block a user