fix(agent): panic recovery on every background goroutine (B1+B2)
Fixes B1 and B2 of plans/2026-07-11-nomos-agent-code-review.md together, since the right granularity for B1 in the auto-continuation worker turned out to require B2's restructuring anyway (see below). B1: grep -rn "recover()" cmd/nomos/ internal/mcp/ internal/httpapi/ returned nothing before this — every explicitly-spawned goroutine (continuation worker, resumed chat turns, async execution dispatch, the SSE listener, two duplicate sshExec implementations' output-collector goroutines) crashed the whole process on an unhandled panic, not just that one goroutine. More consequential post-concurrency: more simultaneous unattended background work means more surface area for one bad input to end every running task. New internal/safego package: Go(label, fn) launches fn in a goroutine with a recover-and-log wrapper. Applied at every bare `go` spawn site across the three packages. Two sites needed bespoke handling instead of the generic helper because their callers block on a channel and a silent recover would just make them hang until timeout: sshExec's output-collector goroutine (two near-identical copies, internal/mcp/server.go and internal/httpapi/phase3.go) and httpapi's ListenAndServe goroutine — both now recover AND send a synthetic error result so the waiting select unblocks immediately instead of waiting out the full timeout. httpapi's sseListener got extra treatment: its per-notification handling was extracted into handleNotification with its own recover, so a panic decoding ONE malformed pg_notify payload can't kill the listener goroutine for every connected SSE client — the outer goroutine spawn only needs to guard the connection setup/reconnect code around it. B2: cmd/nomos/continue.go's processContinuations used to run every pending continuation SEQUENTIALLY in a plain for loop, in the SAME goroutine as the ticker — meaning (a) task B's continuation waited for task A's full (up to 10-minute) resumed turn to finish first, undercutting this session's earlier concurrency work on exactly the path autonomous tasks depend on most, and (b) an unrecovered panic anywhere in that call chain didn't just crash the process (B1) — even WITH B1's recovery wrapped only at the top-level worker spawn, the panic would still unwind the ENTIRE ticker-loop goroutine, silently ending auto-continuation for every task until nomos restarted. Fixed by spawning each pending item via safego.Go individually: real parallelism, and a bad item can now only ever take down its own goroutine. Added internal/safego/safego_test.go: TestGo_RecoversPanic is the concrete proof — a deliberate panic inside Go() that would otherwise crash the whole test binary; reaching the assertion after it IS the evidence recovery works. Verified live against the rebuilt containers: full chat turn round-tripped correctly (hostname lookup, 2 iterations, normal completion) — no regression from threading safego.Go through the tool-dispatch/continuation paths. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
35
internal/safego/safego.go
Normal file
35
internal/safego/safego.go
Normal file
@@ -0,0 +1,35 @@
|
||||
// Package safego provides a goroutine launcher that recovers panics instead
|
||||
// of letting them crash the whole process.
|
||||
//
|
||||
// Go's default behavior for a panic in ANY goroutine — not just the one
|
||||
// serving an HTTP request, which net/http recovers automatically per
|
||||
// request — is to take down the entire process. This codebase runs several
|
||||
// long-lived or unattended background goroutines (the nomos auto-
|
||||
// continuation worker, resumed chat turns, async execution dispatch, the SSE
|
||||
// event listener) that do real work — JSON parsing of model/tool output,
|
||||
// map/slice indexing — with no operator watching. Before this package, a
|
||||
// single edge case in any of them (a malformed tool result, an unexpected
|
||||
// nil) would crash nomos or the api process outright, taking down every
|
||||
// concurrently-running task or request, not just the one that hit it.
|
||||
package safego
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"runtime/debug"
|
||||
)
|
||||
|
||||
// Go runs fn in a new goroutine. A panic inside fn is recovered and logged
|
||||
// (with a stack trace) instead of crashing the process. label identifies the
|
||||
// goroutine in logs — use something a reader can trace back to the call
|
||||
// site, e.g. "nomos:continuation-worker" or "mcp:executeApprovedViaAPI".
|
||||
func Go(label string, fn func()) {
|
||||
go func() {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
slog.Error("panic recovered in background goroutine",
|
||||
"goroutine", label, "panic", r, "stack", string(debug.Stack()))
|
||||
}
|
||||
}()
|
||||
fn()
|
||||
}()
|
||||
}
|
||||
39
internal/safego/safego_test.go
Normal file
39
internal/safego/safego_test.go
Normal file
@@ -0,0 +1,39 @@
|
||||
package safego
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestGo_RecoversPanic is the concrete proof for the B1 fix in
|
||||
// plans/2026-07-11-nomos-agent-code-review.md: a panic inside a goroutine
|
||||
// launched via Go must not crash the process (or, here, the test binary —
|
||||
// the same guarantee). Before this package existed, every background
|
||||
// goroutine in cmd/nomos/internal/mcp/internal/httpapi used a bare `go`
|
||||
// statement; an unhandled panic in any of them takes down the entire Go
|
||||
// process, not just that goroutine.
|
||||
func TestGo_RecoversPanic(t *testing.T) {
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
|
||||
Go("test:deliberate-panic", func() {
|
||||
defer wg.Done()
|
||||
panic("this must be recovered, not crash the test binary")
|
||||
})
|
||||
|
||||
// If the panic weren't recovered, the whole test binary would crash
|
||||
// before ever reaching this line (a Go panic in any goroutine terminates
|
||||
// the process, full stop) — Wait() returning normally IS the proof.
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// TestGo_RunsFnNormally confirms the non-panic path still just runs fn.
|
||||
func TestGo_RunsFnNormally(t *testing.T) {
|
||||
done := make(chan bool, 1)
|
||||
Go("test:normal", func() {
|
||||
done <- true
|
||||
})
|
||||
if !<-done {
|
||||
t.Fatal("fn did not run")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user