fix: sshExec had no timeout — a hung remote command blocked forever
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled

Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).

Two compounding bugs made this unrecoverable without manual intervention:

1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
   NO execution timeout — `session.CombinedOutput()` blocks until the remote
   command exits, with no deadline. A hung remote process blocks the Go
   goroutine forever; the execution can never leave 'running', and the
   operator has no way to make it stop. Fixed: both now race the SSH call
   against a 10-minute hard timeout, closing the session/client and
   returning a clear "timed out after 10m0s" error if exceeded. (The
   mcp/server.go copy also still had the original "swallowed non-zero exit"
   bug from before that fix was applied to httpapi's copy only — fixed here
   too.)

2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
   connectivity — it doesn't; a black-holed network can make each call hang
   far past the resolver's nominal timeout, so the documented "~90s" budget
   was never real. Wrapped every attempt in `timeout 3` so the wall-clock
   budget is now actually enforced (~2min worst case), and the failure
   message now suggests checking the net0 gateway.

Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):

- InlineApproval's track() polling loop had its own ~6min ceiling and simply
  STOPPED polling after that — silently going stale before the backend (now
  correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
  with margin, and added a distinct 'stalled' state if that's ever exceeded
  (explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
  execution's created_at), the actual command being run, and the execution
  ID — previously just a static "this can take a minute" with zero
  information. Also added command display to the destructive pending-
  approval card for full transparency before confirming.

Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-10 10:45:47 +02:00
parent f936098364
commit 8950bada44
4 changed files with 188 additions and 39 deletions

View File

@@ -1048,6 +1048,12 @@ func initSSH() {
}
}
// sshExecTimeout bounds how long a single remote command may run — see the
// matching constant/comment in httpapi/phase3.go. Without it, a hung remote
// command (piped install script stuck retrying DNS, etc.) blocks this
// goroutine forever with no way for the caller to ever get an answer.
const sshExecTimeout = 10 * time.Minute
func sshExec(ctx context.Context, host, user, command string) (string, error) {
initSSH()
if len(sshKey) == 0 {
@@ -1082,11 +1088,40 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
}
defer session.Close()
out, err := session.CombinedOutput(command)
if err != nil && out == nil {
return "", fmt.Errorf("exec: %w", err)
type result struct {
out []byte
err error
}
done := make(chan result, 1)
go func() {
out, err := session.CombinedOutput(command)
done <- result{out, err}
}()
select {
case r := <-done:
text := strings.TrimSpace(string(r.out))
// A non-zero exit MUST surface as an error — matching the fix
// applied to httpapi's sshExec (this copy still had the original
// bug: only erroring when there was no output at all, so a command
// that failed but printed something was silently reported as
// success).
if r.err != nil {
if text != "" {
return text, fmt.Errorf("%w: %s", r.err, text)
}
return text, fmt.Errorf("exec: %w", r.err)
}
return text, nil
case <-time.After(sshExecTimeout):
session.Close()
client.Close()
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
case <-ctx.Done():
session.Close()
client.Close()
return "", ctx.Err()
}
return strings.TrimSpace(string(out)), nil
}
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {