Root cause of "running for 10+ minutes without stopping": a real production
execution (TypeType pct_create) was found genuinely stuck 17+ minutes into a
single blocking SSH call. The container's post_install script was looping on
`getent hosts deb.debian.org`, waiting on a network that could never come up
— the operator's static IP config used gw:192.168.8.1, but the actual gateway
on that subnet is 192.168.8.2, so every network call hung instead of failing
fast (packets dropped, not rejected).
Two compounding bugs made this unrecoverable without manual intervention:
1. sshExec (both internal/httpapi/phase3.go and internal/mcp/server.go) had
NO execution timeout — `session.CombinedOutput()` blocks until the remote
command exits, with no deadline. A hung remote process blocks the Go
goroutine forever; the execution can never leave 'running', and the
operator has no way to make it stop. Fixed: both now race the SSH call
against a 10-minute hard timeout, closing the session/client and
returning a clear "timed out after 10m0s" error if exceeded. (The
mcp/server.go copy also still had the original "swallowed non-zero exit"
bug from before that fix was applied to httpapi's copy only — fixed here
too.)
2. provisionScript's DNS-wait loop assumed `getent hosts` fails fast on no
connectivity — it doesn't; a black-holed network can make each call hang
far past the resolver's nominal timeout, so the documented "~90s" budget
was never real. Wrapped every attempt in `timeout 3` so the wall-clock
budget is now actually enforced (~2min worst case), and the failure
message now suggests checking the net0 gateway.
Also fixes the matching UI-side gap (operator's literal question: "is there
a way to get more details? it has been running for 10+ minutes without
stopping"):
- InlineApproval's track() polling loop had its own ~6min ceiling and simply
STOPPED polling after that — silently going stale before the backend (now
correctly capped at 10min) could ever resolve. Raised to a 14min ceiling
with margin, and added a distinct 'stalled' state if that's ever exceeded
(explicitly says something's wrong, rather than freezing silently).
- The running-card now shows live elapsed time (ticking, from the
execution's created_at), the actual command being run, and the execution
ID — previously just a static "this can take a minute" with zero
information. Also added command display to the destructive pending-
approval card for full transparency before confirming.
Verified live end-to-end in a real browser (dev server proxying to
production): queued a real command via chat, approved via the button,
watched the elapsed-time counter tick in real time, and saw it transition to
a completed card with real output once the command finished.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dtoro
8950bada44