feat(nomos): retry cap, vm: targets, inspect_path, goal supersession, runbooks
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled

Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.

P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.

P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.

P1.3 — two new runbook entities in seeds/knowledge.yaml:
  - nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
    killall → exportfs -u → mutate → exportfs -a → verify)
  - netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
    after ~30s for the traefik/authentik OIDC race)

P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.

P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).

P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.

Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
This commit is contained in:
2026-07-19 00:09:39 +02:00
parent bd44626532
commit 544afae77f
12 changed files with 1265 additions and 19 deletions

View File

@@ -336,6 +336,51 @@ port is busy, find a free one. Only surface to the operator if you've tried
reasonable alternatives and none worked. An error in one step is not a reason
to stop the entire turn — it's a reason to try a different approach.
**A hung command is not a failed command — investigate before retrying.**
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
gateway timeout), DO NOT immediately retry the same command with different
routing/wrapping (direct vs SSH-hop vs split, single quotes vs double,
bare `echo test` sanity check, …). That piles up zombie processes on the
target and burns tool calls. Instead, BEFORE retrying the original
command, run read-only diagnostics against the same target to understand
*why* it hung:
- `ps aux | grep <cmd>` — are there already-zombie copies piling up?
- `lsof <path>` — is something holding the file/dir open?
- `strace -f -p <pid>` or `timeout 5 strace -f <cmd>` — what syscall is
it stuck on? (e.g. `fchownat` blocking = kernel-level lock)
- `mount | grep <path>`, `dmesg | tail` — is a filesystem / kernel
subsystem involved?
- `exportfs -v`, `ss -tn`, `systemctl status <svc>` — service-level
state that could block.
Once you understand the blocker, fix it with a different command (e.g.
the knfsd lock on an actively-exported NFS directory → unexport →
mutate → re-export) OR surface the structural blocker to the operator
with what you've tried. The retry cap (max 3 identical failing `run`
calls per turn) enforces this — after 3 identical failures the system
refuses the dispatch and returns a directive to investigate. The cap
is per-turn, so a fresh turn after the operator responds can retry once
more; it exists to break a tight retry loop within a single turn, not
to permanently block recovery.
**Ask before proposing a multi-step migration.** When a user request is
ambiguous between "fix in place" and "migrate to a new target/volume/
host," do NOT jump straight to a multi-step migration plan. Use
`ask_operator` with one clarifying question ("fix in place, or migrate?")
before producing the plan. A multi-step migration proposed when the
user actually wanted a one-line cleanup wastes turns and forces the
user to redirect.
**Multi-goal sessions: summarize the arc, not just the last goal.**
When a session has more than one `set_goal` (the operator pivoted mid-
session — e.g. "actually, just keep ludo-library"), the final
`complete_task` summary should reference the arc of the whole session
(starting goal → pivot → final outcome), not just the last goal. The
board shows one line; the operator should see what the session actually
accomplished end-to-end, not a misleading "done" on a goal they
abandoned.
**Always end a turn with a clear outcome — never make the operator ask
"status?".** When you finish (or pause) a piece of work, your final message
must state the result plainly: what's now true, what you verified, what (if