feat(nomos): retry cap, vm: targets, inspect_path, goal supersession, runbooks
Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.
P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.
P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.
P1.3 — two new runbook entities in seeds/knowledge.yaml:
- nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
killall → exportfs -u → mutate → exportfs -a → verify)
- netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
after ~30s for the traefik/authentik OIDC race)
P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.
P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).
P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.
Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
This commit is contained in:
414
plans/2026-07-18-session-review-three-sessions.md
Normal file
414
plans/2026-07-18-session-review-three-sessions.md
Normal file
@@ -0,0 +1,414 @@
|
||||
# 2026-07-18 — Session review (three recent sessions)
|
||||
|
||||
**Status:** Implemented — P0.1, P0.2, P1.3, P1.4, P1.5, P1.6, P1.8, P2.10
|
||||
landed in v0.7.12. P1.7 and P2.9 deferred (retry cap addresses the same
|
||||
symptom at lower cost); see "Deferred" section at the bottom.
|
||||
**Updated:** 2026-07-18 — session 1 continued after initial audit; outcome
|
||||
upgraded from ⚠️ partial to ✅ success, root cause revised (knfsd kernel
|
||||
lock, not gateway timeout). Implementation landed same day.
|
||||
|
||||
Review of the last three Nomos sessions against the protocol in
|
||||
`.agents/skills/session-review/SKILL.md`. Data pulled from the local
|
||||
sessions API (`http://localhost:8092/sessions`).
|
||||
|
||||
---
|
||||
|
||||
## Session 1 — `1e9c7691` (2026-07-18T11:28)
|
||||
|
||||
**"Diagnose and fix ZimaOS folder move/delete failures on ludo-library"**
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Messages | 13 (6 user / 7 assistant) |
|
||||
| Tool calls | 97 across 7 turns |
|
||||
| Top tools | `run` ×58, `update_plan_step` ×7, `get_execution_status` ×7, `search_knowledge` ×3, `list_entities` ×3, `get_entity` ×3, `whoami` ×2 |
|
||||
| Objective | Diagnose and fix ZimaOS folder move/delete failures on the ludo-library NFS mount |
|
||||
| Outcome | ✅ success — root cause found and fix applied; verified from ZimaOS |
|
||||
| Severity | friction |
|
||||
|
||||
### What worked
|
||||
- Root-cause analysis was fast and correct at the **NFS permissions** layer:
|
||||
the export on `strong` uses `all_squash,anonuid=33,anongid=10000`, mapping
|
||||
every NFS client to `www-data:media`. The export root `/mnt/media_local`
|
||||
was owned `root:root / 755` while subdirs were `2775 media`. Subdir-level
|
||||
ops worked, root-level (rename/delete top-level entries) failed.
|
||||
- After the user prompted "the command just keeps running? does not
|
||||
complete," the agent dug deeper and found the **real root cause**:
|
||||
`knfsd` (kernel NFS server) holds a lock on actively-exported directories,
|
||||
causing `chown` to hang indefinitely at the `fchownat()` syscall.
|
||||
`strace -f chown :10000 /mnt/media_local` confirmed the hang point.
|
||||
25+ zombie `chgrp`/`chown` processes had piled up from the session's
|
||||
repeated attempts.
|
||||
- The correct fix sequence was identified and applied: `killall -9 chgrp
|
||||
chown` to clear zombies, then unexport → `chown :10000` + `chmod 2775`
|
||||
→ re-export. Routed via SSH-hop from `host:hubris` (the `host:strong`
|
||||
direct path kept timing out because the commands genuinely hang, not
|
||||
because of a network issue).
|
||||
- Verification was done from the client side: `touch`, `mv`, `rm`,
|
||||
`mkdir`, `rmdir` all confirmed working at the NFS root from ZimaOS.
|
||||
- Knowledge writeback was good: `upsert_knowledge` recorded an
|
||||
`investigation` linked to `vm:zimaos`, `host:strong`, `pool:ludo-lvm`,
|
||||
with the fix. `complete_task` was called with a clear summary.
|
||||
- Plan lifecycle was followed: `set_goal` → `propose_plan` →
|
||||
`update_plan_step` (running/done) → `complete_task`.
|
||||
|
||||
### What didn't
|
||||
- **20+ blind retries before investigating why.** The agent retried the
|
||||
same one-line `chown`/`chmod` roughly 20 times across direct runs,
|
||||
SSH-hop-via-hubris, wrapping in a shell script, splitting into smaller
|
||||
commands, and bare `echo test` sanity checks — all hung. Each retry
|
||||
piled up another zombie process on `strong`. The agent only
|
||||
investigated *why* the command hung after the user explicitly asked
|
||||
"the command just keeps running?"
|
||||
- **Misdiagnosed the timeout as a gateway/network issue.** The agent's
|
||||
own narrative said "API seems to be struggling with timeouts," "API
|
||||
keeps timing out on strong," "Strong mutations are consistently timing
|
||||
out — read-only works." This framed the problem as the control plane,
|
||||
when in fact the commands were genuinely hanging at the kernel level
|
||||
on the target host. A `strace` on the first failure would have
|
||||
revealed this immediately.
|
||||
- **Approval window kept expiring between retries.** User had to say "go
|
||||
ahead" twice and "proceed" + "status" once each because the assent
|
||||
window closed while the agent was looping on the hung commands.
|
||||
- **No back-off / cap on retries.** 58 `run` calls in 7 turns, of which
|
||||
~20 are essentially the same `chown :10000 /mnt/media_local && chmod
|
||||
2775 …`. Once a command has timed out 3× in a row, the agent should
|
||||
stop retrying and investigate *or* surface the blocker to the operator.
|
||||
|
||||
### Fixes needed
|
||||
- (friction) **Retry cap + "investigate before retry" rule.** In
|
||||
`cmd/nomos/agent.go`, hash each outgoing `run` command; if the same
|
||||
hash has failed 3× in the session, refuse to issue it again. Force the
|
||||
agent to either change approach (e.g. `strace`, `ps`, `lsof` to see
|
||||
*why*) or surface the blocker to the operator. This single change
|
||||
would have turned session 1 from 58 `run` calls into ~8 and produced
|
||||
the knfsd finding on the first failure instead of the 20th.
|
||||
- (friction) **SOUL.md guidance: a hung command is not a failed command.**
|
||||
When a `run` times out, the agent's first instinct should be to
|
||||
inspect the target (`ps aux | grep <cmd>`, `strace -f -p <pid>`,
|
||||
`lsof <path>`) — not to retry the same command. The current default
|
||||
(retry with different routing/wrapping) wasted 20 calls.
|
||||
- (friction) **Capture the unexport → mutate → re-export pattern as a
|
||||
runbook.** "Mutating an actively-exported NFS directory hangs at
|
||||
`fchownat()`" is a reusable finding. It belongs as a `runbook` entity
|
||||
linked to `host:strong` / `lxc:nfs-export` so the next time someone
|
||||
needs to chown/chmod an exported path, the agent finds it via
|
||||
`get_entity_knowledge` and unexports first.
|
||||
- (friction) Approval window robustness: when an execution times out,
|
||||
extend the assent window for the same plan step automatically — the
|
||||
operator already approved it; we shouldn't make them re-approve
|
||||
because *our* command hung.
|
||||
|
||||
---
|
||||
|
||||
## Session 2 — `55927f0a` (2026-07-18T09:45)
|
||||
|
||||
**"Add NFS export of ludo-lvm (/mnt/library) from strong to ZimaOS, so ZimaOS
|
||||
can see downloads/usenet/movies/ alongside the existing old-library NFS mount"**
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Messages | 25 (12 user / 13 assistant) |
|
||||
| Tool calls | 108 across 13 turns |
|
||||
| Top tools | `run` ×49, `update_plan_step` ×13, `get_entity` ×10, `list_entities` ×6, `update_entity_attributes` ×3, `search_knowledge` ×3, `propose_plan` ×3, `get_relations` ×3, `get_knowledge_content` ×3, `get_execution_status` ×3, `upsert_knowledge` ×2, `set_goal` ×2, `create_relationship` ×2 |
|
||||
| Objective | Originally: fix sabnzbd download folder to use ludo-lvm. Pivoted to: add NFS export of ludo-lvm to ZimaOS. Final outcome: just keep ludo-library (drop redundant /media/media mount) |
|
||||
| Outcome | ✅ success — ZimaOS reduced to two clean tiles, fstab cleaned, knowledge + entity attrs written back |
|
||||
| Severity | friction |
|
||||
|
||||
### What worked
|
||||
- Writeback was thorough: `upsert_knowledge` ×2, `update_entity_attributes`
|
||||
on `vm:zimaos` and `lxc:nfs-export`, `create_relationship` ×2. The
|
||||
knowledge graph is current.
|
||||
- The final cleanup was small and safe: unmount `/media/media` on ZimaOS,
|
||||
remove the fstab entry, `rmdir` the empty directory, clear CasaOS caching
|
||||
artifacts. Each step got its own `run` with a clear result.
|
||||
- Agent correctly noticed the pivot: "wait — `/media/media` IS ludo-lvm
|
||||
too, just via a double NFS hop through nfs-export. Redundant." That
|
||||
insight is what turned a complex migration into a one-step cleanup.
|
||||
|
||||
### What didn't
|
||||
- **Goal pivots were not closed cleanly.** `set_goal` was called twice —
|
||||
once for the sabnzbd fix, once for the NFS export. The first goal was
|
||||
implicitly abandoned when the user said "lets just keep ludo-library
|
||||
then"; there's no `complete_task` for it. If the session state is keyed
|
||||
on the latest `set_goal`, the first goal is orphaned in the UI.
|
||||
- **Excessive fan-out on `run`.** 49 `run` calls, many of which repeat the
|
||||
same diagnostic (`mount | grep`, `cat /etc/exports`, `exportfs -v`,
|
||||
`ls -la /mnt/...`) across `lxc:arriman`, `lxc:jellyfin`, `lxc:nfs-export`,
|
||||
`host:hubris`, `host:strong`. A single bulk "inventory this path across
|
||||
these targets" tool would have collapsed 15+ runs into 1.
|
||||
- **`vm:` targets aren't directly runnable.** Every ZimaOS command had to
|
||||
be `ssh -o StrictHostKeyChecking=no root@192.168.8.195 '…'` from
|
||||
`host:hubris`. Nested shell quoting broke once and the agent had to
|
||||
re-escape. This was called out in the 2026-07-14 review and is still
|
||||
open.
|
||||
- **Agent over-scoped before checking with the user.** After "explain how
|
||||
the migration would work", the agent produced a full multi-LXC migration
|
||||
plan (move `/dev/mapper/library-library` consumers off the old volume,
|
||||
migrate ZimaOS NFS export, etc.). The user replied "lets just keep
|
||||
ludo-library then." A clarifying question — "do you want to migrate, or
|
||||
just clean up the redundant mount?" — would have saved 4 turns.
|
||||
- **Approval friction.** One execution came back with "status=cancelled,
|
||||
but the assent window for this session is not active. The agent will not
|
||||
auto-continue. Reply 'continue' or re-approve the plan to resume." User
|
||||
had to type "proceed" to resume. This is the same assent-window-expiry
|
||||
pattern from session 1.
|
||||
|
||||
### Fixes needed
|
||||
- (friction) Track `set_goal` history per session. When a new goal is set,
|
||||
the previous one should be auto-marked `complete` (or `superseded`) so
|
||||
the UI doesn't show an orphaned active goal.
|
||||
- (friction) Add a bulk inspection tool — e.g. `inspect_path(path, targets)`
|
||||
that returns `mount`, `df`, `ls -la`, and ownership for the same path
|
||||
across multiple entities in one call. Sessions like this routinely spend
|
||||
15+ `run` calls gathering the same facts across hosts.
|
||||
- (friction) `vm:` target support in `run`. Either expose a `qm guest exec`
|
||||
wrapper or accept `vm:<slug>` as a target and route through the host. The
|
||||
manual SSH-hop pattern is error-prone (nested quoting) and slow.
|
||||
- (friction) SOUL.md guidance: before proposing a multi-step migration
|
||||
plan, ask the user "migrate or clean up?" when both are plausible from
|
||||
the prompt. This was a single extra clarification question away from a
|
||||
4-turn savings.
|
||||
|
||||
---
|
||||
|
||||
## Session 3 — `2926de4e` (2026-07-15T22:22)
|
||||
|
||||
**"Deploy apt updates to host:netbird-vps — 92 pending packages including
|
||||
Docker CE, netbird, kernel, ZFS, and security patches."**
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| Messages | 9 (4 user / 5 assistant) |
|
||||
| Tool calls | 27 across 5 turns |
|
||||
| Top tools | `update_plan_step` ×7, `run` ×6, `set_goal` ×2, `search_knowledge` ×2, `complete_task` ×2, `upsert_knowledge` ×1, `update_entity_attributes` ×1, `propose_plan` ×1, `list_lxcs` ×1, `get_relations` ×1, `get_knowledge_content` ×1, `get_entity` ×1, `get_execution_status` ×1 |
|
||||
| Objective | Two-phase: (a) fleet-wide update audit by criticality, (b) deploy the 92-package upgrade to host:netbird-vps |
|
||||
| Outcome | ✅ success — 92→0 packages pending; netbird-mgmt OIDC race caught and fixed; knowledge + entity attrs written back |
|
||||
| Severity | cosmetic |
|
||||
|
||||
### What worked
|
||||
- **Two goals, two clean lifecycles.** `set_goal` → `propose_plan` →
|
||||
`update_plan_step` (running/done) → `complete_task` ran twice, once for
|
||||
the audit and once for the upgrade. The session is the model for how
|
||||
multi-goal sessions should look.
|
||||
- **Pre-existing knowledge reuse.** First `search_knowledge` found a
|
||||
today-dated audit; agent used `get_knowledge_content` and presented it
|
||||
without needing any `run` for the audit half. Zero wasted tool calls.
|
||||
- **Long-running upgrade handled correctly.** The 92-package `apt upgrade`
|
||||
hit the HTTP gateway timeout mid-run. Agent didn't retry it — it called
|
||||
`get_execution_status` and then ran a verification `run`
|
||||
(`apt list --upgradable | wc -l`, `uname -r`, `docker ps`) to confirm
|
||||
completion server-side despite the timeout. This is the right pattern;
|
||||
session 1 should have done the same.
|
||||
- **Gotcha caught.** After the upgrade, `docker logs netbird-mgmt`
|
||||
revealed the management container was crash-looping because it tried to
|
||||
fetch OIDC config from `auth.hubris.network` before traefik/authentik
|
||||
were ready. Fix: `docker restart netbird-mgmt` after ~30s. Captured in
|
||||
`upsert_knowledge` as an `investigation` tagged `apt, upgrade, netbird,
|
||||
docker, gotcha` linked to `host:netbird-vps`.
|
||||
- `update_entity_attributes` was called on `host:netbird-vps` to record the
|
||||
new kernel version. Good writeback hygiene.
|
||||
|
||||
### What didn't
|
||||
- (cosmetic) The HTTP timeout on long-running upgrades surfaced as a
|
||||
transient error to the operator. The agent handled it correctly but the
|
||||
UX would be cleaner if `run` returned `PENDING` immediately for known
|
||||
long-running command patterns (`apt upgrade`, `pct migrate`, `rclone
|
||||
sync`, etc.) instead of timing out at the gateway.
|
||||
- (cosmetic) Two `complete_task` calls in one session produced two "task
|
||||
complete" bubbles. Fine, but the second one could have noted the
|
||||
first-task outcome as well in its summary so the chat reads as one
|
||||
coherent arc.
|
||||
|
||||
### Fixes needed
|
||||
- (cosmetic) Long-running command detection in `run`: if the command
|
||||
matches a known-long pattern, return a `PENDING` execution id with a
|
||||
hint to poll `get_execution_status`, rather than blocking at the HTTP
|
||||
layer for 30s and timing out. Session 3 already proved the
|
||||
poll-after-timeout pattern works — make it the default for these
|
||||
commands.
|
||||
- (cosmetic) Encourage the agent to fold the prior task's outcome into
|
||||
the next `complete_task` summary when a session has multiple goals.
|
||||
|
||||
---
|
||||
|
||||
## Cross-session patterns
|
||||
|
||||
| # | Pattern | Sessions | Severity |
|
||||
|---|---|---|---|
|
||||
| 1 | Agent retries hung commands 20× before investigating *why* | 1 | friction |
|
||||
| 2 | Approval window expires between turns forcing re-approval | 1, 2 | friction |
|
||||
| 3 | `vm:` targets not directly runnable — must SSH-hop via `host:hubris` | 1, 2 | friction |
|
||||
| 4 | N+1 fan-out on `run` for cross-entity fact-gathering | 1, 2 | friction |
|
||||
| 5 | No retry cap — agent retries identical failing `run` 10–20× | 1 | friction |
|
||||
| 6 | Goal pivots not closed (`set_goal` called twice without closing prior) | 2 | friction |
|
||||
| 7 | Long-running commands hit HTTP timeout instead of returning PENDING | 3 | cosmetic |
|
||||
| 8 | Agent over-scopes migration plans before checking intent | 2 | friction |
|
||||
| 9 | Reusable operational gotchas (knfsd lock, OIDC race) captured as investigations, not runbooks | 1, 3 | friction |
|
||||
|
||||
**What consistently works well**
|
||||
- Plan lifecycle: `set_goal` → `propose_plan` → `update_plan_step` →
|
||||
`complete_task` is now followed in all three sessions.
|
||||
- Knowledge writeback: `upsert_knowledge`, `update_entity_attributes`,
|
||||
`create_relationship` are used in every session. The graph is kept
|
||||
current.
|
||||
- Root-cause analysis quality is high once the agent digs in (NFS
|
||||
all_squash + root dir perms → knfsd fchownat hang; double NFS hop;
|
||||
OIDC race condition). The problem is getting the agent to dig in
|
||||
*before* the 20th retry.
|
||||
|
||||
**What consistently breaks**
|
||||
- **Hung commands get retried instead of investigated.** Session 1's
|
||||
`chown` was blocked by knfsd for 30+ minutes while the agent retried
|
||||
with different routing/wrapping. Session 3's `apt upgrade` timed out
|
||||
and the agent correctly polled — but that's the exception, not the
|
||||
rule. The default behavior is "retry the same thing differently."
|
||||
- Approval window lifetime vs. agent retry loops — when execution times
|
||||
out, the assent window lapses and the operator has to re-approve even
|
||||
though the *intent* was never withdrawn.
|
||||
- Reusable operational fixes (unexport → mutate → re-export for NFS
|
||||
dirs; `docker restart netbird-mgmt` after stack upgrade) get recorded
|
||||
as `investigation` entities. They should be `runbook` entities so the
|
||||
agent finds them via `get_entity_knowledge` next time and applies the
|
||||
procedure instead of rediscovering it.
|
||||
|
||||
---
|
||||
|
||||
## Improvement plan
|
||||
|
||||
### P0 — Friction (was blocker; downgraded after session 1 resolved)
|
||||
|
||||
1. **Retry cap + "investigate before retry" rule.** In
|
||||
`cmd/nomos/agent.go`, hash each outgoing `run` command; if the same
|
||||
hash has failed 3× in the session, refuse to issue it again. Force
|
||||
the agent to either change approach (e.g. `strace`, `ps aux | grep`,
|
||||
`lsof` to see *why*) or surface the blocker to the operator. This
|
||||
single change would have turned session 1 from 58 `run` calls into
|
||||
~8 and produced the knfsd finding on the first failure instead of
|
||||
the 20th.
|
||||
2. **SOUL.md guidance: a hung command is not a failed command.** When a
|
||||
`run` times out, the agent's first instinct should be to inspect the
|
||||
target (`ps aux | grep <cmd>`, `strace -f -p <pid>`, `lsof <path>`)
|
||||
— not to retry the same command with different routing/wrapping. The
|
||||
current default wasted 20 calls in session 1.
|
||||
|
||||
### P1 — Friction
|
||||
|
||||
3. **Capture operational gotchas as `runbook` entities, not just
|
||||
`investigation`.** Two candidates from these sessions:
|
||||
- **"Mutating an actively-exported NFS directory hangs at
|
||||
`fchownat()`"** — procedure: `killall -9 chgrp chown` →
|
||||
`exportfs -u <client>:<path>` → `chown`/`chmod` → `exportfs -a`.
|
||||
Linked to `host:strong`, `lxc:nfs-export`.
|
||||
- **"netbird-mgmt crash-loops after stack upgrade"** — procedure:
|
||||
wait ~30s for traefik/authentik to come up, then
|
||||
`docker restart netbird-mgmt`. Linked to `host:netbird-vps`.
|
||||
Today both are `investigation` entries; the agent records them but
|
||||
won't proactively apply them next time.
|
||||
4. **Auto-close prior `set_goal` when a new one is set.** Mark the
|
||||
previous goal `superseded` and emit a synthetic `complete_task`
|
||||
summary so the UI doesn't show an orphaned active goal. (Session 2
|
||||
had this.)
|
||||
5. **Bulk inspection tool.** Add an MCP tool like
|
||||
`inspect_path(path, targets[])` that runs `mount | grep`, `df`,
|
||||
`ls -la`, and `stat` against a list of entity slugs in one call.
|
||||
Sessions 1 and 2 each spent ~15 `run` calls gathering identical
|
||||
facts across hosts/LXCs.
|
||||
6. **`vm:` target support in `run`.** Accept `vm:<slug>` as a target
|
||||
and route via `qm guest exec` on the host that owns the VM.
|
||||
Eliminates the nested-quoting SSH-hop pattern that broke once in
|
||||
session 2 and required manual SSH-hop workarounds in session 1.
|
||||
7. **Approval window robustness.** When an execution times out, extend
|
||||
the assent window for the same plan step automatically — the
|
||||
operator already approved it; we shouldn't make them re-approve
|
||||
because *our* command hung. Affects sessions 1 and 2.
|
||||
8. **SOUL.md guidance: ask-before-migrating.** When a user request is
|
||||
ambiguous between "fix in place" and "migrate," the agent should
|
||||
ask one clarifying question before producing a multi-step migration
|
||||
plan. Session 2 would have saved ~4 turns.
|
||||
|
||||
### P2 — Cosmetic
|
||||
|
||||
9. **Long-running command detection.** Maintain a small regex list
|
||||
(`apt (upgrade|install)`, `pct migrate`, `rclone (sync|copy)`,
|
||||
`dd if=`, `docker compose pull`) for commands that are known to
|
||||
exceed 30s. Return `PENDING` immediately with an `execution_id`
|
||||
instead of blocking at the gateway. Session 3 already uses the
|
||||
poll pattern — make it the default.
|
||||
10. **Multi-goal `complete_task` summaries.** When a session has more
|
||||
than one `set_goal`, the final `complete_task` summary should
|
||||
reference the arc of the whole session, not just the last goal.
|
||||
|
||||
---
|
||||
|
||||
## Revised note on the original P0
|
||||
|
||||
The original P0 ("Diagnose `host:strong` config_mutation timeouts —
|
||||
suspect SSH latency / mesh routing, raise timeout") was **wrong**. The
|
||||
timeouts were not a gateway or network issue — the commands were
|
||||
genuinely hanging at the kernel level because `knfsd` holds a lock on
|
||||
actively-exported directories. Raising the HTTP timeout would not have
|
||||
helped; the `chown` would simply hang longer. The real fix is (a) the
|
||||
retry-cap/investigate-before-retry rule (P0.1 above) and (b) the
|
||||
unexport → mutate → re-export runbook (P1.3).
|
||||
|
||||
---
|
||||
|
||||
## Deferred
|
||||
|
||||
**P1.7 — Approval window auto-extends on execution timeout.** The assent
|
||||
window lives in `autonomy_settings` and is read by `classifyAndGate`
|
||||
(`internal/mcp/server.go:607`); timeout detection lives in `sshExec`
|
||||
(`internal/mcp/server.go:332`). Wiring them requires the SSH-execution
|
||||
path to signal back into the approval-state machine across the nomos ↔
|
||||
api process boundary, and a future implementation needs to distinguish
|
||||
"command genuinely hung" (knfsd case — don't extend, the command is
|
||||
stuck) from "command is long-running" (apt upgrade — extend). Without
|
||||
that distinction, auto-extending on every timeout would mask real hang
|
||||
symptoms — exactly the misdiagnosis session 1 made. **The retry cap
|
||||
(P0.1) addresses the same symptom at lower cost**: after 3 failures
|
||||
the agent is forced to investigate or surface, which removes the
|
||||
cascading retry storm that made the assent expiry visible in the first
|
||||
place. Revisit if future sessions show the operator re-approving a
|
||||
plan they never withdrew in intent (not just retrying a hung command).
|
||||
|
||||
**P2.9 — Long-running command PENDING detection.** A regex list of
|
||||
known-long commands (`apt (upgrade|install)`, `pct migrate`, `rclone
|
||||
(sync|copy)`, `dd if=`, `docker compose pull`) so `run` returns
|
||||
`PENDING` immediately with an `execution_id` instead of blocking at
|
||||
the HTTP gateway for 30s and timing out. **Session 3 already proved
|
||||
the current poll pattern works:** the `apt upgrade` timed out at the
|
||||
gateway, the agent called `get_execution_status`, then ran a
|
||||
verification `run` (`apt list --upgradable | wc -l`, `uname -r`,
|
||||
`docker ps`) — clean 92→0 packages result. The agent did the right
|
||||
thing without any new machinery, and the retry cap (P0.1) protects
|
||||
against the failure mode of this path (blind retry on timeout).
|
||||
Implementing PENDING detection well requires a classifier extension
|
||||
(`internal/policy`) plus a new return shape from `classifyAndGate`
|
||||
that the agent loop has to learn to handle (poll instead of retry) —
|
||||
a real protocol change, not a small fix. Worth doing if the
|
||||
poll-after-timeout pattern proves fragile over the next few sessions;
|
||||
not worth doing speculatively right now.
|
||||
|
||||
---
|
||||
|
||||
## Verification commands
|
||||
|
||||
```bash
|
||||
# Re-pull any session for follow-up
|
||||
curl -s http://localhost:8092/sessions/1e9c7691-5815-48d1-acb4-91a6a39691c9 | jq .
|
||||
curl -s http://localhost:8092/sessions/55927f0a-597e-4561-aaef-077623051432 | jq .
|
||||
curl -s http://localhost:8092/sessions/2926de4e-0b73-4c3d-a2cd-ee9a42089b46 | jq .
|
||||
|
||||
# Confirm host:strong mutation timeout reproduces
|
||||
curl -s http://localhost:8092/sessions | jq -r '.sessions[].id' | head -1 # latest session id
|
||||
```
|
||||
|
||||
## Related files
|
||||
|
||||
- `cmd/nomos/agent.go` — agent loop, retry behavior, goal state
|
||||
- `cmd/nomos/store.go` — `set_goal` / `complete_task` persistence
|
||||
- `internal/mcp/server.go` — `run` tool, timeout handling, `get_execution_status`
|
||||
- `internal/httpapi/server.go` — HTTP gateway timeout for mutations
|
||||
- `nomos/SOUL.md` — agent persona, ask-before-migrate guidance candidate
|
||||
- `plans/2026-07-14-activity-gaps.md` — prior session review (same patterns recurring)
|
||||
@@ -17,6 +17,7 @@ went sideways, open an investigation.
|
||||
| 2026-07-14 | [Activity gaps](2026-07-14-activity-gaps.md) | In Progress |
|
||||
| 2026-07-14 | [Activity timeline](2026-07-14-activity-timeline.md) | In Progress |
|
||||
| 2026-07-17 | [Codebase review, lint audit, and documentation maintenance](2026-07-17-codebase-review-and-cleanup.md) | Report delivered — doc/tooling fixes applied; code refactors pending |
|
||||
| 2026-07-18 | [Session review: three recent sessions](2026-07-18-session-review-three-sessions.md) | Implemented in v0.7.12 — P0.1/P0.2/P1.3/P1.4/P1.5/P1.6/P1.8/P2.10; P1.7 and P2.9 deferred (retry cap covers) |
|
||||
|
||||
## Done
|
||||
|
||||
|
||||
Reference in New Issue
Block a user