feat(nomos): retry cap, vm: targets, inspect_path, goal supersession, runbooks
Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.
P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.
P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.
P1.3 — two new runbook entities in seeds/knowledge.yaml:
- nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
killall → exportfs -u → mutate → exportfs -a → verify)
- netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
after ~30s for the traefik/authentik OIDC race)
P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.
P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).
P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.
Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
This commit is contained in:
@@ -6713,3 +6713,143 @@ runbooks:
|
||||
tags:
|
||||
- skill
|
||||
- runbook
|
||||
- slug: nfs-exported-dir-mutation-hang
|
||||
name: Mutating an actively-exported NFS directory hangs at fchownat
|
||||
risk_class: config_mutation
|
||||
entity_type: host
|
||||
procedure: {}
|
||||
content: |
|
||||
---
|
||||
name: nfs-exported-dir-mutation-hang
|
||||
risk_class: config_mutation
|
||||
inputs: [target_host, exported_path, mutation_command]
|
||||
verification: "stat -c '%a %U:%G' <exported_path> on the target host"
|
||||
docs_update_checklist: [investigation_entry]
|
||||
---
|
||||
|
||||
# Mutating an actively-exported NFS directory hangs at fchownat
|
||||
|
||||
Symptom: a `chown` / `chgrp` / `chmod` against a directory that is
|
||||
actively exported via `nfs-kernel-server` (knfsd) hangs indefinitely — the
|
||||
command appears to run but never returns. SSH/gateway time out waiting for it.
|
||||
`ps aux | grep chown` shows the process in interruptible sleep (D state);
|
||||
repeated retries pile up zombies (25+ observed in one session).
|
||||
|
||||
Root cause: knfsd holds a kernel lock on the directory while it's exported.
|
||||
`fchownat()` blocks waiting for the lock. This is NOT a gateway or SSH issue —
|
||||
raising the timeout just makes the hang longer.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. **Clear the zombies** from prior failed attempts:
|
||||
```
|
||||
killall -9 chgrp chown chmod 2>/dev/null
|
||||
```
|
||||
2. **Temporarily unexport** the path for each client/network that has it exported:
|
||||
```
|
||||
exportfs -u <client>:<path> # e.g. exportfs -u 192.168.8.0/24:/mnt/media_local
|
||||
```
|
||||
3. **Apply the mutation** (now that knfsd has released the lock):
|
||||
```
|
||||
chown :<gid> <path> && chmod <mode> <path>
|
||||
```
|
||||
4. **Re-export** (restore the exports):
|
||||
```
|
||||
exportfs -a
|
||||
```
|
||||
5. **Verify** from an NFS client that the new permissions are visible and operations work end-to-end:
|
||||
```
|
||||
stat -c '%a %U:%G' /media/<mountpoint> # on a client
|
||||
touch /media/<mountpoint>/.test && mv /media/<mountpoint>/.test /media/<mountpoint>/.moved && rm /media/<mountpoint>/.moved
|
||||
```
|
||||
|
||||
## Detection signature (for agents)
|
||||
|
||||
A `run` call against a host that contains `chown|chgrp|chmod` of a path
|
||||
exported by `nfs-kernel-server` AND the call times out → assume this runbook.
|
||||
Don't retry the same command; run `strace -f <cmd>` (it will block at
|
||||
`fchownat`) to confirm, then apply the procedure above.
|
||||
|
||||
## Caveats
|
||||
|
||||
- `exportfs -u` may emit a format-mismatch warning if the export was defined
|
||||
via `/etc/exports` with a different option string than `exportfs -v`
|
||||
reports. The unexport still succeeds; verify with `exportfs -v` afterward
|
||||
that the path is gone, then re-add with `exportfs -a`.
|
||||
- This applies to ANY mutating op on the exported dir (chown, chmod, rename,
|
||||
rmdir of the root). Subdirectory mutations are fine as long as they don't
|
||||
touch the exported root itself.
|
||||
|
||||
Recorded after session 1e9c7691 (2026-07-18) — 20+ retries of a
|
||||
`chown :10000 /mnt/media_local` that hung for 30+ minutes before this
|
||||
procedure was identified.
|
||||
tags:
|
||||
- runbook
|
||||
- nfs
|
||||
- knfsd
|
||||
- gotcha
|
||||
- slug: netbird-mgmt-oidc-race-after-upgrade
|
||||
name: netbird-mgmt crash-loops after stack upgrade (OIDC race)
|
||||
risk_class: reversible_low
|
||||
entity_type: host
|
||||
procedure: {}
|
||||
content: |
|
||||
---
|
||||
name: netbird-mgmt-oidc-race-after-upgrade
|
||||
risk_class: reversible_low
|
||||
inputs: []
|
||||
verification: "docker ps --filter name=netbird-mgmt --format '{{.Status}}' shows Up"
|
||||
docs_update_checklist: [investigation_entry]
|
||||
---
|
||||
|
||||
# netbird-mgmt crash-loops after stack upgrade (OIDC race)
|
||||
|
||||
Symptom: after a full-stack restart on `host:netbird-vps` (e.g. following an
|
||||
apt upgrade that touched Docker, traefik, authentik, or the netbird
|
||||
packages), `netbird-mgmt` enters a crash loop. `docker logs netbird-mgmt
|
||||
--tail 30` shows repeated failed attempts to fetch OIDC config from
|
||||
`auth.hubris.network` (connection refused / i/o timeout).
|
||||
|
||||
Root cause: startup ordering race. `netbird-mgmt` tries to fetch its OIDC
|
||||
configuration from `auth.hubris.network` before traefik and authentik are
|
||||
ready to serve. Connection refused → mgmt exits → docker restarts it →
|
||||
same failure.
|
||||
|
||||
## Procedure
|
||||
|
||||
1. Confirm the race (not a real config breakage):
|
||||
```
|
||||
docker logs netbird-mgmt --tail 30 2>&1 | grep -E 'auth.hubris.network|OIDC|connection refused'
|
||||
curl -fsS -o /dev/null -w '%{http_code}' https://auth.hubris.network/application/o/netbird/.well-known/openid-configuration
|
||||
```
|
||||
If the curl now returns 200, the race has already self-resolved — just restart mgmt.
|
||||
2. Wait ~30s for traefik + authentik to finish coming up.
|
||||
3. Restart just the management container:
|
||||
```
|
||||
docker restart netbird-mgmt
|
||||
```
|
||||
4. Verify:
|
||||
```
|
||||
docker ps --filter name=netbird-mgmt --format '{{.Names}} {{.Status}}'
|
||||
docker logs netbird-mgmt --tail 10 2>&1 # should show clean startup, no OIDC errors
|
||||
```
|
||||
5. Check the rest of the stack is healthy too:
|
||||
```
|
||||
docker ps --format 'table {{.Names}}\t{{.Status}}'
|
||||
```
|
||||
|
||||
## Detection signature (for agents)
|
||||
|
||||
After a `run` that upgraded anything Docker/traefik/authentik/netbird on
|
||||
`host:netbird-vps`, run `docker ps` and `docker logs netbird-mgmt --tail 30`.
|
||||
If mgmt is Restarting + logs mention auth.hubris.network OIDC fetch failure,
|
||||
apply this procedure before declaring the upgrade complete.
|
||||
|
||||
Recorded after session 2926de4e (2026-07-15) — 92-package apt upgrade on
|
||||
netbird-vps; mgmt crash-loop caught and fixed with `docker restart
|
||||
netbird-mgmt` after ~30s.
|
||||
tags:
|
||||
- runbook
|
||||
- netbird
|
||||
- docker
|
||||
- gotcha
|
||||
|
||||
Reference in New Issue
Block a user