feat(nomos): retry cap, vm: targets, inspect_path, goal supersession, runbooks
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
ci / web (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled

Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.

P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.

P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.

P1.3 — two new runbook entities in seeds/knowledge.yaml:
  - nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
    killall → exportfs -u → mutate → exportfs -a → verify)
  - netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
    after ~30s for the traefik/authentik OIDC race)

P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.

P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).

P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.

Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
This commit is contained in:
2026-07-19 00:09:39 +02:00
parent bd44626532
commit 544afae77f
12 changed files with 1265 additions and 19 deletions

View File

@@ -6713,3 +6713,143 @@ runbooks:
tags:
- skill
- runbook
- slug: nfs-exported-dir-mutation-hang
name: Mutating an actively-exported NFS directory hangs at fchownat
risk_class: config_mutation
entity_type: host
procedure: {}
content: |
---
name: nfs-exported-dir-mutation-hang
risk_class: config_mutation
inputs: [target_host, exported_path, mutation_command]
verification: "stat -c '%a %U:%G' <exported_path> on the target host"
docs_update_checklist: [investigation_entry]
---
# Mutating an actively-exported NFS directory hangs at fchownat
Symptom: a `chown` / `chgrp` / `chmod` against a directory that is
actively exported via `nfs-kernel-server` (knfsd) hangs indefinitely — the
command appears to run but never returns. SSH/gateway time out waiting for it.
`ps aux | grep chown` shows the process in interruptible sleep (D state);
repeated retries pile up zombies (25+ observed in one session).
Root cause: knfsd holds a kernel lock on the directory while it's exported.
`fchownat()` blocks waiting for the lock. This is NOT a gateway or SSH issue —
raising the timeout just makes the hang longer.
## Procedure
1. **Clear the zombies** from prior failed attempts:
```
killall -9 chgrp chown chmod 2>/dev/null
```
2. **Temporarily unexport** the path for each client/network that has it exported:
```
exportfs -u <client>:<path> # e.g. exportfs -u 192.168.8.0/24:/mnt/media_local
```
3. **Apply the mutation** (now that knfsd has released the lock):
```
chown :<gid> <path> && chmod <mode> <path>
```
4. **Re-export** (restore the exports):
```
exportfs -a
```
5. **Verify** from an NFS client that the new permissions are visible and operations work end-to-end:
```
stat -c '%a %U:%G' /media/<mountpoint> # on a client
touch /media/<mountpoint>/.test && mv /media/<mountpoint>/.test /media/<mountpoint>/.moved && rm /media/<mountpoint>/.moved
```
## Detection signature (for agents)
A `run` call against a host that contains `chown|chgrp|chmod` of a path
exported by `nfs-kernel-server` AND the call times out → assume this runbook.
Don't retry the same command; run `strace -f <cmd>` (it will block at
`fchownat`) to confirm, then apply the procedure above.
## Caveats
- `exportfs -u` may emit a format-mismatch warning if the export was defined
via `/etc/exports` with a different option string than `exportfs -v`
reports. The unexport still succeeds; verify with `exportfs -v` afterward
that the path is gone, then re-add with `exportfs -a`.
- This applies to ANY mutating op on the exported dir (chown, chmod, rename,
rmdir of the root). Subdirectory mutations are fine as long as they don't
touch the exported root itself.
Recorded after session 1e9c7691 (2026-07-18) — 20+ retries of a
`chown :10000 /mnt/media_local` that hung for 30+ minutes before this
procedure was identified.
tags:
- runbook
- nfs
- knfsd
- gotcha
- slug: netbird-mgmt-oidc-race-after-upgrade
name: netbird-mgmt crash-loops after stack upgrade (OIDC race)
risk_class: reversible_low
entity_type: host
procedure: {}
content: |
---
name: netbird-mgmt-oidc-race-after-upgrade
risk_class: reversible_low
inputs: []
verification: "docker ps --filter name=netbird-mgmt --format '{{.Status}}' shows Up"
docs_update_checklist: [investigation_entry]
---
# netbird-mgmt crash-loops after stack upgrade (OIDC race)
Symptom: after a full-stack restart on `host:netbird-vps` (e.g. following an
apt upgrade that touched Docker, traefik, authentik, or the netbird
packages), `netbird-mgmt` enters a crash loop. `docker logs netbird-mgmt
--tail 30` shows repeated failed attempts to fetch OIDC config from
`auth.hubris.network` (connection refused / i/o timeout).
Root cause: startup ordering race. `netbird-mgmt` tries to fetch its OIDC
configuration from `auth.hubris.network` before traefik and authentik are
ready to serve. Connection refused → mgmt exits → docker restarts it →
same failure.
## Procedure
1. Confirm the race (not a real config breakage):
```
docker logs netbird-mgmt --tail 30 2>&1 | grep -E 'auth.hubris.network|OIDC|connection refused'
curl -fsS -o /dev/null -w '%{http_code}' https://auth.hubris.network/application/o/netbird/.well-known/openid-configuration
```
If the curl now returns 200, the race has already self-resolved — just restart mgmt.
2. Wait ~30s for traefik + authentik to finish coming up.
3. Restart just the management container:
```
docker restart netbird-mgmt
```
4. Verify:
```
docker ps --filter name=netbird-mgmt --format '{{.Names}} {{.Status}}'
docker logs netbird-mgmt --tail 10 2>&1 # should show clean startup, no OIDC errors
```
5. Check the rest of the stack is healthy too:
```
docker ps --format 'table {{.Names}}\t{{.Status}}'
```
## Detection signature (for agents)
After a `run` that upgraded anything Docker/traefik/authentik/netbird on
`host:netbird-vps`, run `docker ps` and `docker logs netbird-mgmt --tail 30`.
If mgmt is Restarting + logs mention auth.hubris.network OIDC fetch failure,
apply this procedure before declaring the upgrade complete.
Recorded after session 2926de4e (2026-07-15) — 92-package apt upgrade on
netbird-vps; mgmt crash-loop caught and fixed with `docker restart
netbird-mgmt` after ~30s.
tags:
- runbook
- netbird
- docker
- gotcha