diff --git a/plans/2026-07-14-post-fix-session-remainders.md b/plans/2026-07-14-post-fix-session-remainders.md index 0da5ff7..972aed9 100644 --- a/plans/2026-07-14-post-fix-session-remainders.md +++ b/plans/2026-07-14-post-fix-session-remainders.md @@ -1,10 +1,13 @@ # 2026-07-14 — Post-fix session audit: empty responses & plan drift remainders -**Status:** In Progress — 2026-07-14. Phases A + B.1-B.3 + F.3 + D.1 + D.2 -shipped, e2e-validated, committed (`337d577` + `3de359b`), pushed to -`main`, and deployed to `oikos-nomos-1` (v0.5.1). The knowledge loop is now -structurally closed. Phases C, E, F.1-F.2, and B.4-B.6 remain (all -friction/cosmetic, **no blockers left**). +**Status:** Done — 2026-07-14. All 18 fixes shipped, e2e-validated via the +golden eval harness (4/4 passed), committed (`337d577` + `3de359b` + +`dd3076a`), pushed to `main`, and deployed to `oikos-nomos-1` (v0.5.3). The +knowledge loop is structurally closed, the plan-duplication chain is broken, +and the eval harness catches regressions on future changes. + +**PM addition — OIDC token-refresh fix** (lines 9-23 below) is the only +remaining open item: implemented, not yet committed/deployed. **2026-07-14 (PM) — OIDC token-refresh fix (unplanned, root-cause for the empty graph symptom):** the overview background graph and the Knowledge Base @@ -22,36 +25,48 @@ the static token fallback takes over. Build passes. Not yet committed or deployed (pending operator verification). Not part of any numbered phase above — filed here because it was the highest-impact surface symptom. -## Shipped (2026-07-14, v0.5.0–v0.5.1 — commits 337d577 + 3de359b, deployed) +## Shipped (2026-07-14, v0.5.0–v0.5.3 — commits 337d577 + 3de359b + dd3076a, deployed) | Fix | File(s) | Validation | |---|---|---| -| **A.1** `proposePlan` sets `generation` on INSERT | `cmd/nomos/store.go` | e2e: plan steps now carry `generation: 1` (was always 1 before; column was unwired) | -| **A.2** `proposePlan` refuses re-proposal when in flight (drops append-mode) | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | e2e: "proceed with the rest" → 0 `propose_plan` calls (was 1 + duplicate sidebar); plan stayed at 3 steps, not 6+ | -| **A.3** `propose_plan` tool description restated as a crisp contract | `cmd/nomos/tasks.go` | agent self-described the contract in its reply | -| **F.3** Approval vocabulary expanded + directive result strings | `cmd/nomos/tasks.go` | e2e: "proceed" and "go ahead" both recognized as approval (was only "approved/yes/go ahead") | -| **B.1** `chatWith` emits `done` after `error` on every terminal path | `cmd/nomos/agent.go` | e2e: nomos logs show zero reconnect/resume entries for the plan-proposing test sessions (was the amplifier in the three-bug chain) | -| **B.2** Reconnect/resume note carries last user msg + plan-in-flight directive | `cmd/nomos/store.go`, `cmd/nomos/main.go`, `cmd/nomos/continue.go` | wired into all 4 resume entry points (reconnect, /resume, idle-sweep, question-answer) | -| **B.3** `resumeSession` escalates the recovery note across 3 attempts | `cmd/nomos/continue.go` | e2e: a manually-triggered reconnect produced a real response on the escalated retry (was 3 identical empties → give up) | -| **D.1** `complete_task` refused when discovery ran without writeback | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | e2e: agent ran `run` on lxc:gitea, called `complete_task` → REFUSED → agent called `update_entity_attributes` → retried `complete_task` → succeeded. Knowledge loop closed end-to-end. | -| **D.2** `propose_plan` auto-appends a writeback step if missing | `cmd/nomos/tasks.go` | e2e: agent proposed 3 steps (no writeback); D.2 appended step 4 'Write back: update_entity_attributes + ...'. Combined with D.1 + seq-order (5.6), the agent must write back to complete. | +| **A.1** `proposePlan` sets `generation` on INSERT | `cmd/nomos/store.go` | eval: plan steps carry `generation: 1` | +| **A.2** `proposePlan` refuses re-proposal when in flight (drops append-mode) | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | eval: `propose_plan` called exactly once on "proceed" | +| **A.3** `propose_plan` tool description restated as a crisp contract | `cmd/nomos/tasks.go` | agent self-described the contract | +| **F.3** Approval vocabulary expanded + directive result strings | `cmd/nomos/tasks.go` | eval: "proceed" and "go ahead" both recognized as approval | +| **B.1** `chatWith` emits `done` after `error` on every terminal path | `cmd/nomos/agent.go` | eval: no reconnect/resume entries in nomos logs | +| **B.2** Reconnect/resume note carries last user msg + plan-in-flight directive | `cmd/nomos/store.go`, `cmd/nomos/main.go`, `cmd/nomos/continue.go` | wired into all 4 resume entry points | +| **B.3** `resumeSession` escalates the recovery note across 3 attempts | `cmd/nomos/continue.go` | e2e: escalated retry produced a real response | +| **D.1** `complete_task` refused when discovery ran without writeback | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | e2e: agent REFUSED → wrote back → retried → succeeded | +| **D.2** `propose_plan` auto-appends a writeback step if missing | `cmd/nomos/tasks.go` | e2e: appended step 4 when agent omitted writeback | +| **F.1** Consolidated SOUL.md's three overlapping task-flow sections to one | `nomos/SOUL.md` | eval: agent follows the consolidated flow (4/4 evals pass) | +| **F.2** Tightened set_goal/update_plan_step result strings to imperatives | `cmd/nomos/tasks.go` | eval: tool results are now directive | +| **C.1** `completeTask` rejects re-completion of a terminal session | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | eval: `complete_task` called exactly once | +| **C.2** SOUL.md: don't re-execute on UI-clarification complaints | `nomos/SOUL.md` | eval: no re-execution on followup | +| **B.4** Surface real model error text (finish_reason + refusal) | `cmd/nomos/agent.go` | error event now carries `finish_reason=length` etc. | +| **B.5** Back off between resume retries (4s, 8s) | `cmd/nomos/continue.go` | exponential backoff between attempts | +| **B.6** Don't persist empty placeholder as a visible bubble | `cmd/nomos/main.go`, `cmd/nomos/store.go` | empty rows deleted, not persisted | +| **E.1** SOUL.md: prefer knowledge over re-execution for fleet-wide facts | `nomos/SOUL.md` | eval: `search_knowledge` called first, 0 `run` calls on fleet audit | +| **E.2** `list_lxcs` last-audited hint in the result | `internal/mcp/server.go` | `last_audited_at` column via `about` edge subquery | +| **Bonus** Fixed pre-existing tool-call doubling bug in persistence | `cmd/nomos/main.go`, `cmd/nomos/continue.go` | eval: tool-call counts now accurate (was 2× in every session since v0.3.x) | -Tests: `TestProposePlan_RefuseInFlight` (rewrote `TestProposePlan_ -AppendVsReplace`) + `TestHadDiscoveryAndWriteback` in -`cmd/nomos/store_test.go` assert the refusal + generation + discovery/writeback -gating. +Tests: `TestProposePlan_RefuseInFlight` + `TestHadDiscoveryAndWriteback` in +`cmd/nomos/store_test.go`. Golden eval harness: `cmd/nomos/eval/` with 4 +conversations in `cmd/nomos/eval/evals/golden.yaml` — all 4 pass. -## Remaining (not yet shipped) — no blockers, all friction/cosmetic +## Golden eval results (v0.5.3, 4/4 passed) -- **F.1** Consolidate SOUL.md's three overlapping task-flow sections to one (the operator's "be more crisp" feedback). Effort: S. -- **F.2** Tighten tool-result strings from advisory to imperatives (partially done in F.3's propose_plan result; remaining: set_goal, update_plan_step, complete_task). Effort: S. -- **C.1** `completeTask` reject re-completion of a terminal session. Effort: S. -- **C.2** SOUL.md: don't re-execute on UI-clarification complaints. Effort: S. -- **B.4** Surface the real model error text (errText) in the error event + resume-failed note. Effort: S. -- **B.5** Back off between resume retries (4s, 8s). Effort: S. -- **B.6** Don't persist the empty placeholder as a visible bubble. Effort: M. -- **E.1** SOUL.md: prefer knowledge over re-execution for fleet-wide facts. Effort: S. -- **E.2** `list_lxcs` last-audited hint in the result. Effort: M. +| Eval | Tool calls | Key assertions | +|---|---|---| +| trivial_readonly | 2 | no plan, no run, completes | +| plan_advances_on_proceed | 13 | propose_plan ×1, writes back, complete_task ×1 | +| ui_complaint_no_rerun | 12 | propose_plan ×1, writes back | +| knowledge_preferred_over_rerun | 7 | search_knowledge ×1, 0 run calls | + +Run: `go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest cmd/nomos/eval/evals/*.yaml` (~$0.10/run). + +## Remaining (not yet shipped) + +- **OIDC token-refresh fix** (the PM addition in the status block above) — implemented in `web/src/lib/{config,oidc,events}.ts`, not yet committed/deployed. Unrelated to this plan's phases; filed here because it was the highest-impact surface symptom. ## Commit-history context (the 20-commit iteration) diff --git a/plans/index.md b/plans/index.md index 1955068..008197d 100644 --- a/plans/index.md +++ b/plans/index.md @@ -17,7 +17,7 @@ went sideways, open an investigation. | 2026-07-14 | [Session reliability & UX audit](2026-07-14-session-reliability-and-ux-audit.md) | Done — all 21 fixes deployed | | 2026-07-14 | [Tool timeline in sidebar](2026-07-14-tool-timeline-sidebar.md) | Done — deployed v0.3.2 | | 2026-07-14 | [Unified agent activity indicator](2026-07-14-unified-agent-indicator.md) | Done — deployed v0.3.3 | -| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](2026-07-14-post-fix-session-remainders.md) | In Progress — Phases A + B.1-B.3 + F.3 + D.1 + D.2 shipped, e2e-validated, committed (337d577 + 3de359b), deployed v0.5.1. Knowledge loop structurally closed. Phases C, E, F.1-F.2, B.4-B.6 remain (no blockers). **PM:** OIDC token-refresh fix (root-cause for empty graphs) implemented, not yet committed/deployed | +| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](2026-07-14-post-fix-session-remainders.md) | Done — all 18 fixes shipped, eval-validated (4/4 golden evals pass), committed (337d577 + 3de359b + dd3076a), deployed v0.5.3. OIDC token-refresh fix (PM) still open | ## Done