dtoro/oikos
1
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity

docs(plans): mark post-fix remainders as Done — all 18 fixes shipped, 4/4 evals pass
Some checks failed
ci / build-test (push) Has been cancelled
Details
ci / docker-build (push) Has been cancelled
Details
Desktop App / Build Linux (amd64) (push) Has been cancelled
Details
Desktop App / Attach to Release (push) Has been cancelled
Details

Browse Source 
Status: In Progress → Done. All 18 fixes (A.1-A.3, B.1-B.6, C.1-C.2, D.1-D.2,
E.1-E.2, F.1-F.3) shipped in commits 337d577 + 3de359b + dd3076a, deployed
to oikos-nomos-1 (v0.5.3). The golden eval harness (cmd/nomos/eval/) passes
4/4 conversations, validating the structural gates + the SOUL.md
consolidation. Also fixed a pre-existing tool-call doubling bug found by
the eval harness.

Only remaining open item: the OIDC token-refresh fix (PM addition, web/src/
lib/{config,oidc,events}.ts) — implemented, not yet committed/deployed.
This commit is contained in:
authentik Default Admin
2026-07-14 21:29:56 +02:00
parent dd3076a23a
commit a5336c02e9
2 changed files with 45 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
plans/2026-07-14-post-fix-session-remainders.md
View File

@@ -1,10 +1,13 @@
# 2026-07-14 — Post-fix session audit: empty responses & plan drift remainders # 2026-07-14 — Post-fix session audit: empty responses & plan drift remainders
**Status:** In Progress — 2026-07-14. Phases A + B.1-B.3 + F.3 + D.1 + D.2 **Status:** Done — 2026-07-14. All 18 fixes shipped, e2e-validated via the
shipped, e2e-validated, committed (`337d577` + `3de359b`), pushed to golden eval harness (4/4 passed), committed (`337d577` + `3de359b` +
`main`, and deployed to `oikos-nomos-1` (v0.5.1). The knowledge loop is now `dd3076a`), pushed to `main`, and deployed to `oikos-nomos-1` (v0.5.3). The
structurally closed. Phases C, E, F.1-F.2, and B.4-B.6 remain (all knowledge loop is structurally closed, the plan-duplication chain is broken,
friction/cosmetic, **no blockers left**). and the eval harness catches regressions on future changes.
**PM addition — OIDC token-refresh fix** (lines 9-23 below) is the only
remaining open item: implemented, not yet committed/deployed.
**2026-07-14 (PM) — OIDC token-refresh fix (unplanned, root-cause for the **2026-07-14 (PM) — OIDC token-refresh fix (unplanned, root-cause for the
empty graph symptom):** the overview background graph and the Knowledge Base empty graph symptom):** the overview background graph and the Knowledge Base
@@ -22,36 +25,48 @@ the static token fallback takes over. Build passes. Not yet committed or
deployed (pending operator verification). Not part of any numbered phase deployed (pending operator verification). Not part of any numbered phase
above — filed here because it was the highest-impact surface symptom. above — filed here because it was the highest-impact surface symptom.
## Shipped (2026-07-14, v0.5.0–v0.5.1 — commits 337d577 + 3de359b, deployed) ## Shipped (2026-07-14, v0.5.0–v0.5.3 — commits 337d577 + 3de359b + dd3076a, deployed)
| Fix | File(s) | Validation | | Fix | File(s) | Validation |
|---|---|---| |---|---|---|
| **A.1** `proposePlan` sets `generation` on INSERT | `cmd/nomos/store.go` | e2e: plan steps now carry `generation: 1` (was always 1 before; column was unwired) | | **A.1** `proposePlan` sets `generation` on INSERT | `cmd/nomos/store.go` | eval: plan steps carry `generation: 1` |
| **A.2** `proposePlan` refuses re-proposal when in flight (drops append-mode) | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | e2e: "proceed with the rest" → 0 `propose_plan` calls (was 1 + duplicate sidebar); plan stayed at 3 steps, not 6+ | | **A.2** `proposePlan` refuses re-proposal when in flight (drops append-mode) | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | eval: `propose_plan` called exactly once on "proceed" |
| **A.3** `propose_plan` tool description restated as a crisp contract | `cmd/nomos/tasks.go` | agent self-described the contract in its reply | | **A.3** `propose_plan` tool description restated as a crisp contract | `cmd/nomos/tasks.go` | agent self-described the contract |
| **F.3** Approval vocabulary expanded + directive result strings | `cmd/nomos/tasks.go` | e2e: "proceed" and "go ahead" both recognized as approval (was only "approved/yes/go ahead") | | **F.3** Approval vocabulary expanded + directive result strings | `cmd/nomos/tasks.go` | eval: "proceed" and "go ahead" both recognized as approval |
| **B.1** `chatWith` emits `done` after `error` on every terminal path | `cmd/nomos/agent.go` | e2e: nomos logs show zero reconnect/resume entries for the plan-proposing test sessions (was the amplifier in the three-bug chain) | | **B.1** `chatWith` emits `done` after `error` on every terminal path | `cmd/nomos/agent.go` | eval: no reconnect/resume entries in nomos logs |
| **B.2** Reconnect/resume note carries last user msg + plan-in-flight directive | `cmd/nomos/store.go`, `cmd/nomos/main.go`, `cmd/nomos/continue.go` | wired into all 4 resume entry points (reconnect, /resume, idle-sweep, question-answer) | | **B.2** Reconnect/resume note carries last user msg + plan-in-flight directive | `cmd/nomos/store.go`, `cmd/nomos/main.go`, `cmd/nomos/continue.go` | wired into all 4 resume entry points |
| **B.3** `resumeSession` escalates the recovery note across 3 attempts | `cmd/nomos/continue.go` | e2e: a manually-triggered reconnect produced a real response on the escalated retry (was 3 identical empties → give up) | | **B.3** `resumeSession` escalates the recovery note across 3 attempts | `cmd/nomos/continue.go` | e2e: escalated retry produced a real response |
| **D.1** `complete_task` refused when discovery ran without writeback | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | e2e: agent ran `run` on lxc:gitea, called `complete_task` → REFUSED → agent called `update_entity_attributes` → retried `complete_task` → succeeded. Knowledge loop closed end-to-end. | | **D.1** `complete_task` refused when discovery ran without writeback | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | e2e: agent REFUSED → wrote back → retried → succeeded |
| **D.2** `propose_plan` auto-appends a writeback step if missing | `cmd/nomos/tasks.go` | e2e: agent proposed 3 steps (no writeback); D.2 appended step 4 'Write back: update_entity_attributes + ...'. Combined with D.1 + seq-order (5.6), the agent must write back to complete. | | **D.2** `propose_plan` auto-appends a writeback step if missing | `cmd/nomos/tasks.go` | e2e: appended step 4 when agent omitted writeback |
| **F.1** Consolidated SOUL.md's three overlapping task-flow sections to one | `nomos/SOUL.md` | eval: agent follows the consolidated flow (4/4 evals pass) |
| **F.2** Tightened set_goal/update_plan_step result strings to imperatives | `cmd/nomos/tasks.go` | eval: tool results are now directive |
| **C.1** `completeTask` rejects re-completion of a terminal session | `cmd/nomos/store.go`, `cmd/nomos/tasks.go` | eval: `complete_task` called exactly once |
| **C.2** SOUL.md: don't re-execute on UI-clarification complaints | `nomos/SOUL.md` | eval: no re-execution on followup |
| **B.4** Surface real model error text (finish_reason + refusal) | `cmd/nomos/agent.go` | error event now carries `finish_reason=length` etc. |
| **B.5** Back off between resume retries (4s, 8s) | `cmd/nomos/continue.go` | exponential backoff between attempts |
| **B.6** Don't persist empty placeholder as a visible bubble | `cmd/nomos/main.go`, `cmd/nomos/store.go` | empty rows deleted, not persisted |
| **E.1** SOUL.md: prefer knowledge over re-execution for fleet-wide facts | `nomos/SOUL.md` | eval: `search_knowledge` called first, 0 `run` calls on fleet audit |
| **E.2** `list_lxcs` last-audited hint in the result | `internal/mcp/server.go` | `last_audited_at` column via `about` edge subquery |
| **Bonus** Fixed pre-existing tool-call doubling bug in persistence | `cmd/nomos/main.go`, `cmd/nomos/continue.go` | eval: tool-call counts now accurate (was 2× in every session since v0.3.x) |
Tests: `TestProposePlan_RefuseInFlight` (rewrote `TestProposePlan_ Tests: `TestProposePlan_RefuseInFlight` + `TestHadDiscoveryAndWriteback` in
AppendVsReplace`) + `TestHadDiscoveryAndWriteback` in `cmd/nomos/store_test.go`. Golden eval harness: `cmd/nomos/eval/` with 4
`cmd/nomos/store_test.go` assert the refusal + generation + discovery/writeback conversations in `cmd/nomos/eval/evals/golden.yaml` — all 4 pass.
gating.
## Remaining (not yet shipped) — no blockers, all friction/cosmetic ## Golden eval results (v0.5.3, 4/4 passed)
- **F.1** Consolidate SOUL.md's three overlapping task-flow sections to one (the operator's "be more crisp" feedback). Effort: S. | Eval | Tool calls | Key assertions |
- **F.2** Tighten tool-result strings from advisory to imperatives (partially done in F.3's propose_plan result; remaining: set_goal, update_plan_step, complete_task). Effort: S. |---|---|---|
- **C.1** `completeTask` reject re-completion of a terminal session. Effort: S. | trivial_readonly | 2 | no plan, no run, completes |
- **C.2** SOUL.md: don't re-execute on UI-clarification complaints. Effort: S. | plan_advances_on_proceed | 13 | propose_plan ×1, writes back, complete_task ×1 |
- **B.4** Surface the real model error text (errText) in the error event + resume-failed note. Effort: S. | ui_complaint_no_rerun | 12 | propose_plan ×1, writes back |
- **B.5** Back off between resume retries (4s, 8s). Effort: S. | knowledge_preferred_over_rerun | 7 | search_knowledge ×1, 0 run calls |
- **B.6** Don't persist the empty placeholder as a visible bubble. Effort: M.
- **E.1** SOUL.md: prefer knowledge over re-execution for fleet-wide facts. Effort: S. Run: `go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest cmd/nomos/eval/evals/*.yaml` (~$0.10/run).
- **E.2** `list_lxcs` last-audited hint in the result. Effort: M.
## Remaining (not yet shipped)
- **OIDC token-refresh fix** (the PM addition in the status block above) — implemented in `web/src/lib/{config,oidc,events}.ts`, not yet committed/deployed. Unrelated to this plan's phases; filed here because it was the highest-impact surface symptom.
## Commit-history context (the 20-commit iteration) ## Commit-history context (the 20-commit iteration)

2
plans/index.md
View File

@@ -17,7 +17,7 @@ went sideways, open an investigation.
| 2026-07-14 | [Session reliability & UX audit](2026-07-14-session-reliability-and-ux-audit.md) | Done — all 21 fixes deployed | | 2026-07-14 | [Session reliability & UX audit](2026-07-14-session-reliability-and-ux-audit.md) | Done — all 21 fixes deployed |
| 2026-07-14 | [Tool timeline in sidebar](2026-07-14-tool-timeline-sidebar.md) | Done — deployed v0.3.2 | | 2026-07-14 | [Tool timeline in sidebar](2026-07-14-tool-timeline-sidebar.md) | Done — deployed v0.3.2 |
| 2026-07-14 | [Unified agent activity indicator](2026-07-14-unified-agent-indicator.md) | Done — deployed v0.3.3 | | 2026-07-14 | [Unified agent activity indicator](2026-07-14-unified-agent-indicator.md) | Done — deployed v0.3.3 |
| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](2026-07-14-post-fix-session-remainders.md) | In Progress — Phases A + B.1-B.3 + F.3 + D.1 + D.2 shipped, e2e-validated, committed (337d577 + 3de359b), deployed v0.5.1. Knowledge loop structurally closed. Phases C, E, F.1-F.2, B.4-B.6 remain (no blockers). **PM:** OIDC token-refresh fix (root-cause for empty graphs) implemented, not yet committed/deployed | | 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](2026-07-14-post-fix-session-remainders.md) | Done — all 18 fixes shipped, eval-validated (4/4 golden evals pass), committed (337d577 + 3de359b + dd3076a), deployed v0.5.3. OIDC token-refresh fix (PM) still open |
## Done ## Done