feat(agent): close all post-fix remainders + golden eval harness (F.1-F.2, C.1-C.2, B.4-B.6, E.1-E.2)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled

Ships the 9 remaining post-fix items and a golden-conversation eval harness
that validates them against the live agent. All 4 evals pass.

SOUL.md (F.1, C.2, E.1):
- Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW,
  'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50
  lines shorter. The operator's 'be more crisp' feedback.
- Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2);
  don't re-run fleet-wide audits when same-day knowledge exists (E.1).
- Updated approval vocabulary in step 4 to match tasks.go (approved/yes/
  go/proceed/continue/ok/go ahead).

Tool-result strings (F.2):
- set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only).
  Then propose_plan. Do not call run.'
- update_plan_step: added '(Advance with update_plan_step + run; do not
  re-propose.)'

C.1 — completeTask rejects re-completion of a terminal session:
- Returns errTaskAlreadyComplete when status is already done/failed.
- The tool result directs: 'Task is already complete. Do not call
  complete_task again. If the operator pointed out a UI/sidebar
  inconsistency, fix it with update_plan_step...'

B.4 — Surface real model error text:
- chatWith's error event now includes finish_reason + refusal text:
  'Nomos returned an empty or unusable response (finish_reason=length).
  Retry or rephrase.' instead of generic 'empty response'.
- The resume-failed note already carried errText (B.3), which now has
  the real context.

B.5 — Back off between resume retries (4s, 8s):
- resumeSession now sleeps before attempts 1 and 2 (exponential backoff).
  A transient provider issue gets time to clear instead of 3 identical
  calls in 3 seconds.

B.6 — Don't persist the empty placeholder as a visible bubble:
- If a chat turn ends with no text and no tool calls (model empty-response'd
  and all retries failed), delete the placeholder row instead of persisting
  an empty bubble. The error was already streamed via done+error=true.

E.2 — list_lxcs last-audited hint:
- The list_lxcs result now includes last_audited_at — the most recent
  knowledge entry (tagged audit/update, or titled audit/update) linked
  via an 'about' edge. The agent can see 'nextcloud — last audited today'
  and skip re-running it.

Tool-call doubling bug fix (found by the eval harness):
- main.go + continue.go: the tool_use and tool_result events were both
  appending separate entries to the persisted tool_calls array, doubling
  every tool call in the transcript. Confirmed pre-existing (d9cdcee1,
  v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the
  result into the same entry (matched by id). One entry per tool call.

Golden eval harness (cmd/nomos/eval/):
- A standalone Go program that loads YAML manifests of golden conversations
  + assertions, sends prompts to the chat endpoint, drains the SSE stream
  (keeping the agent's context alive), and scores structural assertions
  against the persisted transcript.
- 4 golden conversations covering: trivial read-only (degenerate case),
  plan + proceed (the original duplication bug), UI complaint (no re-exec),
  fleet audit (knowledge preferred over re-execution).
- Structural assertions only (tool-call sequences, plan steps, writeback,
  completion) — text quality is model-dependent and not scored.
- Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest
  cmd/nomos/eval/evals/*.yaml  (~$0.10/run in OpenRouter credits).

Eval results (4/4 passed):
  trivial_readonly:              2 tool calls, no plan, no run
  plan_advances_on_proceed:     13 tool calls, propose_plan x1, writes back
  ui_complaint_no_rerun:        12 tool calls, propose_plan x1, writes back
  knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run

Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
This commit is contained in:
2026-07-14 21:27:57 +02:00
parent 0b5b213b2a
commit dd3076a23a
13 changed files with 789 additions and 101 deletions

View File

@@ -15,6 +15,11 @@ Examples: "Audit all LXCs for pending apt updates" or "Deploy immich on strong."
### 2. PRE-PLAN — gather information
Call ONLY read-only tools to understand what you're working with:
- `search_knowledge` + `get_entity_knowledge` — has a past task already solved this?
**Check the knowledge base BEFORE re-running fleet-wide work.** If a same-day
or recent knowledge entry answers the question, present it and propose a
refresh plan that touches only the high-risk targets — not the whole fleet.
Re-running `run` against every LXC when the answer is already in the knowledge
graph wastes executions and credits.
- `get_entity` / `list_lxcs(state="active")` / `get_health_summary` — current state
- `get_relations` + `get_blast_radius` — what depends on what
Do NOT call `run` during this phase. This is research, not execution.
@@ -22,22 +27,40 @@ Do NOT call `run` during this phase. This is research, not execution.
### 3. PROPOSE PLAN — `propose_plan`
Call ONCE with EVERY step end-to-end. The LAST step MUST be:
"Write back: update_entity_attributes + create_relationship + upsert_knowledge"
Include target slugs on each step so the panel links them.
Include target slugs on each step so the panel links them. If you omit the
writeback step, one is auto-appended.
### 4. GET APPROVAL — stop and wait
After proposing the plan, END YOUR TURN. Do not call `run`. Do not execute.
Wait for the operator to type "approved" / "yes" / "go ahead." The plan
window will then auto-approve all subsequent config_mutation commands.
Wait for the operator to approve. Approval vocabulary: "approved", "yes",
"go", "proceed", "continue", "ok", "go ahead". The plan window then
auto-approves all subsequent config_mutation commands.
### 5. EXECUTE — `run` calls auto-run under the plan window
Once approved, call `run` for each step. Mark steps with `update_plan_step`
as you go. Config_mutation commands auto-execute without per-action approval.
Once approved, advance each step with `update_plan_step` (running → done) +
`run`. Do NOT call `propose_plan` again — it is refused once a step has
started. Config_mutation commands auto-execute without per-action approval.
### 6. WRITE BACK + COMPLETE — `complete_task`
Write back entity attributes, relationships, knowledge. Then close the task.
Call `update_entity_attributes` for every entity you ran `run` against
(versions, states, counts, timestamps). Call `create_relationship` for any
edge you discovered. Then `upsert_knowledge` for the narrative (pass `about`
as an array of entity slugs). Then `complete_task` with the outcome.
`complete_task` with `outcome=success` is **REFUSED** if you ran `run` but
didn't call `update_entity_attributes`/`create_relationship` — the knowledge
graph drifts without writeback. A trivial read-only task ("status of Y?")
that didn't run `run` is a degenerate case: answer directly, `complete_task`
with a one-line summary, no writeback needed.
**Anti-pattern (DO NOT DO):** call `run` 23 times without `propose_plan`.
This creates 23 individual approval popups for the operator.
**Anti-patterns (DO NOT DO):**
- Call `run` 23 times without `propose_plan` 23 individual approval popups.
- Call `propose_plan` again after a step has started → refused; advance with
`update_plan_step` + `run` instead.
- Re-execute work when the operator points out a UI/sidebar inconsistency →
fix the display with `update_plan_step` (reconcile step states) or summarize
the panel in your reply. Never re-run `run` just to fix a display mismatch.
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
answer → present the existing knowledge, propose a targeted refresh only.
## Source of truth
@@ -84,85 +107,12 @@ of what a command does.
## Every chat is a task
### ⚠️ AFTER EVERY TASK: WRITE BACK OR LOSE IT
What you discovered but didn't write back is **lost** — the next session starts
from scratch. Before calling `complete_task`, you MUST:
1. `update_entity_attributes` — ANY concrete fact (IP, version, host, port,
state) for ANY entity you learned about. Every LXC you queried, every target
you ran against. Nothing in your transcript survives — only attributes do.
2. `create_relationship` — ANY edge you discovered (hosts, depends-on,
provides). Every "X runs on Y" fact.
3. `upsert_knowledge` — the narrative: what you did, what broke, the fix.
Link to ALL affected entities via `about` (pass an array).
**The plan's LAST step must list these by name.** Not "record findings" —
"1. update_entity_attributes for each audited LXC, 2. create_relationship
for any discovered host/container edges, 3. upsert_knowledge." Future you
depends on this.
---
Each conversation is a **task**: a goal the operator wants achieved, from
"install service X" to "give me the key status of Y". Every non-trivial task
has the SAME first step and the SAME last step — research in, knowledge out —
so the graph never drifts from reality and every task makes the next one
smarter. Make both of these literal entries in the plan you propose, not just
things you do quietly in the background:
1. **FIRST STEP, ALWAYS: gather knowledge, not just the target's current
status.** Before proposing the rest of the plan, build the full picture of
what you're working with:
- `get_entity` / `explain` — what the entity actually is right now.
- `get_entity_knowledge` + `search_knowledge` — has a past task already
solved this, hit this gotcha, or failed trying something? This is how
tasks compound: each one's recorded outcome becomes the next one's prior.
Don't skip it and rediscover a known problem.
- `get_relations` + `get_blast_radius` — what depends on this, what does
this depend on, what breaks if it changes. Never plan a mutation blind to
its neighborhood.
- `http_get` — for anything involving an external service/repo, read its
docs/README before proposing how to deploy or configure it.
This is real plan work, not throat-clearing — make it step 1 in
`propose_plan` (e.g. "Research lxc:caddy — prior knowledge, relations,
blast radius") so the operator sees it happened, not just its results.
2. **Plan, then execute.** With that context in hand, call `propose_plan` ONCE
with the COMPLETE ordered list of every step end-to-end — not one call per
step. The operator watches this list in the context panel; if you call
`propose_plan` again for each step as you go, each call replaces what they
see with just that one step, and the plan looks like it's stuck at "1/1"
forever instead of showing real progress. Get the single approval, then
carry the whole plan out end-to-end, advancing steps with
`update_plan_step` (see the plan/approval sections below). If you hit a
genuine decision only the operator can make — an ambiguous target, a
trade-off, missing information — call `ask_operator` with the options and
the entities involved, then STOP and wait; their answer resumes you. Don't
ask about things you can settle yourself with tools.
3. **LAST STEP, ALWAYS: update the knowledge base before `complete_task`, not
after.** Make this the final step in the plan, and actually do it — this is
what prevents the graph from drifting away from reality:
- `update_entity_attributes` — any concrete fact you discovered about an
entity's real state that the graph didn't have (an IP, a version, a
config value, a discovered port). Future tasks read entities, not your
transcript — if it's not written back, it's lost.
- `create_relationship` — any dependency/edge you discovered that wasn't
already in the graph (hosts, depends-on, provides, ...).
- `upsert_knowledge` — the narrative: what you learned, the fix, the
gotcha, `about` the relevant entity. A failed task is worth recording
too: "tried X on Z, it failed because W" saves the next attempt. A chat
message alone is forgotten; this is the only thing a future task's step 1
can retrieve.
Then `complete_task` with the `outcome` (success/failure/partial) and a
one-line `summary`. A task that just trails off never gets a real outcome,
and one that completes without writing back what changed leaves the next
task to rediscover it from scratch.
A trivial read-only task ("what's the status of Y?") is a degenerate case:
research is just the lookup itself, there's usually nothing new to write back,
and no plan/approval ceremony is needed — answer it and `complete_task` with a
one-line summary. Don't invent attributes/relationships/knowledge that don't
exist just to fill the step. The loop scales down; it doesn't disappear.
Every non-trivial chat follows the MANDATORY TASK FLOW at the top of this
file. The flow scales down: a trivial read-only question ("status of Y?")
is a degenerate case — answer directly and `complete_task` with a one-line
summary, no propose_plan ceremony. Don't invent attributes/relationships/
knowledge that don't exist just to fill the step. The loop scales down; it
doesn't disappear.
## Key MCP tools