dtoro
dd3076a23a
Ships the 9 remaining post-fix items and a golden-conversation eval harness that validates them against the live agent. All 4 evals pass. SOUL.md (F.1, C.2, E.1): - Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW, 'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50 lines shorter. The operator's 'be more crisp' feedback. - Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2); don't re-run fleet-wide audits when same-day knowledge exists (E.1). - Updated approval vocabulary in step 4 to match tasks.go (approved/yes/ go/proceed/continue/ok/go ahead). Tool-result strings (F.2): - set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only). Then propose_plan. Do not call run.' - update_plan_step: added '(Advance with update_plan_step + run; do not re-propose.)' C.1 — completeTask rejects re-completion of a terminal session: - Returns errTaskAlreadyComplete when status is already done/failed. - The tool result directs: 'Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step...' B.4 — Surface real model error text: - chatWith's error event now includes finish_reason + refusal text: 'Nomos returned an empty or unusable response (finish_reason=length). Retry or rephrase.' instead of generic 'empty response'. - The resume-failed note already carried errText (B.3), which now has the real context. B.5 — Back off between resume retries (4s, 8s): - resumeSession now sleeps before attempts 1 and 2 (exponential backoff). A transient provider issue gets time to clear instead of 3 identical calls in 3 seconds. B.6 — Don't persist the empty placeholder as a visible bubble: - If a chat turn ends with no text and no tool calls (model empty-response'd and all retries failed), delete the placeholder row instead of persisting an empty bubble. The error was already streamed via done+error=true. E.2 — list_lxcs last-audited hint: - The list_lxcs result now includes last_audited_at — the most recent knowledge entry (tagged audit/update, or titled audit/update) linked via an 'about' edge. The agent can see 'nextcloud — last audited today' and skip re-running it. Tool-call doubling bug fix (found by the eval harness): - main.go + continue.go: the tool_use and tool_result events were both appending separate entries to the persisted tool_calls array, doubling every tool call in the transcript. Confirmed pre-existing (d9cdcee1, v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the result into the same entry (matched by id). One entry per tool call. Golden eval harness (cmd/nomos/eval/): - A standalone Go program that loads YAML manifests of golden conversations + assertions, sends prompts to the chat endpoint, drains the SSE stream (keeping the agent's context alive), and scores structural assertions against the persisted transcript. - 4 golden conversations covering: trivial read-only (degenerate case), plan + proceed (the original duplication bug), UI complaint (no re-exec), fleet audit (knowledge preferred over re-execution). - Structural assertions only (tool-call sequences, plan steps, writeback, completion) — text quality is model-dependent and not scored. - Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest cmd/nomos/eval/evals/*.yaml (~$0.10/run in OpenRouter credits). Eval results (4/4 passed): trivial_readonly: 2 tool calls, no plan, no run plan_advances_on_proceed: 13 tool calls, propose_plan x1, writes back ui_complaint_no_rerun: 12 tool calls, propose_plan x1, writes back knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
56 lines
2.7 KiB
YAML
56 lines
2.7 KiB
YAML
# Golden conversation evals for the nomos agent.
|
||
# Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest cmd/nomos/eval/evals/*.yaml
|
||
#
|
||
# Each conversation costs real OpenRouter credits (~$0.01–0.05). The runner
|
||
# sends the prompt, waits for the turn to finish, optionally sends a followup,
|
||
# and scores assertions against the final persisted transcript.
|
||
#
|
||
# These are STRUCTURAL assertions only — tool-call sequences, plan steps,
|
||
# writeback, completion. Text quality is model-dependent and not scored.
|
||
|
||
# --- eval 1: trivial read-only task (degenerate case) ---
|
||
- name: trivial_readonly
|
||
prompt: "What is the state of lxc:dns? One line, no plan needed."
|
||
assertions:
|
||
- kind: completes
|
||
- kind: no_propose_plan # trivial — no ceremony
|
||
- kind: max_tool_calls
|
||
value: 5 # get_entity + complete_task + maybe one more
|
||
- kind: no_run # read-only, no `run` needed
|
||
|
||
# --- eval 2: the operator's original bug — plan + proceed ---
|
||
- name: plan_advances_on_proceed
|
||
prompt: "Check the uptime of lxc:gitea. Plan it out, propose the plan, then wait for my approval before running anything."
|
||
followup: "proceed with the rest"
|
||
assertions:
|
||
- kind: completes
|
||
- kind: proposes_plan_once # propose_plan called exactly once
|
||
- kind: no_duplicate_proposal # the original bug: re-propose on "proceed"
|
||
- kind: writes_back # ran `run` → must update_entity_attributes (D.1)
|
||
- kind: no_duplicate_complete # C.1 — complete_task called at most once
|
||
|
||
# --- eval 3: UI complaint should not re-execute (C.2) ---
|
||
- name: ui_complaint_no_rerun
|
||
prompt: "Check the uptime of lxc:dns. Plan it out and wait for my approval."
|
||
followup: "go ahead"
|
||
assertions:
|
||
- kind: completes
|
||
- kind: proposes_plan_once
|
||
- kind: writes_back
|
||
# (The followup "go ahead" is approval, not a UI complaint — we'd test the
|
||
# complaint path separately with a second followup, but that needs the
|
||
# session to stay open after completion, which the runner doesn't support yet.
|
||
# For now this validates the approval-vocabulary path.)
|
||
|
||
# --- eval 4: knowledge preferred over fleet re-execution (E.1) ---
|
||
# A same-day fleet audit knowledge entry exists in the DB. The agent should
|
||
# search_knowledge first and NOT run `run` against 20 LXCs.
|
||
- name: knowledge_preferred_over_rerun
|
||
prompt: "Give me an overview of what needs updating across the homelab, categorize by criticality. There may be a recent audit already."
|
||
assertions:
|
||
- kind: completes
|
||
- kind: calls_tool
|
||
value: search_knowledge # E.1 — must check the knowledge base first
|
||
- kind: max_run_calls
|
||
value: 4 # NOT 20+ — a targeted refresh only
|