feat(agent): close all post-fix remainders + golden eval harness (F.1-F.2, C.1-C.2, B.4-B.6, E.1-E.2)
Some checks failed
ci / build-test (push) Has been cancelled
ci / docker-build (push) Has been cancelled
Desktop App / Build Linux (amd64) (push) Has been cancelled
Desktop App / Attach to Release (push) Has been cancelled

Ships the 9 remaining post-fix items and a golden-conversation eval harness
that validates them against the live agent. All 4 evals pass.

SOUL.md (F.1, C.2, E.1):
- Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW,
  'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50
  lines shorter. The operator's 'be more crisp' feedback.
- Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2);
  don't re-run fleet-wide audits when same-day knowledge exists (E.1).
- Updated approval vocabulary in step 4 to match tasks.go (approved/yes/
  go/proceed/continue/ok/go ahead).

Tool-result strings (F.2):
- set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only).
  Then propose_plan. Do not call run.'
- update_plan_step: added '(Advance with update_plan_step + run; do not
  re-propose.)'

C.1 — completeTask rejects re-completion of a terminal session:
- Returns errTaskAlreadyComplete when status is already done/failed.
- The tool result directs: 'Task is already complete. Do not call
  complete_task again. If the operator pointed out a UI/sidebar
  inconsistency, fix it with update_plan_step...'

B.4 — Surface real model error text:
- chatWith's error event now includes finish_reason + refusal text:
  'Nomos returned an empty or unusable response (finish_reason=length).
  Retry or rephrase.' instead of generic 'empty response'.
- The resume-failed note already carried errText (B.3), which now has
  the real context.

B.5 — Back off between resume retries (4s, 8s):
- resumeSession now sleeps before attempts 1 and 2 (exponential backoff).
  A transient provider issue gets time to clear instead of 3 identical
  calls in 3 seconds.

B.6 — Don't persist the empty placeholder as a visible bubble:
- If a chat turn ends with no text and no tool calls (model empty-response'd
  and all retries failed), delete the placeholder row instead of persisting
  an empty bubble. The error was already streamed via done+error=true.

E.2 — list_lxcs last-audited hint:
- The list_lxcs result now includes last_audited_at — the most recent
  knowledge entry (tagged audit/update, or titled audit/update) linked
  via an 'about' edge. The agent can see 'nextcloud — last audited today'
  and skip re-running it.

Tool-call doubling bug fix (found by the eval harness):
- main.go + continue.go: the tool_use and tool_result events were both
  appending separate entries to the persisted tool_calls array, doubling
  every tool call in the transcript. Confirmed pre-existing (d9cdcee1,
  v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the
  result into the same entry (matched by id). One entry per tool call.

Golden eval harness (cmd/nomos/eval/):
- A standalone Go program that loads YAML manifests of golden conversations
  + assertions, sends prompts to the chat endpoint, drains the SSE stream
  (keeping the agent's context alive), and scores structural assertions
  against the persisted transcript.
- 4 golden conversations covering: trivial read-only (degenerate case),
  plan + proceed (the original duplication bug), UI complaint (no re-exec),
  fleet audit (knowledge preferred over re-execution).
- Structural assertions only (tool-call sequences, plan steps, writeback,
  completion) — text quality is model-dependent and not scored.
- Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest
  cmd/nomos/eval/evals/*.yaml  (~$0.10/run in OpenRouter credits).

Eval results (4/4 passed):
  trivial_readonly:              2 tool calls, no plan, no run
  plan_advances_on_proceed:     13 tool calls, propose_plan x1, writes back
  ui_complaint_no_rerun:        12 tool calls, propose_plan x1, writes back
  knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run

Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
This commit is contained in:
2026-07-14 21:27:57 +02:00
parent 0b5b213b2a
commit dd3076a23a
13 changed files with 789 additions and 101 deletions

View File

@@ -227,13 +227,47 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
// without this outer retry the operator would see nothing at all.
cctx, cancel := context.WithTimeout(ctx, 10*time.Minute)
defer cancel()
// B.3: escalate the recovery note across attempts — a transient flake
// needs a different prompt than a model that's stuck no-op'ing. The
// final attempt is maximally directive ("do this specific thing now").
// B.5: back off between retries (4s, 8s) so a transient provider issue
// has time to clear — 3 identical calls in 3 seconds just get 3
// identical empties.
notes := []string{
note, // attempt 0: the original (already enriched per B.2) note
fmt.Sprintf("[System: your previous turn produced no response. %s. Produce a response now — call the next tool or report progress in one sentence.]", note),
fmt.Sprintf("[System: two consecutive empty responses. Stop trying to be clever. The next action is: pick the lowest-pending plan step, mark it running with update_plan_step, and call run for its target. Do that now.]"),
}
for attempt := 0; attempt < 3; attempt++ {
if attempt > 0 {
select {
case <-cctx.Done():
return
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
}
}
toolCalls, finalText, errText = nil, "", ""
emit := func(ev agentEvent) {
if ev.Type == "tool_use" || ev.Type == "tool_result" {
if m, ok := ev.Data.(map[string]any); ok {
m["type"] = ev.Type
toolCalls = append(toolCalls, m)
// One entry per tool call: tool_use creates it,
// tool_result merges the result into the same entry
// (matched by id). Before this fix, both events
// appended separate entries, doubling every tool call.
id, _ := m["id"].(string)
if id != "" && ev.Type == "tool_result" {
for _, existing := range toolCalls {
if eID, _ := existing["id"].(string); eID == id {
for k, v := range m {
existing[k] = v
}
break
}
}
} else {
toolCalls = append(toolCalls, m)
}
}
persist() // live: a poller sees this step land within seconds
}
@@ -244,12 +278,12 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
errText, _ = ev.Data.(string)
}
}
a.chatWith(cctx, sessionID, "", note, emit)
a.chatWith(cctx, sessionID, "", notes[attempt], emit)
if finalText != "" || len(toolCalls) > 0 {
break
}
if attempt == 0 {
slog.Warn("nomos: resume produced nothing, retrying once", "session", sessionID, "error", errText)
if attempt < 2 {
slog.Warn("nomos: resume produced nothing, retrying", "session", sessionID, "error", errText, "attempt", attempt+1)
}
}