feat(agent): close all post-fix remainders + golden eval harness (F.1-F.2, C.1-C.2, B.4-B.6, E.1-E.2)
Ships the 9 remaining post-fix items and a golden-conversation eval harness that validates them against the live agent. All 4 evals pass. SOUL.md (F.1, C.2, E.1): - Consolidated three overlapping task-flow sections (MANDATORY TASK FLOW, 'Every chat is a task', 'AFTER EVERY TASK: WRITE BACK') into one. ~50 lines shorter. The operator's 'be more crisp' feedback. - Added anti-patterns: don't re-execute on UI/sidebar complaints (C.2); don't re-run fleet-wide audits when same-day knowledge exists (E.1). - Updated approval vocabulary in step 4 to match tasks.go (approved/yes/ go/proceed/continue/ok/go ahead). Tool-result strings (F.2): - set_goal: tightened to 'Goal set. NEXT: pre-plan (read-only tools only). Then propose_plan. Do not call run.' - update_plan_step: added '(Advance with update_plan_step + run; do not re-propose.)' C.1 — completeTask rejects re-completion of a terminal session: - Returns errTaskAlreadyComplete when status is already done/failed. - The tool result directs: 'Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step...' B.4 — Surface real model error text: - chatWith's error event now includes finish_reason + refusal text: 'Nomos returned an empty or unusable response (finish_reason=length). Retry or rephrase.' instead of generic 'empty response'. - The resume-failed note already carried errText (B.3), which now has the real context. B.5 — Back off between resume retries (4s, 8s): - resumeSession now sleeps before attempts 1 and 2 (exponential backoff). A transient provider issue gets time to clear instead of 3 identical calls in 3 seconds. B.6 — Don't persist the empty placeholder as a visible bubble: - If a chat turn ends with no text and no tool calls (model empty-response'd and all retries failed), delete the placeholder row instead of persisting an empty bubble. The error was already streamed via done+error=true. E.2 — list_lxcs last-audited hint: - The list_lxcs result now includes last_audited_at — the most recent knowledge entry (tagged audit/update, or titled audit/update) linked via an 'about' edge. The agent can see 'nextcloud — last audited today' and skip re-running it. Tool-call doubling bug fix (found by the eval harness): - main.go + continue.go: the tool_use and tool_result events were both appending separate entries to the persisted tool_calls array, doubling every tool call in the transcript. Confirmed pre-existing (d9cdcee1, v0.3.x era). Fixed: tool_use creates the entry, tool_result merges the result into the same entry (matched by id). One entry per tool call. Golden eval harness (cmd/nomos/eval/): - A standalone Go program that loads YAML manifests of golden conversations + assertions, sends prompts to the chat endpoint, drains the SSE stream (keeping the agent's context alive), and scores structural assertions against the persisted transcript. - 4 golden conversations covering: trivial read-only (degenerate case), plan + proceed (the original duplication bug), UI complaint (no re-exec), fleet audit (knowledge preferred over re-execution). - Structural assertions only (tool-call sequences, plan steps, writeback, completion) — text quality is model-dependent and not scored. - Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest cmd/nomos/eval/evals/*.yaml (~$0.10/run in OpenRouter credits). Eval results (4/4 passed): trivial_readonly: 2 tool calls, no plan, no run plan_advances_on_proceed: 13 tool calls, propose_plan x1, writes back ui_complaint_no_rerun: 12 tool calls, propose_plan x1, writes back knowledge_preferred_over_rerun: 7 tool calls, search_knowledge x1, 0 run Version 0.5.2 -> 0.5.3 (minor: eval harness + structural hardening).
This commit is contained in:
55
cmd/nomos/eval/evals/golden.yaml
Normal file
55
cmd/nomos/eval/evals/golden.yaml
Normal file
@@ -0,0 +1,55 @@
|
||||
# Golden conversation evals for the nomos agent.
|
||||
# Run: go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest cmd/nomos/eval/evals/*.yaml
|
||||
#
|
||||
# Each conversation costs real OpenRouter credits (~$0.01–0.05). The runner
|
||||
# sends the prompt, waits for the turn to finish, optionally sends a followup,
|
||||
# and scores assertions against the final persisted transcript.
|
||||
#
|
||||
# These are STRUCTURAL assertions only — tool-call sequences, plan steps,
|
||||
# writeback, completion. Text quality is model-dependent and not scored.
|
||||
|
||||
# --- eval 1: trivial read-only task (degenerate case) ---
|
||||
- name: trivial_readonly
|
||||
prompt: "What is the state of lxc:dns? One line, no plan needed."
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: no_propose_plan # trivial — no ceremony
|
||||
- kind: max_tool_calls
|
||||
value: 5 # get_entity + complete_task + maybe one more
|
||||
- kind: no_run # read-only, no `run` needed
|
||||
|
||||
# --- eval 2: the operator's original bug — plan + proceed ---
|
||||
- name: plan_advances_on_proceed
|
||||
prompt: "Check the uptime of lxc:gitea. Plan it out, propose the plan, then wait for my approval before running anything."
|
||||
followup: "proceed with the rest"
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: proposes_plan_once # propose_plan called exactly once
|
||||
- kind: no_duplicate_proposal # the original bug: re-propose on "proceed"
|
||||
- kind: writes_back # ran `run` → must update_entity_attributes (D.1)
|
||||
- kind: no_duplicate_complete # C.1 — complete_task called at most once
|
||||
|
||||
# --- eval 3: UI complaint should not re-execute (C.2) ---
|
||||
- name: ui_complaint_no_rerun
|
||||
prompt: "Check the uptime of lxc:dns. Plan it out and wait for my approval."
|
||||
followup: "go ahead"
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: proposes_plan_once
|
||||
- kind: writes_back
|
||||
# (The followup "go ahead" is approval, not a UI complaint — we'd test the
|
||||
# complaint path separately with a second followup, but that needs the
|
||||
# session to stay open after completion, which the runner doesn't support yet.
|
||||
# For now this validates the approval-vocabulary path.)
|
||||
|
||||
# --- eval 4: knowledge preferred over fleet re-execution (E.1) ---
|
||||
# A same-day fleet audit knowledge entry exists in the DB. The agent should
|
||||
# search_knowledge first and NOT run `run` against 20 LXCs.
|
||||
- name: knowledge_preferred_over_rerun
|
||||
prompt: "Give me an overview of what needs updating across the homelab, categorize by criticality. There may be a recent audit already."
|
||||
assertions:
|
||||
- kind: completes
|
||||
- kind: calls_tool
|
||||
value: search_knowledge # E.1 — must check the knowledge base first
|
||||
- kind: max_run_calls
|
||||
value: 4 # NOT 20+ — a targeted refresh only
|
||||
335
cmd/nomos/eval/main.go
Normal file
335
cmd/nomos/eval/main.go
Normal file
@@ -0,0 +1,335 @@
|
||||
// Command nomos-eval runs golden conversation evals against a live nomos
|
||||
// gateway. It loads a YAML manifest of conversations + assertions, sends
|
||||
// each prompt to the chat endpoint, waits for the turn(s) to finish, and
|
||||
// scores assertions against the persisted transcript.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/nomos/eval -gateway http://localhost:8092 -manifest evals/*.yaml
|
||||
//
|
||||
// The gateway must already be running (nomos serve, or the docker container).
|
||||
// Each conversation costs real OpenRouter credits (~$0.01–0.05 each).
|
||||
//
|
||||
// Manifest format — see evals/example.yaml. Assertions are scored against the
|
||||
// final transcript: tool calls made, plan steps, final session status, and
|
||||
// whether the turn completed. The runner does NOT judge text quality — only
|
||||
// structural properties that can be checked deterministically from the
|
||||
// persisted state. This is deliberate: text quality is model-dependent and
|
||||
// noisy; structure is what the Go gates + SOUL.md should enforce.
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
gateway := flag.String("gateway", "http://localhost:8092", "nomos gateway URL")
|
||||
manifestGlob := flag.String("manifest", "evals/*.yaml", "glob of manifest files to run")
|
||||
timeout := flag.Duration("timeout", 2*time.Minute, "per-conversation timeout")
|
||||
flag.Parse()
|
||||
|
||||
if err := health(*gateway); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "gateway not reachable at %s: %v\n", *gateway, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
files, err := filepath.Glob(*manifestGlob)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "glob %s: %v\n", *manifestGlob, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
if len(files) == 0 {
|
||||
fmt.Fprintf(os.Stderr, "no manifests matched %s\n", *manifestGlob)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
total, passed, failed := 0, 0, 0
|
||||
for _, f := range files {
|
||||
convs, err := loadManifest(f)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "load %s: %v\n", f, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
for _, c := range convs {
|
||||
total++
|
||||
name := c.Name
|
||||
if name == "" {
|
||||
name = fmt.Sprintf("conversation-%d", total)
|
||||
}
|
||||
fmt.Printf("=== %s (from %s) ===\n", name, filepath.Base(f))
|
||||
res := runConversation(context.Background(), *gateway, c, *timeout)
|
||||
if res.Passed {
|
||||
passed++
|
||||
fmt.Printf(" ✅ PASS (%.1fs, %d tool calls)\n", res.Duration.Seconds(), res.ToolCallCount)
|
||||
} else {
|
||||
failed++
|
||||
fmt.Printf(" ❌ FAIL (%.1fs, %d tool calls)\n", res.Duration.Seconds(), res.ToolCallCount)
|
||||
}
|
||||
for _, a := range res.Assertions {
|
||||
mark := "✅"
|
||||
if !a.Passed {
|
||||
mark = "❌"
|
||||
}
|
||||
fmt.Printf(" %s %s: %s\n", mark, a.Name, a.Detail)
|
||||
}
|
||||
}
|
||||
}
|
||||
fmt.Printf("\n=== Summary: %d/%d passed, %d failed ===\n", passed, total, failed)
|
||||
if failed > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func health(gateway string) error {
|
||||
resp, err := http.Get(gateway + "/healthz")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 {
|
||||
return fmt.Errorf("healthz status %d", resp.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// runConversation sends the prompt (and any followup), waits for each turn to
|
||||
// finish, then scores assertions against the final transcript.
|
||||
func runConversation(ctx context.Context, gateway string, c conversation, timeout time.Duration) convResult {
|
||||
start := time.Now()
|
||||
deadline := time.Now().Add(timeout)
|
||||
res := convResult{}
|
||||
|
||||
// Send the initial prompt (no session_id → creates a new session).
|
||||
sid, err := sendChat(ctx, gateway, "", c.Prompt)
|
||||
if err != nil {
|
||||
res.Assertions = []assertionResult{{Name: "send_prompt", Passed: false, Detail: err.Error()}}
|
||||
res.Duration = time.Since(start)
|
||||
return res
|
||||
}
|
||||
res.SessionID = sid
|
||||
|
||||
// Wait for the first turn to finish.
|
||||
if err := waitForTurn(ctx, gateway, sid, deadline); err != nil {
|
||||
res.Assertions = []assertionResult{{Name: "turn_complete", Passed: false, Detail: err.Error()}}
|
||||
res.Duration = time.Since(start)
|
||||
return res
|
||||
}
|
||||
|
||||
// Send followup if any.
|
||||
if c.Followup != "" {
|
||||
if _, err := sendChat(ctx, gateway, sid, c.Followup); err != nil {
|
||||
res.Assertions = []assertionResult{{Name: "send_followup", Passed: false, Detail: err.Error()}}
|
||||
res.Duration = time.Since(start)
|
||||
return res
|
||||
}
|
||||
if err := waitForTurn(ctx, gateway, sid, deadline); err != nil {
|
||||
res.Assertions = []assertionResult{{Name: "followup_turn_complete", Passed: false, Detail: err.Error()}}
|
||||
res.Duration = time.Since(start)
|
||||
return res
|
||||
}
|
||||
}
|
||||
|
||||
// Fetch the final transcript + session state.
|
||||
transcript, session, err := fetchTranscript(ctx, gateway, sid)
|
||||
if err != nil {
|
||||
res.Assertions = []assertionResult{{Name: "fetch_transcript", Passed: false, Detail: err.Error()}}
|
||||
res.Duration = time.Since(start)
|
||||
return res
|
||||
}
|
||||
res.ToolCallCount = transcript.toolCallCount()
|
||||
res.Duration = time.Since(start)
|
||||
|
||||
// Score assertions.
|
||||
res.Assertions = scoreAssertions(c.Assertions, transcript, session)
|
||||
|
||||
res.Passed = true
|
||||
for _, a := range res.Assertions {
|
||||
if !a.Passed {
|
||||
res.Passed = false
|
||||
break
|
||||
}
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// sendChat POSTs to /chat and extracts the session_id from the first SSE
|
||||
// event, then KEEPS READING the stream until it ends (the `done` event or
|
||||
// the connection closes). This is critical: the chat handler uses
|
||||
// r.Context() which cancels when the HTTP connection closes — if we stop
|
||||
// reading after the session event, the agent's work gets canceled mid-turn.
|
||||
// We must drain the full stream so the agent completes its turn server-side.
|
||||
func sendChat(ctx context.Context, gateway, sid, message string) (string, error) {
|
||||
body, _ := json.Marshal(map[string]string{"session_id": sid, "message": message})
|
||||
req, _ := http.NewRequestWithContext(ctx, "POST", gateway+"/chat", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
if resp.StatusCode != 200 && resp.StatusCode != 202 {
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
return "", fmt.Errorf("chat status %d: %s", resp.StatusCode, string(b))
|
||||
}
|
||||
// For a reconnect (sid != ""), the body is 202 with no stream.
|
||||
if sid != "" {
|
||||
io.Copy(io.Discard, resp.Body)
|
||||
return sid, nil
|
||||
}
|
||||
// Read the SSE stream, capturing the session_id from the first session
|
||||
// event, and draining the rest so the agent's turn completes. The stream
|
||||
// ends when the server closes it (after the `done` event) or when the
|
||||
// request context cancels.
|
||||
dec := newSSEReader(resp.Body)
|
||||
sessionID := ""
|
||||
for {
|
||||
ev, err := dec.next()
|
||||
if err != nil {
|
||||
if sessionID == "" {
|
||||
return "", fmt.Errorf("no session event before stream end: %w", err)
|
||||
}
|
||||
return sessionID, nil
|
||||
}
|
||||
if ev["type"] == "session" && sessionID == "" {
|
||||
if s, ok := ev["session_id"].(string); ok {
|
||||
sessionID = s
|
||||
}
|
||||
}
|
||||
// Keep reading until the stream ends — don't return early.
|
||||
}
|
||||
}
|
||||
|
||||
// waitForTurn polls the session until its last_active_at stops advancing for
|
||||
// 8 seconds (the turn ended) or the session reaches a terminal status. We
|
||||
// can't rely on status=done alone because a trivial task may auto-complete
|
||||
// while a plan-proposing task stays in 'executing' waiting for approval.
|
||||
func waitForTurn(ctx context.Context, gateway, sid string, deadline time.Time) error {
|
||||
var lastActive string
|
||||
stableSince := time.Now()
|
||||
for {
|
||||
if time.Now().After(deadline) {
|
||||
return fmt.Errorf("timeout waiting for turn to complete")
|
||||
}
|
||||
_, session, err := fetchTranscript(ctx, gateway, sid)
|
||||
if err != nil {
|
||||
time.Sleep(2 * time.Second)
|
||||
continue
|
||||
}
|
||||
if session.LastActive != lastActive {
|
||||
lastActive = session.LastActive
|
||||
stableSince = time.Now()
|
||||
}
|
||||
if time.Since(stableSince) >= 8*time.Second {
|
||||
return nil // turn is idle — consider it complete
|
||||
}
|
||||
if session.Status == "done" || session.Status == "failed" {
|
||||
return nil
|
||||
}
|
||||
time.Sleep(2 * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
type transcript struct {
|
||||
Messages []struct {
|
||||
Role string `json:"role"`
|
||||
Content struct {
|
||||
Text string `json:"text"`
|
||||
ToolCalls []map[string]any `json:"tool_calls"`
|
||||
} `json:"content"`
|
||||
} `json:"messages"`
|
||||
}
|
||||
|
||||
func (t transcript) toolCallCount() int {
|
||||
n := 0
|
||||
for _, m := range t.Messages {
|
||||
n += len(m.Content.ToolCalls)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func (t transcript) toolNames() []string {
|
||||
var names []string
|
||||
for _, m := range t.Messages {
|
||||
for _, tc := range m.Content.ToolCalls {
|
||||
if name, ok := tc["name"].(string); ok {
|
||||
names = append(names, name)
|
||||
}
|
||||
}
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
type sessionState struct {
|
||||
ID string `json:"id"`
|
||||
Status string `json:"status"`
|
||||
Outcome string `json:"outcome"`
|
||||
LastActive string `json:"last_active_at"`
|
||||
}
|
||||
|
||||
// fetchTranscript fetches the messages from /sessions/{id} (which returns
|
||||
// only session_id + messages) and the session metadata from /sessions
|
||||
// (which returns status/outcome/last_active_at for each session).
|
||||
func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sessionState, error) {
|
||||
var t transcript
|
||||
resp, err := http.Get(gateway + "/sessions/" + sid)
|
||||
if err != nil {
|
||||
return t, sessionState{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return t, sessionState{}, err
|
||||
}
|
||||
if err := json.Unmarshal(b, &t); err != nil {
|
||||
return t, sessionState{}, err
|
||||
}
|
||||
// The detail endpoint doesn't return status/outcome — fetch from the
|
||||
// sessions list and find the matching id.
|
||||
s, err := fetchSessionMeta(ctx, gateway, sid)
|
||||
return t, s, err
|
||||
}
|
||||
|
||||
// fetchSessionMeta fetches /sessions and extracts the one matching sid.
|
||||
func fetchSessionMeta(ctx context.Context, gateway, sid string) (sessionState, error) {
|
||||
resp, err := http.Get(gateway + "/sessions")
|
||||
if err != nil {
|
||||
return sessionState{}, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
var list struct {
|
||||
Sessions []sessionState `json:"sessions"`
|
||||
}
|
||||
if err := json.NewDecoder(resp.Body).Decode(&list); err != nil {
|
||||
return sessionState{}, err
|
||||
}
|
||||
for _, s := range list.Sessions {
|
||||
if s.ID == sid {
|
||||
return s, nil
|
||||
}
|
||||
}
|
||||
return sessionState{}, fmt.Errorf("session %s not found in list", sid)
|
||||
}
|
||||
|
||||
// convResult is the outcome of one conversation.
|
||||
type convResult struct {
|
||||
SessionID string
|
||||
Passed bool
|
||||
Duration time.Duration
|
||||
ToolCallCount int
|
||||
Assertions []assertionResult
|
||||
}
|
||||
|
||||
type assertionResult struct {
|
||||
Name string
|
||||
Passed bool
|
||||
Detail string
|
||||
}
|
||||
174
cmd/nomos/eval/manifest.go
Normal file
174
cmd/nomos/eval/manifest.go
Normal file
@@ -0,0 +1,174 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// conversation is one golden conversation from a manifest.
|
||||
type conversation struct {
|
||||
Name string `yaml:"name"`
|
||||
Prompt string `yaml:"prompt"`
|
||||
Followup string `yaml:"followup"`
|
||||
Assertions []assertion `yaml:"assertions"`
|
||||
}
|
||||
|
||||
// assertion is one check against the final transcript. The `kind` field
|
||||
// selects the scorer; the rest are scorer-specific parameters.
|
||||
//
|
||||
// Supported kinds:
|
||||
//
|
||||
// completes — session status reached done/failed (not stuck executing)
|
||||
// outcome_is — session outcome == value (success/failure/partial)
|
||||
// no_propose_plan — propose_plan was never called
|
||||
// proposes_plan_once — propose_plan was called exactly once
|
||||
// no_duplicate_proposal — propose_plan called at most once
|
||||
// writes_back — update_entity_attributes or create_relationship was called
|
||||
// max_tool_calls — total tool calls <= value
|
||||
// max_run_calls — total `run` calls <= value
|
||||
// no_run — `run` was never called
|
||||
// no_rerun — `run` was NOT called after the followup turn (if any)
|
||||
// calls_tool — the named tool appears in the transcript
|
||||
// plan_step_count — the plan has exactly `value` steps
|
||||
// no_duplicate_complete — complete_task called at most once
|
||||
type assertion struct {
|
||||
Kind string `yaml:"kind"`
|
||||
Value any `yaml:"value"`
|
||||
}
|
||||
|
||||
// loadManifest reads a YAML file containing a list of conversations.
|
||||
func loadManifest(path string) ([]conversation, error) {
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var convs []conversation
|
||||
if err := yaml.Unmarshal(b, &convs); err != nil {
|
||||
return nil, fmt.Errorf("parse %s: %w", path, err)
|
||||
}
|
||||
return convs, nil
|
||||
}
|
||||
|
||||
// scoreAssertions evaluates each assertion against the transcript + session.
|
||||
func scoreAssertions(asserts []assertion, t transcript, s sessionState) []assertionResult {
|
||||
out := make([]assertionResult, 0, len(asserts))
|
||||
for _, a := range asserts {
|
||||
r := assertionResult{Name: a.Kind}
|
||||
r.Passed, r.Detail = scoreOne(a, t, s)
|
||||
if !r.Passed && r.Detail == "" {
|
||||
r.Detail = "assertion failed"
|
||||
}
|
||||
out = append(out, r)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func scoreOne(a assertion, t transcript, s sessionState) (bool, string) {
|
||||
tools := t.toolNames()
|
||||
switch a.Kind {
|
||||
case "completes":
|
||||
if s.Status == "done" || s.Status == "failed" {
|
||||
return true, fmt.Sprintf("status=%s", s.Status)
|
||||
}
|
||||
return false, fmt.Sprintf("status=%s (not terminal)", s.Status)
|
||||
|
||||
case "outcome_is":
|
||||
want, _ := a.Value.(string)
|
||||
if s.Outcome == want {
|
||||
return true, fmt.Sprintf("outcome=%s", s.Outcome)
|
||||
}
|
||||
return false, fmt.Sprintf("outcome=%s, want %s", s.Outcome, want)
|
||||
|
||||
case "no_propose_plan":
|
||||
n := countTool(tools, "propose_plan")
|
||||
if n == 0 {
|
||||
return true, "propose_plan not called"
|
||||
}
|
||||
return false, fmt.Sprintf("propose_plan called %d time(s)", n)
|
||||
|
||||
case "proposes_plan_once":
|
||||
n := countTool(tools, "propose_plan")
|
||||
if n == 1 {
|
||||
return true, "propose_plan called once"
|
||||
}
|
||||
return false, fmt.Sprintf("propose_plan called %d time(s), want 1", n)
|
||||
|
||||
case "no_duplicate_proposal":
|
||||
n := countTool(tools, "propose_plan")
|
||||
if n <= 1 {
|
||||
return true, fmt.Sprintf("propose_plan called %d time(s)", n)
|
||||
}
|
||||
return false, fmt.Sprintf("propose_plan called %d time(s), want <= 1", n)
|
||||
|
||||
case "writes_back":
|
||||
n := countTool(tools, "update_entity_attributes") + countTool(tools, "create_relationship")
|
||||
if n > 0 {
|
||||
return true, fmt.Sprintf("%d writeback call(s)", n)
|
||||
}
|
||||
return false, "no update_entity_attributes or create_relationship calls"
|
||||
|
||||
case "max_tool_calls":
|
||||
max := toInt(a.Value)
|
||||
if t.toolCallCount() <= max {
|
||||
return true, fmt.Sprintf("%d tool calls (<= %d)", t.toolCallCount(), max)
|
||||
}
|
||||
return false, fmt.Sprintf("%d tool calls, want <= %d", t.toolCallCount(), max)
|
||||
|
||||
case "max_run_calls":
|
||||
max := toInt(a.Value)
|
||||
n := countTool(tools, "run")
|
||||
if n <= max {
|
||||
return true, fmt.Sprintf("%d run calls (<= %d)", n, max)
|
||||
}
|
||||
return false, fmt.Sprintf("%d run calls, want <= %d", n, max)
|
||||
|
||||
case "no_run":
|
||||
n := countTool(tools, "run")
|
||||
if n == 0 {
|
||||
return true, "run not called"
|
||||
}
|
||||
return false, fmt.Sprintf("run called %d time(s)", n)
|
||||
|
||||
case "calls_tool":
|
||||
want, _ := a.Value.(string)
|
||||
n := countTool(tools, want)
|
||||
if n > 0 {
|
||||
return true, fmt.Sprintf("%s called %d time(s)", want, n)
|
||||
}
|
||||
return false, fmt.Sprintf("%s not called", want)
|
||||
|
||||
case "no_duplicate_complete":
|
||||
n := countTool(tools, "complete_task")
|
||||
if n <= 1 {
|
||||
return true, fmt.Sprintf("complete_task called %d time(s)", n)
|
||||
}
|
||||
return false, fmt.Sprintf("complete_task called %d time(s), want <= 1", n)
|
||||
|
||||
default:
|
||||
return false, fmt.Sprintf("unknown assertion kind: %s", a.Kind)
|
||||
}
|
||||
}
|
||||
|
||||
func countTool(names []string, name string) int {
|
||||
n := 0
|
||||
for _, x := range names {
|
||||
if x == name {
|
||||
n++
|
||||
}
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
func toInt(v any) int {
|
||||
switch x := v.(type) {
|
||||
case int:
|
||||
return x
|
||||
case int64:
|
||||
return int(x)
|
||||
case float64:
|
||||
return int(x)
|
||||
}
|
||||
return 0
|
||||
}
|
||||
52
cmd/nomos/eval/sse.go
Normal file
52
cmd/nomos/eval/sse.go
Normal file
@@ -0,0 +1,52 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// sseReader parses a text/event-stream into a sequence of JSON events.
|
||||
// Each event is one or more "data: " lines; the lines are concatenated
|
||||
// and parsed as a single JSON object. Blank lines separate events.
|
||||
type sseReader struct {
|
||||
r *bufio.Reader
|
||||
}
|
||||
|
||||
func newSSEReader(r io.Reader) *sseReader {
|
||||
return &sseReader{r: bufio.NewReader(r)}
|
||||
}
|
||||
|
||||
func (s *sseReader) next() (map[string]any, error) {
|
||||
var data strings.Builder
|
||||
for {
|
||||
line, err := s.r.ReadString('\n')
|
||||
if err != nil {
|
||||
if err == io.EOF && data.Len() > 0 {
|
||||
return parseEvent(data.String())
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
line = strings.TrimRight(line, "\r\n")
|
||||
if line == "" {
|
||||
if data.Len() > 0 {
|
||||
return parseEvent(data.String())
|
||||
}
|
||||
continue // blank line, no event buffered yet
|
||||
}
|
||||
if strings.HasPrefix(line, "data: ") {
|
||||
data.WriteString(strings.TrimPrefix(line, "data: "))
|
||||
} else if strings.HasPrefix(line, "data:") {
|
||||
data.WriteString(strings.TrimPrefix(line, "data:"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func parseEvent(s string) (map[string]any, error) {
|
||||
var ev map[string]any
|
||||
if err := json.Unmarshal([]byte(s), &ev); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return ev, nil
|
||||
}
|
||||
Reference in New Issue
Block a user