83 lines
3.2 KiB
Markdown
83 lines
3.2 KiB
Markdown
---
|
|
name: session-review
|
|
description: "Examine a Nomos chat session, compare the user's objective with the actual outcome, identify causes of failure (missing tools, excessive tool calls, blocked actions, model behavior), and propose concrete fixes."
|
|
risk_class: reversible_low
|
|
inputs: [session_id]
|
|
---
|
|
# Session review
|
|
|
|
Analyze Nomos chat sessions from the live database, diff objectives
|
|
against outcomes, and propose fixes.
|
|
|
|
## 1. Retrieve session data
|
|
|
|
```bash
|
|
# List recent sessions
|
|
curl -s http://localhost:8092/sessions | jq '.sessions[:5]'
|
|
|
|
# Fetch one session with messages
|
|
curl -s http://localhost:8092/sessions/{session_id} | jq .
|
|
```
|
|
|
|
## 2. Classify the session
|
|
|
|
For each session determine:
|
|
|
|
| Dimension | Check |
|
|
|-----------|-------|
|
|
| Objective | What was the user trying to accomplish? |
|
|
| Outcome | Was it achieved? (read final assistant text) |
|
|
| Tool calls | Count, unique tools, redundancy (e.g., N+1 fan-out) |
|
|
| Blockers | Missing action? Missing tool? Model refusal? Empty response? |
|
|
| User frustration | Did the user need to clarify/correct/repeat? |
|
|
| Message sizes | Content blob sizes — truncation needed? |
|
|
|
|
## 3. Key failure signatures
|
|
|
|
| Signature | Root cause | Fix |
|
|
|-----------|-----------|-----|
|
|
| Agent: "I can't run X" | Missing target or capability | Use `run` with shell command — there is no fixed action enum anymore |
|
|
| Agent: "No local knowledge on that" + no web tool | Missing `http_get` / web fetch MCP tool | Add MCP tool |
|
|
| Empty assistant bubble (text="", no tools) | Model returned blank completion | Retry + error surfacing |
|
|
| Non-English boilerplate refusal | Flash-tier model degradation | Response quality guard |
|
|
| >30 tool calls per turn, same tool repeated | N+1 fan-out instead of bulk tool | Enrich bulk tools + tighten SOUL.md |
|
|
| Message >50KB in DB | Raw tool results persisted verbatim | Truncation in `store.go` |
|
|
|
|
## 4. Extract patterns across sessions
|
|
|
|
```bash
|
|
# All sessions summary
|
|
curl -s http://localhost:8092/sessions | jq -r '.sessions[] | "\(.id[:8]) \(.title[:80]) \(.created_at[:16])"'
|
|
|
|
# Message count + tool count per session
|
|
for id in $(curl -s http://localhost:8092/sessions | jq -r '.sessions[].id'); do
|
|
msgs=$(curl -s "http://localhost:8092/sessions/$id" | jq '.messages | length')
|
|
tools=$(curl -s "http://localhost:8092/sessions/$id" | jq '[.messages[].content.tool_calls | length] | add')
|
|
echo "$id $msgs msgs $tools tools"
|
|
done
|
|
```
|
|
|
|
## 5. Output format
|
|
|
|
```
|
|
Session: {id[:8]} — "{title[:60]}"
|
|
Messages: {N} ({user}/{assistant})
|
|
Tool calls: {total} across {turns} turns
|
|
Top tools: {name:count, name:count, ...}
|
|
Objective: {one-line summary}
|
|
Outcome: ✅ / ❌ / ⚠️
|
|
Blockers: {list or "none"}
|
|
Fixes needed: {concrete actions}
|
|
Severity: blocker | friction | cosmetic
|
|
```
|
|
|
|
## Related files
|
|
|
|
- `cmd/nomos/agent.go` — agent loop, tool building, response guards
|
|
- `cmd/nomos/store.go` — session + message persistence
|
|
- `internal/mcp/server.go` — all tool implementations (`run`, `list_lxcs`, …)
|
|
- `web/src/lib/components/ToolCallGroup.svelte` — tool result display
|
|
- `nomos/SOUL.md` — agent persona and tool selection rules
|
|
- `plans/done/2026-07-09-chat-sessions-improvements.md` — prior session findings
|
|
- `plans/done/2026-07-09-session-execution-and-ux-fixes.md` — latest plan
|