From 614c38ea7c957eb7297513b5df107698f05c0c8c Mon Sep 17 00:00:00 2001 From: dtoro Date: Thu, 9 Jul 2026 10:09:34 +0200 Subject: [PATCH] docs: plan chat-sessions fixes from real production usage data Inspected the live agent_sessions/agent_messages tables on mac-mini and found silent empty responses, a canned non-English refusal after 22 tool calls, 70-call fan-out for simple fleet questions, 100KB+ persisted messages, and no session delete/title hygiene. Co-Authored-By: Claude Sonnet 5 --- .../2026-07-09-chat-sessions-improvements.md | 160 ++++++++++++++++++ plans/index.md | 1 + 2 files changed, 161 insertions(+) create mode 100644 plans/2026-07-09-chat-sessions-improvements.md diff --git a/plans/2026-07-09-chat-sessions-improvements.md b/plans/2026-07-09-chat-sessions-improvements.md new file mode 100644 index 0000000..77eec3b --- /dev/null +++ b/plans/2026-07-09-chat-sessions-improvements.md @@ -0,0 +1,160 @@ +# 2026-07-09 — Chat sessions: reliability, cost, and session-management fixes + +**Status:** Planned + +## Goal + +Fix concrete problems found by inspecting the *actual* production Nomos chat +data (`agent_sessions`/`agent_messages` on the `oikos` Postgres, 24 sessions / +52 messages as of 2026-07-09), not a code-review of the UI in the abstract. +Findings below are backed by real rows, not hypotheticals. + +## How this was investigated + +Queried `oikos-postgres-1` directly (`docker exec oikos-postgres-1 psql -U oikos +-d oikos`) since this runs on mac-mini, the same host as the production +containers. Pulled session list, per-message content sizes, tool-call +breakdowns, and cross-checked against `cmd/nomos/agent.go` to explain what was +observed. + +--- + +## Findings + +### 1. ~17% of turns come back completely empty, silently + +4 of 24 sessions ("hi" ×3, "what services are healthy?") have an assistant +message with `text=""` and zero tool calls — the model returned a blank +completion. `cmd/nomos/agent.go:175-183` treats `len(msg.ToolCalls) == 0` as a +normal final answer and emits `text: ""` + `done`. No `error` event fires, so +[chat.ts](../web/src/lib/stores/chat.ts) never sets `$error`, and +[Chat.svelte](../web/src/pages/Chat.svelte) renders a permanently-blank +assistant bubble (the "…" typing dots only show while `$streaming` is true; +once `done` fires they vanish, leaving nothing). The user has no idea the turn +failed and no obvious way to retry — they have to notice the silence and +retype. + +**Fix:** in `agent.chat()`, if `msg.Content == "" && len(msg.ToolCalls) == 0`, +treat it as a retryable failure: log it, retry once against the provider +before giving up, and if still empty, emit a real `error` event instead of an +empty `text`/`done` pair. On the frontend, surface a "Nomos didn't respond — +retry?" affordance on empty assistant messages rather than a silent blank +bubble. + +### 2. A tool-heavy turn came back as a canned non-English refusal + +The session "what are the termals of hubris?" ran 22 tool calls and then +returned, verbatim: `关于这个问题，我没有相关信息，您可以尝试问我其它问题，我会尽力为您解答~` +("I don't have relevant information on this, try asking me something else"). +This is after successfully gathering data via tools — the model discarded its +own tool results and emitted a boilerplate deflection in the wrong language. +`NOMOS_MODEL` is currently a DeepSeek flash-tier model on OpenRouter, which is +consistent with this kind of degraded-tier fallback text leaking through. + +**Fix:** add a response-quality guard in `agent.chat()` — if the final text +doesn't match the conversation's language/looks like a canned refusal (simple +heuristic: non-ASCII-majority reply to an ASCII-majority conversation, or +matches a small denylist of known refusal boilerplate), treat it like the +empty-response case (retry, then surface an error rather than showing it to +the operator as a real answer). Separately, reconsider whether the flash-tier +model is worth the latency/cost tradeoff given it's producing failures like +this in a small sample — worth an eval pass against a couple of alternative +OpenRouter models on the same 24 real prompts before deciding. + +### 3. Simple fleet questions fan out into dozens of individual tool calls + +"Are any of the proxmox hosts saturated?" (2 hosts) triggered **70 tool +calls** in one turn, 21 of them individual `get_lxc_state` calls — one per LXC +container — instead of using the already-available `list_lxcs()` bulk tool. +Similar pattern in "What should be updated with high priority?" (68 calls) and +"What needs updating?" (54 calls). Each `get_lxc_state` is a live `pct status` +SSH round-trip to the Proxmox host, so this is 21 sequential SSH round trips +to answer a question `list_lxcs()` already answers in one call. This is the +direct cause of both slow responses and the huge persisted payloads in +finding 4. + +**Fix:** two angles, not mutually exclusive: +- **Prompt-level**: tighten the Nomos system prompt (`nomos/SOUL.md`) to + explicitly prefer bulk tools (`list_lxcs`, `get_state_snapshot`, + `query_metrics`) over per-entity tools when the question is fleet-wide, and + only fall back to `get_lxc_state`/`tail_log` for a specific named entity. +- **Tool-level**: `get_lxc_state` already exists per-slug; consider whether + `list_lxcs()`'s summary is actually sufficient for "saturated" (CPU/mem % + per container) — if it's missing that field, that's *why* the model loops + per-container, and the real fix is enriching `list_lxcs()` rather than + prompting around the gap. + +### 4. Tool results are persisted raw and unbounded, inflating messages to 100KB+ + +Message content sizes in `agent_messages.content` (JSONB) range up to +**106KB** for a single assistant turn. Even a plain "hi" greeting produced a +44KB message, because `get_state_snapshot()`'s full result — every entity in +the DB, including ~15 `document:containers/*` rows that are all +`state: , health: unknown` and contribute nothing — gets embedded +verbatim in the `tool_calls[].result` field and stored as-is +(`cmd/nomos/store.go:68-76` just JSON-inserts whatever the tool returned). +This bloats the DB, and every time a session is opened via +[loadSessionMessages](../web/src/lib/stores/chat.ts#L56) or the +[SessionRail](../web/src/lib/components/SessionRail.svelte)/ +[Sessions](../web/src/pages/Sessions.svelte) page loads history, the browser +downloads and parses all of it just to render a collapsed tool-call summary. + +**Fix:** +- Filter `get_state_snapshot()`'s result server-side (in the MCP tool, not + the agent) to drop entities with no meaningful state/health signal, or add + a `type` filter param the agent can pass. +- In `store.saveMessage`, cap persisted tool-result size (e.g. truncate to a + few KB with a `"...truncated, N bytes"` marker) — the full result already + served its purpose informing that turn's answer; historical replay + (`agent.chat()`'s history-replay loop at `agent.go:122-137`) doesn't need + the full blob, just enough for the model to know what it already checked. + +### 5. No session hygiene: duplicate/typo'd titles, no delete/archive + +Session titles are the raw, unprocessed first user message +(`store.createSession`), with no dedup, normalization, or cleanup. Real +production titles include **6 sessions titled "hi"**, **2 titled "say ok"**, +and typos preserved verbatim ("what are the **termals** of hubris?", "whats +the **termans** of strong", "Are any of the **proxomox** hosts saturated?"). +Neither [Sessions.svelte](../web/src/pages/Sessions.svelte) nor +[SessionRail.svelte](../web/src/lib/components/SessionRail.svelte) nor the +`store`/API layer (`cmd/nomos/store.go`, `web/src/lib/api.ts`) has any delete +or archive path — grepped the whole stack, confirmed absent. Throwaway test +sessions accumulate forever with no way to clean them from the UI. + +**Fix:** +- Add `DELETE /sessions/{id}` to the nomos gateway + a matching store method + and wire a delete affordance into `SessionRail`/`Sessions` (hover trash + icon, confirm on click). +- Generate titles from the assistant's actual answer once the turn completes + (or a cheap follow-up summarization call) instead of the raw first message, + so distinct "hi" sessions become distinguishable by what was actually + discussed. + +--- + +## Implementation order + +1. **Empty-response + refusal-leak guard** (`cmd/nomos/agent.go`) — highest + user-visible impact, smallest change, no schema/API changes. +2. **Bulk-tool prompting fix** (`nomos/SOUL.md`) — cheap, directly cuts + latency and tool-call volume; re-run the same 24 real prompts against the + updated prompt to confirm `get_lxc_state` fan-out drops. +3. **Tool-result truncation on persist** (`cmd/nomos/store.go`) — bounds + future DB growth; pair with a one-off cleanup pass on the 52 existing rows + if the table needs to be shrunk immediately. +4. **`get_state_snapshot` filtering** — coordinate with whichever MCP tool + file defines it; verify with `jsonb_pretty` on a fresh "hi" session that + payload drops well below the current ~44KB. +5. **Session delete + title generation** — UI + gateway change, lowest risk, + can ship independently of 1-4. + +## Verification + +- Re-run the same 24 real user prompts (recorded in this plan's investigation) + against the patched agent; confirm zero empty/refusal-leak responses and + `get_lxc_state`-style fan-out drops to O(hosts) not O(containers). +- `docker exec oikos-postgres-1 psql -U oikos -d oikos -c "SELECT max(length(content::text)) FROM agent_messages;"` + before/after — expect the ceiling to move from ~106KB to low single-digit KB. +- Manually delete a test session via the new UI affordance, confirm it's gone + from both `SessionRail` and the `agent_sessions` table. diff --git a/plans/index.md b/plans/index.md index 9c607b1..057fd16 100644 --- a/plans/index.md +++ b/plans/index.md @@ -13,6 +13,7 @@ went sideways, open an investigation. | 2026-07-08 | [Oikos gaps, broken things, and improvements](2026-07-08-oikos-gaps-and-improvements.md) | Planned | | 2026-07-08 | [Control room web UI](2026-07-08-control-room-webui.md) | In Progress | | 2026-07-08 | [Nomos resident agent (renames Hermes)](2026-07-08-nomos-resident-agent.md) | In Progress | +| 2026-07-09 | [Chat sessions: reliability, cost, and session-management fixes](2026-07-09-chat-sessions-improvements.md) | Planned | ## Done