# 2026-07-08 — Liveness, drift, and UX cohesion **Status:** In Progress — Phases 1–4 code complete and now deployed (re-verified 2026-07-12: mac-mini was redeployed from `main` that day for unrelated auth work — plans/2026-07-12-wails-desktop-app.md — which carried every commit up to that point, including this plan's, so "not yet deployed" below is stale). Phase 5 deferred. (Audited 2026-07-11 — still accurate; prompt caching within Phase 4 also confirmed not implemented.) - **Phase 1 (drift/staleness):** done. Health/metrics/events misattribution fix, staleness sweep, `/entities` health+freshness, dashboard/fleet-health `check`-entity exclusion, Entities table health column. Migration 016 applied to the live dev DB (safe cleanup, additive-only). - **Phase 2 (detail sidebar + legibility):** done. `EntityDetailContent` extracted and shared between the full `#/entity/:slug` page and a new `EntitySheet` opened from the Entities table (master-detail, no navigation). Adds a **Monitoring** card (per-entity `check_defs`: kind, interval, enabled/disabled with click-to-toggle) and renders attributes as a key/value list instead of raw JSON. - **Phase 3 (sessions rejoin chat):** done. Fixed the click-does-nothing bug, added a session rail inside Chat, fixed the local dev proxy to match production's `/agent` prefix-stripping. **Also found and fixed a real latent bug**: persisted `tool_calls` store the `tool_use`/`tool_result` as two entries sharing one `id`; Chat.svelte's keyed `{#each tool (tool.id)}` threw on the duplicate key and silently blanked the entire message list. This had presumably never been noticed because sessions were never clickable before this fix. Fixed in `chat.ts` by merging tool_calls by id before rendering. - **Phase 4 (agent efficiency):** core piece done — prior turns' tool calls/results are now replayed into the conversation (previously dropped entirely), and a compact live fleet-health snapshot is injected into the system prompt each turn so the agent starts oriented. Prompt caching is **not done** (lower priority, no measured regression without it). Reconsidering the default model — done, but not by this plan: switched to `deepseek/deepseek-v4-pro` on 2026-07-10 (`cmd/nomos/agent.go:70`) for reliability, per that commit's own comment ("the flash tier over-narrates, occasionally emits canned refusals, and is unreliable at multi-step tool use"). - **Phase 5 (CRUD):** `PatchEntity` and a full `/checks` CRUD API (list/create/patch, including enable/disable) already existed server-side; the new Monitoring card's toggle uses `PatchCheck`. **Not done**: a "run check now" endpoint (no scheduler on-demand entrypoint exists yet), relationship editing, and an entity attribute editor UI. Addresses five felt problems with the current system: (1) the agent reports stale machine state as if it were fresh, (2) sessions can't be opened and feel disconnected from chat, (3) the Nomos agent re-derives state every turn and wastes iterations, (4) the UI feels dead — tables with no context, no sense of what is monitored, (5) no way to inspect or customize entities and their checks. The unifying UX principle for this plan: **master-detail with a detail sidebar**, not full-page navigation. Selecting an entity, session, or signal opens a right-hand detail panel over the current list, so the operator keeps context and drills in without losing their place. Full pages remain addressable (deep links) but are no longer the primary way to inspect a row. Sequencing is driven by pain: **drift/staleness is Phase 1.** --- ## Root causes (verified in code) ### Drift / staleness — root cause was worse than a missing TTL Live-DB inspection (`oikos-postgres-1`) found the real cause: `check_defs` has two entity references — `entity_id` (the internal probe/"check" entity) and `target_id` (the host/service actually being observed). The scheduler wrote `entity_status`, `metric_samples`, and scheduler-sourced `events` keyed by `cd.EntityID` (the probe) instead of `cd.TargetID` (the target) — [scheduler.go:110-171](../internal/scheduler/scheduler.go) (pre-fix). Verified against the live database: ``` entity_status by type: only type='check' rows ever had real health (24 healthy, 1 down); every host/service/lxc/vm/proxmox-host was frozen at 'unknown' since creation. metric_samples: 17,559 rows, 100% attached to type='check' entities — zero attached to any real host or service. events: 45 of 46 scheduler-sourced rows attached to type='check' entities. ``` So this wasn't staleness in the TTL sense — the entities you actually care about (`host:hubris`, `service:authentik`, etc.) **never received an observation at all**. Every health check, metric, and event the scheduler produced was filed under an internal bookkeeping entity the UI doesn't even surface distinctly. This is the literal mechanism behind "the agent tells me stale/wrong state." **Fixed** (this session): `runCheck`/`resolveSignal` now resolve `targetID := cd.TargetID` and write status/metrics/events there, falling back to the check's own id only if `target_id` is unset. Signals remain keyed by the check entity (unchanged, matches their existing resolution logic). A new migration ([016_fix_check_status_misattribution](../migrations/016_fix_check_status_misattribution.up.sql)) deletes the orphaned check-entity `entity_status` rows so rollups stop double-counting probes as monitored entities; historical `metric_samples` on check entities are left as-is (time-series data, not safe to reattribute). On top of the misattribution fix, a genuine staleness gap also existed and is now closed: `entity_status.health` was written only when a check ran, with no TTL — a stalled scheduler or disabled check_def would leave the last health value looking current forever. - `last_check_at` is recorded but was never surfaced. The Entities table showed `entity.updated_at` (row mutation time), not observation time ([Entities.svelte:108](../web/src/pages/Entities.svelte), pre-fix). - The `/entities` list endpoint returned neither `health` nor `last_check_at` — only `/graph?include=status` and `/fleet/health` did ([impl.go:344](../internal/httpapi/impl.go), pre-fix). ### Sessions - Clicking a session calls `loadSessionMessages()` but never navigates to the chat page ([Sessions.svelte:17](../web/src/pages/Sessions.svelte)); it mutates the chat store while the user stays on the session list, so nothing appears to happen. There is also no session switcher inside Chat. ### Agent efficiency - Multi-turn history replay **drops all `tool_use`/`tool_result` pairs**; only prior final text is replayed ([agent.go:108-127](../cmd/nomos/agent.go)). Each new turn re-discovers the fleet from scratch, re-calling tools already run. - Cold start: the system prompt injects no fleet snapshot ([agent.go:81](../cmd/nomos/agent.go)); default model is `deepseek/deepseek-v4-flash` ([agent.go:34](../cmd/nomos/agent.go)); tool schema + system prompt are rebuilt each call with no prompt caching. ### Dead UI / no inspection - Entities table = slug/type/name/state/updated; no health, no last-seen, no signal count. - EntityDetail dumps `JSON.stringify(attributes)` raw ([EntityDetail.svelte:123](../web/src/pages/EntityDetail.svelte)) and never shows the entity's `check_defs` — the operator cannot see *what is monitored*, when it last ran, or what it returned. - No CRUD anywhere: no entity editor, no check management (enable/disable/edit/ run-now), no relationship editing. `check_defs` do not appear in the web app. --- ## Phase 1 — Kill the drift (highest priority) Goal: the system never presents stale observations as fresh, and freshness is visible everywhere health is. **Backend** - Add a staleness sweep to `housekeeping()` ([scheduler.go:243](../internal/scheduler/scheduler.go)): for each `entity_status` where `now() - last_check_at > staleAfter` (default `max(3 × check interval, 5m)`), transition health to a new `stale` value and emit a `health.stale` event once (not every pass). - Treat `stale` as a first-class health in dashboard rollups ([dashboard.go:57](../internal/httpapi/dashboard.go)) and fleet health ([impl.go:516](../internal/httpapi/impl.go)) — do not fold it into `unknown`. - Extend the `/entities` list response with `health` and `last_check_at` (join `entity_status`), so the table can show freshness without N graph calls. - Nomos: when answering about state, tool results should carry `last_check_at` and a stale flag so the agent can hedge ("healthy as of 4m ago") instead of asserting stale data. (Verify the MCP topology/health tools include it.) **Frontend** - Entities table: replace the `Updated` column with **health dot + relative "checked 2m ago"**, and add an **open-signal count** badge per row. Stale rows get a distinct muted/amber treatment, not a green dot. - Global header: add an "as of {time}" and make the SSE connection dot a real liveness indicator (last event received, reconnect state). **Acceptance:** disable a check or stop the scheduler → within one stale window the affected entity shows `stale` in the table and dashboard, an event fires, and asking Nomos "is X healthy?" yields a freshness-qualified answer. --- ## Phase 2 — Detail sidebar + entity legibility (less navigation) Goal: inspect any row in place; make an entity's monitoring self-evident. - Introduce a reusable **DetailSheet** (right-side panel) used across Entities, Signals, Sessions, Executions. Row click opens the sheet; URL hash updates for deep-linking; Esc / click-away closes. Full `#/entity/:slug` page remains for direct links but reuses the same detail component. - Entity detail content (in the sheet): - Header: slug, type, **health + freshness** ("checked 2m ago" / "stale 18m"). - **Monitoring card**: the entity's `check_defs` — kind, schedule, enabled, last result + evidence, next run. This is the missing "what is watched." - Attributes rendered as a key/value panel, not raw JSON. - Relations, open signals, recent executions, metrics sparklines (reuse existing EntityDetail sections). - Backend: endpoint to list `check_defs` for an entity with last-result join (currently checks are only visible to the scheduler). **Acceptance:** from the Entities list, one click reveals what an entity is, what's monitoring it, when it was last seen, and its open signals — without a full page load or losing the list. --- ## Phase 3 — Sessions rejoin chat Goal: sessions are openable and live next to the conversation. - Fix: clicking a session navigates to `#/chat` and loads it ([Sessions.svelte:17](../web/src/pages/Sessions.svelte)). - Add a **session rail inside Chat** (collapsible left list: title, last-active, active highlight) so switching sessions never leaves the chat surface. The standalone Sessions page becomes a thin wrapper / can be retired from nav. - Show session metadata (message count, last actor) and allow rename/delete. **Acceptance:** clicking any past session opens its transcript in the chat view; starting a new chat and switching back and forth works without navigation. --- ## Phase 4 — Agent efficiency Goal: stop re-deriving state; start each turn already oriented. - Persist and replay tool evidence across turns ([agent.go:108-127](../cmd/nomos/agent.go)): either replay `tool_use`/ `tool_result` pairs with consistent ids, or persist a compacted per-turn "evidence summary" and replay that. Removes redundant re-querying. - Inject a compact fleet snapshot (counts by health, open signals, stale set) into the system prompt ([agent.go:81](../cmd/nomos/agent.go)) so the agent starts oriented instead of spending iterations on discovery. - Add prompt caching for the system prompt + tool schema (rebuilt every call today); revisit the default model ([agent.go:34](../cmd/nomos/agent.go)) — evaluate a stronger default for fewer, better tool calls. - Surface per-turn iteration/token/cost in the chat UI (data already logged to `agent_activity`) so inefficiency is visible and measurable. **Acceptance:** a 3-turn conversation about the same entity does not re-call the same read tools each turn; median iterations-per-answer drops. --- ## Phase 5 — Customize & inspect (CRUD) Goal: manage the system from the UI, not just observe it. - Entity editor (attributes, state) via existing mutation endpoints. - Check management from the entity detail sheet: enable/disable, edit config/ thresholds, and **run-now** (trigger a single check pass on demand — new scheduler entrypoint). - Relationship add/remove. - Raw DB-row view toggle in the detail sheet for power inspection. --- ## Suggested order of work 1. Phase 1 backend (staleness sweep + `/entities` health/freshness) → Phase 1 frontend (table freshness + liveness header). 2. Phase 2 DetailSheet + entity monitoring card. 3. Phase 3 sessions fix (small; can slot in earlier if desired). 4. Phase 4 agent efficiency. 5. Phase 5 CRUD. Phases 1–3 are the ones that most directly turn "the system feels dead and I don't trust it" into "it's alive and I can see and act on it."