oikos/plans/2026-07-08-liveness-drift-and-ux-cohesion.md

# 2026-07-08 — Liveness, drift, and UX cohesion

**Status:** In Progress — Phases 1–4 code complete; not yet deployed. Phase 5 deferred.

- **Phase 1 (drift/staleness):** done. Health/metrics/events misattribution
  fix, staleness sweep, `/entities` health+freshness, dashboard/fleet-health
  `check`-entity exclusion, Entities table health column. Migration 016
  applied to the live dev DB (safe cleanup, additive-only).
- **Phase 2 (detail sidebar + legibility):** done. `EntityDetailContent`
  extracted and shared between the full `#/entity/:slug` page and a new
  `EntitySheet` opened from the Entities table (master-detail, no
  navigation). Adds a **Monitoring** card (per-entity `check_defs`: kind,
  interval, enabled/disabled with click-to-toggle) and renders attributes as
  a key/value list instead of raw JSON.
- **Phase 3 (sessions rejoin chat):** done. Fixed the click-does-nothing bug,
  added a session rail inside Chat, fixed the local dev proxy to match
  production's `/agent` prefix-stripping. **Also found and fixed a real
  latent bug**: persisted `tool_calls` store the `tool_use`/`tool_result` as
  two entries sharing one `id`; Chat.svelte's keyed `{#each tool (tool.id)}`
  threw on the duplicate key and silently blanked the entire message list.
  This had presumably never been noticed because sessions were never
  clickable before this fix. Fixed in `chat.ts` by merging tool_calls by id
  before rendering.
- **Phase 4 (agent efficiency):** core piece done — prior turns' tool
  calls/results are now replayed into the conversation (previously dropped
  entirely), and a compact live fleet-health snapshot is injected into the
  system prompt each turn so the agent starts oriented. Prompt caching and
  reconsidering the default model are **not done** (lower priority, no
  measured regression without them).
- **Phase 5 (CRUD):** `PatchEntity` and a full `/checks` CRUD API
  (list/create/patch, including enable/disable) already existed server-side;
  the new Monitoring card's toggle uses `PatchCheck`. **Not done**: a
  "run check now" endpoint (no scheduler on-demand entrypoint exists yet),
  relationship editing, and an entity attribute editor UI.

**Not yet deployed** — the live `oikos-api`/`oikos-scheduler`/nomos
containers still run the pre-fix binaries; rebuilding and restarting them
needs an explicit go-ahead since it touches the running homelab control
plane.

Addresses five felt problems with the current system: (1) the agent reports
stale machine state as if it were fresh, (2) sessions can't be opened and feel
disconnected from chat, (3) the Nomos agent re-derives state every turn and
wastes iterations, (4) the UI feels dead — tables with no context, no sense of
what is monitored, (5) no way to inspect or customize entities and their checks.

The unifying UX principle for this plan: **master-detail with a detail
sidebar**, not full-page navigation. Selecting an entity, session, or signal
opens a right-hand detail panel over the current list, so the operator keeps
context and drills in without losing their place. Full pages remain
addressable (deep links) but are no longer the primary way to inspect a row.

Sequencing is driven by pain: **drift/staleness is Phase 1.**

---

## Root causes (verified in code)

### Drift / staleness — root cause was worse than a missing TTL
Live-DB inspection (`oikos-postgres-1`) found the real cause: `check_defs` has
two entity references — `entity_id` (the internal probe/"check" entity) and
`target_id` (the host/service actually being observed). The scheduler wrote
`entity_status`, `metric_samples`, and scheduler-sourced `events` keyed by
`cd.EntityID` (the probe) instead of `cd.TargetID` (the target) —
[scheduler.go:110-171](../internal/scheduler/scheduler.go) (pre-fix). Verified
against the live database:

```
entity_status by type: only type='check' rows ever had real health (24
healthy, 1 down); every host/service/lxc/vm/proxmox-host was frozen at
'unknown' since creation.
metric_samples: 17,559 rows, 100% attached to type='check' entities — zero
attached to any real host or service.
events: 45 of 46 scheduler-sourced rows attached to type='check' entities.
```

So this wasn't staleness in the TTL sense — the entities you actually care
about (`host:hubris`, `service:authentik`, etc.) **never received an
observation at all**. Every health check, metric, and event the scheduler
produced was filed under an internal bookkeeping entity the UI doesn't even
surface distinctly. This is the literal mechanism behind "the agent tells me
stale/wrong state."

**Fixed** (this session): `runCheck`/`resolveSignal` now resolve
`targetID := cd.TargetID` and write status/metrics/events there, falling back
to the check's own id only if `target_id` is unset. Signals remain keyed by
the check entity (unchanged, matches their existing resolution logic). A new
migration ([016_fix_check_status_misattribution](../migrations/016_fix_check_status_misattribution.up.sql))
deletes the orphaned check-entity `entity_status` rows so rollups stop
double-counting probes as monitored entities; historical `metric_samples` on
check entities are left as-is (time-series data, not safe to reattribute).

On top of the misattribution fix, a genuine staleness gap also existed and is
now closed: `entity_status.health` was written only when a check ran, with no
TTL — a stalled scheduler or disabled check_def would leave the last health
value looking current forever.
- `last_check_at` is recorded but was never surfaced. The Entities table
  showed `entity.updated_at` (row mutation time), not observation time
  ([Entities.svelte:108](../web/src/pages/Entities.svelte), pre-fix).
- The `/entities` list endpoint returned neither `health` nor `last_check_at`
  — only `/graph?include=status` and `/fleet/health` did
  ([impl.go:344](../internal/httpapi/impl.go), pre-fix).

### Sessions
- Clicking a session calls `loadSessionMessages()` but never navigates to the
  chat page ([Sessions.svelte:17](../web/src/pages/Sessions.svelte)); it mutates
  the chat store while the user stays on the session list, so nothing appears to
  happen. There is also no session switcher inside Chat.

### Agent efficiency
- Multi-turn history replay **drops all `tool_use`/`tool_result` pairs**; only
  prior final text is replayed ([agent.go:108-127](../cmd/nomos/agent.go)). Each
  new turn re-discovers the fleet from scratch, re-calling tools already run.
- Cold start: the system prompt injects no fleet snapshot
  ([agent.go:81](../cmd/nomos/agent.go)); default model is
  `deepseek/deepseek-v4-flash` ([agent.go:34](../cmd/nomos/agent.go)); tool
  schema + system prompt are rebuilt each call with no prompt caching.

### Dead UI / no inspection
- Entities table = slug/type/name/state/updated; no health, no last-seen, no
  signal count.
- EntityDetail dumps `JSON.stringify(attributes)` raw
  ([EntityDetail.svelte:123](../web/src/pages/EntityDetail.svelte)) and never
  shows the entity's `check_defs` — the operator cannot see *what is monitored*,
  when it last ran, or what it returned.
- No CRUD anywhere: no entity editor, no check management (enable/disable/edit/
  run-now), no relationship editing. `check_defs` do not appear in the web app.

---

## Phase 1 — Kill the drift (highest priority)

Goal: the system never presents stale observations as fresh, and freshness is
visible everywhere health is.

**Backend**
- Add a staleness sweep to `housekeeping()`
  ([scheduler.go:243](../internal/scheduler/scheduler.go)): for each
  `entity_status` where `now() - last_check_at > staleAfter` (default
  `max(3 × check interval, 5m)`), transition health to a new `stale` value and
  emit a `health.stale` event once (not every pass).
- Treat `stale` as a first-class health in dashboard rollups
  ([dashboard.go:57](../internal/httpapi/dashboard.go)) and fleet health
  ([impl.go:516](../internal/httpapi/impl.go)) — do not fold it into `unknown`.
- Extend the `/entities` list response with `health` and `last_check_at`
  (join `entity_status`), so the table can show freshness without N graph calls.
- Nomos: when answering about state, tool results should carry `last_check_at`
  and a stale flag so the agent can hedge ("healthy as of 4m ago") instead of
  asserting stale data. (Verify the MCP topology/health tools include it.)

**Frontend**
- Entities table: replace the `Updated` column with **health dot + relative
  "checked 2m ago"**, and add an **open-signal count** badge per row. Stale rows
  get a distinct muted/amber treatment, not a green dot.
- Global header: add an "as of {time}" and make the SSE connection dot a real
  liveness indicator (last event received, reconnect state).

**Acceptance:** disable a check or stop the scheduler → within one stale window
the affected entity shows `stale` in the table and dashboard, an event fires,
and asking Nomos "is X healthy?" yields a freshness-qualified answer.

---

## Phase 2 — Detail sidebar + entity legibility (less navigation)

Goal: inspect any row in place; make an entity's monitoring self-evident.

- Introduce a reusable **DetailSheet** (right-side panel) used across Entities,
  Signals, Sessions, Executions. Row click opens the sheet; URL hash updates for
  deep-linking; Esc / click-away closes. Full `#/entity/:slug` page remains for
  direct links but reuses the same detail component.
- Entity detail content (in the sheet):
  - Header: slug, type, **health + freshness** ("checked 2m ago" / "stale
    18m").
  - **Monitoring card**: the entity's `check_defs` — kind, schedule, enabled,
    last result + evidence, next run. This is the missing "what is watched."
  - Attributes rendered as a key/value panel, not raw JSON.
  - Relations, open signals, recent executions, metrics sparklines (reuse
    existing EntityDetail sections).
- Backend: endpoint to list `check_defs` for an entity with last-result join
  (currently checks are only visible to the scheduler).

**Acceptance:** from the Entities list, one click reveals what an entity is,
what's monitoring it, when it was last seen, and its open signals — without a
full page load or losing the list.

---

## Phase 3 — Sessions rejoin chat

Goal: sessions are openable and live next to the conversation.

- Fix: clicking a session navigates to `#/chat` and loads it
  ([Sessions.svelte:17](../web/src/pages/Sessions.svelte)).
- Add a **session rail inside Chat** (collapsible left list: title, last-active,
  active highlight) so switching sessions never leaves the chat surface. The
  standalone Sessions page becomes a thin wrapper / can be retired from nav.
- Show session metadata (message count, last actor) and allow rename/delete.

**Acceptance:** clicking any past session opens its transcript in the chat view;
starting a new chat and switching back and forth works without navigation.

---

## Phase 4 — Agent efficiency

Goal: stop re-deriving state; start each turn already oriented.

- Persist and replay tool evidence across turns
  ([agent.go:108-127](../cmd/nomos/agent.go)): either replay `tool_use`/
  `tool_result` pairs with consistent ids, or persist a compacted per-turn
  "evidence summary" and replay that. Removes redundant re-querying.
- Inject a compact fleet snapshot (counts by health, open signals, stale set)
  into the system prompt ([agent.go:81](../cmd/nomos/agent.go)) so the agent
  starts oriented instead of spending iterations on discovery.
- Add prompt caching for the system prompt + tool schema (rebuilt every call
  today); revisit the default model
  ([agent.go:34](../cmd/nomos/agent.go)) — evaluate a stronger default for
  fewer, better tool calls.
- Surface per-turn iteration/token/cost in the chat UI (data already logged to
  `agent_activity`) so inefficiency is visible and measurable.

**Acceptance:** a 3-turn conversation about the same entity does not re-call the
same read tools each turn; median iterations-per-answer drops.

---

## Phase 5 — Customize & inspect (CRUD)

Goal: manage the system from the UI, not just observe it.

- Entity editor (attributes, state) via existing mutation endpoints.
- Check management from the entity detail sheet: enable/disable, edit config/
  thresholds, and **run-now** (trigger a single check pass on demand — new
  scheduler entrypoint).
- Relationship add/remove.
- Raw DB-row view toggle in the detail sheet for power inspection.

---

## Suggested order of work

1. Phase 1 backend (staleness sweep + `/entities` health/freshness) →
   Phase 1 frontend (table freshness + liveness header).
2. Phase 2 DetailSheet + entity monitoring card.
3. Phase 3 sessions fix (small; can slot in earlier if desired).
4. Phase 4 agent efficiency.
5. Phase 5 CRUD.

Phases 1–3 are the ones that most directly turn "the system feels dead and I
don't trust it" into "it's alive and I can see and act on it."