Files

dtoro 0d29b1db81 fix: move completed signal-triggers plan to done/, add missing liveness-drift to index, add plan-consistency lint checks

2026-07-09 10:47:39 +02:00

13 KiB

Raw Permalink Blame History

2026-07-08 — Liveness, drift, and UX cohesion

Status: In Progress — Phases 1–4 code complete; not yet deployed. Phase 5 deferred.

Phase 1 (drift/staleness): done. Health/metrics/events misattribution fix, staleness sweep, /entities health+freshness, dashboard/fleet-health check-entity exclusion, Entities table health column. Migration 016 applied to the live dev DB (safe cleanup, additive-only).
Phase 2 (detail sidebar + legibility): done. EntityDetailContent extracted and shared between the full #/entity/:slug page and a new EntitySheet opened from the Entities table (master-detail, no navigation). Adds a Monitoring card (per-entity check_defs: kind, interval, enabled/disabled with click-to-toggle) and renders attributes as a key/value list instead of raw JSON.
Phase 3 (sessions rejoin chat): done. Fixed the click-does-nothing bug, added a session rail inside Chat, fixed the local dev proxy to match production's /agent prefix-stripping. Also found and fixed a real latent bug: persisted tool_calls store the tool_use/tool_result as two entries sharing one id; Chat.svelte's keyed {#each tool (tool.id)} threw on the duplicate key and silently blanked the entire message list. This had presumably never been noticed because sessions were never clickable before this fix. Fixed in chat.ts by merging tool_calls by id before rendering.
Phase 4 (agent efficiency): core piece done — prior turns' tool calls/results are now replayed into the conversation (previously dropped entirely), and a compact live fleet-health snapshot is injected into the system prompt each turn so the agent starts oriented. Prompt caching and reconsidering the default model are not done (lower priority, no measured regression without them).
Phase 5 (CRUD): PatchEntity and a full /checks CRUD API (list/create/patch, including enable/disable) already existed server-side; the new Monitoring card's toggle uses PatchCheck. Not done: a "run check now" endpoint (no scheduler on-demand entrypoint exists yet), relationship editing, and an entity attribute editor UI.

Not yet deployed — the live oikos-api/oikos-scheduler/nomos containers still run the pre-fix binaries; rebuilding and restarting them needs an explicit go-ahead since it touches the running homelab control plane.

Addresses five felt problems with the current system: (1) the agent reports stale machine state as if it were fresh, (2) sessions can't be opened and feel disconnected from chat, (3) the Nomos agent re-derives state every turn and wastes iterations, (4) the UI feels dead — tables with no context, no sense of what is monitored, (5) no way to inspect or customize entities and their checks.

The unifying UX principle for this plan: master-detail with a detail sidebar, not full-page navigation. Selecting an entity, session, or signal opens a right-hand detail panel over the current list, so the operator keeps context and drills in without losing their place. Full pages remain addressable (deep links) but are no longer the primary way to inspect a row.

Sequencing is driven by pain: drift/staleness is Phase 1.

Root causes (verified in code)

Drift / staleness — root cause was worse than a missing TTL

Live-DB inspection (oikos-postgres-1) found the real cause: check_defs has two entity references — entity_id (the internal probe/"check" entity) and target_id (the host/service actually being observed). The scheduler wrote entity_status, metric_samples, and scheduler-sourced events keyed by cd.EntityID (the probe) instead of cd.TargetID (the target) — scheduler.go:110-171 (pre-fix). Verified against the live database:

entity_status by type: only type='check' rows ever had real health (24
healthy, 1 down); every host/service/lxc/vm/proxmox-host was frozen at
'unknown' since creation.
metric_samples: 17,559 rows, 100% attached to type='check' entities — zero
attached to any real host or service.
events: 45 of 46 scheduler-sourced rows attached to type='check' entities.

So this wasn't staleness in the TTL sense — the entities you actually care about (host:hubris, service:authentik, etc.) never received an observation at all. Every health check, metric, and event the scheduler produced was filed under an internal bookkeeping entity the UI doesn't even surface distinctly. This is the literal mechanism behind "the agent tells me stale/wrong state."

Fixed (this session): runCheck/resolveSignal now resolve targetID := cd.TargetID and write status/metrics/events there, falling back to the check's own id only if target_id is unset. Signals remain keyed by the check entity (unchanged, matches their existing resolution logic). A new migration (016_fix_check_status_misattribution) deletes the orphaned check-entity entity_status rows so rollups stop double-counting probes as monitored entities; historical metric_samples on check entities are left as-is (time-series data, not safe to reattribute).

On top of the misattribution fix, a genuine staleness gap also existed and is now closed: entity_status.health was written only when a check ran, with no TTL — a stalled scheduler or disabled check_def would leave the last health value looking current forever.

last_check_at is recorded but was never surfaced. The Entities table showed entity.updated_at (row mutation time), not observation time (Entities.svelte:108, pre-fix).
The /entities list endpoint returned neither health nor last_check_at — only /graph?include=status and /fleet/health did (impl.go:344, pre-fix).

Sessions

Clicking a session calls loadSessionMessages() but never navigates to the chat page (Sessions.svelte:17); it mutates the chat store while the user stays on the session list, so nothing appears to happen. There is also no session switcher inside Chat.

Agent efficiency

Multi-turn history replay drops all tool_use/tool_result pairs; only prior final text is replayed (agent.go:108-127). Each new turn re-discovers the fleet from scratch, re-calling tools already run.
Cold start: the system prompt injects no fleet snapshot (agent.go:81); default model is deepseek/deepseek-v4-flash (agent.go:34); tool schema + system prompt are rebuilt each call with no prompt caching.

Dead UI / no inspection

Entities table = slug/type/name/state/updated; no health, no last-seen, no signal count.
EntityDetail dumps JSON.stringify(attributes) raw (EntityDetail.svelte:123) and never shows the entity's check_defs — the operator cannot see what is monitored, when it last ran, or what it returned.
No CRUD anywhere: no entity editor, no check management (enable/disable/edit/ run-now), no relationship editing. check_defs do not appear in the web app.

Phase 1 — Kill the drift (highest priority)

Goal: the system never presents stale observations as fresh, and freshness is visible everywhere health is.

Backend

Add a staleness sweep to housekeeping() (scheduler.go:243): for each entity_status where now() - last_check_at > staleAfter (default max(3 × check interval, 5m)), transition health to a new stale value and emit a health.stale event once (not every pass).
Treat stale as a first-class health in dashboard rollups (dashboard.go:57) and fleet health (impl.go:516) — do not fold it into unknown.
Extend the /entities list response with health and last_check_at (join entity_status), so the table can show freshness without N graph calls.
Nomos: when answering about state, tool results should carry last_check_at and a stale flag so the agent can hedge ("healthy as of 4m ago") instead of asserting stale data. (Verify the MCP topology/health tools include it.)

Frontend

Entities table: replace the Updated column with health dot + relative "checked 2m ago", and add an open-signal count badge per row. Stale rows get a distinct muted/amber treatment, not a green dot.
Global header: add an "as of {time}" and make the SSE connection dot a real liveness indicator (last event received, reconnect state).

Acceptance: disable a check or stop the scheduler → within one stale window the affected entity shows stale in the table and dashboard, an event fires, and asking Nomos "is X healthy?" yields a freshness-qualified answer.

Goal: inspect any row in place; make an entity's monitoring self-evident.

Introduce a reusable DetailSheet (right-side panel) used across Entities, Signals, Sessions, Executions. Row click opens the sheet; URL hash updates for deep-linking; Esc / click-away closes. Full #/entity/:slug page remains for direct links but reuses the same detail component.
Entity detail content (in the sheet):
- Header: slug, type, health + freshness ("checked 2m ago" / "stale 18m").
- Monitoring card: the entity's check_defs — kind, schedule, enabled, last result + evidence, next run. This is the missing "what is watched."
- Attributes rendered as a key/value panel, not raw JSON.
- Relations, open signals, recent executions, metrics sparklines (reuse existing EntityDetail sections).
Backend: endpoint to list check_defs for an entity with last-result join (currently checks are only visible to the scheduler).

Acceptance: from the Entities list, one click reveals what an entity is, what's monitoring it, when it was last seen, and its open signals — without a full page load or losing the list.

Phase 3 — Sessions rejoin chat

Goal: sessions are openable and live next to the conversation.

Fix: clicking a session navigates to #/chat and loads it (Sessions.svelte:17).
Add a session rail inside Chat (collapsible left list: title, last-active, active highlight) so switching sessions never leaves the chat surface. The standalone Sessions page becomes a thin wrapper / can be retired from nav.
Show session metadata (message count, last actor) and allow rename/delete.

Acceptance: clicking any past session opens its transcript in the chat view; starting a new chat and switching back and forth works without navigation.

Phase 4 — Agent efficiency

Goal: stop re-deriving state; start each turn already oriented.

Persist and replay tool evidence across turns (agent.go:108-127): either replay tool_use/ tool_result pairs with consistent ids, or persist a compacted per-turn "evidence summary" and replay that. Removes redundant re-querying.
Inject a compact fleet snapshot (counts by health, open signals, stale set) into the system prompt (agent.go:81) so the agent starts oriented instead of spending iterations on discovery.
Add prompt caching for the system prompt + tool schema (rebuilt every call today); revisit the default model (agent.go:34) — evaluate a stronger default for fewer, better tool calls.
Surface per-turn iteration/token/cost in the chat UI (data already logged to agent_activity) so inefficiency is visible and measurable.

Acceptance: a 3-turn conversation about the same entity does not re-call the same read tools each turn; median iterations-per-answer drops.

Phase 5 — Customize & inspect (CRUD)

Goal: manage the system from the UI, not just observe it.

Entity editor (attributes, state) via existing mutation endpoints.
Check management from the entity detail sheet: enable/disable, edit config/ thresholds, and run-now (trigger a single check pass on demand — new scheduler entrypoint).
Relationship add/remove.
Raw DB-row view toggle in the detail sheet for power inspection.

Suggested order of work

Phase 1 backend (staleness sweep + /entities health/freshness) → Phase 1 frontend (table freshness + liveness header).
Phase 2 DetailSheet + entity monitoring card.
Phase 3 sessions fix (small; can slot in earlier if desired).
Phase 4 agent efficiency.
Phase 5 CRUD.

Phases 1–3 are the ones that most directly turn "the system feels dead and I don't trust it" into "it's alive and I can see and act on it."

13 KiB Raw Permalink Blame History Unescape Escape