feat(observability): restore monitoring coverage, make gaps visible, stream executions

Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:

- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
  (slug) DO NOTHING, then wrote a check_defs row referencing it. On any
  re-seed the slug already existed, the entity insert no-oped, and the FK
  violated — aborting the ingest transaction and surfacing as an unrelated
  failure several entities later. Re-seeding has been broken since; prod's
  coverage was frozen at its first successful seed. This is what
  TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
  to ".network" and overwrote each other; service:jellyfin collided with
  lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
  process_check.sh always ran without its unit name and returned "unknown".

Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.

Also:
- entity_types.attribute_schema was never ingested: the seed loader read
  "attribute_schema" but the YAML says "attributes", so all 60 types stored
  JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
  paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
  execution reported NULL for its whole life. The three MCP auto-run copies
  wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
  Both sshExec copies now stream through a shared execlog sink into
  execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
  nothing; it is now the chat session id, which is what lets the chat tail
  live output.
- reversible_low had no auto-run branch despite policy declaring it
  unattended. Since computeCommandRisk never returns it, the class only arises
  when an agent declares it over a read_only command — so gating it penalised
  candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
  the first target is on macOS), resolving its host by walking backs-up-to
  backwards. The pre-deploy pg_dump is now a tracked backup target.

UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.

Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 13:51:14 +02:00
parent 873b00ac42
commit 1dca2cfd7a
39 changed files with 3105 additions and 273 deletions

View File

@@ -0,0 +1,35 @@
-- 022_entity_type_monitoring.up.sql
-- Declare, per entity type, what monitoring that type warrants.
--
-- Motivation: only 3 of 89 active entities had an enabled check_def, because
-- checkdefaults.forEntityType hardcoded a Go `switch` over five entity types
-- and resolveHost looked for attribute shapes the seed data never used. The
-- failure was silent — every tx.Exec in that file discarded its error.
--
-- Fixing coverage alone is not enough: coverage is NOT uniform. Some types
-- (site, cluster, lan, mesh) are topological groupings with nothing to probe;
-- their health is implied by their members. Without an explicit declaration,
-- the "unmonitored" signal added alongside this migration would fire
-- permanently and unresolvably against entities that are working as intended.
--
-- So monitoring becomes a property of the TYPE, resolved through the existing
-- `parent_type` is-a hierarchy (declaring it on abstract `machine` covers
-- proxmox-host / standalone-server / workstation).
--
-- Three states, deliberately distinguishable:
-- NULL — undeclared. An ontology gap; reported at info severity,
-- not as a fleet gap. This is why the column is nullable
-- rather than defaulting to '[]'.
-- '[]' — explicitly none. Excluded from coverage signalling.
-- '["http", ...]' — the check kinds this type warrants.
--
-- The column holds check KINDS only. Deriving each check's config (host,
-- script, url, thresholds) stays in Go, in internal/checkdefaults — a config
-- template language in YAML is the natural follow-on, not this change.
ALTER TABLE entity_types ADD COLUMN IF NOT EXISTS monitoring_spec JSONB;
-- Kept on one line and free of semicolons: the migration runner splits on ';'
-- without tracking string literals, so both a newline and an inner semicolon
-- would truncate this statement mid-quote.
COMMENT ON COLUMN entity_types.monitoring_spec IS 'Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.';

View File

@@ -0,0 +1,18 @@
-- 023_executions_created_at_index.up.sql
-- Support newest-first execution history.
--
-- ListExecutions previously ordered by the target entity's slug, which is
-- neither useful for a history view nor unique enough to paginate on. It now
-- orders by (created_at DESC, entity_id DESC) -- the compound key the cursor
-- carries -- and executions had indexes only on target_entity_id and status.
--
-- The same ordering backs /activity/recent, which was doing this unindexed.
CREATE INDEX IF NOT EXISTS idx_executions_created_at
ON executions (created_at DESC, entity_id DESC);
-- Per-entity history ("what has run against this host?") filters on the target
-- and then sorts, so give it a composite rather than making the planner sort
-- every row for a target with a long history.
CREATE INDEX IF NOT EXISTS idx_executions_target_created_at
ON executions (target_entity_id, created_at DESC);

View File

@@ -0,0 +1,43 @@
-- 024_execution_logs.up.sql
-- Incremental command output for executions.
--
-- Until now `executions.result` was a single JSONB blob written once, at the
-- terminal state: {"output": "...everything..."}. Two consequences:
--
-- 1. Nothing could be seen while a command ran. A ten-minute apt upgrade
-- showed an empty row until it finished.
-- 2. On the sshExecTimeout path the output was discarded entirely — the
-- code returned "" — so the executions most worth inspecting (the ones
-- that hung) were the ones that left no trace at all.
--
-- Chunks land here as they arrive. `executions.result` still gets the full
-- output at the end, so existing readers keep working unchanged and this
-- table is purely additive.
CREATE TABLE IF NOT EXISTS execution_logs (
execution_id UUID NOT NULL,
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
-- Monotonic per execution. ts alone cannot order chunks: several arrive
-- within the same microsecond on a fast command.
seq INTEGER NOT NULL,
-- 'stdout' or 'stderr'. Both are also concatenated into the combined
-- output, matching what CombinedOutput used to return.
stream TEXT NOT NULL,
chunk TEXT NOT NULL,
PRIMARY KEY (execution_id, seq, ts)
);
SELECT create_hypertable('execution_logs', 'ts',
chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE);
-- The only query that matters: replay one execution's output in order.
CREATE INDEX IF NOT EXISTS idx_execution_logs_exec_seq
ON execution_logs (execution_id, seq);
-- Matches the events table's 90 days. Command output is bulkier than events,
-- but keeping it exactly as long as the event stream that references it avoids
-- dangling 'execution.output' events pointing at rows that no longer exist.
DO $$ BEGIN
PERFORM add_retention_policy('execution_logs', INTERVAL '90 days');
EXCEPTION WHEN OTHERS THEN NULL;
END $$;