feat(observability): restore monitoring coverage, make gaps visible, stream executions

Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:

- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
  (slug) DO NOTHING, then wrote a check_defs row referencing it. On any
  re-seed the slug already existed, the entity insert no-oped, and the FK
  violated — aborting the ingest transaction and surfacing as an unrelated
  failure several entities later. Re-seeding has been broken since; prod's
  coverage was frozen at its first successful seed. This is what
  TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
  to ".network" and overwrote each other; service:jellyfin collided with
  lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
  process_check.sh always ran without its unit name and returned "unknown".

Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.

Also:
- entity_types.attribute_schema was never ingested: the seed loader read
  "attribute_schema" but the YAML says "attributes", so all 60 types stored
  JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
  paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
  execution reported NULL for its whole life. The three MCP auto-run copies
  wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
  Both sshExec copies now stream through a shared execlog sink into
  execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
  nothing; it is now the chat session id, which is what lets the chat tail
  live output.
- reversible_low had no auto-run branch despite policy declaring it
  unattended. Since computeCommandRisk never returns it, the class only arises
  when an agent declares it over a read_only command — so gating it penalised
  candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
  the first target is on macOS), resolving its host by walking backs-up-to
  backwards. The pre-deploy pg_dump is now a tracked backup target.

UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.

Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 13:51:14 +02:00
parent 873b00ac42
commit 1dca2cfd7a
39 changed files with 3105 additions and 273 deletions

View File

@@ -16,6 +16,7 @@ import (
"regexp"
"runtime"
"strconv"
"strings"
"time"
"github.com/dtoro/oikos/internal/config"
@@ -73,7 +74,12 @@ func runCheckPass(ctx context.Context, pool *db.Pool) {
return
}
if len(defs) == 0 {
// Still run housekeeping: a fleet with no enabled check_defs is
// precisely the case coverageSweep exists to report, and returning
// here would mean the one situation that most needs reporting is the
// one situation that stays silent.
slog.Debug("scheduler: no enabled check_defs")
housekeeping(ctx, pool)
return
}
@@ -248,6 +254,8 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
return checkPing(ctx, cd)
case "ssh-script":
return checkSSHScript(ctx, cd)
case "backup-freshness":
return checkBackupFreshness(ctx, cd)
default:
return checkResult{health: "unknown"}
}
@@ -265,6 +273,7 @@ func housekeeping(ctx context.Context, pool *db.Pool) {
}
staleSweep(ctx, pool)
coverageSweep(ctx, pool)
// Log housekeeping completion
slog.Debug("scheduler: housekeeping done", "pruned_idempotency_before", cutoff.Format(time.RFC3339))
@@ -337,9 +346,14 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
cfg := struct {
URL string `json:"url"`
ExpectedStatus int `json:"expected_status"`
Insecure bool `json:"insecure"`
// MaxStatus accepts a range instead of one exact code. Most services
// sit behind Authentik and answer 302 or 401 — a working service, but
// an exact-match on 200 reports it degraded and raises a signal.
// Unset expected_status means "any response below MaxStatus is fine".
MaxStatus int `json:"max_status"`
Insecure bool `json:"insecure"`
}{
ExpectedStatus: 200,
MaxStatus: 500,
}
if len(cd.Config) > 0 {
_ = json.Unmarshal(cd.Config, &cfg)
@@ -379,10 +393,17 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
}
defer resp.Body.Close()
if resp.StatusCode != cfg.ExpectedStatus {
if cfg.ExpectedStatus != 0 {
if resp.StatusCode != cfg.ExpectedStatus {
return checkResult{
health: "degraded", signalKind: "http",
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
}
}
} else if resp.StatusCode >= cfg.MaxStatus {
return checkResult{
health: "degraded", signalKind: "http",
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
evidence: fmt.Sprintf("GET %s returned %d (expected below %d)", cfg.URL, resp.StatusCode, cfg.MaxStatus),
}
}
@@ -616,6 +637,11 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
Port int `json:"port"`
User string `json:"user"`
Script string `json:"script"`
// Args is a single positional argument for the script. checkdefaults
// has always written it for process_check.sh, but nothing read it —
// so every process check ran argument-less and process_check.sh
// answered "no service name provided" with health unknown.
Args string `json:"args"`
}{}
if len(cd.Config) > 0 {
_ = json.Unmarshal(cd.Config, &cfg)
@@ -646,6 +672,11 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
defer cancel()
scriptPath := "/opt/oikos/checks/" + cfg.Script
if cfg.Args != "" {
// Single-quote the argument so an entity name can never break out of
// the remote command. The script name itself is allowlisted above.
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
}
port := strconv.Itoa(cfg.Port)
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)