feat(observability): restore monitoring coverage, make gaps visible, stream executions

Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by discarded errors in checkdefaults: - writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT (slug) DO NOTHING, then wrote a check_defs row referencing it. On any re-seed the slug already existed, the entity insert no-oped, and the FK violated — aborting the ingest transaction and surfacing as an unrelated failure several entities later. Re-seeding has been broken since; prod's coverage was frozen at its first successful seed. This is what TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting. - shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed to ".network" and overwrote each other; service:jellyfin collided with lxc:jellyfin. - The ssh-script checker never read the `args` config checkdefaults wrote, so process_check.sh always ran without its unit name and returned "unknown". Coverage is now 75/89. Monitoring is declared per entity type in seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap. coverageSweep raises an `unmonitored` signal only where a type declares monitoring it lacks — 8 real gaps, no false positives. Also: - entity_types.attribute_schema was never ingested: the seed loader read "attribute_schema" but the YAML says "attributes", so all 60 types stored JSON null. - ListExecutions ignored its declared target/action/correlation_id filters and paginated on a non-unique target slug, dropping and repeating rows. - started_at was captured but only written at terminal state, so a running execution reported NULL for its whole life. The three MCP auto-run copies wrote no timing at all; they are now one autoRun helper. - SSH output was buffered to completion and discarded entirely on timeout. Both sshExec copies now stream through a shared execlog sink into execution_logs, and keep partial output when a command is cancelled. - executions.correlation_id was a random per-execution uuid that correlated nothing; it is now the chat session id, which is what lets the chat tail live output. - reversible_low had no auto-run branch despite policy declaring it unattended. Since computeCommandRisk never returns it, the class only arises when an agent declares it over a read_only command — so gating it penalised candor without adding safety. - backup-target gains a backup-freshness checker (portable find -mmin, since the first target is on macOS), resolving its host by walking backs-up-to backwards. The pre-deploy pg_dump is now a tracked backup target. UI: an Executions section on entity detail with live output tailing, and streamed output under a running `run` call in the chat timeline. Migrations 022-024. Ops.svelte and context.ts exclude execution.output from their refetch triggers, which would otherwise fire once a second per command. Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-28 13:51:14 +02:00
parent 873b00ac42
commit 1dca2cfd7a
39 changed files with 3105 additions and 273 deletions
--- a/seeds/inventory.yaml
+++ b/seeds/inventory.yaml
@@ -201,7 +201,17 @@ entities:
  - {slug: "volume:media-local", type: volume, name: media-local,
     attributes: {path: /mnt/media_local}}
  - {slug: "backup:proton-drive", type: backup-target, name: proton-drive,
-     attributes: {provider: proton, encrypted: true}}
+     attributes: {provider: proton, encrypted: true,
+                  path: /mnt/backup,
+                  note: "rclone stages here before pushing to Proton; freshness is checked on lxc:rclone via the backs-up-to edge"}}
+  # The pre-deploy pg_dump written by scripts/deploy.sh on every push to main.
+  # It was the lab's only untracked backup: its failure path is `|| echo
+  # WARNING` inside the deploy script, so a broken dump was invisible until a
+  # rollback needed it.
+  - {slug: "backup:oikos-predeploy", type: backup-target, name: oikos-predeploy,
+     attributes: {provider: local, encrypted: false,
+                  path: /opt/oikos/backups,
+                  note: "pre-deploy pg_dump on the mac-mini; one per deployed SHA"}}

  # ─── Services ──────────────────────────────────────────────────────
  - {slug: "service:proxmox-ui", type: service, name: proxmox_ui,
@@ -536,6 +546,10 @@ relationships:
  - {source: "lxc:romm", target: "pool:ludo-lvm", type: stores-on}
  - {source: "lxc:teddycloud", target: "pool:local-lvm-hubris", type: stores-on}
  - {source: "lxc:rclone", target: "backup:proton-drive", type: backs-up-to}
+  # The mac-mini writes the pre-deploy dumps, so it is also where the freshness
+  # check runs — checkdefaults resolves a backup-target's host by walking this
+  # edge backwards.
+  - {source: "ws:mac-mini", target: "backup:oikos-predeploy", type: backs-up-to}

  # ─── Governance ────────────────────────────────────────────────────
  - {source: "person:dtoro", target: "agent:nomos", type: owns}
--- a/seeds/ontology.yaml
+++ b/seeds/ontology.yaml
@@ -176,6 +176,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Physical location (home, VPS datacenter).
+    monitoring: none              # topological — health is its members'
    attributes: {type: object, properties: {address: {type: string}}}
  ups:
    parent: entity
@@ -183,6 +184,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Uninterruptible power supply.
+    monitoring: none              # warranted, but no SNMP/NUT checker exists yet
    attributes: {type: object, properties: {vendor: {type: string}, va: {type: integer}}}
  sensor:
    parent: entity
@@ -190,12 +192,14 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Environmental sensor.
+    monitoring: none              # readings are metrics, not health
  peripheral:
    parent: entity
    domain: physical
    layer: infrastructure
    lifecycle: infrastructure
    description: Attached hardware (GPU, e-ink display, dongle).
+    monitoring: none              # visible only through its host

  # ── Infrastructure / compute ──
  compute-entity:
@@ -210,6 +214,8 @@ entity_types:
    domain: compute
    layer: infrastructure
    description: Physical machine. Always instantiated as a subtype.
+    monitoring: [ping, resource, updates]  # inherited by proxmox-host /
+                                  # standalone-server / workstation / appliance
    attributes:
      type: object
      properties:
@@ -266,6 +272,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Virtual machine.
+    monitoring: [ping]            # no guest agent assumed; reachability only
    attributes:
      type: object
      properties:
@@ -281,6 +288,7 @@ entity_types:
    domain: compute
    layer: infrastructure
    description: OS-level container (LXC or Docker).
+    monitoring: [resource]        # inherited by lxc / docker-container
    attributes:
      type: object
      properties: {runtime: {type: string}}
@@ -317,6 +325,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Hypervisor software running on a machine (PVE, KVM, OrbStack).
+    monitoring: none              # the hosting machine's checks cover it
    attributes:
      type: object
      properties: {type: {type: string}, version: {type: string}}
@@ -328,6 +337,8 @@ entity_types:
    domain: network
    layer: infrastructure
    description: A network things connect to.
+    monitoring: none              # inherited by lan / mesh / vlan — a network's
+                                  # reachability is a property of its members
  lan:
    parent: network
    domain: network
@@ -360,6 +371,7 @@ entity_types:
    layer: infrastructure
    description: Optional per-interface refinement (mac, ip). The seed uses
      coarse connects-via edges; interfaces can be backfilled later.
+    monitoring: none              # covered by its machine's ping/resource checks
    attributes: {type: object, properties: {mac: {type: string}, ip: {type: string}}}
  dns-zone:
    parent: entity
@@ -367,12 +379,15 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: DNS zone (e.g. split-horizon hubris.network).
+    monitoring: [dns]             # NOTE: no `dns` checker exists yet — this is a
+                                  # real gap and coverageSweep will report it
    attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
  dns-record:
    parent: entity
    domain: network
    layer: infrastructure
    description: Individual DNS record.
+    monitoring: none              # the zone is the unit of monitoring
    attributes:
      type: object
      properties: {name: {type: string}, record_type: {type: string}, value: {type: string}}
@@ -382,6 +397,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Public hostname → upstream mapping (Caddy).
+    monitoring: [http]            # end-to-end: exercises Caddy + DNS + TLS + upstream
    attributes:
      type: object
      properties:
@@ -393,12 +409,14 @@ entity_types:
    domain: network
    layer: infrastructure
    description: TLS certificate.
+    monitoring: [cert-expiry]
    attributes: {type: object, properties: {issuer: {type: string}, expires: {type: string}}}
  firewall-rule:
    parent: entity
    domain: network
    layer: infrastructure
    description: Firewall / port-forward rule.
+    monitoring: none              # declarative config, not a running thing

  # ── Infrastructure / storage ──
  storage-pool:
@@ -407,6 +425,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Storage pool (LVM, ZFS, NFS).
+    monitoring: [capacity]
    attributes:
      type: object
      properties:
@@ -419,6 +438,7 @@ entity_types:
    lifecycle: infrastructure
    description: Named volume / dataset within a pool. Mount details live as
      attributes on `mounts` edges.
+    monitoring: [capacity]
    attributes: {type: object, properties: {size_gb: {type: number}, path: {type: string}}}
  backup-target:
    parent: entity
@@ -426,6 +446,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Where backups land (Proton Drive, PBS).
+    monitoring: [backup-freshness]  # checker lands in Phase 4
    attributes: {type: object, properties: {provider: {type: string}, encrypted: {type: boolean}}}
  dataset:
    parent: entity
@@ -433,6 +454,7 @@ entity_types:
    layer: infrastructure
    description: Logical data collection worth tracking independently of its
      volume (e.g. paperless documents).
+    monitoring: none              # its volume and owning service carry the checks

  # ── Infrastructure / software ──
  service:
@@ -441,6 +463,8 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: A running service with consumers.
+    monitoring: [http, process]   # http when it has a `url`, else a process check
+                                  # on the host resolved through its hosting edge
    attributes:
      type: object
      properties:
@@ -457,12 +481,14 @@ entity_types:
    domain: software
    layer: infrastructure
    description: Deployed application/package a service runs.
+    monitoring: none              # the service in front of it is the probe target
    attributes: {type: object, properties: {version: {type: string}}}
  config-repo:
    parent: entity
    domain: software
    layer: infrastructure
    description: Git repo holding tracked configuration.
+    monitoring: none              # its Gitea service carries the checks
    attributes:
      type: object
      properties: {url: {type: string}, branch: {type: string}}
@@ -471,6 +497,7 @@ entity_types:
    domain: software
    layer: infrastructure
    description: Automated deploy path (webhook → script).
+    monitoring: none              # health is per-deploy, tracked as executions
    attributes:
      type: object
      properties: {trigger: {type: string}, target_path: {type: string}}
@@ -479,12 +506,14 @@ entity_types:
    domain: software
    layer: infrastructure
    description: Managed package baseline for a host class.
+    monitoring: none              # drift shows up via each host's updates check
  cluster:
    parent: entity
    domain: software
    layer: infrastructure
    lifecycle: infrastructure
    description: Proxmox cluster.
+    monitoring: none              # topological — its member hosts carry the checks
    attributes: {type: object, properties: {quorum: {type: string}}}
  compose-stack:
    parent: entity
@@ -492,6 +521,7 @@ entity_types:
    layer: infrastructure
    lifecycle: infrastructure
    description: Docker Compose stack (the Oikos OS itself is one).
+    monitoring: [process]
    attributes: {type: object, properties: {path: {type: string}}}

  # ── Infrastructure / external ──
@@ -500,22 +530,26 @@ entity_types:
    domain: external
    layer: infrastructure
    description: Registered public domain.
+    monitoring: none              # expiry is a calendar concern, not a probe
    attributes: {type: object, properties: {registrar: {type: string}, expires: {type: string}}}
  cloud-service:
    parent: entity
    domain: external
    layer: infrastructure
    description: External SaaS/cloud dependency.
+    monitoring: [http]            # only when the entity carries a `url`
  isp-link:
    parent: entity
    domain: external
    layer: infrastructure
    description: Internet uplink.
+    monitoring: none              # no probe target; reachability shows up fleet-wide
  vendor-dependency:
    parent: entity
    domain: external
    layer: infrastructure
    description: Vendor the lab depends on (registrar, IONOS, Proton).
+    monitoring: none              # a commercial relationship, not a running thing

  # ── Governance / identity ──
  person:
@@ -532,6 +566,8 @@ entity_types:
    layer: governance
    lifecycle: infrastructure   # agents are deployed/retired like infrastructure
    description: Software agent actor (Nomos, the Oikos control loop).
+    monitoring: [http]            # governance layer, but genuinely probeable —
+                                  # Nomos serves a gateway on :8092
    attributes:
      type: object
      properties:
--- a/seeds/policy.yaml
+++ b/seeds/policy.yaml
@@ -44,6 +44,23 @@ risk_classes:
    autonomy_allowed: false

 approval_rules:
+  # Signal kinds are looked up as `action` by internal/policy/classify.go, and
+  # an unmatched kind silently falls back to reversible_low/operator. Declare
+  # `unmonitored` so its routing is intentional: it reports a coverage gap and
+  # there is nothing to remediate automatically — closing it means an operator
+  # adding a check_def, which is its own deliberate change.
+  - {entity_type: entity, action: unmonitored, risk_class: read_only, autonomy_level: auto}
+
+  # Backup freshness signals. Reporting-only for the same reason: the fix for a
+  # stale or missing backup is a deliberate human change (re-run the job, fix
+  # the mount, correct the path), never something to auto-remediate. Declared
+  # so the routing is intentional rather than the reversible_low/operator
+  # fallback an unmatched kind would otherwise get.
+  - {entity_type: backup-target, action: backup-stale, risk_class: read_only, autonomy_level: auto}
+  - {entity_type: backup-target, action: backup-missing, risk_class: read_only, autonomy_level: auto}
+  - {entity_type: backup-target, action: backup-misconfigured, risk_class: read_only, autonomy_level: auto}
+  - {entity_type: backup-target, action: backup-unreachable, risk_class: read_only, autonomy_level: auto}
+
  # ── Generic rules on (possibly abstract) types ──
  - {entity_type: service, action: restart, risk_class: reversible_low, autonomy_level: auto}
  - {entity_type: service, action: cache-clear, risk_class: reversible_low, autonomy_level: auto}