feat(observability): restore monitoring coverage, make gaps visible, stream executions
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by discarded errors in checkdefaults: - writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT (slug) DO NOTHING, then wrote a check_defs row referencing it. On any re-seed the slug already existed, the entity insert no-oped, and the FK violated — aborting the ingest transaction and surfacing as an unrelated failure several entities later. Re-seeding has been broken since; prod's coverage was frozen at its first successful seed. This is what TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting. - shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed to ".network" and overwrote each other; service:jellyfin collided with lxc:jellyfin. - The ssh-script checker never read the `args` config checkdefaults wrote, so process_check.sh always ran without its unit name and returned "unknown". Coverage is now 75/89. Monitoring is declared per entity type in seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap. coverageSweep raises an `unmonitored` signal only where a type declares monitoring it lacks — 8 real gaps, no false positives. Also: - entity_types.attribute_schema was never ingested: the seed loader read "attribute_schema" but the YAML says "attributes", so all 60 types stored JSON null. - ListExecutions ignored its declared target/action/correlation_id filters and paginated on a non-unique target slug, dropping and repeating rows. - started_at was captured but only written at terminal state, so a running execution reported NULL for its whole life. The three MCP auto-run copies wrote no timing at all; they are now one autoRun helper. - SSH output was buffered to completion and discarded entirely on timeout. Both sshExec copies now stream through a shared execlog sink into execution_logs, and keep partial output when a command is cancelled. - executions.correlation_id was a random per-execution uuid that correlated nothing; it is now the chat session id, which is what lets the chat tail live output. - reversible_low had no auto-run branch despite policy declaring it unattended. Since computeCommandRisk never returns it, the class only arises when an agent declares it over a read_only command — so gating it penalised candor without adding safety. - backup-target gains a backup-freshness checker (portable find -mmin, since the first target is on macOS), resolving its host by walking backs-up-to backwards. The pre-deploy pg_dump is now a tracked backup target. UI: an Executions section on entity detail with live output tailing, and streamed output under a running `run` call in the chat timeline. Migrations 022-024. Ops.svelte and context.ts exclude execution.output from their refetch triggers, which would otherwise fire once a second per command. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
90
internal/ontology/monitoring.go
Normal file
90
internal/ontology/monitoring.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package ontology
|
||||
|
||||
// Monitoring resolution: which check kinds an entity type warrants.
|
||||
//
|
||||
// Coverage is not uniform. A `service` warrants an HTTP probe; a `site` is a
|
||||
// physical location with nothing to probe; a `dns-zone` warrants a check whose
|
||||
// checker does not exist yet. Collapsing those three into "has no check_def"
|
||||
// is what made the fleet's monitoring gap invisible — 86 of 89 active entities
|
||||
// had no check, and staleSweep's INNER JOIN against check_defs meant none of
|
||||
// them could ever be marked stale.
|
||||
//
|
||||
// So the declaration lives on the entity TYPE, in seeds/ontology.yaml, and
|
||||
// resolves through the same is-a hierarchy the validator already walks:
|
||||
// declaring `monitoring: [ping, resource]` on abstract `machine` covers
|
||||
// proxmox-host, standalone-server, workstation and appliance.
|
||||
|
||||
// MonitoringResolution is the outcome of resolving a type's monitoring
|
||||
// declaration. The three states are deliberately distinguishable:
|
||||
//
|
||||
// Declared=false — nobody in the chain said anything. An ontology
|
||||
// gap: report it, but as a modelling problem
|
||||
// rather than as a fleet monitoring problem.
|
||||
// Declared=true, len(0) — explicitly unmonitorable. Working as intended;
|
||||
// never raise an `unmonitored` signal for it.
|
||||
// Declared=true, len(n) — these kinds are expected to exist.
|
||||
type MonitoringResolution struct {
|
||||
Kinds []string
|
||||
|
||||
// Declared reports whether anything in the chain (or the layer default)
|
||||
// settled the question.
|
||||
Declared bool
|
||||
|
||||
// Source names the type that supplied the answer — the type itself, an
|
||||
// ancestor, or "" when the layer default applied. Useful in log lines
|
||||
// that explain why an entity has the checks it has.
|
||||
Source string
|
||||
}
|
||||
|
||||
// None reports an explicit "this type is not monitored".
|
||||
func (m MonitoringResolution) None() bool {
|
||||
return m.Declared && len(m.Kinds) == 0
|
||||
}
|
||||
|
||||
// Wants reports whether the type expects a check of this kind.
|
||||
func (m MonitoringResolution) Wants(kind string) bool {
|
||||
for _, k := range m.Kinds {
|
||||
if k == kind {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Monitoring resolves the check kinds a type warrants, walking parent types
|
||||
// until one carries a declaration.
|
||||
//
|
||||
// Types outside the infrastructure layer (governance, cognition, meta) fall
|
||||
// back to an implicit "none": a signal, an approval, a document and a person
|
||||
// are records, not running things. That default keeps ~20 record types out of
|
||||
// the ontology without needing an explicit `monitoring: none` on each, while
|
||||
// still treating an undeclared *infrastructure* type as a genuine gap — those
|
||||
// are the ones somebody should have made a decision about. A non-infrastructure
|
||||
// type that really is probeable (agent, which serves a gateway on :8092) just
|
||||
// declares its kinds explicitly and wins on the first rule.
|
||||
func (t *TypeTree) Monitoring(typ string) MonitoringResolution {
|
||||
seen := map[string]bool{}
|
||||
for cur := typ; cur != ""; cur = t.Types[cur].Parent {
|
||||
info, ok := t.Types[cur]
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
if seen[cur] {
|
||||
break // cycle guard — ingest rejects cycles, belt and braces
|
||||
}
|
||||
seen[cur] = true
|
||||
|
||||
if info.Monitoring != nil {
|
||||
return MonitoringResolution{
|
||||
Kinds: *info.Monitoring,
|
||||
Declared: true,
|
||||
Source: cur,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if info, ok := t.Types[typ]; ok && info.Layer != "infrastructure" {
|
||||
return MonitoringResolution{Declared: true}
|
||||
}
|
||||
return MonitoringResolution{}
|
||||
}
|
||||
121
internal/ontology/monitoring_test.go
Normal file
121
internal/ontology/monitoring_test.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package ontology
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func kinds(v ...string) *[]string {
|
||||
s := append([]string{}, v...)
|
||||
return &s
|
||||
}
|
||||
|
||||
// monitoringTree mirrors the real shape of seeds/ontology.yaml: a declaration
|
||||
// on an abstract type that concrete subtypes inherit, an explicit none on a
|
||||
// topological type, a probeable type outside the infrastructure layer, and an
|
||||
// undeclared infrastructure type (the ontology gap this is meant to catch).
|
||||
func monitoringTree() *TypeTree {
|
||||
return &TypeTree{
|
||||
Types: map[string]TypeInfo{
|
||||
"entity": {IsAbstract: true, Layer: "meta"},
|
||||
"compute-entity": {Parent: "entity", IsAbstract: true, Layer: "infrastructure"},
|
||||
"machine": {Parent: "compute-entity", IsAbstract: true, Layer: "infrastructure",
|
||||
Monitoring: kinds("ping", "resource")},
|
||||
"proxmox-host": {Parent: "machine", Layer: "infrastructure"},
|
||||
"workstation": {Parent: "machine", Layer: "infrastructure"},
|
||||
"service": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds("http", "process")},
|
||||
"site": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds()},
|
||||
"vlan": {Parent: "entity", Layer: "infrastructure"}, // undeclared: a gap
|
||||
"agent": {Parent: "entity", Layer: "governance", Monitoring: kinds("http")},
|
||||
"signal": {Parent: "entity", Layer: "cognition"},
|
||||
"document": {Parent: "entity", Layer: "governance"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringResolvesThroughHierarchy(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
cases := []struct {
|
||||
typ string
|
||||
wantKinds []string
|
||||
wantDecl bool
|
||||
wantSource string
|
||||
desc string
|
||||
}{
|
||||
{"machine", []string{"ping", "resource"}, true, "machine", "declared on itself"},
|
||||
{"proxmox-host", []string{"ping", "resource"}, true, "machine", "inherited from abstract parent"},
|
||||
{"workstation", []string{"ping", "resource"}, true, "machine", "inherited by a sibling too"},
|
||||
{"service", []string{"http", "process"}, true, "service", "declared on itself"},
|
||||
{"site", nil, true, "site", "explicitly none — not a gap"},
|
||||
{"agent", []string{"http"}, true, "agent", "explicit declaration beats the layer default"},
|
||||
{"signal", nil, true, "", "cognition layer is implicitly none"},
|
||||
{"document", nil, true, "", "governance layer is implicitly none"},
|
||||
{"vlan", nil, false, "", "undeclared infrastructure type is a genuine gap"},
|
||||
{"nonexistent", nil, false, "", "unknown type resolves to undeclared"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
got := tree.Monitoring(c.typ)
|
||||
if got.Declared != c.wantDecl {
|
||||
t.Errorf("%s (%s): Declared = %v, want %v", c.typ, c.desc, got.Declared, c.wantDecl)
|
||||
}
|
||||
if got.Source != c.wantSource {
|
||||
t.Errorf("%s (%s): Source = %q, want %q", c.typ, c.desc, got.Source, c.wantSource)
|
||||
}
|
||||
if len(got.Kinds) != len(c.wantKinds) {
|
||||
t.Errorf("%s (%s): Kinds = %v, want %v", c.typ, c.desc, got.Kinds, c.wantKinds)
|
||||
continue
|
||||
}
|
||||
for i, k := range c.wantKinds {
|
||||
if got.Kinds[i] != k {
|
||||
t.Errorf("%s (%s): Kinds[%d] = %q, want %q", c.typ, c.desc, i, got.Kinds[i], k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The distinction between these two is what keeps coverageSweep from raising
|
||||
// permanent, unresolvable signals against entities that are working as intended.
|
||||
func TestMonitoringNoneIsNotTheSameAsUndeclared(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
site := tree.Monitoring("site")
|
||||
if !site.None() {
|
||||
t.Error("site declared `monitoring: none`, expected None() to report true")
|
||||
}
|
||||
|
||||
vlan := tree.Monitoring("vlan")
|
||||
if vlan.None() {
|
||||
t.Error("vlan declared nothing at all — None() must not claim it opted out")
|
||||
}
|
||||
if vlan.Declared {
|
||||
t.Error("vlan is an undeclared infrastructure type; it should read as a gap")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringWants(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
svc := tree.Monitoring("service")
|
||||
if !svc.Wants("http") {
|
||||
t.Error("service should want an http check")
|
||||
}
|
||||
if svc.Wants("resource") {
|
||||
t.Error("service should not want a resource check")
|
||||
}
|
||||
if tree.Monitoring("site").Wants("http") {
|
||||
t.Error("an explicitly unmonitorable type wants nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringSurvivesParentCycle(t *testing.T) {
|
||||
// The ingest rejects cycles; this guards the walker regardless.
|
||||
tree := &TypeTree{Types: map[string]TypeInfo{
|
||||
"a": {Parent: "b", Layer: "infrastructure"},
|
||||
"b": {Parent: "a", Layer: "infrastructure"},
|
||||
}}
|
||||
got := tree.Monitoring("a")
|
||||
if got.Declared {
|
||||
t.Errorf("cyclic chain declared nothing, got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,13 @@ type TypeInfo struct {
|
||||
Parent string
|
||||
IsAbstract bool
|
||||
LifecycleID string
|
||||
Layer string
|
||||
|
||||
// Monitoring is this type's own `monitoring:` declaration, or nil if it
|
||||
// declared nothing (in which case the answer comes from an ancestor, or
|
||||
// from the layer default). A non-nil pointer to an empty slice means
|
||||
// "explicitly unmonitorable" — see TypeTree.Monitoring.
|
||||
Monitoring *[]string
|
||||
}
|
||||
|
||||
// RelTypeInfo is the subset of a relationship type the validator needs.
|
||||
|
||||
Reference in New Issue
Block a user