diff --git a/.agents/skills/knowledge-graph-audit/SKILL.md b/.agents/skills/knowledge-graph-audit/SKILL.md new file mode 100644 index 0000000..2e3ce6d --- /dev/null +++ b/.agents/skills/knowledge-graph-audit/SKILL.md @@ -0,0 +1,83 @@ +--- +name: knowledge-graph-audit +risk_class: read_only +inputs: [] +verification: "audit_knowledge_graph returns a report with summary.total_findings" +docs_update_checklist: [] +--- + +# Knowledge-graph audit + +Goal: validate that the knowledge graph (entities, relationships, checks) and +the monitoring built on it reflect live reality — without mutating anything. +Read-only. Run this before trusting health, blast-radius, or coverage answers, +and whenever something feels off (a healthy host reports `down`, a retired +service still alarms, the graph looks thin). + +## 1. Run the drift report + +Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a +ranked list of findings, each with `{category, severity, count, entities, +evidence, suggested_runbook}`, plus a `summary` with totals by category. + +The DB-side categories: + +- **orphan_checks** — check entities with truncated/random slugs left by the + old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`. +- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/ + `destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node` + (the scheduler already skips these, but the rows should be retired). +- **down_checks** — enabled probes reporting `down`. Remediation: + `service-health-check` (then check whether the failure is real or a + probe-config/routing problem — see step 3). +- **unknown_checks** — probes that ran but reported `unknown` (usually a + misconfigured or not-yet-deployed probe script). +- **unmonitored** — active entities whose type declares monitoring but have no + enabled `check_def`. +- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at + destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`. + +## 2. Triage + +`severity: critical` (down_checks) first. For each finding, read `evidence` and +open the entities with `get_entity` / `get_relations` to confirm the diagnosis +before acting — the report is a pointer, not a verdict. + +## 3. Common probe-failure causes + +A `down_checks` finding that is NOT a real outage is usually one of: + +- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of + routing through its Proxmox host. Confirm with `get_relations` that a `hosts` + edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest + exec` automatically when both are present. +- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/` + inside the target. Remediation: redeploy via `tools/deploy-checks.sh`. +- **macOS host** — a workstation check used the wrong SSH user or a Linux-only + script flag. The scheduler resolves `user: dtoro` from the entity attribute. + +## 4. What this audit does NOT cover (follow-ups) + +Live-infrastructure discovery is out of scope for the DB report and must be done +manually until that machinery lands: + +- **Ghost vs missing entities** — cross-check `pct list` / `qm list` (on + `host:hubris`, `host:strong`) and `docker ps` against `list_entities`. A guest + with no entity, or an entity with no guest, is drift. +- **Misplaced parent** — compare each guest's actual Proxmox host against its + `hosts` edge (migrations leave these stale). +- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence. +- **Unmodeled certs** — Caddy-managed TLS certs with no `certificate` entity. +- **Seed drift** — run `oikos export` and `git diff seeds/` to find + runtime-created entities not in version control. + +## 5. Acting on findings + +This skill is read-only — make no changes here. Route each confirmed finding to +its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and +proceed through the normal lifecycle/approval flow. Re-run the audit afterward +to confirm the finding cleared. + +Docs-update checklist: none — the audit reads state; it changes nothing. If a +finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml` +in that remediation session. diff --git a/internal/audit/audit.go b/internal/audit/audit.go new file mode 100644 index 0000000..be30051 --- /dev/null +++ b/internal/audit/audit.go @@ -0,0 +1,125 @@ +// Package audit produces read-only drift reports over the knowledge graph and +// monitoring state. It is the shared engine behind the +// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool. +// +// It surfaces the structural gaps an operator otherwise discovers only by +// accident: orphan check entities, checks targeting retired entities, probes +// stuck down/unknown, unmonitored declared types, and live edges pointing at +// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a +// follow-up that needs host-hop execution; these categories are pure DB +// queries, so the report is cheap, safe to run unattended, and testable. +package audit + +import ( + "context" + + "github.com/dtoro/oikos/internal/db" +) + +// Finding is one drift item the operator should look at. +type Finding struct { + Category string `json:"category"` + Severity string `json:"severity"` // info | warning | critical + Count int `json:"count"` + Entities []string `json:"entities"` + Evidence string `json:"evidence"` + SuggestedRunbook string `json:"suggested_runbook"` +} + +// Summary tallies findings by category. +type Summary struct { + TotalFindings int `json:"total_findings"` + ByCategory map[string]int `json:"by_category"` +} + +// Report runs every drift check and returns the findings plus a summary. +func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) { + specs := []struct { + finding Finding + query string + }{ + { + Finding{Category: "orphan_checks", Severity: "warning", + Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target", + SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"}, + `SELECT e.slug FROM entities e + WHERE e.type = 'check' + AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`, + }, + { + Finding{Category: "dead_checks", Severity: "warning", + Evidence: "enabled check_defs whose target entity is deprecated/destroyed", + SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"}, + `SELECT e.slug FROM check_defs cd + JOIN entities e ON e.id = cd.entity_id + JOIN entities tgt ON tgt.id = cd.target_id + WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`, + }, + { + Finding{Category: "down_checks", Severity: "critical", + Evidence: "enabled checks reporting health=down", + SuggestedRunbook: "service-health-check"}, + `SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id + WHERE cd.enabled AND cd.last_health = 'down'`, + }, + { + Finding{Category: "unknown_checks", Severity: "warning", + Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)", + SuggestedRunbook: "knowledge-graph-audit"}, + `SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id + WHERE cd.enabled AND cd.last_health = 'unknown'`, + }, + { + Finding{Category: "unmonitored", Severity: "warning", + Evidence: "active entities whose type declares monitoring but have no enabled check_def", + SuggestedRunbook: "knowledge-graph-audit"}, + `SELECT DISTINCT e.slug FROM signals sg + JOIN entities e ON e.id = sg.target_entity_id + WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`, + }, + { + Finding{Category: "dangling_edges", Severity: "warning", + Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets", + SuggestedRunbook: "lifecycle-destroy-node"}, + `SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r + JOIN entities src ON src.id = r.source_id + JOIN entities tgt ON tgt.id = r.target_id + WHERE r.valid_to IS NULL + AND src.state NOT IN ('destroyed','deprecated') + AND tgt.state IN ('destroyed','deprecated')`, + }, + } + + findings := make([]Finding, 0, len(specs)) + summary := Summary{ByCategory: map[string]int{}} + for _, sp := range specs { + f := runFinding(ctx, pool, sp.finding, sp.query) + findings = append(findings, f) + summary.TotalFindings += f.Count + summary.ByCategory[f.Category] = f.Count + } + return findings, summary +} + +const entityCap = 50 + +// runFinding runs a single-column slug query and folds the rows into a Finding. +func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding { + rows, err := pool.Query(ctx, query) + if err != nil { + f.Evidence = f.Evidence + " (query error: " + err.Error() + ")" + return f + } + defer rows.Close() + for rows.Next() { + var slug string + if err := rows.Scan(&slug); err != nil { + continue + } + f.Count++ + if len(f.Entities) < entityCap { + f.Entities = append(f.Entities, slug) + } + } + return f +} diff --git a/internal/audit/audit_test.go b/internal/audit/audit_test.go new file mode 100644 index 0000000..bd674dc --- /dev/null +++ b/internal/audit/audit_test.go @@ -0,0 +1,69 @@ +package audit + +import ( + "context" + "testing" + + "github.com/dtoro/oikos/internal/db" + "github.com/google/uuid" +) + +// Integration tests against a real Postgres, guarded by +// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler). + +func newAuditPool(t *testing.T) *db.Pool { + t.Helper() + base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "") + if base == "" { + t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test") + } + return createTestDB(t, base) +} + +func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) { + pool := newAuditPool(t) + ctx := context.Background() + + // An orphan check entity (truncated random slug, the legacy bug shape). + mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at) + VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New()) + + // An active entity + a check_def on it stuck down. + target := uuid.New() + mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at) + VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target) + checkE := uuid.New() + mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at) + VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE) + mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health) + VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target) + + // A deprecated entity still carrying an enabled check (dead_checks). + dep := uuid.New() + mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at) + VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep) + depCheck := uuid.New() + mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at) + VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck) + mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at) + VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep) + + findings, summary := Report(ctx, pool) + + byCat := map[string]int{} + for _, f := range findings { + byCat[f.Category] = f.Count + } + if byCat["orphan_checks"] < 1 { + t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"]) + } + if byCat["down_checks"] < 1 { + t.Errorf("down_checks = %d, want >=1", byCat["down_checks"]) + } + if byCat["dead_checks"] < 1 { + t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"]) + } + if summary.TotalFindings < 3 { + t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings) + } +} diff --git a/internal/audit/testutil_test.go b/internal/audit/testutil_test.go new file mode 100644 index 0000000..619ebe3 --- /dev/null +++ b/internal/audit/testutil_test.go @@ -0,0 +1,67 @@ +package audit + +import ( + "context" + "fmt" + "math/rand" + "os" + "strings" + "testing" + + "github.com/dtoro/oikos/internal/db" + "github.com/jackc/pgx/v5" +) + +// createTestDB provisions a throwaway migrated database, same convention as +// internal/scheduler/coverage_test.go. +func createTestDB(t *testing.T, baseURL string) *db.Pool { + t.Helper() + ctx := context.Background() + + admin, err := pgx.Connect(ctx, baseURL) + if err != nil { + t.Fatalf("connect admin: %v", err) + } + dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63()) + if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil { + admin.Close(ctx) + t.Fatalf("create test db: %v", err) + } + admin.Close(ctx) + + at := strings.LastIndex(baseURL, "/") + testURL := baseURL[:at+1] + dbName + if q := strings.Index(baseURL[at:], "?"); q >= 0 { + testURL += baseURL[at+q:] + } + + pool, err := db.New(ctx, testURL) + if err != nil { + t.Fatalf("connect test db: %v", err) + } + if err := pool.Migrate(ctx); err != nil { + t.Fatalf("migrate: %v", err) + } + t.Cleanup(func() { + pool.Close() + if admin, err := pgx.Connect(ctx, baseURL); err == nil { + admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)") + admin.Close(ctx) + } + }) + return pool +} + +func getenvOrDefault(key, def string) string { + if v := os.Getenv(key); v != "" { + return v + } + return def +} + +func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) { + t.Helper() + if _, err := pool.Exec(ctx, q, args...); err != nil { + t.Fatalf("exec %s: %v", q, err) + } +} diff --git a/internal/httpapi/audit.go b/internal/httpapi/audit.go new file mode 100644 index 0000000..4506dd1 --- /dev/null +++ b/internal/httpapi/audit.go @@ -0,0 +1,20 @@ +package httpapi + +import ( + "net/http" + + "github.com/dtoro/oikos/internal/audit" +) + +// serveAuditDrift returns a read-only DB-side drift report: orphan check +// entities, checks on retired targets, probes stuck down/unknown, unmonitored +// declared types, and dangling edges. Companion to the knowledge-graph-audit +// skill. Live-infra discovery (pct/docker/certs) is a follow-up. +func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) { + findings, summary := audit.Report(req.Context(), s.pool) + writeJSON(w, map[string]any{ + "findings": findings, + "summary": summary, + "note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up", + }) +} diff --git a/internal/httpapi/server.go b/internal/httpapi/server.go index 6402f04..cc55d11 100644 --- a/internal/httpapi/server.go +++ b/internal/httpapi/server.go @@ -237,6 +237,11 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans) r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge) + // Drift audit: read-only DB-side report of orphan checks, checks on + // retired targets, stuck down/unknown probes, unmonitored declared types, + // and dangling edges. Companion to the knowledge-graph-audit skill. + r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift) + // Custom (non-OpenAPI) routes: the global activity feed (recency-ordered, // unlike ListExecutions which sorts by target for pagination) and the // per-session "what did this session do" digest. diff --git a/internal/mcp/tools.go b/internal/mcp/tools.go index 20c0004..70e75ee 100644 --- a/internal/mcp/tools.go +++ b/internal/mcp/tools.go @@ -6,6 +6,7 @@ import ( "fmt" "strings" + "github.com/dtoro/oikos/internal/audit" "github.com/dtoro/oikos/internal/db" "github.com/dtoro/oikos/internal/policy" "github.com/google/uuid" @@ -782,6 +783,14 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg { `), "fleet_snapshot"), nil }}, + {tool: &mcp.Tool{Name: "audit_knowledge_graph", Description: "Read-only drift report over the knowledge graph and monitoring: orphan check entities, checks targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored declared entity types, and live edges pointing at destroyed targets. Returns ranked findings with a suggested remediation runbook each. Use this to validate the graph is complete and consistent before trusting health/blast-radius answers. Does NOT mutate anything.", + InputSchema: objSchema(), + }, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) { + findings, summary := audit.Report(ctx, pool) + b, _ := json.Marshal(map[string]any{"findings": findings, "summary": summary}) + return textResult(string(b)), nil + }}, + {tool: &mcp.Tool{Name: "list_my_secrets", Description: "List secrets accessible to this client by public key", InputSchema: objSchema(prop{"caller_pubkey", "string", "Age public key of the caller (optional)"}), }, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) { diff --git a/seeds/knowledge.yaml b/seeds/knowledge.yaml index 6e1fec4..bfb6720 100644 --- a/seeds/knowledge.yaml +++ b/seeds/knowledge.yaml @@ -6372,6 +6372,23 @@ investigations: tags: - investigation runbooks: +- slug: knowledge-graph-audit + name: Knowledge-graph audit + risk_class: read_only + entity_type: entity + procedure: {} + content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\ + \ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\ + \ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\ + \ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\ + \ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\ + \ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\ + \ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\ + \ each with `get_entity`/`get_relations` before acting. This skill makes no changes\ + \ \u2014 route confirmed findings to their remediation runbook and re-run the audit\ + \ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\ + \ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\ + \ follow-up until that machinery lands.\n" - slug: client-enrollment name: Client enrollment risk_class: read_only