feat(audit): read-only knowledge-graph drift report + skill

Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.

Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.

.agents/skills/knowledge-graph-audit/SKILL.md

@@ -0,0 +1,83 @@
+---
+name: knowledge-graph-audit
+risk_class: read_only
+inputs: []
+verification: "audit_knowledge_graph returns a report with summary.total_findings"
+docs_update_checklist: []
+---
+
+# Knowledge-graph audit
+
+Goal: validate that the knowledge graph (entities, relationships, checks) and
+the monitoring built on it reflect live reality — without mutating anything.
+Read-only. Run this before trusting health, blast-radius, or coverage answers,
+and whenever something feels off (a healthy host reports `down`, a retired
+service still alarms, the graph looks thin).
+
+## 1. Run the drift report
+
+Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
+ranked list of findings, each with `{category, severity, count, entities,
+evidence, suggested_runbook}`, plus a `summary` with totals by category.
+
+The DB-side categories:
+
+- **orphan_checks** — check entities with truncated/random slugs left by the
+  old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
+- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
+  `destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
+  (the scheduler already skips these, but the rows should be retired).
+- **down_checks** — enabled probes reporting `down`. Remediation:
+  `service-health-check` (then check whether the failure is real or a
+  probe-config/routing problem — see step 3).
+- **unknown_checks** — probes that ran but reported `unknown` (usually a
+  misconfigured or not-yet-deployed probe script).
+- **unmonitored** — active entities whose type declares monitoring but have no
+  enabled `check_def`.
+- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
+  destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
+
+## 2. Triage
+
+`severity: critical` (down_checks) first. For each finding, read `evidence` and
+open the entities with `get_entity` / `get_relations` to confirm the diagnosis
+before acting — the report is a pointer, not a verdict.
+
+## 3. Common probe-failure causes
+
+A `down_checks` finding that is NOT a real outage is usually one of:
+
+- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
+  routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
+  edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
+  exec` automatically when both are present.
+- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
+  inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
+- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
+  script flag. The scheduler resolves `user: dtoro` from the entity attribute.
+
+## 4. What this audit does NOT cover (follow-ups)
+
+Live-infrastructure discovery is out of scope for the DB report and must be done
+manually until that machinery lands:
+
+- **Ghost vs missing entities** — cross-check `pct list` / `qm list` (on
+  `host:hubris`, `host:strong`) and `docker ps` against `list_entities`. A guest
+  with no entity, or an entity with no guest, is drift.
+- **Misplaced parent** — compare each guest's actual Proxmox host against its
+  `hosts` edge (migrations leave these stale).
+- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
+- **Unmodeled certs** — Caddy-managed TLS certs with no `certificate` entity.
+- **Seed drift** — run `oikos export` and `git diff seeds/` to find
+  runtime-created entities not in version control.
+
+## 5. Acting on findings
+
+This skill is read-only — make no changes here. Route each confirmed finding to
+its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
+proceed through the normal lifecycle/approval flow. Re-run the audit afterward
+to confirm the finding cleared.
+
+Docs-update checklist: none — the audit reads state; it changes nothing. If a
+finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
+in that remediation session.

internal/audit/audit.go

@@ -0,0 +1,125 @@
+// Package audit produces read-only drift reports over the knowledge graph and
+// monitoring state. It is the shared engine behind the
+// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
+//
+// It surfaces the structural gaps an operator otherwise discovers only by
+// accident: orphan check entities, checks targeting retired entities, probes
+// stuck down/unknown, unmonitored declared types, and live edges pointing at
+// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
+// follow-up that needs host-hop execution; these categories are pure DB
+// queries, so the report is cheap, safe to run unattended, and testable.
+package audit
+
+import (
+	"context"
+
+	"github.com/dtoro/oikos/internal/db"
+)
+
+// Finding is one drift item the operator should look at.
+type Finding struct {
+	Category         string   `json:"category"`
+	Severity         string   `json:"severity"` // info | warning | critical
+	Count            int      `json:"count"`
+	Entities         []string `json:"entities"`
+	Evidence         string   `json:"evidence"`
+	SuggestedRunbook string   `json:"suggested_runbook"`
+}
+
+// Summary tallies findings by category.
+type Summary struct {
+	TotalFindings int            `json:"total_findings"`
+	ByCategory    map[string]int `json:"by_category"`
+}
+
+// Report runs every drift check and returns the findings plus a summary.
+func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
+	specs := []struct {
+		finding Finding
+		query   string
+	}{
+		{
+			Finding{Category: "orphan_checks", Severity: "warning",
+				Evidence:         "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
+				SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
+			`SELECT e.slug FROM entities e
+			 WHERE e.type = 'check'
+			   AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
+		},
+		{
+			Finding{Category: "dead_checks", Severity: "warning",
+				Evidence:         "enabled check_defs whose target entity is deprecated/destroyed",
+				SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
+			`SELECT e.slug FROM check_defs cd
+			 JOIN entities e ON e.id = cd.entity_id
+			 JOIN entities tgt ON tgt.id = cd.target_id
+			 WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
+		},
+		{
+			Finding{Category: "down_checks", Severity: "critical",
+				Evidence:         "enabled checks reporting health=down",
+				SuggestedRunbook: "service-health-check"},
+			`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
+			 WHERE cd.enabled AND cd.last_health = 'down'`,
+		},
+		{
+			Finding{Category: "unknown_checks", Severity: "warning",
+				Evidence:         "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
+				SuggestedRunbook: "knowledge-graph-audit"},
+			`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
+			 WHERE cd.enabled AND cd.last_health = 'unknown'`,
+		},
+		{
+			Finding{Category: "unmonitored", Severity: "warning",
+				Evidence:         "active entities whose type declares monitoring but have no enabled check_def",
+				SuggestedRunbook: "knowledge-graph-audit"},
+			`SELECT DISTINCT e.slug FROM signals sg
+			 JOIN entities e ON e.id = sg.target_entity_id
+			 WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
+		},
+		{
+			Finding{Category: "dangling_edges", Severity: "warning",
+				Evidence:         "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
+				SuggestedRunbook: "lifecycle-destroy-node"},
+			`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
+			 JOIN entities src ON src.id = r.source_id
+			 JOIN entities tgt ON tgt.id = r.target_id
+			 WHERE r.valid_to IS NULL
+			   AND src.state NOT IN ('destroyed','deprecated')
+			   AND tgt.state IN ('destroyed','deprecated')`,
+		},
+	}
+
+	findings := make([]Finding, 0, len(specs))
+	summary := Summary{ByCategory: map[string]int{}}
+	for _, sp := range specs {
+		f := runFinding(ctx, pool, sp.finding, sp.query)
+		findings = append(findings, f)
+		summary.TotalFindings += f.Count
+		summary.ByCategory[f.Category] = f.Count
+	}
+	return findings, summary
+}
+
+const entityCap = 50
+
+// runFinding runs a single-column slug query and folds the rows into a Finding.
+func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
+	rows, err := pool.Query(ctx, query)
+	if err != nil {
+		f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
+		return f
+	}
+	defer rows.Close()
+	for rows.Next() {
+		var slug string
+		if err := rows.Scan(&slug); err != nil {
+			continue
+		}
+		f.Count++
+		if len(f.Entities) < entityCap {
+			f.Entities = append(f.Entities, slug)
+		}
+	}
+	return f
+}

internal/audit/audit_test.go

@@ -0,0 +1,69 @@
+package audit
+
+import (
+	"context"
+	"testing"
+
+	"github.com/dtoro/oikos/internal/db"
+	"github.com/google/uuid"
+)
+
+// Integration tests against a real Postgres, guarded by
+// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
+
+func newAuditPool(t *testing.T) *db.Pool {
+	t.Helper()
+	base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
+	if base == "" {
+		t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
+	}
+	return createTestDB(t, base)
+}
+
+func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
+	pool := newAuditPool(t)
+	ctx := context.Background()
+
+	// An orphan check entity (truncated random slug, the legacy bug shape).
+	mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
+		VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
+
+	// An active entity + a check_def on it stuck down.
+	target := uuid.New()
+	mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
+		VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
+	checkE := uuid.New()
+	mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
+		VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
+	mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
+		VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
+
+	// A deprecated entity still carrying an enabled check (dead_checks).
+	dep := uuid.New()
+	mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
+		VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
+	depCheck := uuid.New()
+	mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
+		VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
+	mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
+		VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
+
+	findings, summary := Report(ctx, pool)
+
+	byCat := map[string]int{}
+	for _, f := range findings {
+		byCat[f.Category] = f.Count
+	}
+	if byCat["orphan_checks"] < 1 {
+		t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
+	}
+	if byCat["down_checks"] < 1 {
+		t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
+	}
+	if byCat["dead_checks"] < 1 {
+		t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
+	}
+	if summary.TotalFindings < 3 {
+		t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
+	}
+}

internal/audit/testutil_test.go

@@ -0,0 +1,67 @@
+package audit
+
+import (
+	"context"
+	"fmt"
+	"math/rand"
+	"os"
+	"strings"
+	"testing"
+
+	"github.com/dtoro/oikos/internal/db"
+	"github.com/jackc/pgx/v5"
+)
+
+// createTestDB provisions a throwaway migrated database, same convention as
+// internal/scheduler/coverage_test.go.
+func createTestDB(t *testing.T, baseURL string) *db.Pool {
+	t.Helper()
+	ctx := context.Background()
+
+	admin, err := pgx.Connect(ctx, baseURL)
+	if err != nil {
+		t.Fatalf("connect admin: %v", err)
+	}
+	dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
+	if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
+		admin.Close(ctx)
+		t.Fatalf("create test db: %v", err)
+	}
+	admin.Close(ctx)
+
+	at := strings.LastIndex(baseURL, "/")
+	testURL := baseURL[:at+1] + dbName
+	if q := strings.Index(baseURL[at:], "?"); q >= 0 {
+		testURL += baseURL[at+q:]
+	}
+
+	pool, err := db.New(ctx, testURL)
+	if err != nil {
+		t.Fatalf("connect test db: %v", err)
+	}
+	if err := pool.Migrate(ctx); err != nil {
+		t.Fatalf("migrate: %v", err)
+	}
+	t.Cleanup(func() {
+		pool.Close()
+		if admin, err := pgx.Connect(ctx, baseURL); err == nil {
+			admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
+			admin.Close(ctx)
+		}
+	})
+	return pool
+}
+
+func getenvOrDefault(key, def string) string {
+	if v := os.Getenv(key); v != "" {
+		return v
+	}
+	return def
+}
+
+func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
+	t.Helper()
+	if _, err := pool.Exec(ctx, q, args...); err != nil {
+		t.Fatalf("exec %s: %v", q, err)
+	}
+}

internal/httpapi/audit.go

@@ -0,0 +1,20 @@
+package httpapi
+
+import (
+	"net/http"
+
+	"github.com/dtoro/oikos/internal/audit"
+)
+
+// serveAuditDrift returns a read-only DB-side drift report: orphan check
+// entities, checks on retired targets, probes stuck down/unknown, unmonitored
+// declared types, and dangling edges. Companion to the knowledge-graph-audit
+// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
+func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
+	findings, summary := audit.Report(req.Context(), s.pool)
+	writeJSON(w, map[string]any{
+		"findings": findings,
+		"summary":  summary,
+		"note":     "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
+	})
+}

internal/httpapi/server.go

@@ -237,6 +237,11 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
 	r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
 	r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
 
+	// Drift audit: read-only DB-side report of orphan checks, checks on
+	// retired targets, stuck down/unknown probes, unmonitored declared types,
+	// and dangling edges. Companion to the knowledge-graph-audit skill.
+	r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
+
 	// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
 	// unlike ListExecutions which sorts by target for pagination) and the
 	// per-session "what did this session do" digest.

internal/mcp/tools.go

@@ -6,6 +6,7 @@ import (
 	"fmt"
 	"strings"
 
+	"github.com/dtoro/oikos/internal/audit"
 	"github.com/dtoro/oikos/internal/db"
 	"github.com/dtoro/oikos/internal/policy"
 	"github.com/google/uuid"
@@ -782,6 +783,14 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
 		`), "fleet_snapshot"), nil
 		}},
 
+		{tool: &mcp.Tool{Name: "audit_knowledge_graph", Description: "Read-only drift report over the knowledge graph and monitoring: orphan check entities, checks targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored declared entity types, and live edges pointing at destroyed targets. Returns ranked findings with a suggested remediation runbook each. Use this to validate the graph is complete and consistent before trusting health/blast-radius answers. Does NOT mutate anything.",
+			InputSchema: objSchema(),
+		}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
+			findings, summary := audit.Report(ctx, pool)
+			b, _ := json.Marshal(map[string]any{"findings": findings, "summary": summary})
+			return textResult(string(b)), nil
+		}},
+
 		{tool: &mcp.Tool{Name: "list_my_secrets", Description: "List secrets accessible to this client by public key",
 			InputSchema: objSchema(prop{"caller_pubkey", "string", "Age public key of the caller (optional)"}),
 		}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {

seeds/knowledge.yaml

@@ -6372,6 +6372,23 @@ investigations:
   tags:
   - investigation
 runbooks:
+- slug: knowledge-graph-audit
+  name: Knowledge-graph audit
+  risk_class: read_only
+  entity_type: entity
+  procedure: {}
+  content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\
+    \ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\
+    \ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\
+    \ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\
+    \ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\
+    \ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\
+    \ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\
+    \ each with `get_entity`/`get_relations` before acting. This skill makes no changes\
+    \ \u2014 route confirmed findings to their remediation runbook and re-run the audit\
+    \ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\
+    \ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\
+    \ follow-up until that machinery lands.\n"
 - slug: client-enrollment
   name: Client enrollment
   risk_class: read_only