dtoro
0929c17cbb
The DB-only audit_knowledge_graph can't see guests running in Proxmox that have no entity, or entities whose pve_id is no longer live — the drift that the stray test LXCs were a symptom of. discover_infra_drift enumerates running guests via pct/qm list on every proxmox host (over the same SSH/pct path the checks use) and diffs against the DB: returns missing (live, no entity) and ghost (DB, not live). Read-only. Companion to audit_knowledge_graph; the skill now runs both and treats the remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
88 lines
4.0 KiB
Markdown
88 lines
4.0 KiB
Markdown
---
|
|
name: knowledge-graph-audit
|
|
risk_class: read_only
|
|
inputs: []
|
|
verification: "audit_knowledge_graph returns a report with summary.total_findings"
|
|
docs_update_checklist: []
|
|
---
|
|
|
|
# Knowledge-graph audit
|
|
|
|
Goal: validate that the knowledge graph (entities, relationships, checks) and
|
|
the monitoring built on it reflect live reality — without mutating anything.
|
|
Read-only. Run this before trusting health, blast-radius, or coverage answers,
|
|
and whenever something feels off (a healthy host reports `down`, a retired
|
|
service still alarms, the graph looks thin).
|
|
|
|
## 1. Run the drift report
|
|
|
|
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
|
|
ranked list of findings, each with `{category, severity, count, entities,
|
|
evidence, suggested_runbook}`, plus a `summary` with totals by category.
|
|
|
|
The DB-side categories:
|
|
|
|
- **orphan_checks** — check entities with truncated/random slugs left by the
|
|
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
|
|
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
|
|
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
|
|
(the scheduler already skips these, but the rows should be retired).
|
|
- **down_checks** — enabled probes reporting `down`. Remediation:
|
|
`service-health-check` (then check whether the failure is real or a
|
|
probe-config/routing problem — see step 3).
|
|
- **unknown_checks** — probes that ran but reported `unknown` (usually a
|
|
misconfigured or not-yet-deployed probe script).
|
|
- **unmonitored** — active entities whose type declares monitoring but have no
|
|
enabled `check_def`.
|
|
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
|
|
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
|
|
|
|
## 2. Triage
|
|
|
|
`severity: critical` (down_checks) first. For each finding, read `evidence` and
|
|
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
|
|
before acting — the report is a pointer, not a verdict.
|
|
|
|
## 3. Common probe-failure causes
|
|
|
|
A `down_checks` finding that is NOT a real outage is usually one of:
|
|
|
|
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
|
|
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
|
|
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
|
|
exec` automatically when both are present.
|
|
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
|
|
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
|
|
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
|
|
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
|
|
|
|
## 4. What this audit does NOT cover (follow-ups)
|
|
|
|
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
|
|
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
|
|
proxmox host) against the DB graph and returns:
|
|
|
|
- **missing entities** — a guest running in Proxmox with no DB entity.
|
|
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
|
|
|
|
Still manual until that machinery lands:
|
|
|
|
- **Misplaced parent** — compare each guest's actual Proxmox host against its
|
|
`hosts` edge (migrations leave these stale).
|
|
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
|
|
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
|
|
the cert-expiry checks.
|
|
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
|
|
runtime-created entities not in version control.
|
|
|
|
## 5. Acting on findings
|
|
|
|
This skill is read-only — make no changes here. Route each confirmed finding to
|
|
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
|
|
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
|
|
to confirm the finding cleared.
|
|
|
|
Docs-update checklist: none — the audit reads state; it changes nothing. If a
|
|
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
|
|
in that remediation session.
|