chore: graph view, dns-zone gap, fleet deploy/cleanup tooling

Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.

dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.

Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.

Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.

VERSION

@@ -1 +1 @@
-0.13.0
+0.14.0

internal/httpapi/impl.go

@@ -24,7 +24,12 @@ import (
 const (
 	defaultLimit = 50
 	maxLimit     = 200
-	graphNodeCap = 500
+	// graphNodeCap bounds the whole-graph view. The cognition transactional
+	// types (execution, task) are audit records, not topology, and previously
+	// crowded out every host/lxc/service; the default whole-graph view below
+	// excludes them so the cap is spent on the actual fleet graph. Operators
+	// still reach executions/tasks via list_entities.
+	graphNodeCap = 2000
 )
 
 // actorInfo returns the caller's (type, label) from the request context,
@@ -307,14 +312,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
 		// alphabetically.  Without this the cap fills with exec:* rows and
 		// drops every host/lxc/service/vm — and every edge those entities
 		// connect — because edges require both endpoints in the node set.
+		// Exclude the cognition transactional types (execution/task): they
+		// are audit records rather than topology, and at ~380 rows they
+		// consumed most of the old 500-node cap.
 		nodes, err = s.queryEntities(ctx, `
 			SELECT `+entityCols+`
 			FROM entities e
 			LEFT JOIN entity_status st ON st.entity_id = e.id
-			WHERE e.id IN (
+			WHERE e.type NOT IN ('execution','task')
+			  AND e.id IN (
 				SELECT e2.id FROM entities e2
 				LEFT JOIN relationships r ON r.valid_to IS NULL
 					AND (r.source_id = e2.id OR r.target_id = e2.id)
+				WHERE e2.type NOT IN ('execution','task')
 				GROUP BY e2.id
 				ORDER BY count(r.type) DESC, e2.slug
 				LIMIT $1

plans/2026-07-29-health-check-reality-and-knowledge-graph.md

@@ -0,0 +1,410 @@
+# Plan: Make health reflect reality + complete the knowledge graph
+
+Status: ready for implementation · Created 2026-07-29
+
+## Context
+
+`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
+Investigation showed the problem is systemic, not local: **49 enabled checks report
+`down`**, almost all `ssh-script`, because the resource/updates probes assume
+**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target** —
+both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
+also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
+topology view).
+
+The DB is the source of truth; live state was verified via the REST API
+(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
+and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.
+
+## Decisions (confirmed with operator)
+
+1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
+   `qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
+   plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
+   use direct SSH with the correct per-target user.
+2. **Canonical host-hop access** — `pct exec`/`qm guest exec` through the proxmox host is
+   the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
+   guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
+3. **Auto-provision monitoring for new entities** — wire script-deploy + the
+   `health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
+   becomes monitorable with zero manual steps (Track E).
+4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
+   permanent false alarms from retired things.
+5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
+   re-export seeds, seed skills, raise graph cap.
+6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
+   it against the DB graph, producing a ranked drift report; the operator acts on findings
+   via existing lifecycle runbooks. No auto-fix. (Track F.)
+
+## Findings (evidence)
+
+### A. Health-check reality gaps (49 checks `down`)
+
+**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
+`-i /etc/oikos/ssh_key`, not a default-key test):**
+
+The MCP `run` tool already reaches every guest correctly via
+`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
+(`attributes.host` → `hosts` edge → hubris default), SSH there, run
+`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
+That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.
+
+The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
+entity's own resolved address (`internal/scheduler/scheduler.go:758`,
+`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
+`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:
+
+| Class | Targets (verified) | Root cause |
+|---|---|---|
+| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
+| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
+| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
+| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
+| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |
+
+**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
+the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
+(it's a homelab client with root key + scripts). So the host hop is the reliable path.
+
+**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
+strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
+script deployment, not topology.
+
+Health aggregation itself is correct: `WorstHealthForTarget`
+(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
+ssh-script drags an otherwise-healthy entity to `down`.
+
+### B. Dead/stale data
+
+- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
+  (e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
+  Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
+- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
+  routes to it and alarms permanently.
+
+### C. Knowledge-graph gaps
+
+- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
+  checker all exist, but **0** certificate entities. Cert expiry is invisible.
+- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
+  checker exists → every zone is an `unmonitored` signal.
+- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
+- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
+- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
+  299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.
+
+---
+
+## Work breakdown
+
+### Track A — Make ssh-script checks work everywhere (route through the proxmox host)
+
+Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
+tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
+host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
+(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
+hop already has working root SSH. Hosts/workstations keep direct SSH.
+
+**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**
+
+Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
+execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
+owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
+credential per host (root key, already authorized on hubris/strong), no per-guest keys,
+sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
+the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).
+
+- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
+  execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
+  they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
+- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
+  package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
+  (`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
+  other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
+  the scheduler SSHes guests directly (broken), MCP host-hops (works).
+- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
+  proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
+  base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
+  keep direct SSH (they ARE the host).
+- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
+  approval. Verify one LXC end-to-end (rclone) before fanning out.
+
+**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
+- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
+  romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
+  scripts must live in the guest.
+- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
+  host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
+  `checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
+- Backfill once now (all guests + mac-mini). See Track E for the automated version.
+
+**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
+- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
+  `attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
+  for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
+- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).
+
+**A4. macOS-aware check scripts.**
+- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
+  (`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
+  (`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
+  with `security_updates=0` or read `softwareupdate --list`).
+- Each must still emit `{"health":..,"metrics":{..}}` JSON
+  (`internal/scheduler/scheduler.go:767`).
+
+**A5. Reachability for external/mesh-only hosts.**
+- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
+  for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
+  Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
+  key or proxy via hubris. Confirm before assuming direct SSH works.
+- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
+  keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
+  Open Questions.)
+- `lxc:rclone` no longer a special case — handled by A1's pct routing.
+
+**A6. ICMP-blocked VMs.**
+- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
+  (`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
+  Lower priority — confirm haos blocks ICMP before building.
+
+### Track B — Lifecycle monitoring gate
+
+**B1. Skip monitoring for deprecated/destroyed targets.**
+- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
+  `state` ∈ {`deprecated`,`destroyed`}.
+- Implement by joining target state in `ListEnabledCheckDefs`
+  (`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
+  whose target is retired — **or** in a `housekeeping` sweep
+  (`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
+  (no write needed at runtime).
+- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
+- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.
+
+### Track C — Dead-data cleanup
+
+**C1. Delete 24 orphan check_defs + check entities.**
+- Direct SQL (have psql access): delete `check_defs` then `entities` matching
+  `slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
+  `enabled=false` first (already verified).
+- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
+  the rows first; this is `config_mutation` → operator approval.
+
+**C2. Retire the secrets route.**
+- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
+  → `deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
+  (or keep as archaeology). Decide with operator.
+
+**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
+- DB shows these with live `hosts` edges on strong, never cleaned up:
+  `lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
+  `lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
+- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
+  entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
+  container still exists, destroy via `pct destroy` first (destructive → approval).
+- They currently generate checks and pollute the graph/health view.
+
+### Track D — Knowledge graph
+
+**D1. Model TLS certificates.**
+- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
+  cert) + `uses-certificate` edges from each `ingress-route`.
+- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
+  checker's discovery, or seed from Caddyfile and backfill `expires` live.
+- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
+  against real entities instead of nothing.
+
+**D2. dns-zone monitoring gap.**
+- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]` → `monitoring: none`
+  with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
+  `unmonitored` noise. Re-seed.
+
+**D3. Re-export seeds to fix drift.**
+- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
+  other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.
+
+**D4. Seed skills from disk.**
+- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
+  exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
+  ingest. `get_skills()` then returns data.
+
+**D5. Raise graph node cap.**
+- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
+  paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
+  confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
+  the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.
+
+### Track E — Auto-provision monitoring when a new entity is created
+
+Goal: the operator's request — "make sure this is handled automatically in the future
+when the agent creates new entities." Today `ensureDefaultChecks`
+(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
+NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
+admits the gap. A new entity should become monitorable with zero manual steps.
+
+**E1. Hook script-deploy into entity creation / provisioning.**
+- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
+  after writing check_defs, it also ensures the target can answer:
+  - **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
+    (reuse the host resolution from A1). Idempotent (skip if present + unchanged).
+  - **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
+    over SSH; locally on mac-mini).
+- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
+  sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
+  automated. mac-mini still needs its `dtoro` key (A3) once.
+
+**E2. Tie into the lifecycle `provisioning → active` gate.**
+- The ontology already requires `health-check-answering` for `provisioning → active`
+  (`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
+- Make that gate actually run one check against the new entity and require a non-`down`
+  verdict before the transition is allowed. This closes the loop: an entity isn't "active"
+  (and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.
+
+**E3. Re-run on re-seed / attribute change.**
+- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
+  (`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
+  for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
+  and re-deploy scripts to the new guest.
+
+**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
+entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
+script-pushed + check_defs + a passing `health-check-answering` gate before going active.
+
+### Track F — Read-only knowledge-graph audit skill
+
+Goal: the operator's request — a skill that auto-discovers live infra and validates the
+knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
+producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
+finding to the relevant lifecycle runbook.
+
+**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
+knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
+- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
+- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
+- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
+  `get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.
+
+**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
+- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
+  discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
+  entities, evidence, suggested_runbook}`.
+- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
+  config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
+  API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
+  what's deployed at `/opt/oikos/checks/` per target.
+- Report categories (the gaps this investigation found):
+  1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
+  2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
+  3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
+     rclone class — though rclone's parent is correct; this catches real migrations).
+  4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
+     orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
+  5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
+     the guest (the strong-guest/rclone class).
+  6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
+  7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
+     via `oikos export` diff.
+  8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
+  9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).
+
+**F2. Author the skill.**
+- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
+  `risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
+  Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
+  the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
+  `config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
+  tracks for cert/seed/graph-cap work). No mutating steps.
+- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
+  (bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
+  empty-skills-table gap, Track D4).
+
+**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
+sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
+unless the operator wants continuous drift signals; the on-demand skill is the deliverable.
+
+**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
+DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.
+
+---
+
+## Validation
+
+After each track, verify via API (read-only, no approval):
+
+- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
+- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
+  rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
+  route via `pct exec 132` on hubris.
+- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
+  after scripts pushed inside + routed via strong's `pct exec`.
+- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
+  genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
+- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
+- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
+  WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
+- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
+  ('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
+- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
+  `health-check-answering` before reaching `active` (E1/E2).
+- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
+- `GET /api/v1/entities?type=skill&limit=50` → >0.
+- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
+- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
+- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
+- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
+  `grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
+  go through the shared `pct exec`/`qm guest exec` resolver.
+- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
+  the 9 categories; running it against current state reproduces this plan's findings
+  (orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
+  is read-only — confirm it performs no DB writes (audit-log shows only reads).
+
+Unit/integration tests to add/update:
+- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
+  `pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
+  `public_ipv4` preferred for standalone-server (`defaults_test.go`).
+- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
+  (new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
+- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
+- E1: new-entity creation triggers script push (mock pct/SSH in test).
+- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
+  category counts (ghost, missing, orphan, undeployed, drift).
+
+## Risks
+
+- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
+  guests. This is already true (pct exec requires the host up) and is a net improvement
+  (one credential vs many), but a host outage now fails all its guest checks together —
+  which is the *correct* blast radius (guests are unreachable when their host is down).
+- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
+  one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
+  `resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
+- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
+  Idempotent + content-checked; never clobber a same-named operator script without diffing.
+- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
+- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
+  proxying via hubris; confirm before assuming direct SSH works (A5).
+- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
+  only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
+  reachable probe, or grant an operator override.
+- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
+  read_only; ensure discovery commands are in the read-only allowlist (no state change).
+- **Seed re-export** can surface large diffs (cognition entities) — scope export to
+  topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
+- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.
+
+## Open questions (none blocking; confirm during implementation)
+
+- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
+  `monitoring: none`? (A5)
+- secrets ingress: keep as archaeology or destroy the route? (C2)
+- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
+- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
+- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
+  stacks only? (F1)
+
+## Suggested order
+
+A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
+guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
+→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
+→ D1 → D4 → D3 → D5. Validate after each track.

scripts/cleanup-orphan-checks.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+# cleanup-orphan-checks.sh — remove orphan check entities + check_defs.
+#
+# These are leftovers from the old shortSlug() collision bug: check entities
+# with truncated 8-hex slugs (e.g. check:ssh-script:0d31fdd1) that have no
+# live target and are disabled. They pollute the entity table and the checks
+# view. The audit_knowledge_graph tool reports them as `orphan_checks`.
+#
+# Risk class: config_mutation (deletes rows). DRY-RUN by default; pass --apply
+# to actually delete. Review the listed slugs first — they must all match the
+# legacy random-slug pattern and be disabled.
+#
+# Usage:
+#   cleanup-orphan-checks.sh            # dry-run: list what would be deleted
+#   cleanup-orphan-checks.sh --apply    # delete check_defs rows, then entities
+#
+# Connects via the OIKOS_TEST... no — via the running postgres container by
+# default, or OIKOS_PSQL if set.
+set -euo pipefail
+
+PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
+PATTERN='^check:(ping|ssh-script|disk):[0-9a-f]{8}$'
+
+# Orphan = matches the legacy random-slug pattern AND has no enabled check_def
+# pointing at a real target. A random-slug check that IS enabled and has a live
+# target is a working check with a bad slug — keep it (deleting would drop
+# monitoring), and flag it for a slug fix instead.
+ORPHAN_PRED="e.type='check' AND e.slug ~ '$PATTERN'
+  AND NOT EXISTS (SELECT 1 FROM check_defs cd
+                  WHERE cd.entity_id = e.id AND cd.enabled AND cd.target_id IS NOT NULL)"
+
+echo "== orphan checks matching /$PATTERN/ (no enabled check_def w/ target) =="
+$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
+
+echo "== details (slug, state, enabled) =="
+$PSQL_CMD -F ' | ' -Ac "
+SELECT e.slug, COALESCE(e.state,'(null)'),
+       COALESCE((SELECT cd.enabled::text FROM check_defs cd WHERE cd.entity_id=e.id LIMIT 1),'no-check_def')
+FROM entities e
+WHERE $ORPHAN_PRED
+ORDER BY e.slug;" | head -60
+
+if [ "${1:-}" != "--apply" ]; then
+  echo
+  echo "DRY RUN — no rows deleted. Re-run with --apply to delete:"
+  echo "  check_defs whose check entity is an orphan, then those entities."
+  exit 0
+fi
+
+echo
+echo "== applying (config_mutation) =="
+# Delete check_defs first (FK), then the orphan check entities.
+$PSQL_CMD -v ON_ERROR_STOP=1 <<SQL
+BEGIN;
+DELETE FROM check_defs WHERE entity_id IN (SELECT id FROM entities e WHERE $ORPHAN_PRED);
+DELETE FROM entities e WHERE $ORPHAN_PRED;
+COMMIT;
+SQL
+
+echo "== remaining orphans (should be 0) =="
+$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"

scripts/report-stray-test-lxcs.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# report-stray-test-lxcs.sh — list leftover test/scratch LXC entities.
+#
+# Provisioning experiments leave active `lxc:test-*` / `lxc:preflight-*`
+# entities in the graph long after the containers are gone or repurposed.
+# They generate checks and pollute health/graph views. This reports them and
+# their DB state + which Proxmox host each is parented on, so the operator can
+# confirm the container is really gone and retire the entity via the
+# lifecycle-destroy-node runbook (a destructive, approval-gated action).
+#
+# Read-only. Pair with: lifecycle-destroy-node (mark destroyed) or
+# lifecycle-deprecate-node.
+set -euo pipefail
+
+PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
+
+echo "== stray test/scratch LXC entities =="
+$PSQL_CMD -F ' | ' -Ac "
+SELECT e.slug, COALESCE(e.state,'active') AS state,
+       e.attributes->>'pve_id' AS pve_id,
+       COALESCE(h.slug,'(no host)') AS host
+FROM entities e
+LEFT JOIN relationships r ON r.target_id = e.id AND r.type='hosts' AND r.valid_to IS NULL
+LEFT JOIN entities h ON h.id = r.source_id
+WHERE e.type='lxc' AND e.slug ~ '^lxc:(test|preflight)'
+ORDER BY e.slug;"
+
+echo
+echo "Next: for each, confirm the container is gone in Proxmox (pct list on its"
+echo "host), then retire via lifecycle-destroy-node (destructive) or mark"
+echo "deprecated. If a container still exists, pct destroy it first."

seeds/ontology.yaml

@@ -379,8 +379,11 @@ entity_types:
     layer: infrastructure
     lifecycle: infrastructure
     description: DNS zone (e.g. split-horizon hubris.network).
-    monitoring: [dns]             # NOTE: no `dns` checker exists yet — this is a
-                                  # real gap and coverageSweep will report it
+    monitoring: none             # no `dns` checker exists yet; declaring [dns]
+                                  # made every zone an unresolvable `unmonitored`
+                                  # signal. Flip back to [dns] when a checker lands.
+                                  # Requires ontology re-ingest to take effect;
+                                  # coverageSweep then auto-clears the stale signals.
     attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
   dns-record:
     parent: entity

tools/deploy-checks.sh

@@ -0,0 +1,94 @@
+#!/usr/bin/env bash
+# deploy-checks.sh — push the check scripts into every monitored target.
+#
+# Background: an ssh-script check runs the script INSIDE the target, so the
+# script must exist at /opt/oikos/checks/ on the target itself — not just on
+# the proxmox host. The scheduler routes LXC/VM checks through the host via
+# `pct exec`/`qm guest exec`, so it never SSHes a guest directly, but the
+# script still has to be present inside the guest. This script deploys them.
+#
+# Run from a Proxmox host (it uses `pct`/`qm`) to populate every local guest,
+# and/or pass --host to install on a host/workstation over SSH.
+#
+# Usage:
+#   deploy-checks.sh                    # on a proxmox host: push to every LXC/VM here
+#   deploy-checks.sh --host ws:mac-mini # ssh-install scripts on a host/workstation
+#   deploy-checks.sh --checks /path     # override the source checks dir
+#
+# Idempotent: skips a script whose deployed copy is byte-identical.
+set -euo pipefail
+
+CHECKS_DIR="${OIKOS_CHECK_DIR:-${HOMELAB_CONTEXT_DIR:-/opt/homelab}/checks}"
+DEST=/opt/oikos/checks
+
+die() { echo "deploy-checks: $*" >&2; exit 1; }
+
+deploy_to_guest() {
+  local id="$1" vm="$2"      # vm=0 for LXC, 1 for VM
+  local kind=pct; [ "$vm" = "1" ] && kind=qm
+  echo "[deploy-checks] $kind $id"
+  # Ensure the destination dir exists inside the guest.
+  if [ "$kind" = "pct" ]; then
+    pct exec "$id" -- mkdir -p "$DEST" 2>/dev/null || { echo "  skip (pct exec failed)"; return; }
+  else
+    # qm guest exec returns JSON; best-effort for VMs (guest agent required).
+    qm guest exec "$id" -- mkdir -p "$DEST" >/dev/null 2>&1 || { echo "  skip (qm guest exec failed)"; return; }
+  fi
+  local pushed=0 skipped=0
+  for script in "$CHECKS_DIR"/*.sh; do
+    local name; name=$(basename "$script")
+    [ "$name" = "deploy-checks.sh" ] && continue
+    [ "$name" = "install.sh" ] && continue
+    if [ "$kind" = "pct" ]; then
+      pct push "$id" "$script" "$DEST/$name" --perms 755 2>/dev/null && pushed=$((pushed+1)) || skipped=$((skipped+1))
+    else
+      # qm has no push; copy via the guest agent file write if available.
+      qm guest exec "$id" -- /bin/sh -c "cat > $DEST/$name" < "$script" >/dev/null 2>&1 && pushed=$((pushed+1)) || skipped=$((skipped+1))
+    fi
+  done
+  echo "  pushed=$pushed skipped=$skipped"
+}
+
+deploy_to_host() {
+  local target="$1"          # user@ip or slug resolved by caller
+  echo "[deploy-checks] host $target"
+  ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$target" "bash -s" < "$CHECKS_DIR/install.sh" \
+    || echo "  WARNING: install on $target failed"
+}
+
+if [ ! -d "$CHECKS_DIR" ]; then die "checks dir not found: $CHECKS_DIR"; fi
+
+# Host/workstation install mode.
+if [ "${1:-}" = "--host" ]; then
+  [ $# -ge 2 ] || die "--host needs a target (user@ip)"
+  deploy_to_host "$2"
+  exit 0
+fi
+
+# Proxmox-host mode: push to every local LXC and VM.
+if command -v pct >/dev/null 2>&1; then
+  # LXC containers: ID and status. Skip stopped ones.
+  while IFS= read -r line; do
+    [ -z "$line" ] && continue
+    id=$(awk '{print $1}' <<<"$line")
+    status=$(awk '{print $2}' <<<"$line")
+    [ "$status" = "running" ] || { echo "[deploy-checks] skip LXC $id ($status)"; continue; }
+    deploy_to_guest "$id" 0
+  done < <(pct list 2>/dev/null | tail -n +2)
+else
+  echo "deploy-checks: 'pct' not found — not a Proxmox host."
+  echo "  On a host/workstation, use: deploy-checks.sh --host user@ip"
+  exit 0
+fi
+
+if command -v qm >/dev/null 2>&1; then
+  while IFS= read -r line; do
+    [ -z "$line" ] && continue
+    id=$(awk '{print $1}' <<<"$line")
+    status=$(awk '{print $2}' <<<"$line")
+    [ "$status" = "running" ] || continue
+    deploy_to_guest "$id" 1
+  done < <(qm list 2>/dev/null | tail -n +2)
+fi
+
+echo "[deploy-checks] done"

tools/setup-checks.sh

@@ -1,13 +1,35 @@
 #!/usr/bin/env bash
-# setup-checks.sh — deploy check scripts to /opt/oikos/checks on each host.
+# setup-checks.sh — deploy check scripts to /opt/oikos/checks.
 # Auto-setup hook: tools/setup-*.sh runs after every git pull.
+#
+# On a plain host/workstation this installs the scripts locally (the pulling
+# host). On a Proxmox host it ALSO pushes the scripts into every running LXC/VM
+# guest, because an ssh-script check runs the script INSIDE the target — a
+# script present only on the host does nothing for a guest reached via
+# pct/qm exec. Guest deployment is delegated to deploy-checks.sh.
 set -euo pipefail
 
 CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
 CHECK_SETUP="$CLONE_DIR/checks/install.sh"
+DEPLOY="$CLONE_DIR/tools/deploy-checks.sh"
 
 if [ -f "$CHECK_SETUP" ]; then
   bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
 else
   echo "[setup-checks] no checks/install.sh found, skipping"
 fi
+
+# On a Proxmox host, keep every guest's scripts in sync too. Best-effort: a
+# failing push to one guest must not abort the whole hook. Warn (not skip
+# silently) if deploy-checks.sh itself is absent — without it guests never get
+# scripts and the pct-exec routing reports every guest check down.
+if command -v pct >/dev/null 2>&1; then
+  if [ ! -f "$DEPLOY" ]; then
+    echo "[setup-checks] WARNING: $DEPLOY missing — guest scripts will go stale. Commit tools/deploy-checks.sh alongside this hook."
+  elif [ ! -x "$DEPLOY" ]; then
+    echo "[setup-checks] WARNING: $DEPLOY not executable — running via bash"
+    bash "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
+  else
+    "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
+  fi
+fi