oikos/plans/done/2026-07-29-health-check-reality-and-knowledge-graph.md

# Plan: Make health reflect reality + complete the knowledge graph

Status: ready for implementation · Created 2026-07-29

## Context

`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
Investigation showed the problem is systemic, not local: **49 enabled checks report
`down`**, almost all `ssh-script`, because the resource/updates probes assume
**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target** —
both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
topology view).

The DB is the source of truth; live state was verified via the REST API
(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.

## Decisions (confirmed with operator)

1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
   `qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
   plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
   use direct SSH with the correct per-target user.
2. **Canonical host-hop access** — `pct exec`/`qm guest exec` through the proxmox host is
   the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
   guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
3. **Auto-provision monitoring for new entities** — wire script-deploy + the
   `health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
   becomes monitorable with zero manual steps (Track E).
4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
   permanent false alarms from retired things.
5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
   re-export seeds, seed skills, raise graph cap.
6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
   it against the DB graph, producing a ranked drift report; the operator acts on findings
   via existing lifecycle runbooks. No auto-fix. (Track F.)

## Findings (evidence)

### A. Health-check reality gaps (49 checks `down`)

**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
`-i /etc/oikos/ssh_key`, not a default-key test):**

The MCP `run` tool already reaches every guest correctly via
`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
(`attributes.host` → `hosts` edge → hubris default), SSH there, run
`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.

The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
entity's own resolved address (`internal/scheduler/scheduler.go:758`,
`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:

| Class | Targets (verified) | Root cause |
|---|---|---|
| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |

**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
(it's a homelab client with root key + scripts). So the host hop is the reliable path.

**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
script deployment, not topology.

Health aggregation itself is correct: `WorstHealthForTarget`
(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
ssh-script drags an otherwise-healthy entity to `down`.

### B. Dead/stale data

- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
  (e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
  Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
  routes to it and alarms permanently.

### C. Knowledge-graph gaps

- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
  checker all exist, but **0** certificate entities. Cert expiry is invisible.
- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
  checker exists → every zone is an `unmonitored` signal.
- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
  299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.

---

## Work breakdown

### Track A — Make ssh-script checks work everywhere (route through the proxmox host)

Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
hop already has working root SSH. Hosts/workstations keep direct SSH.

**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**

Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
credential per host (root key, already authorized on hubris/strong), no per-guest keys,
sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).

- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
  execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
  they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
  package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
  (`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
  other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
  the scheduler SSHes guests directly (broken), MCP host-hops (works).
- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
  proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
  base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
  keep direct SSH (they ARE the host).
- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
  approval. Verify one LXC end-to-end (rclone) before fanning out.

**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
  romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
  scripts must live in the guest.
- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
  host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
  `checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
- Backfill once now (all guests + mac-mini). See Track E for the automated version.

**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
  `attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
  for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).

**A4. macOS-aware check scripts.**
- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
  (`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
  (`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
  with `security_updates=0` or read `softwareupdate --list`).
- Each must still emit `{"health":..,"metrics":{..}}` JSON
  (`internal/scheduler/scheduler.go:767`).

**A5. Reachability for external/mesh-only hosts.**
- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
  for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
  Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
  key or proxy via hubris. Confirm before assuming direct SSH works.
- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
  keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
  Open Questions.)
- `lxc:rclone` no longer a special case — handled by A1's pct routing.

**A6. ICMP-blocked VMs.**
- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
  (`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
  Lower priority — confirm haos blocks ICMP before building.

### Track B — Lifecycle monitoring gate

**B1. Skip monitoring for deprecated/destroyed targets.**
- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
  `state` ∈ {`deprecated`,`destroyed`}.
- Implement by joining target state in `ListEnabledCheckDefs`
  (`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
  whose target is retired — **or** in a `housekeeping` sweep
  (`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
  (no write needed at runtime).
- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.

### Track C — Dead-data cleanup

**C1. Delete 24 orphan check_defs + check entities.**
- Direct SQL (have psql access): delete `check_defs` then `entities` matching
  `slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
  `enabled=false` first (already verified).
- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
  the rows first; this is `config_mutation` → operator approval.

**C2. Retire the secrets route.**
- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
  → `deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
  (or keep as archaeology). Decide with operator.

**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
- DB shows these with live `hosts` edges on strong, never cleaned up:
  `lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
  `lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
  entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
  container still exists, destroy via `pct destroy` first (destructive → approval).
- They currently generate checks and pollute the graph/health view.

### Track D — Knowledge graph

**D1. Model TLS certificates.**
- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
  cert) + `uses-certificate` edges from each `ingress-route`.
- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
  checker's discovery, or seed from Caddyfile and backfill `expires` live.
- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
  against real entities instead of nothing.

**D2. dns-zone monitoring gap.**
- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]` → `monitoring: none`
  with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
  `unmonitored` noise. Re-seed.

**D3. Re-export seeds to fix drift.**
- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
  other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.

**D4. Seed skills from disk.**
- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
  exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
  ingest. `get_skills()` then returns data.

**D5. Raise graph node cap.**
- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
  paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
  confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
  the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.

### Track E — Auto-provision monitoring when a new entity is created

Goal: the operator's request — "make sure this is handled automatically in the future
when the agent creates new entities." Today `ensureDefaultChecks`
(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
admits the gap. A new entity should become monitorable with zero manual steps.

**E1. Hook script-deploy into entity creation / provisioning.**
- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
  after writing check_defs, it also ensures the target can answer:
  - **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
    (reuse the host resolution from A1). Idempotent (skip if present + unchanged).
  - **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
    over SSH; locally on mac-mini).
- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
  sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
  automated. mac-mini still needs its `dtoro` key (A3) once.

**E2. Tie into the lifecycle `provisioning → active` gate.**
- The ontology already requires `health-check-answering` for `provisioning → active`
  (`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
- Make that gate actually run one check against the new entity and require a non-`down`
  verdict before the transition is allowed. This closes the loop: an entity isn't "active"
  (and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.

**E3. Re-run on re-seed / attribute change.**
- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
  (`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
  for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
  and re-deploy scripts to the new guest.

**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
script-pushed + check_defs + a passing `health-check-answering` gate before going active.

### Track F — Read-only knowledge-graph audit skill

Goal: the operator's request — a skill that auto-discovers live infra and validates the
knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
finding to the relevant lifecycle runbook.

**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
  `get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.

**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
  discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
  entities, evidence, suggested_runbook}`.
- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
  config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
  API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
  what's deployed at `/opt/oikos/checks/` per target.
- Report categories (the gaps this investigation found):
  1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
  2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
  3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
     rclone class — though rclone's parent is correct; this catches real migrations).
  4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
     orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
  5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
     the guest (the strong-guest/rclone class).
  6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
  7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
     via `oikos export` diff.
  8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
  9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).

**F2. Author the skill.**
- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
  `risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
  Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
  the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
  `config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
  tracks for cert/seed/graph-cap work). No mutating steps.
- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
  (bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
  empty-skills-table gap, Track D4).

**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
unless the operator wants continuous drift signals; the on-demand skill is the deliverable.

**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.

---

## Validation

After each track, verify via API (read-only, no approval):

- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
  rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
  route via `pct exec 132` on hubris.
- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
  after scripts pushed inside + routed via strong's `pct exec`.
- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
  genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
  WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
  ('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
  `health-check-answering` before reaching `active` (E1/E2).
- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
- `GET /api/v1/entities?type=skill&limit=50` → >0.
- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
  `grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
  go through the shared `pct exec`/`qm guest exec` resolver.
- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
  the 9 categories; running it against current state reproduces this plan's findings
  (orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
  is read-only — confirm it performs no DB writes (audit-log shows only reads).

Unit/integration tests to add/update:
- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
  `pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
  `public_ipv4` preferred for standalone-server (`defaults_test.go`).
- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
  (new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
- E1: new-entity creation triggers script push (mock pct/SSH in test).
- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
  category counts (ghost, missing, orphan, undeployed, drift).

## Risks

- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
  guests. This is already true (pct exec requires the host up) and is a net improvement
  (one credential vs many), but a host outage now fails all its guest checks together —
  which is the *correct* blast radius (guests are unreachable when their host is down).
- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
  one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
  `resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
  Idempotent + content-checked; never clobber a same-named operator script without diffing.
- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
  proxying via hubris; confirm before assuming direct SSH works (A5).
- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
  only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
  reachable probe, or grant an operator override.
- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
  read_only; ensure discovery commands are in the read-only allowlist (no state change).
- **Seed re-export** can surface large diffs (cognition entities) — scope export to
  topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.

## Open questions (none blocking; confirm during implementation)

- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
  `monitoring: none`? (A5)
- secrets ingress: keep as archaeology or destroy the route? (C2)
- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
  stacks only? (F1)

## Suggested order

A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
→ D1 → D4 → D3 → D5. Validate after each track.