Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.
dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.
Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.
Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.
Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
Two things the entity window redesign surfaced but deliberately left alone.
**blast_radius answered the wrong question.** It walked source→target for every
relationship type, but which end of an edge is the dependent differs per type:
"machine hosts container" means the target breaks, while "service depends-on
service" and "ingress routes-to service" mean the SOURCE breaks. Walking
everything forwards was right for hosts/provides and backwards for everything
else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping,
so the result contained tasks and executions that cannot break.
Direction is now declared per relationship type in seeds/ontology.yaml
(blast_direction: forward | backward | none), the same shape as the entity
types' monitoring: declaration, and defaults to none so an undeclared edge
contributes nothing rather than a confidently wrong answer.
It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so
nothing recorded that all 21 public hostnames are terminated by caddy. A
`served-by` edge type now says so.
pool:ludo-lvm 2 -> 23 (every container storing on it, then their services)
lxc:caddy 4 -> 22 (service:caddy, then all 21 ingress routes)
service:authentik 7 (what authenticates via it)
**Every ping check was reporting down.** Not a host:strong false positive: all
seven, including ws:mac-mini — the Docker host itself. The scheduler runs in
Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed
and every LAN ping fails. Under health aggregation each broken probe dragged its
entity to down.
The question the check exists to answer is "is it reachable", and ICMP is only
one way to ask it. checkPing now falls back to a TCP connect before concluding
anything, which restores an honest verdict for the four hosts that are genuinely
up while leaving the genuinely unreachable ones down.
TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1,
authentik=2 — the cycle walked the wrong way); it now asserts the corrected
depths, and its exact-node-count check is relaxed because walking the right way
also surfaces the seed's own real dependents, which are correct answers.
Co-Authored-By: Claude <noreply@anthropic.com>
host:strong logged 226 health.changed events in one hour, oscillating
down/healthy while the host was fine throughout. host:hubris did it 126 times.
runCheck wrote entity_status.health on every check completion, so an entity's
health was simply whichever of its checks finished most recently. A host with
six checks reported whichever facet happened to be sampled last, and one
failing probe alternating with five passing ones flapped forever. resolveSignal
forced "healthy" too, a second path by which one passing probe erased another
probe's genuine failure.
On this fleet the trigger is a known false positive: the scheduler's network
vantage point cannot ICMP host:strong, so its ping check fails while every
ssh-script check succeeds. Under last-writer-wins that single probe declared
the whole host down, twice a minute.
Each check now records its own verdict (check_defs.last_health, migration 027)
and the entity's health is the worst across its enabled checks. A failing probe
now degrades the entity honestly and *stably*, without erasing what the other
five report, and health.changed fires only when that aggregate actually moves.
Checks that have never run are ignored rather than counted as unknown, so
adding a check cannot drag a known-good entity down before it has a verdict.
Also declares service:oikos in the seed. The previous commit re-pointed the mcp
ingress at it, but the entity only ever existed in the production database — so
a fresh seed (a new install, or a DR restore) failed on an unresolvable edge.
Caught by seeding an empty database rather than a copy of prod, which is the
only way that class of bug shows up.
Co-Authored-By: Claude <noreply@anthropic.com>
Rehearsing the deploy against a full copy of prod surfaced 40+ cardinality
violations that would have failed the seed. Since api/scheduler/notifier all
depend on `seed: service_completed_successfully`, and this change alters the
seed files (so the content hash changes and a full re-ingest runs for the
first time in months), that failure would have stopped those services from
starting at all.
None of them are new. The foreign-key bug in checkdefaults was aborting the
ingest earlier, during entity ingest, so ValidateCardinality at the end never
got the chance to run. Fixing the first failure revealed the next.
- `documents` was declared many-to-one, meaning a document may document at
most one entity. Nomos has been writing docs that cover several (a
fleet-wide apt audit documents every host it touched) for months, which is
reasonable — the ontology was the strict one. Now many-to-many.
- The mcp ingress still routed to service:homelab-mcp, which prod marks
deprecated: the Python MCP server on apps/105 was stopped at the Go cutover.
Nomos re-pointed it at service:oikos on 2026-07-12 and was right; the seed
was stale, and re-asserting the old edge alongside the new one is what made
it a violation.
Remaining after this: one genuine drift, `hosts target=lxc:caddy (2 edges)`,
which needs a prod data fix rather than a code change — see the follow-up.
Co-Authored-By: Claude <noreply@anthropic.com>
Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by
discarded errors in checkdefaults:
- writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT
(slug) DO NOTHING, then wrote a check_defs row referencing it. On any
re-seed the slug already existed, the entity insert no-oped, and the FK
violated — aborting the ingest transaction and surfacing as an unrelated
failure several entities later. Re-seeding has been broken since; prod's
coverage was frozen at its first successful seed. This is what
TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting.
- shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed
to ".network" and overwrote each other; service:jellyfin collided with
lxc:jellyfin.
- The ssh-script checker never read the `args` config checkdefaults wrote, so
process_check.sh always ran without its unit name and returned "unknown".
Coverage is now 75/89. Monitoring is declared per entity type in
seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say
it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap.
coverageSweep raises an `unmonitored` signal only where a type declares
monitoring it lacks — 8 real gaps, no false positives.
Also:
- entity_types.attribute_schema was never ingested: the seed loader read
"attribute_schema" but the YAML says "attributes", so all 60 types stored
JSON null.
- ListExecutions ignored its declared target/action/correlation_id filters and
paginated on a non-unique target slug, dropping and repeating rows.
- started_at was captured but only written at terminal state, so a running
execution reported NULL for its whole life. The three MCP auto-run copies
wrote no timing at all; they are now one autoRun helper.
- SSH output was buffered to completion and discarded entirely on timeout.
Both sshExec copies now stream through a shared execlog sink into
execution_logs, and keep partial output when a command is cancelled.
- executions.correlation_id was a random per-execution uuid that correlated
nothing; it is now the chat session id, which is what lets the chat tail
live output.
- reversible_low had no auto-run branch despite policy declaring it
unattended. Since computeCommandRisk never returns it, the class only arises
when an agent declares it over a read_only command — so gating it penalised
candor without adding safety.
- backup-target gains a backup-freshness checker (portable find -mmin, since
the first target is on macOS), resolving its host by walking backs-up-to
backwards. The pre-deploy pg_dump is now a tracked backup target.
UI: an Executions section on entity detail with live output tailing, and
streamed output under a running `run` call in the chat timeline.
Migrations 022-024. Ops.svelte and context.ts exclude execution.output from
their refetch triggers, which would otherwise fire once a second per command.
Co-Authored-By: Claude <noreply@anthropic.com>
Session-review implementation for the three sessions audited in
plans/2026-07-18-session-review-three-sessions.md. v0.7.11 → v0.7.12.
P0.1 — retry cap + investigate-before-retry (cmd/nomos/retrycap.go,
agent.go): after 3 identical failing run calls in a single turn, refuse
to dispatch the call again and return a directive to investigate *why*
(ps/strace/lsof) or surface the blocker. Per-turn scope so a fresh turn
after the operator responds can retry once more. Session 1e9c7691's 20+
identical chown retries (knfsd held a kernel lock on the exported NFS
dir) is the direct motivation.
P0.2 + P1.8 + P2.10 — SOUL.md guidance: hung command is not a failed
command (investigate before retry); ask before proposing a multi-step
migration; multi-goal sessions summarize the arc not just the last goal.
P1.3 — two new runbook entities in seeds/knowledge.yaml:
- nfs-exported-dir-mutation-hang (the knfsd fchownat lock procedure:
killall → exportfs -u → mutate → exportfs -a → verify)
- netbird-mgmt-oidc-race-after-upgrade (docker restart netbird-mgmt
after ~30s for the traefik/authentik OIDC race)
P1.4 — setGoal emits task.superseded event when prior goal is overwritten
by a different goal (store.go, TestSetGoal_SupersededEvent). Session
55927f0a had two set_goal calls with the first silently abandoned.
P1.5 — inspect_path MCP tool: runs mount/df/ls/stat for one path across
up to 8 targets in one parallel call, replacing the 15+ run-call
fact-gathering fan-out sessions 1 and 2 each spent on cross-target path
tracing (tools.go, server.go: inspectPathAcrossTargets, inspectOneTarget).
P1.6 — vm: target support in run via qm guest exec (no more SSH-hop
with nested quoting). Extracted shared resolveProxmoxHostSlug for
LXC + VM, with hosts-relationship fallback when attributes.host is
absent (server.go, tools.go). Session 55927f0a's SSH-hop workarounds
for vm:zimaos are the direct motivation.
Deferred (documented in plan): P1.7 (approval window auto-extend on
timeout) and P2.9 (long-running command PENDING detection) — both
addressed at lower cost by the retry cap. Session 3's poll-after-timeout
pattern already works; the cap protects against the failure mode.
Migration 018 adds goal/status/outcome/summary/entity_id to agent_sessions
and creates session_plan_steps + session_questions. Registers a 'task'
entity type and an 'involves' (task→entity) relationship in the ontology
so each session anchors its knowledge and involved-entity edges on the
existing relationships graph.
nomos createSession now mints a task:<session-id> entity (type task) and
links it via agent_sessions.entity_id — best-effort so chat never blocks on
it. listSessions/GET /sessions surface the new task fields.
No behaviour change yet; this is the data foundation for the task board and
live context panel. Verified end-to-end against the local stack: migration
applied, ontology ingested (60 types/47 rels), a new session mints a linked
task entity and the API returns status/goal/entity_id.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- knowledge.go: scan tags as []string from pgx (not JSON)
- seed.go: convert tags to PG array format, fix runbook applies_to_type
- convert-wiki.py: fix entity slug prefixes to match inventory.yaml
(host: not proxmox-host:, ws: not workstation:, service:homelab-mcp with hyphen)
- convert-wiki.py: read from archive/knowledge/ since wiki was archived
- lxc:rclone verified live as LXC 132 on hubris (pct list via MCP);
hosts edge added
- /mnt/library backing storage identified from hosts/hubris.md: 'library'
lvmthin pool, 3.7T, 2nd NVMe — added pool:library-hubris + contains edge
- all 8 derived services (books/seanime/roms/house/jellyseerr/qbit/sab/
teddy) confirmed responding over their ingress URLs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
