Problem: entity/relationship/graph/blast-radius reads were embedded
as inline SQL in the httpapi handlers, duplicating the recursive type
tree CTE, the blast_radius function call, and the topology-picking
query across the REST and MCP surfaces with no port abstraction.
Change:
- ports.ReadModels interface: ListEntities, GetEntity, GetEntityBySlug,
GetEntityRelations, GetBlastRadius, GetGraph (with health),
ListEntityTypes. Returns EntityWithHealth (domain.Entity + health
from entity_status join) and domain.Relationship — no gen types
in the port.
- adapters/postgres/readmodels.go: EntityReader implements ReadModels
with the existing SQL verbatim (recursive type filter, blast_radius,
most-connected-first topology, graph edge listing).
- httpapi/entities.go: ListEntities, GetEntity, GetEntityRelations,
GetBlastRadius, GetGraph rewired to ReadModels. SQL moved to the
adapter; handlers map domain/ports types to gen wire shapes.
entityWithHealthToGen, sqlcEntityToGen helpers added.
- Old sqlcEntityToGen (sqlcgen.Entity → gen.Entity) preserved for
client_lifecycle.go; mutation handlers use domainToGen.
Verification: go build/vet, full test suite (19 pkgs), DB integration
(postgres + mcp — both green). httpapi DB tests have the pre-existing
set of failures (TestAPIEndToEnd entity_types=60/501, TestPhase3*)
verified at ec11956.
Problem: entity mutations (create/update/state) existed as three drifted
copies — HTTP CreateEntity/PatchEntity, MCP create_entity/
update_entity_attributes/set_entity_state — each with its own inline
SQL, its own validation subset (MCP validated lifecycle states, HTTP
did not; HTTP patched attributes without regenerating derived checks,
MCP did; MCP wrote no audit trail), the exact drift ADR 0016's first
vertical slice exists to collapse.
Change:
- internal/core/ports: DerivedCheck, Idempotency (adapter-owned request
hash + cached-body renderer so the replay record commits in the
create's transaction), IdempotentResponse + GetIdempotent read,
AuditEntry gains Method/Path/CorrelationID, Event gains
CorrelationID; EntityUpdateInput carries ExpectedVersion +
RederiveChecks (derivation for updates runs repo-side: the graph
host fallback reads relationships through the open tx).
- internal/adapters/postgres/repositories.go: EntityRepo (Create/
Update/SetState/reads/idempotency) preserving the load-bearing
check-then-act invariants in-tx: version WHERE-clause, declared
transitions + preconditions (ValidateTransition), duplicate-slug
mapping, audit/event/writeCheck all inside one BEGIN…COMMIT.
OntologyRepo: TTL-cached OntologyStore.
- internal/core/app/entities.go: EntityService — ontology validation
(type exists, concrete, state declared — the stricter MCP rule now
governs both surfaces), default-state resolution, id generation,
derivation for creates, audit/event construction, idempotency
pass-through.
- httpapi CreateEntity/PatchEntity rewired to the service; PATCH now
regenerates derived checks (the A2 parity gap). MCP create/update/
set-state tools call the same service — and now write audit + event
rows like the HTTP surface always did.
- Integration-test seed paths fixed for the adapters/postgres package
depth (../../../seeds).
Pre-existing failures documented: TestAPIEndToEnd (entity_types 60 vs
59; 501-endpoint now 200), TestClientLifecycleEndToEnd, TestPhase3*
rows — verified failing identically at ec11956 (scratch approval-
notifier commit test drift), unrelated to this change. All mutation
integration tests (create/patch/idempotency/audit/regeneration) pass.
Verification: make test-db (postgres + mcp green, httpapi green except
the pre-existing set), full non-DB suite (19 pkgs), golangci on new
packages — 0 issues.
Problem: check derivation logic lived in internal/checkdefaults with
the pure decision logic (buildKind, address/user/port resolution)
interleaved with tx I/O (entity_status insert, graph host fallback,
check upserts) — and internal/db importing it was the plan's called-out
inverted dependency.
Change:
- internal/core/app/checkdefaults.go: Derive(tree, target, lookup) —
the full derivation (monitoring overrides, host fallback via an
injected HostLookup thunk, per-kind builders) with zero I/O imports.
Types renamed for the app surface: CheckTarget, CheckDef,
DeriveResult, Skip; LogDeriveResult.
- internal/adapters/postgres/checks.go absorbs the I/O half:
EnsureChecks (entity_status row + upsert loop), writeCheck, and
hostViaGraph. The db→checkdefaults edge is gone — adapters→core is
the ADR 0016 direction (the Phase 7 SeedService note anticipated
this; the inversion is fixed a phase early).
- seed.go pending-checks loop uses app.CheckTarget + EnsureChecks;
mcp formatting/tests follow the renamed types; both test files moved
to internal/core/app.
- Deliberate behavior note: a hostViaGraph read failure inside the
thunk now logs a warning and degrades to 'skipped: no address'
instead of aborting the whole entity-create tx — a monitoring
derivation gap is visible (warn log + coverage sweep) and self-heals
on the next mutation; failing the create over a graph-read blip was
disproportionate.
Verification: go build/vet, full test suite green (app tests exercise
every buildKind branch at their new home).
Problem: the hexagon's Phase 2 (plans/2026-08-15-hexagonal-architecture.md)
must give the use-cases-to-be their contract surface: driven-port
interfaces, test fakes, the secrets interface moved into core, and the
postgres package inside the adapters tree — before the first vertical
slice (Phase 3) can wire a composition root.
Change:
- internal/core/ports: full driven-port catalog per plan §3.3 —
repositories as transaction-scoped aggregates whose inputs carry
derived checks, audit, and events (§3.6), plus CommandExecutor,
TargetResolver, Checker, Secrets, EventPublisher, Provisioner.
Port-local payload types (Event, AuditEntry, CheckDef, KnowledgeEntry,
ExecResult) keep signatures off infrastructure; TypeTree aliases
internal/ontology (pure over domain) until checkdefaults is absorbed.
ReadModels intentionally not declared yet — it materializes with the
Phase 3 slice and grows as report handlers rewire.
- secrets.Backend is now an alias of ports.Secrets; implementations
(Infisical, SOPS, Manager) unchanged. mcp's local secretBackend
subset is deleted; tool constructors take ports.Secrets.
- internal/db → internal/adapters/postgres (mechanical import rewrite;
package identifier stays db until the Phase 3 repository split).
sqlc.yaml, Makefile, golangci exclusions, and docs follow the move;
make generate-check verified.
- internal/adapters/ssh: Executor implements ports.CommandExecutor over
the actuator dial pool + RunStreaming (10-min default timeout carried
over from the httpapi path).
- internal/adapters/remote: Resolver implements ports.TargetResolver
delegating to internal/remote (still pool-based; drops onto
ports.EntityRepository when repositories land in Phase 3 — documented
transitional import).
- internal/core/ports/portstest: importable fakes — in-memory
EntityRepo (with check-then-act SetState, side-effect recording),
RecordingExecutor, FakeChecker, SpyPublisher; port-satisfaction
guards; tests.
Risk: ports are declared ahead of implementations — signatures firm up
per phase as slices land (documented in the package doc); the
remote→postgres transitional import is explicit and dissolves in
Phase 3.
Verification: go vet, make test (race, 19 packages), generate-check,
golangci on core+adapters — 0 issues; full-repo baseline down
365→344.
Problem: the hexagonal refactor churns the backend tree for nine more
phases; the UI delivery stack (web/ SPA, cmd/desktop Wails wrapper,
compose/web image) must move to its own repo first so doc/layout
rewrites land once on a backend-only tree.
Change:
- New repo git.hubris.network/dtoro/oikos-web (v0.33.0): web/, desktop/
(updateURL repointed to oikos-web releases), compose/, own CI (web +
desktop jobs), own deploy script (CI-green gate, TOCTOU guard,
version-tagged images, prune-to-3), own webhook receiver on :9798 +
launchd unit, own compose project publishing the same 8091:80.
- Cutover executed on mac-mini in order: oikos stack's web service
stopped+removed, oikos-web project brought up on 8091; outer Caddy
untouched (targets the published port) — serving + Authentik flow +
/wails 404 quirk verified post-cutover.
- Stripped from oikos: web/, cmd/desktop/, compose/web/, desktop CI
workflow, ci.yml web job, Makefile ui/desktop/desktop-package/install
targets, the compose web service, oikos-web from deploy.sh's fallback
prune list; wails + go-keyring dropped from go.mod, vendor synced.
- README / CONTRIBUTING / AGENTS.md / .agents dev+operations docs now
point at the new repo; mbse + mascot design docs carry a path note.
Risk: production SPA serving depends on the new pipeline now; rollback
is versioned-image re-up of the old web service from a pre-split
checkout (port 8091). Desktop builds installed before the split still
check dtoro/oikos releases — one manual reinstall, noted in the
oikos-web release notes.
Verification: go vet, make test (race), make generate-check, golangci
(no new findings; baseline down 400→365); post-cutover curls —
localhost:8091 200, /wails/runtime.js 404, outer Caddy 302 Authentik.
Problem: the hexagonal-architecture plan (plans/2026-08-15-hexagonal-
architecture.md) needs its foundation — an accepted ADR, the target
directory tree, and machine-checked dependency rules — before any
service extraction starts. Also folds the four outstanding review
findings (F3.1/F5/F6/F7) into the plan: ObservationService owns the
bounded probe-concurrency contract (scheduler.go:133), Phase 9 gates
ExecutionService+PolicyService ≥ 90% with a gating-matrix test,
per-phase abort criteria, and the §3.2 internal/config note.
Change:
- docs/adr/0016-hexagonal-ports-adapters.md records context, decision,
and consequences of the ports & adapters migration.
- internal/domain → internal/core/domain (mechanical import rewrite,
20 files), new internal/core/{ports,app}, internal/adapters trees
with package docs.
- .golangci.yml: depguard rules for §3.1 (core purity, no agent-client
tech in core, nomos isolation — the nomos rules self-activate when
internal/nomos exists in Phase 8). Config migrated to golangci-lint
v2 format so it loads at all (the v1 config errored under v2, masked
by CI's advisory continue-on-error). Verified depguard fires on a
planted openai-go import in internal/core/app.
- CONTRIBUTING.md layout section now shows the core/adapters tree.
Risk: import path churn is mechanical and tests pass unchanged; the
lint config migration surfaces the pre-existing 400-issue baseline
(advisory in CI, unchanged policy) — new/moved packages lint clean.
Verification: go vet ./..., make test (race, core/domain at 100%
coverage), make generate-check, golangci-lint on internal/core/... and
internal/adapters/... — 0 issues; depguard violation probe confirmed.
Removes the entire Matrix-based notifier (internal/notifier/) that polled
for pending approvals, sent Matrix alerts, and checked for reaction-based
approve/deny. Approval decisions now work on any chat platform (Hermes
desktop, Telegram, Discord, WhatsApp, CLI) via two new MCP tools:
- list_approvals — query pending/recent approvals by status or entity
- decide_approval — approve/deny via same API endpoint as UI + nomos
Config fields removed: MatrixHomeserver, MatrixUserID, MatrixToken,
MatrixRoomID, ApprovalHMACSecret. Docker notifier: service removed.
Approval HMAC token generation removed (unused by code).
The existing chat-assent path in nomos (cmd/nomos/assent.go) and the
control-room Approve button keep working unchanged — both call the
shared POST /api/v1/approvals/{id}/decision endpoint.
New tool batches session discoveries into the knowledge graph:
- Creates a session-audit knowledge entry with summary
- Links it to all touched entities via 'documents' relationships
- Creates individual discovery knowledge entries
- Stamps each entity with last_agent_session attribute
- Updates AGENTS.md with tool listing
P0: Transport escalation in classifyAndGate (server.go:745) now exempts
log-inspection commands (tail/head/cat/journalctl on *.log or /logs/)
from the /opt//etc//var/lib/ gating on LXC targets. Fixes the
'tail -3 /opt/seanime/data/logs/seanime.log queued for approval' bug.
P1: queryEntity returns actionable error when slug_or_id param is empty
('slug_or_id is required') instead of silent 'entity not found: '.
P2: Added slugArg() helper (server.go) so get_entity, get_relations,
get_blast_radius, and explain accept 'slug' as an alias for their
declared param name. Solves the discoverability inconsistency where
every tool used a different param name for the same concept.
P3: Two new MCP tools:
- restart_service(target, service) — systemctl restart wrapper,
correctly classified config_mutation (requires approval)
- push_file(target, source_path, dest_path, backup=true) — pct push
from Proxmox host into LXC, with optional backup. Classified
config_mutation. LXC-only for now.
P4: Updated homelab-lxc-ops skill with MCP tools preference table.
Plus: Wails desktop build now uses build-tag approach for frontend embed
(assets_embed.go + assets_stub.go), so go build ./... works on
clean checkout without the frontend built first.
Version: 0.31.0 → 0.32.0
Adds pvecm_quorum_check.sh probe script and wires it into the
checkdefaults system as a new 'quorum' monitoring kind on proxmox-host
entities. Runs every 60s via ssh-script, surfaces unhealthy signal when
cluster loses quorum.
Closes the monitoring blind spot that let the 2026-08-12 3.5h corosync
flapping outage go undetected (ping passed, cluster was non-quorate).
Changes:
- seeds/ontology.yaml: proxmox-host declares monitoring: [quorum]
- internal/checkdefaults/defaults.go: KindQuorum builder
- internal/checkdefaults/build_test.go: 2 new test cases
- checks/pvecm_quorum_check.sh: new probe (deployed to hubris + strong)
- VERSION: 0.30.2 -> 0.31.0
Registers a handler that serves the embedded OpenAPI 3.0 spec
(compiled into the binary via oapi-codegen) at a browseable
endpoint. Uses gen.GetSwagger() to deserialize the embedded
base64+gzip spec and returns it as JSON.
46 paths, 42 schemas — agents and humans can now introspect the
full API surface without reading Go source.
Read-only diagnostic commands ethtool, lsmod, lspci, modinfo, and dkms
were missing from the readOnlyLeadPattern in the command classifier,
causing compound diagnostic commands (e.g. 'uname -r && ethtool -i eno1
&& lsmod | grep r8169') to be misclassified as config_mutation instead
of read_only. This forced operator approval for simple hardware/driver
inspection during the 2026-08-12 hubris NIC cutover session.
Added regression test with the exact compound command from that session.
Every run call creates a classification entity, but it was never
connected to the execution via a graph edge — only via a DB column
(executions.classification_id). The ontology requires:
classification —precedes→ execution
Without this edge, all 53 classification entities had zero
relationships, making them invisible to get_relations and
blast-radius analysis.
Adds an idempotent INSERT into relationships after the existing
classification_id update, matching the same pattern used for
targets edges.
- Wire secretsManager in NewHandler() — instantiate InfisicalBackend
when OIKOS_INFISICAL_SITE_URL is set (previously always nil)
- Add get_secret, list_secrets, set_secret MCP tools with nil-backend
graceful degradation
- Add oikos secret get|set|list CLI subcommands for Infisical
- Fix Set() bug: create-before-update so new keys are created;
add Type: "shared" to Update so it finds the right secret;
disable SDK cache so Get returns fresh data after Set
- Clean enrollment response: remove fake infisical_client_id/
infisical_client_secret stubs, store age key in Infisical for real
- classifyAndGate: escalate read-only commands on lxc: targets that
touch /opt/, /etc/, /var/lib/ to config_mutation. The classifier
scores command text only, not the SSH transport layer — SSH-ing into
a container to read config is riskier than pct exec from the host.
- ontology: standalone-server monitoring override from inherited
[ping, resource, updates] to [http]. VPS-like machines may not be
SSH/ICMP-reachable from the scheduler; HTTP is the LCD liveness
signal. Entities with full SSH can override per-entity.
- get_health_summary: filter out state=destroyed entities (was noise
from 20+ destroyed test LXCs, deprecated services, etc.)
- create_entity: document the monitoring footgun in the tool description
(creating a type=check entity does NOT wire a check_def; the correct
path is update_entity_attributes with monitoring + url attributes)
Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.
Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck
The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
I — run pre-flights QEMU guest agent before queueing VM execution
classifyAndGate now checks vm: targets for qemu_guest_agent attribute.
If not_running/missing, returns immediate error instead of queuing forever.
II — policy.yaml: documented host-mutation classifier rule
Added comment clarifying that host-level package/kernel mutations
(apt-get install, dpkg, systemctl enable) always classify as
config_mutation and thus need operator approval.
III — health attribute read-only in update_entity_attributes
Strips scheduler-owned keys (health, last_check_at, last_check) from
attribute updates with a clear message directing agents to
get_health_summary / list_checks instead.
IV — Recorded discovered dependency edges
vm:zimaos → depends-on → lxc:nfs-export (NFS /media/library mount)
vm:zimaos → depends-on → host:strong (NFS /media/ludo-library mount)
Also updated the run tool description to mention both guardrails.
get_relations now accepts an optional 'types' (comma-separated) parameter
to filter relationship types — filters out the noisy exec/targets edges
that previously drowned useful host/provides edges.
get_health_summary now accepts an optional 'health' (comma-separated)
parameter to return only entities in specific health states (e.g.
'health=down,stale') instead of the full 100+ entity list.
ping_service now:
- Falls back to e.attributes->>'public_host' when 'url' is not set
(covers LXCs that only have public_host in the graph)
- Performs a live HTTP HEAD probe against the resolved URL, returning
the actual status code instead of just the scheduler's stale health
state
Also: fixed matrix.hubris.network DNS record (was pointing to dead VPS),
pruned 6 dead graph edges, wired url attributes on 7 LXCs, added VPS
HTTP monitoring check, and resolved the 18k-occurrence unmonitored signal.
This session's audit is documented as
document:nomos/2026-08-05-dns-monitoring-improvements-for-strong-hosted-services.
- Add MCP tool — lightweight connectivity check returning server
identity, no DB hit (resolves agent connection-test friction)
- Tighten 6 tool descriptions (get_relations, get_health_summary,
query_metrics, get_trend, get_event_timeline, ping) to be searchable
in the first 8-12 words
- Document Hermes MCP client setup in ADR-0012 with token security caveat
- Move completed plan to plans/done/
A `monitoring` attribute on an entity now overrides its type's declaration:
"none" opts out, a list overrides the kinds. service:haos uses it to opt out —
haos blocks SSH (no process probe can reach it) and the VM is already covered
by vm:haos's vm-status check, so the redundant process check only ever reported
false-down. vm:haos -> service:haos via provides confirms the coverage.
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].
matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
The ontology's stated intent was "http when it has a url, else a process
check", but the implementation emitted BOTH for every url-service — so ~17
fronted services carried a redundant process check that, under worst-of
aggregation, let a fragile supplementary probe (wrong unit name, unreachable
host, no guest agent) veto two healthy http checks and report the service
"down" while it was up (authentik, zimaos, house, matrix, ...).
buildKind now emits a process check only for services WITHOUT a url, or when
an explicit probe_unit opts into binary-level depth. http is the canonical
service-liveness probe (tests the real endpoint through the TLS terminator);
the redundant process checks were removed.
process_check.sh ran `systemctl is-active <entity-name>`, but a service's name
is a logical label, not its unit/container name — matrix is matrix-synapse.service
+ element-web/mautrix-* containers, authentik is authentik-server/-worker
containers. So every multi-component or docker service reported "inactive"
while up (authentik, matrix, photos, house, arr-stack, …).
Resolve in order: exact systemd unit, a unit with the name as prefix
(matrix -> matrix-synapse.service), or a running docker container whose name
contains it. checkdefaults passes a declared probe_unit/systemd_unit/container
attribute when set, for precision.
The DB-only audit_knowledge_graph can't see guests running in Proxmox that
have no entity, or entities whose pve_id is no longer live — the drift that
the stray test LXCs were a symptom of. discover_infra_drift enumerates running
guests via pct/qm list on every proxmox host (over the same SSH/pct path the
checks use) and diffs against the DB: returns missing (live, no entity) and
ghost (DB, not live). Read-only.
Companion to audit_knowledge_graph; the skill now runs both and treats the
remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
resolveProxmoxHostSlug trusted attributes.host verbatim, so a value polluted
with prose — lxc:teddycloud carried host="hubris (confirmed via pct config…)" —
became a slug that never resolved, leaving its checks 'down' despite a correct
`hosts` edge. Treat an attribute containing whitespace/parens as invalid and
fall back to the canonical hosts edge.
The audit now reports `polluted_attrs` — entities whose routing-critical
attributes carry prose — so this class is visible instead of a silent
resolution failure.
The B1 target-state filter `tgt.state NOT IN ('deprecated','destroyed')`
evaluates to NULL (unknown) when a target's state is NULL, which the WHERE
clause treats as false — so freshly-seeded entities without an explicit state
(the 20 TLS certificates) were silently dropped from ListEnabledCheckDefs and
never monitored. Treat NULL state as active (only explicit deprecated/
destroyed is excluded): `tgt.state IS NULL OR tgt.state NOT IN (...)`.
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.
Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
The ontology declared monitoring [cert-expiry] on the certificate type and a
working checkCertExpiry probe existed, but checkdefaults had no cert-expiry
builder and no certificate entities were seeded — so certificate expiry, a
real failure mode, was invisible.
Add a KindCertExpiry builder (dials the cert's hostname on :443 hourly, warns
at 30d / crit at 7d) and seed certificate entities for the 20 public
*.hubris.network routes plus uses-certificate edges from each ingress route.
A service check used to bake its hosting LXC's lan_ip and SSH it directly as
root, which failed because the scheduler key is authorized on the Proxmox hosts
but not inside every guest — leaving all 8 service process checks 'down' even
after the guest routing and scripts were fixed.
ResolveExecTargetForCheck now, for a non-guest target, walks the
provides/runs-on/hosts edges to the compute entity that runs it and routes
through that: pct/qm exec if the host is a guest, direct SSH with the host's
correct user (workstation `user` attr) if it's a machine. The guest-resolution
path is shared via resolveGuest, and the scheduler no longer needs an
isMachine special case — one resolver handles guest, machine, and service.
Older writeCheck inserts omitted target_type, so every seed-created check_def
had a NULL/empty target_type. checkSSHScript's IsGuest check then never matched,
and guest checks silently fell back to their baked (often mesh-only) address —
keeping them 'down' even after the pct-exec routing and deployed scripts were
in place. rclone stayed down for exactly this reason after the host-hop fix.
writeCheck now writes target_type, and checkSSHScript resolves the type from
the target_id when the column is blank (a runtime safety net for existing rows;
the seed rows were also backfilled in the live DB).
Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.
dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.
Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.
Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.
Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.
coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
The scheduler SSHed each guest directly and assumed a deployed probe script
plus working root SSH at the guest's address — false for headless (nfs-export),
keyless (teddycloud), mesh-only (rclone), and macOS (mac-mini) targets, which
left 49 enabled checks stuck "down" on a healthy fleet.
Extract the MCP run tool's resolveExecTarget into a shared internal/remote
package and make it the single execution path for both the scheduler and MCP.
LXC/VM checks now host-hop via pct exec / qm guest exec through the owning
Proxmox host (no per-guest lan_ip, sshd, or authorized key needed); hosts and
workstations resolve their address and user live, so mac-mini's `user: dtoro`
is honored without a re-seed. Address preference now prefers public_ipv4 over
mesh, so netbird-vps is probeable from the scheduler container.
cpu_check.sh gains a real Darwin branch (it reported cpu_pct 0 before).
checkdefaults.resolveSSHUser reads the top-level `user` attribute too.
A machine-target resolution failure is now logged before falling back to baked
config, so a broken probe-config is distinguishable from a real outage.
Two things the entity window redesign surfaced but deliberately left alone.
**blast_radius answered the wrong question.** It walked source→target for every
relationship type, but which end of an edge is the dependent differs per type:
"machine hosts container" means the target breaks, while "service depends-on
service" and "ingress routes-to service" mean the SOURCE breaks. Walking
everything forwards was right for hosts/provides and backwards for everything
else — and swept in 2,800+ documents/involves/targets edges of pure bookkeeping,
so the result contained tasks and executions that cannot break.
Direction is now declared per relationship type in seeds/ontology.yaml
(blast_direction: forward | backward | none), the same shape as the entity
types' monitoring: declaration, and defaults to none so an undeclared edge
contributes nothing rather than a confidently wrong answer.
It also needed a modelling fix: `routes-to` names an ingress's BACKEND, so
nothing recorded that all 21 public hostnames are terminated by caddy. A
`served-by` edge type now says so.
pool:ludo-lvm 2 -> 23 (every container storing on it, then their services)
lxc:caddy 4 -> 22 (service:caddy, then all 21 ingress routes)
service:authentik 7 (what authenticates via it)
**Every ping check was reporting down.** Not a host:strong false positive: all
seven, including ws:mac-mini — the Docker host itself. The scheduler runs in
Docker on macOS, whose VM does not route ICMP to the LAN; loopback pings succeed
and every LAN ping fails. Under health aggregation each broken probe dragged its
entity to down.
The question the check exists to answer is "is it reachable", and ICMP is only
one way to ask it. checkPing now falls back to a TCP connect before concluding
anything, which restores an honest verdict for the four hosts that are genuinely
up while leaving the genuinely unreachable ones down.
TestBlastRadiusTerminatesOnCycles asserted the old direction (caddy=1,
authentik=2 — the cycle walked the wrong way); it now asserts the corrected
depths, and its exact-node-count check is relaxed because walking the right way
also surfaces the seed's own real dependents, which are correct answers.
Co-Authored-By: Claude <noreply@anthropic.com>
The window rendered the same 13 collapsible sections for every entity, sorted
only by "does it have content". Audit trail carried the same visual weight as
Health, and the window answered "what data do we hold about X?" rather than
"what do I need to know, and what should I do?".
Measured against prod: host:hubris has 223 relations, 1,601 events, 2.7M metric
samples and 148 executions; an ingress route has three facts. Both got 13
identical headers. Expanding a host put ~540 interactive elements on screen.
- **A verdict header that never collapses.** Not just "down" but *why*:
"ping failing · 5 of 6 checks passing". That line did not previously exist
and could not have — checks rendered as configuration, never as results.
- **Sections composed per type.** A document has no checks, metrics or blast
radius; a signal or execution is a record, not a thing. Infrastructure gets
Status/Impact/Activity/Metrics/Reference, knowledge types lead with Content,
records get a minimal view. Unknown types fall back to infrastructure so a
new entity type is never a blank window.
- **Status replaces Monitoring**, showing each check's own verdict and when it
last ran — the section that answers the header's "why".
- **Impact** finally calls /entities/{id}/blast-radius. The endpoint has existed
since the first API and had no frontend caller anywhere, despite
.agents/OIKOS.md naming blast radius as the reason the ontology exists. Its
outgoing-edges-only limitation is stated in the UI rather than hidden.
- **Activity merges four lists** (executions, signals, events, agent activity)
that were telling one story in four places.
- **Relations cap at 8 with a drill-in** — 540 interactive elements down to 126.
- **Ask Nomos** opens a task pre-scoped to what you are looking at, seeded with
the verdict just computed, via an optional draft threaded through
openNewTaskWindow -> NewTaskChat -> ChatThread.
Requires exposing check_defs.last_health/last_run_at through the API (the
columns landed with the health-aggregation work but were never surfaced).
Adding a fourth enum containing "unknown" made oapi-codegen disambiguate all
enum constants by type prefix, so metrics.go moves to gen.TrendDirection*.
Verdict derivation and type->section composition live in $lib/entityView.ts as
pure functions with 15 unit tests, including the host:strong case that
motivated this.
Co-Authored-By: Claude <noreply@anthropic.com>
host:strong logged 226 health.changed events in one hour, oscillating
down/healthy while the host was fine throughout. host:hubris did it 126 times.
runCheck wrote entity_status.health on every check completion, so an entity's
health was simply whichever of its checks finished most recently. A host with
six checks reported whichever facet happened to be sampled last, and one
failing probe alternating with five passing ones flapped forever. resolveSignal
forced "healthy" too, a second path by which one passing probe erased another
probe's genuine failure.
On this fleet the trigger is a known false positive: the scheduler's network
vantage point cannot ICMP host:strong, so its ping check fails while every
ssh-script check succeeds. Under last-writer-wins that single probe declared
the whole host down, twice a minute.
Each check now records its own verdict (check_defs.last_health, migration 027)
and the entity's health is the worst across its enabled checks. A failing probe
now degrades the entity honestly and *stably*, without erasing what the other
five report, and health.changed fires only when that aggregate actually moves.
Checks that have never run are ignored rather than counted as unknown, so
adding a check cannot drag a known-good entity down before it has a verdict.
Also declares service:oikos in the seed. The previous commit re-pointed the mcp
ingress at it, but the entity only ever existed in the production database — so
a fresh seed (a new install, or a DR restore) failed on an unresolvable edge.
Caught by seeding an empty database rather than a copy of prod, which is the
only way that class of bug shows up.
Co-Authored-By: Claude <noreply@anthropic.com>
ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.
- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
A column rather than scheduler memory because this control plane restarts on
every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
created by the same seed do not stay in lockstep — otherwise ~165 probes
land in the same instant each minute instead of spread across it.
Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.
Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.
Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.
Co-Authored-By: Claude <noreply@anthropic.com>
