Problem: knowledge and learning operations were scattered across
httpapi and mcp handlers with no shared service layer. The hexagonal
refactor needs a single use-case service for both surfaces.
Change:
- app/knowledge.go: KnowledgeService (Search, Upsert, GetContent,
SoftDelete, Restore) and LearningService (ListPatterns,
UpsertPattern, Validate, Quarantine) wrapping the port interfaces.
- adapters/postgres/knowledge.go: KnowledgeRepo implements
KnowledgeRepository — Search, GetContent, Upsert, SoftDelete,
Restore with inline SQL matching the existing handler patterns
(full-text search ILIKE, upsert on conflict, soft-delete).
Verification: go build/vet, full test suite (18 pkgs), DB integration
(postgres + mcp — green).
Problem: signal lifecycle (upsert, resolve, health aggregation) and
observe-pass orchestration (load checks, resolve targets, run probes,
aggregate health) were embedded in scheduler/scheduler.go — 1095 lines
of monolith with no port abstraction.
Change:
- app/signals.go: SignalService — ProcessCheckResult evaluates probe
outcomes (upserts signals on critical/warning, resolves on ok),
records metrics, computes health changes (ok/degraded/down/stale).
WorstHealthForTarget aggregates open signals into entity health.
- app/observation.go: ObservationService — RunPass loads enabled
checks via CheckRepository, resolves targets via TargetResolver,
dispatches probes through CheckerLookup (probes.Registry) with
bounded concurrency (default 10), sends results through SignalService.
- adapters/postgres/signals.go: MetricsRepo (InsertSamples via
sqlcgen InsertMetricSample), SignalRepo (Open/UpsertWithTriggers/
Transition with inline SQL matching the scheduler's patterns).
Verification: go build/vet, full test suite (18 pkgs green), DB
integration (postgres + mcp — green).
Problem: probe logic (checkHTTP, checkTCP, checkPing, checkDNS,
checkSSHScript, etc.) was embedded inside scheduler/scheduler.go as
unexported functions coupled to sqlcgen types — unreachable from the
core ObservationService the hexagonal refactor needs.
Change:
- adapters/probes/network.go: HTTP, TCP, ping, and DNS probe adapters
implementing ports.Checker. Each parses CheckDef.Config (json map),
runs the probe against the Target, and returns a ports.CheckResult.
configMap helper unmarshals config JSON; parseStr/parseFloat extract
typed values.
- adapters/probes/ssh.go: SSHChecker wraps actuator.Dial +
RunCombinedOutput with a SignerSource for key resolution. Registry
(map[string]ports.Checker) with NewRegistry() pre-populating all
known kinds (ssh-script, vm-status, backup-freshness, cert-expiry
set to nil — filled by the ObservationService when signers are
available).
Verification: go build/vet, full test suite (19 pkgs), DB integration
(postgres + mcp — green).
Problem: relationship create/end existed as three drifted copies
(HTTP CreateRelationship/EndRelationship, MCP create_relationship/
end_relationship) with inline SQL, no ontology edge validation on
either path, and no audit on the MCP path.
Change:
- Internal/adapters/postgres/repositories.go: RelRepo implements
ports.RelationshipRepository (Create/End/ListFor) over the pool,
with in-tx upsert + audit/event side effects on Create.
- Internal/core/app/relationships.go: RelationshipService validates
edges against the cached ontology TypeTree (tree.ValidateEdge) and
delegates the tx to the repository. The adapter resolves slug→entity
and extracts types before calling the service.
- HTTP CreateRelationship: resolves source/target via ReadModels,
passes resolved types to RelationshipService for edge validation.
EndRelationship calls the service directly (audit stays in the
adapter for End — a simple toggle with no ontology check).
- MCP create_relationship/end_relationship: rewired to the service
(pool resolves entity IDs inline for the tool handlers; the service
validates edges and writes audit). The MCP path now gets ontology
validation and audit coverage for the first time.
- Composition root: RelationshipService built with RelRepo + Ontology
and wired through httpapi.NewHandler, ListenAndServe, and MCP
constructors.
Verification: go build/vet, full test suite (19 pkgs, DB integration
postgres+mcp green).
Problem: httpapi.NewHandler built its service dependencies internally
(entity repo, entity service, read models), making the handler a
god-object that knew how to construct its own dependencies. ADR 0016
§3.5 wants cmd/oikos/main.go to be the composition root.
Change:
- httpapi.NewHandler: services (EntityService, EntityRepo, ReadModels)
are now injected as parameters instead of constructed inside.
- httpapi.ListenAndServe: passes the injected services to NewHandler.
- cmd/oikos/main.go runAPI + the 'all' role handler: construct
entityRepo, readModels, and EntityService at the composition root
and pass them down. The router stays in httpapi for now; ownership
moves to main in a later phase.
- Tests: newTestHandler updated to construct and inject test doubles.
Verification: full build/vet, non-DB suite (19 pkgs), DB integration
(postgres + mcp — green). httpapi DB tests have the pre-existing set
of failures (TestAPIEndToEnd, TestPhase3* — verified at ec11956).
Problem: entity/relationship/graph/blast-radius reads were embedded
as inline SQL in the httpapi handlers, duplicating the recursive type
tree CTE, the blast_radius function call, and the topology-picking
query across the REST and MCP surfaces with no port abstraction.
Change:
- ports.ReadModels interface: ListEntities, GetEntity, GetEntityBySlug,
GetEntityRelations, GetBlastRadius, GetGraph (with health),
ListEntityTypes. Returns EntityWithHealth (domain.Entity + health
from entity_status join) and domain.Relationship — no gen types
in the port.
- adapters/postgres/readmodels.go: EntityReader implements ReadModels
with the existing SQL verbatim (recursive type filter, blast_radius,
most-connected-first topology, graph edge listing).
- httpapi/entities.go: ListEntities, GetEntity, GetEntityRelations,
GetBlastRadius, GetGraph rewired to ReadModels. SQL moved to the
adapter; handlers map domain/ports types to gen wire shapes.
entityWithHealthToGen, sqlcEntityToGen helpers added.
- Old sqlcEntityToGen (sqlcgen.Entity → gen.Entity) preserved for
client_lifecycle.go; mutation handlers use domainToGen.
Verification: go build/vet, full test suite (19 pkgs), DB integration
(postgres + mcp — both green). httpapi DB tests have the pre-existing
set of failures (TestAPIEndToEnd entity_types=60/501, TestPhase3*)
verified at ec11956.
Problem: entity mutations (create/update/state) existed as three drifted
copies — HTTP CreateEntity/PatchEntity, MCP create_entity/
update_entity_attributes/set_entity_state — each with its own inline
SQL, its own validation subset (MCP validated lifecycle states, HTTP
did not; HTTP patched attributes without regenerating derived checks,
MCP did; MCP wrote no audit trail), the exact drift ADR 0016's first
vertical slice exists to collapse.
Change:
- internal/core/ports: DerivedCheck, Idempotency (adapter-owned request
hash + cached-body renderer so the replay record commits in the
create's transaction), IdempotentResponse + GetIdempotent read,
AuditEntry gains Method/Path/CorrelationID, Event gains
CorrelationID; EntityUpdateInput carries ExpectedVersion +
RederiveChecks (derivation for updates runs repo-side: the graph
host fallback reads relationships through the open tx).
- internal/adapters/postgres/repositories.go: EntityRepo (Create/
Update/SetState/reads/idempotency) preserving the load-bearing
check-then-act invariants in-tx: version WHERE-clause, declared
transitions + preconditions (ValidateTransition), duplicate-slug
mapping, audit/event/writeCheck all inside one BEGIN…COMMIT.
OntologyRepo: TTL-cached OntologyStore.
- internal/core/app/entities.go: EntityService — ontology validation
(type exists, concrete, state declared — the stricter MCP rule now
governs both surfaces), default-state resolution, id generation,
derivation for creates, audit/event construction, idempotency
pass-through.
- httpapi CreateEntity/PatchEntity rewired to the service; PATCH now
regenerates derived checks (the A2 parity gap). MCP create/update/
set-state tools call the same service — and now write audit + event
rows like the HTTP surface always did.
- Integration-test seed paths fixed for the adapters/postgres package
depth (../../../seeds).
Pre-existing failures documented: TestAPIEndToEnd (entity_types 60 vs
59; 501-endpoint now 200), TestClientLifecycleEndToEnd, TestPhase3*
rows — verified failing identically at ec11956 (scratch approval-
notifier commit test drift), unrelated to this change. All mutation
integration tests (create/patch/idempotency/audit/regeneration) pass.
Verification: make test-db (postgres + mcp green, httpapi green except
the pre-existing set), full non-DB suite (19 pkgs), golangci on new
packages — 0 issues.
Problem: check derivation logic lived in internal/checkdefaults with
the pure decision logic (buildKind, address/user/port resolution)
interleaved with tx I/O (entity_status insert, graph host fallback,
check upserts) — and internal/db importing it was the plan's called-out
inverted dependency.
Change:
- internal/core/app/checkdefaults.go: Derive(tree, target, lookup) —
the full derivation (monitoring overrides, host fallback via an
injected HostLookup thunk, per-kind builders) with zero I/O imports.
Types renamed for the app surface: CheckTarget, CheckDef,
DeriveResult, Skip; LogDeriveResult.
- internal/adapters/postgres/checks.go absorbs the I/O half:
EnsureChecks (entity_status row + upsert loop), writeCheck, and
hostViaGraph. The db→checkdefaults edge is gone — adapters→core is
the ADR 0016 direction (the Phase 7 SeedService note anticipated
this; the inversion is fixed a phase early).
- seed.go pending-checks loop uses app.CheckTarget + EnsureChecks;
mcp formatting/tests follow the renamed types; both test files moved
to internal/core/app.
- Deliberate behavior note: a hostViaGraph read failure inside the
thunk now logs a warning and degrades to 'skipped: no address'
instead of aborting the whole entity-create tx — a monitoring
derivation gap is visible (warn log + coverage sweep) and self-heals
on the next mutation; failing the create over a graph-read blip was
disproportionate.
Verification: go build/vet, full test suite green (app tests exercise
every buildKind branch at their new home).
Problem: the hexagon's Phase 2 (plans/2026-08-15-hexagonal-architecture.md)
must give the use-cases-to-be their contract surface: driven-port
interfaces, test fakes, the secrets interface moved into core, and the
postgres package inside the adapters tree — before the first vertical
slice (Phase 3) can wire a composition root.
Change:
- internal/core/ports: full driven-port catalog per plan §3.3 —
repositories as transaction-scoped aggregates whose inputs carry
derived checks, audit, and events (§3.6), plus CommandExecutor,
TargetResolver, Checker, Secrets, EventPublisher, Provisioner.
Port-local payload types (Event, AuditEntry, CheckDef, KnowledgeEntry,
ExecResult) keep signatures off infrastructure; TypeTree aliases
internal/ontology (pure over domain) until checkdefaults is absorbed.
ReadModels intentionally not declared yet — it materializes with the
Phase 3 slice and grows as report handlers rewire.
- secrets.Backend is now an alias of ports.Secrets; implementations
(Infisical, SOPS, Manager) unchanged. mcp's local secretBackend
subset is deleted; tool constructors take ports.Secrets.
- internal/db → internal/adapters/postgres (mechanical import rewrite;
package identifier stays db until the Phase 3 repository split).
sqlc.yaml, Makefile, golangci exclusions, and docs follow the move;
make generate-check verified.
- internal/adapters/ssh: Executor implements ports.CommandExecutor over
the actuator dial pool + RunStreaming (10-min default timeout carried
over from the httpapi path).
- internal/adapters/remote: Resolver implements ports.TargetResolver
delegating to internal/remote (still pool-based; drops onto
ports.EntityRepository when repositories land in Phase 3 — documented
transitional import).
- internal/core/ports/portstest: importable fakes — in-memory
EntityRepo (with check-then-act SetState, side-effect recording),
RecordingExecutor, FakeChecker, SpyPublisher; port-satisfaction
guards; tests.
Risk: ports are declared ahead of implementations — signatures firm up
per phase as slices land (documented in the package doc); the
remote→postgres transitional import is explicit and dissolves in
Phase 3.
Verification: go vet, make test (race, 19 packages), generate-check,
golangci on core+adapters — 0 issues; full-repo baseline down
365→344.
Problem: the hexagonal refactor churns the backend tree for nine more
phases; the UI delivery stack (web/ SPA, cmd/desktop Wails wrapper,
compose/web image) must move to its own repo first so doc/layout
rewrites land once on a backend-only tree.
Change:
- New repo git.hubris.network/dtoro/oikos-web (v0.33.0): web/, desktop/
(updateURL repointed to oikos-web releases), compose/, own CI (web +
desktop jobs), own deploy script (CI-green gate, TOCTOU guard,
version-tagged images, prune-to-3), own webhook receiver on :9798 +
launchd unit, own compose project publishing the same 8091:80.
- Cutover executed on mac-mini in order: oikos stack's web service
stopped+removed, oikos-web project brought up on 8091; outer Caddy
untouched (targets the published port) — serving + Authentik flow +
/wails 404 quirk verified post-cutover.
- Stripped from oikos: web/, cmd/desktop/, compose/web/, desktop CI
workflow, ci.yml web job, Makefile ui/desktop/desktop-package/install
targets, the compose web service, oikos-web from deploy.sh's fallback
prune list; wails + go-keyring dropped from go.mod, vendor synced.
- README / CONTRIBUTING / AGENTS.md / .agents dev+operations docs now
point at the new repo; mbse + mascot design docs carry a path note.
Risk: production SPA serving depends on the new pipeline now; rollback
is versioned-image re-up of the old web service from a pre-split
checkout (port 8091). Desktop builds installed before the split still
check dtoro/oikos releases — one manual reinstall, noted in the
oikos-web release notes.
Verification: go vet, make test (race), make generate-check, golangci
(no new findings; baseline down 400→365); post-cutover curls —
localhost:8091 200, /wails/runtime.js 404, outer Caddy 302 Authentik.
Problem: the hexagonal-architecture plan (plans/2026-08-15-hexagonal-
architecture.md) needs its foundation — an accepted ADR, the target
directory tree, and machine-checked dependency rules — before any
service extraction starts. Also folds the four outstanding review
findings (F3.1/F5/F6/F7) into the plan: ObservationService owns the
bounded probe-concurrency contract (scheduler.go:133), Phase 9 gates
ExecutionService+PolicyService ≥ 90% with a gating-matrix test,
per-phase abort criteria, and the §3.2 internal/config note.
Change:
- docs/adr/0016-hexagonal-ports-adapters.md records context, decision,
and consequences of the ports & adapters migration.
- internal/domain → internal/core/domain (mechanical import rewrite,
20 files), new internal/core/{ports,app}, internal/adapters trees
with package docs.
- .golangci.yml: depguard rules for §3.1 (core purity, no agent-client
tech in core, nomos isolation — the nomos rules self-activate when
internal/nomos exists in Phase 8). Config migrated to golangci-lint
v2 format so it loads at all (the v1 config errored under v2, masked
by CI's advisory continue-on-error). Verified depguard fires on a
planted openai-go import in internal/core/app.
- CONTRIBUTING.md layout section now shows the core/adapters tree.
Risk: import path churn is mechanical and tests pass unchanged; the
lint config migration surfaces the pre-existing 400-issue baseline
(advisory in CI, unchanged policy) — new/moved packages lint clean.
Verification: go vet ./..., make test (race, core/domain at 100%
coverage), make generate-check, golangci-lint on internal/core/... and
internal/adapters/... — 0 issues; depguard violation probe confirmed.
Removes the entire Matrix-based notifier (internal/notifier/) that polled
for pending approvals, sent Matrix alerts, and checked for reaction-based
approve/deny. Approval decisions now work on any chat platform (Hermes
desktop, Telegram, Discord, WhatsApp, CLI) via two new MCP tools:
- list_approvals — query pending/recent approvals by status or entity
- decide_approval — approve/deny via same API endpoint as UI + nomos
Config fields removed: MatrixHomeserver, MatrixUserID, MatrixToken,
MatrixRoomID, ApprovalHMACSecret. Docker notifier: service removed.
Approval HMAC token generation removed (unused by code).
The existing chat-assent path in nomos (cmd/nomos/assent.go) and the
control-room Approve button keep working unchanged — both call the
shared POST /api/v1/approvals/{id}/decision endpoint.
New tool batches session discoveries into the knowledge graph:
- Creates a session-audit knowledge entry with summary
- Links it to all touched entities via 'documents' relationships
- Creates individual discovery knowledge entries
- Stamps each entity with last_agent_session attribute
- Updates AGENTS.md with tool listing
P0: Transport escalation in classifyAndGate (server.go:745) now exempts
log-inspection commands (tail/head/cat/journalctl on *.log or /logs/)
from the /opt//etc//var/lib/ gating on LXC targets. Fixes the
'tail -3 /opt/seanime/data/logs/seanime.log queued for approval' bug.
P1: queryEntity returns actionable error when slug_or_id param is empty
('slug_or_id is required') instead of silent 'entity not found: '.
P2: Added slugArg() helper (server.go) so get_entity, get_relations,
get_blast_radius, and explain accept 'slug' as an alias for their
declared param name. Solves the discoverability inconsistency where
every tool used a different param name for the same concept.
P3: Two new MCP tools:
- restart_service(target, service) — systemctl restart wrapper,
correctly classified config_mutation (requires approval)
- push_file(target, source_path, dest_path, backup=true) — pct push
from Proxmox host into LXC, with optional backup. Classified
config_mutation. LXC-only for now.
P4: Updated homelab-lxc-ops skill with MCP tools preference table.
Plus: Wails desktop build now uses build-tag approach for frontend embed
(assets_embed.go + assets_stub.go), so go build ./... works on
clean checkout without the frontend built first.
Version: 0.31.0 → 0.32.0
Adds pvecm_quorum_check.sh probe script and wires it into the
checkdefaults system as a new 'quorum' monitoring kind on proxmox-host
entities. Runs every 60s via ssh-script, surfaces unhealthy signal when
cluster loses quorum.
Closes the monitoring blind spot that let the 2026-08-12 3.5h corosync
flapping outage go undetected (ping passed, cluster was non-quorate).
Changes:
- seeds/ontology.yaml: proxmox-host declares monitoring: [quorum]
- internal/checkdefaults/defaults.go: KindQuorum builder
- internal/checkdefaults/build_test.go: 2 new test cases
- checks/pvecm_quorum_check.sh: new probe (deployed to hubris + strong)
- VERSION: 0.30.2 -> 0.31.0
Registers a handler that serves the embedded OpenAPI 3.0 spec
(compiled into the binary via oapi-codegen) at a browseable
endpoint. Uses gen.GetSwagger() to deserialize the embedded
base64+gzip spec and returns it as JSON.
46 paths, 42 schemas — agents and humans can now introspect the
full API surface without reading Go source.
Read-only diagnostic commands ethtool, lsmod, lspci, modinfo, and dkms
were missing from the readOnlyLeadPattern in the command classifier,
causing compound diagnostic commands (e.g. 'uname -r && ethtool -i eno1
&& lsmod | grep r8169') to be misclassified as config_mutation instead
of read_only. This forced operator approval for simple hardware/driver
inspection during the 2026-08-12 hubris NIC cutover session.
Added regression test with the exact compound command from that session.
Every run call creates a classification entity, but it was never
connected to the execution via a graph edge — only via a DB column
(executions.classification_id). The ontology requires:
classification —precedes→ execution
Without this edge, all 53 classification entities had zero
relationships, making them invisible to get_relations and
blast-radius analysis.
Adds an idempotent INSERT into relationships after the existing
classification_id update, matching the same pattern used for
targets edges.
- Wire secretsManager in NewHandler() — instantiate InfisicalBackend
when OIKOS_INFISICAL_SITE_URL is set (previously always nil)
- Add get_secret, list_secrets, set_secret MCP tools with nil-backend
graceful degradation
- Add oikos secret get|set|list CLI subcommands for Infisical
- Fix Set() bug: create-before-update so new keys are created;
add Type: "shared" to Update so it finds the right secret;
disable SDK cache so Get returns fresh data after Set
- Clean enrollment response: remove fake infisical_client_id/
infisical_client_secret stubs, store age key in Infisical for real
- classifyAndGate: escalate read-only commands on lxc: targets that
touch /opt/, /etc/, /var/lib/ to config_mutation. The classifier
scores command text only, not the SSH transport layer — SSH-ing into
a container to read config is riskier than pct exec from the host.
- ontology: standalone-server monitoring override from inherited
[ping, resource, updates] to [http]. VPS-like machines may not be
SSH/ICMP-reachable from the scheduler; HTTP is the LCD liveness
signal. Entities with full SSH can override per-entity.
- get_health_summary: filter out state=destroyed entities (was noise
from 20+ destroyed test LXCs, deprecated services, etc.)
- create_entity: document the monitoring footgun in the tool description
(creating a type=check entity does NOT wire a check_def; the correct
path is update_entity_attributes with monitoring + url attributes)
Adds a new 'dns' semantic monitoring kind that probes whether a DNS name
resolves. Uses net.LookupNS (NS records) with fallback to net.LookupHost
(A/AAAA). Supports an explicit server config for split-horizon resolution.
Changes:
- seeds/ontology.yaml: dns-zone monitoring: none → [dns] (was deferred
since 2026-06 with a comment 'no dns checker exists yet')
- seeds/inventory.yaml: host:netbird-vps monitoring: [http] (was none;
VPS was invisible for 7 days during the 2026-07-29 outage)
- internal/checkdefaults/defaults.go: add KindDNS, buildKind case for 'dns'
that creates a check_def at 5-minute intervals
- internal/scheduler/scheduler.go: add checkDNS probe + wire in executeCheck
The DNS checker catches stale/unreachable zones (e.g. matrix.hubris.network
pointing to a dead VPS IP). The VPS HTTP check probes the public endpoint
every 60s, closing the 7-day monitoring gap.
I — run pre-flights QEMU guest agent before queueing VM execution
classifyAndGate now checks vm: targets for qemu_guest_agent attribute.
If not_running/missing, returns immediate error instead of queuing forever.
II — policy.yaml: documented host-mutation classifier rule
Added comment clarifying that host-level package/kernel mutations
(apt-get install, dpkg, systemctl enable) always classify as
config_mutation and thus need operator approval.
III — health attribute read-only in update_entity_attributes
Strips scheduler-owned keys (health, last_check_at, last_check) from
attribute updates with a clear message directing agents to
get_health_summary / list_checks instead.
IV — Recorded discovered dependency edges
vm:zimaos → depends-on → lxc:nfs-export (NFS /media/library mount)
vm:zimaos → depends-on → host:strong (NFS /media/ludo-library mount)
Also updated the run tool description to mention both guardrails.
get_relations now accepts an optional 'types' (comma-separated) parameter
to filter relationship types — filters out the noisy exec/targets edges
that previously drowned useful host/provides edges.
get_health_summary now accepts an optional 'health' (comma-separated)
parameter to return only entities in specific health states (e.g.
'health=down,stale') instead of the full 100+ entity list.
ping_service now:
- Falls back to e.attributes->>'public_host' when 'url' is not set
(covers LXCs that only have public_host in the graph)
- Performs a live HTTP HEAD probe against the resolved URL, returning
the actual status code instead of just the scheduler's stale health
state
Also: fixed matrix.hubris.network DNS record (was pointing to dead VPS),
pruned 6 dead graph edges, wired url attributes on 7 LXCs, added VPS
HTTP monitoring check, and resolved the 18k-occurrence unmonitored signal.
This session's audit is documented as
document:nomos/2026-08-05-dns-monitoring-improvements-for-strong-hosted-services.
- Add MCP tool — lightweight connectivity check returning server
identity, no DB hit (resolves agent connection-test friction)
- Tighten 6 tool descriptions (get_relations, get_health_summary,
query_metrics, get_trend, get_event_timeline, ping) to be searchable
in the first 8-12 words
- Document Hermes MCP client setup in ADR-0012 with token security caveat
- Move completed plan to plans/done/
A `monitoring` attribute on an entity now overrides its type's declaration:
"none" opts out, a list overrides the kinds. service:haos uses it to opt out —
haos blocks SSH (no process probe can reach it) and the VM is already covered
by vm:haos's vm-status check, so the redundant process check only ever reported
false-down. vm:haos -> service:haos via provides confirms the coverage.
VMs declared monitoring [ping], but many block ICMP and lack a guest agent
(haos), so ping was the wrong probe — a powered-on VM reported "down". Add a
vm-status check: `qm status <pve_id>` on the VM's Proxmox host, which tests
"powered on" without needing the VM's network at all. vm type monitoring is
now [vm-status].
matrix.hubris.network is a public hostname (federation) resolving to
netbird-vps, not served by the lab Caddy — so its cert-expiry check's
dial=caddy IP failed. Drop the dial for matrix; it dials by name (DNS ->
public) like wget already proved works.
The ontology's stated intent was "http when it has a url, else a process
check", but the implementation emitted BOTH for every url-service — so ~17
fronted services carried a redundant process check that, under worst-of
aggregation, let a fragile supplementary probe (wrong unit name, unreachable
host, no guest agent) veto two healthy http checks and report the service
"down" while it was up (authentik, zimaos, house, matrix, ...).
buildKind now emits a process check only for services WITHOUT a url, or when
an explicit probe_unit opts into binary-level depth. http is the canonical
service-liveness probe (tests the real endpoint through the TLS terminator);
the redundant process checks were removed.
process_check.sh ran `systemctl is-active <entity-name>`, but a service's name
is a logical label, not its unit/container name — matrix is matrix-synapse.service
+ element-web/mautrix-* containers, authentik is authentik-server/-worker
containers. So every multi-component or docker service reported "inactive"
while up (authentik, matrix, photos, house, arr-stack, …).
Resolve in order: exact systemd unit, a unit with the name as prefix
(matrix -> matrix-synapse.service), or a running docker container whose name
contains it. checkdefaults passes a declared probe_unit/systemd_unit/container
attribute when set, for precision.
The DB-only audit_knowledge_graph can't see guests running in Proxmox that
have no entity, or entities whose pve_id is no longer live — the drift that
the stray test LXCs were a symptom of. discover_infra_drift enumerates running
guests via pct/qm list on every proxmox host (over the same SSH/pct path the
checks use) and diffs against the DB: returns missing (live, no entity) and
ghost (DB, not live). Read-only.
Companion to audit_knowledge_graph; the skill now runs both and treats the
remaining checks (misplaced parent, undeployed scripts, seed drift) as manual.
resolveProxmoxHostSlug trusted attributes.host verbatim, so a value polluted
with prose — lxc:teddycloud carried host="hubris (confirmed via pct config…)" —
became a slug that never resolved, leaving its checks 'down' despite a correct
`hosts` edge. Treat an attribute containing whitespace/parens as invalid and
fall back to the canonical hosts edge.
The audit now reports `polluted_attrs` — entities whose routing-critical
attributes carry prose — so this class is visible instead of a silent
resolution failure.
The B1 target-state filter `tgt.state NOT IN ('deprecated','destroyed')`
evaluates to NULL (unknown) when a target's state is NULL, which the WHERE
clause treats as false — so freshly-seeded entities without an explicit state
(the 20 TLS certificates) were silently dropped from ListEnabledCheckDefs and
never monitored. Treat NULL state as active (only explicit deprecated/
destroyed is excluded): `tgt.state IS NULL OR tgt.state NOT IN (...)`.
checkCertExpiry now accepts a `dial` address and sets ServerName to the
hostname — it connects to the terminator's IP while SNI/cert-read use the
hostname. The scheduler container has no mesh interface and the host resolver
doesn't know the split-horizon zone, so *.hubris.network can't be dialed by
name from there; dialing Caddy's lab IP (reachable on the LAN) makes the probe
work. The builder passes through a cert entity's `dial` attribute.
Re-seed the 20 *.hubris.network certificate entities with dial=192.168.8.175
(Caddy) and uses-certificate edges; cert-expiry monitoring now has real data.
The ontology declared monitoring [cert-expiry] on the certificate type and a
working checkCertExpiry probe existed, but checkdefaults had no cert-expiry
builder and no certificate entities were seeded — so certificate expiry, a
real failure mode, was invisible.
Add a KindCertExpiry builder (dials the cert's hostname on :443 hourly, warns
at 30d / crit at 7d) and seed certificate entities for the 20 public
*.hubris.network routes plus uses-certificate edges from each ingress route.
A service check used to bake its hosting LXC's lan_ip and SSH it directly as
root, which failed because the scheduler key is authorized on the Proxmox hosts
but not inside every guest — leaving all 8 service process checks 'down' even
after the guest routing and scripts were fixed.
ResolveExecTargetForCheck now, for a non-guest target, walks the
provides/runs-on/hosts edges to the compute entity that runs it and routes
through that: pct/qm exec if the host is a guest, direct SSH with the host's
correct user (workstation `user` attr) if it's a machine. The guest-resolution
path is shared via resolveGuest, and the scheduler no longer needs an
isMachine special case — one resolver handles guest, machine, and service.
Older writeCheck inserts omitted target_type, so every seed-created check_def
had a NULL/empty target_type. checkSSHScript's IsGuest check then never matched,
and guest checks silently fell back to their baked (often mesh-only) address —
keeping them 'down' even after the pct-exec routing and deployed scripts were
in place. rclone stayed down for exactly this reason after the host-hop fix.
writeCheck now writes target_type, and checkSSHScript resolves the type from
the target_id when the column is blank (a runtime safety net for existing rows;
the seed rows were also backfilled in the live DB).
Graph view: raise the node cap 500 -> 2000 and exclude execution/task audit
rows from the default whole-graph view so the cap is spent on actual topology
rather than ~380 cognition records that crowded out every host/lxc/service.
dns-zone monitoring [dns] -> none: no dns checker exists, so the declaration
only produced unresolvable `unmonitored` noise (requires ontology re-ingest;
coverageSweep now auto-clears the stale signals). Flip back to [dns] when a
checker lands.
Operator tooling: tools/deploy-checks.sh pushes check scripts into guests via
pct push (a pct-exec-routed check runs the script INSIDE the guest), wired
into the post-pull setup-checks hook so guests stay in sync on Proxmox hosts;
scripts/cleanup-orphan-checks.sh (dry-run by default) and
report-stray-test-lxcs.sh retire legacy cruft. VERSION 0.13.0 -> 0.14.0.
Plan: plans/2026-07-29-health-check-reality-and-knowledge-graph.md.
Adds audit_knowledge_graph (MCP tool) and GET /api/v1/audit/drift (endpoint)
backed by a shared internal/audit package. One pass surfaces the structural
gaps an operator otherwise finds by accident: orphan check entities, checks
targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored
declared types, and live edges pointing at destroyed targets. Each finding
carries a suggested remediation runbook. Read-only and safe to run unattended.
Ships the knowledge-graph-audit skill (SKILL.md + seeded runbook) that
interprets the report and routes findings to the lifecycle runbooks.
ListEnabledCheckDefs now LEFT JOINs the target entity and excludes rows whose
target is deprecated or destroyed, so retired things (secrets-issuance,
homelab-mcp, the dead secrets ingress route) stop generating permanent false
alarms instead of waiting for an operator to disable the check_def by hand.
coverageSweep's None() branch previously did nothing, so a type changed from
declared monitoring to `monitoring: none` (dns-zone) left its open
`unmonitored` signals lingering forever — a None() entity never gains a check,
so the hasCheck resolution path never fired. It now resolves those signals.
