oikos/docs/infrastructure/oikos-check-lifecycle.md

Oikos check lifecycle — how monitoring works

This runbook covers how Oikos health checks are derived, created, and wired so an agent (Nomos) doesn't reverse-engineer source when asked to add monitoring to an entity — the problem that stranded session 23da10db (2026-08-03).

Concepts

• check_defs (scheduler config, table check_defs): the row the scheduler reads to know what to probe and when. One per check instance.
• check entity (type check, slug check:<kind>:<target>:<n>): the knowledge-graph entity for that check. It carries attributes (check_type, target, port, …) and checks edges to the probed target.
• monitoring spec on an entity type (entity_types.monitoring_spec): the default list of check kinds (e.g. [http, process] for service).
  • Per-entity override: set monitoring in the entity's attributes — "none" for zero checks, ["http"] to replace the type defaults.
• checkdefaults.Ensure (internal/checkdefaults/defaults.go): the function that reads the monitoring spec, resolves host/port/URL from attributes + relationships, and writes check_defs rows. Idempotent.

When checks are derived

checkdefaults.Ensure runs in three situations (as of v0.17.1+):

1. Seed/deploy ingest — internal/db/seed.go:231. Every entity gets its default checks once on initial ingest.
2. HTTP POST /api/v1/entities (create) — ensureDefaultChecks at internal/httpapi/impl.go:1012. Creating an entity via the REST API derives its checks in the same transaction.
3. HTTP PATCH /api/v1/entities (patch) — ensureDefaultChecks at internal/httpapi/impl.go:1280. Changing an entity's attributes (especially monitoring) via the REST API regenerates its checks.
4. MCP create_entity — SAME hook. Creating an entity via the MCP tool derives checks. (Added 2026-08-03; previously MCP had no create.)
5. MCP update_entity_attributes — SAME hook. Changing an entity's monitoring attribute via MCP now regenerates checks. (Added 2026-08-03; previously MCP updates silently skipped check derivation — the exact bug that stranded the haos session.)

Check slug grammar

check:<kind>:<target-type>:<target-name>:<n>

Examples: check:http:service:jellyfin:0, check:vm-status:vm:haos:0, check:cert-expiry:cert:house.hubris.network:0.

Adding monitoring to an entity

If the entity already exists:

update_entity_attributes(slug="service:haos", attributes={"monitoring":["http"]})

This regenerates checks via checkdefaults.Ensure. The result message tells you how many checks were derived and whether any kinds were skipped (and why).

If the entity does not exist yet (a new check, ingress, cert, etc.):

create_entity(type="check", name="HAOS http check",
  slug="check:http:service:haos:0",
  attributes={"check_type":"http:service","target":"service:haos","port":"8123"})

This creates the entity AND derives its check_defs. Same for a new ingress (type=ingress, monitoring [http]) or cert (type=cert, monitoring [cert-expiry]).

To remove monitoring: set monitoring:["none"] or transition the entity to a terminal lifecycle state (set_entity_state → deprecated/destroyed).

Caveats

• A service without a url attribute AND without a probe_unit gets no process check (the http check covers liveness; the process check would be redundant without an opt-in probe_unit). The skip is logged.
• A service whose address comes from a hosts edge may produce no checks on initial create because the edge doesn't exist yet — the next inventory ingest (or a later update_entity_attributes after the edge is created) fills it in.
• A not found error from update_entity_attributes means the entity doesn't exist — use create_entity instead.
• check_defs has target columns (target_id, target_type). A check entity needs a checks relationship (create_relationship(source=check:…, target=service:…, type="checks")) so the scheduler can resolve what to probe. create_entity derives the check_def; create_relationship links the check entity to its target in the graph.

Related files

• internal/checkdefaults/defaults.go — Ensure, Target, LogResult
• internal/httpapi/default_checks.go — ensureDefaultChecks (HTTP hook)
• internal/db/checks.go — db.EnsureEntityChecks (shared hook)
• internal/db/seed.go — seed-time check derivation
• internal/mcp/tools.go — create_entity, update_entity_attributes

Revision history

• 2026-08-03: Created after session 23da10db stranded for lack of entity- creation tool and unawareness of check-derivation triggers. Covers the MCP create_entity + update_entity_attributes regen paths added same day.