fix(scheduler): derive entity health from all its checks, not the last one
host:strong logged 226 health.changed events in one hour, oscillating down/healthy while the host was fine throughout. host:hubris did it 126 times. runCheck wrote entity_status.health on every check completion, so an entity's health was simply whichever of its checks finished most recently. A host with six checks reported whichever facet happened to be sampled last, and one failing probe alternating with five passing ones flapped forever. resolveSignal forced "healthy" too, a second path by which one passing probe erased another probe's genuine failure. On this fleet the trigger is a known false positive: the scheduler's network vantage point cannot ICMP host:strong, so its ping check fails while every ssh-script check succeeds. Under last-writer-wins that single probe declared the whole host down, twice a minute. Each check now records its own verdict (check_defs.last_health, migration 027) and the entity's health is the worst across its enabled checks. A failing probe now degrades the entity honestly and *stably*, without erasing what the other five report, and health.changed fires only when that aggregate actually moves. Checks that have never run are ignored rather than counted as unknown, so adding a check cannot drag a known-good entity down before it has a verdict. Also declares service:oikos in the seed. The previous commit re-pointed the mcp ingress at it, but the entity only ever existed in the production database — so a fresh seed (a new install, or a DR restore) failed on an unresolvable edge. Caught by seeding an empty database rather than a copy of prod, which is the only way that class of bug shows up. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -270,6 +270,15 @@ entities:
|
||||
attributes: {url: "https://teddy.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/131-teddycloud.md,
|
||||
risk_notes: "no forward-auth gate — reachable by anyone on LAN/mesh"}}
|
||||
# The Go control plane itself: api/scheduler/notifier/web on the mac-mini,
|
||||
# and what mcp.hubris.network fronts since the cutover. It existed in the
|
||||
# database (created outside the seed) but was never declared here, so a
|
||||
# fresh seed could not resolve the routes-to edge below.
|
||||
- {slug: "service:oikos", type: service, name: oikos,
|
||||
attributes: {url: "https://oikos.hubris.network",
|
||||
host: "ws:mac-mini",
|
||||
ports: {api: 8090, web: 8091, nomos_gateway: 8092},
|
||||
note: "homelab automation platform — api/scheduler/notifier/web on mac-mini docker compose (project name oikos)"}}
|
||||
- {slug: "service:homelab-mcp", type: service, name: homelab_mcp,
|
||||
attributes: {port: 9810, systemd_unit: homelab-mcp,
|
||||
endpoint: "https://mcp.hubris.network/mcp",
|
||||
@@ -451,6 +460,7 @@ relationships:
|
||||
# fronts the Go api. Nomos recorded this correctly on 2026-07-12; the seed
|
||||
# was the stale one, and re-asserting the old edge alongside it is what made
|
||||
# ingress:mcp a cardinality violation.
|
||||
- {source: "ws:mac-mini", target: "service:oikos", type: provides}
|
||||
- {source: "ingress:mcp.hubris.network", target: "service:oikos", type: routes-to}
|
||||
- {source: "ingress:secrets.hubris.network", target: "service:secrets-issuance", type: routes-to}
|
||||
- {source: "ingress:house.hubris.network", target: "service:house", type: routes-to}
|
||||
|
||||
Reference in New Issue
Block a user