dtoro
6ca6d5b352
host:strong logged 226 health.changed events in one hour, oscillating down/healthy while the host was fine throughout. host:hubris did it 126 times. runCheck wrote entity_status.health on every check completion, so an entity's health was simply whichever of its checks finished most recently. A host with six checks reported whichever facet happened to be sampled last, and one failing probe alternating with five passing ones flapped forever. resolveSignal forced "healthy" too, a second path by which one passing probe erased another probe's genuine failure. On this fleet the trigger is a known false positive: the scheduler's network vantage point cannot ICMP host:strong, so its ping check fails while every ssh-script check succeeds. Under last-writer-wins that single probe declared the whole host down, twice a minute. Each check now records its own verdict (check_defs.last_health, migration 027) and the entity's health is the worst across its enabled checks. A failing probe now degrades the entity honestly and *stably*, without erasing what the other five report, and health.changed fires only when that aggregate actually moves. Checks that have never run are ignored rather than counted as unknown, so adding a check cannot drag a known-good entity down before it has a verdict. Also declares service:oikos in the seed. The previous commit re-pointed the mcp ingress at it, but the entity only ever existed in the production database — so a fresh seed (a new install, or a DR restore) failed on an unresolvable edge. Caught by seeding an empty database rather than a copy of prod, which is the only way that class of bug shows up. Co-Authored-By: Claude <noreply@anthropic.com>
29 lines
1.6 KiB
SQL
29 lines
1.6 KiB
SQL
-- 027_check_last_health.up.sql
|
|
-- Aggregate an entity's health across its checks instead of last-writer-wins.
|
|
--
|
|
-- runCheck wrote entity_status.health on every check completion, so an
|
|
-- entity's health was simply whichever of its checks finished most recently.
|
|
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
|
|
-- five passing ones produced a permanent flap: 226 health.changed events for
|
|
-- host:strong in a single hour, oscillating down/healthy, while the host was
|
|
-- fine the whole time.
|
|
--
|
|
-- On this fleet the trigger is a known false positive: the scheduler's network
|
|
-- vantage point cannot ICMP host:strong, so its ping check fails while every
|
|
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
|
|
-- to declare the whole host down, twice a minute.
|
|
--
|
|
-- Storing each check's own verdict lets entity health be derived as the worst
|
|
-- current result across that entity's enabled checks — so a single failing
|
|
-- probe degrades the entity honestly without erasing what the other five say,
|
|
-- and a passing probe cannot mask a genuine failure elsewhere.
|
|
|
|
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
|
|
|
|
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
|
|
|
|
-- The aggregation reads every enabled check for one target on each completion.
|
|
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
|
|
ON check_defs (target_id)
|
|
WHERE enabled AND target_id IS NOT NULL;
|