fix(scheduler): derive entity health from all its checks, not the last one
host:strong logged 226 health.changed events in one hour, oscillating down/healthy while the host was fine throughout. host:hubris did it 126 times. runCheck wrote entity_status.health on every check completion, so an entity's health was simply whichever of its checks finished most recently. A host with six checks reported whichever facet happened to be sampled last, and one failing probe alternating with five passing ones flapped forever. resolveSignal forced "healthy" too, a second path by which one passing probe erased another probe's genuine failure. On this fleet the trigger is a known false positive: the scheduler's network vantage point cannot ICMP host:strong, so its ping check fails while every ssh-script check succeeds. Under last-writer-wins that single probe declared the whole host down, twice a minute. Each check now records its own verdict (check_defs.last_health, migration 027) and the entity's health is the worst across its enabled checks. A failing probe now degrades the entity honestly and *stably*, without erasing what the other five report, and health.changed fires only when that aggregate actually moves. Checks that have never run are ignored rather than counted as unknown, so adding a check cannot drag a known-good entity down before it has a verdict. Also declares service:oikos in the seed. The previous commit re-pointed the mcp ingress at it, but the entity only ever existed in the production database — so a fresh seed (a new install, or a DR restore) failed on an unresolvable edge. Caught by seeding an empty database rather than a copy of prod, which is the only way that class of bug shows up. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -114,6 +114,8 @@ type CheckDef struct {
|
||||
UpdatedAt time.Time
|
||||
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
||||
LastRunAt *time.Time
|
||||
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
type Classification struct {
|
||||
@@ -255,6 +257,20 @@ type KnowledgeEntity struct {
|
||||
UpdatedAt time.Time
|
||||
ContentHash *string
|
||||
Search interface{}
|
||||
EditedBy string
|
||||
DeletedAt *time.Time
|
||||
}
|
||||
|
||||
type KnowledgeRevision struct {
|
||||
ID int64
|
||||
EntityID uuid.UUID
|
||||
Title string
|
||||
Content string
|
||||
Source *string
|
||||
Tags []string
|
||||
EditedBy string
|
||||
VersionAt time.Time
|
||||
RevisedAt time.Time
|
||||
}
|
||||
|
||||
type Ledger struct {
|
||||
|
||||
Reference in New Issue
Block a user