fix(scheduler): honour check_defs.interval_s, and renumber migrations off main
ListEnabledCheckDefs selected interval_s but never filtered on it, so every enabled check ran on every 30s pass and the declared per-check intervals were decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126 SSH connections every 30s (~363k/day) and `apt update` on every machine every 30 seconds — 14,400 mirror hits a day to answer a question that changes daily. - check_defs.last_run_at (migration 026) + a due-ness predicate in the query. A column rather than scheduler memory because this control plane restarts on every deploy, and an in-memory map would re-fire every check on each restart. - runCheck stamps last_run_at before processing the result, so a permanently failing check backs off to its interval instead of re-running every pass. - updates and backup-freshness drop to daily. Both answer questions whose answers change about once a day; 60s was just the shared ssh-script default. - last_run_at is seeded to a random offset within the interval so checks created by the same seed do not stay in lockstep — otherwise ~165 probes land in the same instant each minute instead of spread across it. Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them. Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day (down from 14,400), with each 60s check landing at its own point in the minute. Also renumbers 022→023, 023→024, 024→025: origin/main added its own 022_knowledge_revisions, and prod has already applied version 22. Left colliding, prod would have skipped the monitoring_spec migration entirely and then failed the seed on a missing column. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -183,7 +183,13 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{ssh("updates_check.sh")}, ""
|
||||
// Daily. updates_check.sh runs `apt update` against the distro
|
||||
// mirrors; the shared 60s ssh-script default would have meant 1,440
|
||||
// mirror hits per machine per day to answer a question whose answer
|
||||
// changes about once a day.
|
||||
u := ssh("updates_check.sh")
|
||||
u.interval = 86400
|
||||
return []checkDef{u}, ""
|
||||
|
||||
case KindCapacity:
|
||||
if host == "" {
|
||||
@@ -224,9 +230,9 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
|
||||
if port != 0 && port != 22 {
|
||||
cfg["port"] = port
|
||||
}
|
||||
// Hourly: a daily backup does not need a 60s probe, and each one is an
|
||||
// SSH round trip.
|
||||
return []checkDef{{kind: "backup-freshness", config: cfg, interval: 3600}}, ""
|
||||
// Daily. The freshness budget itself is a day, so probing more often
|
||||
// cannot surface anything sooner — it just costs an SSH round trip.
|
||||
return []checkDef{{kind: "backup-freshness", config: cfg, interval: 86400}}, ""
|
||||
|
||||
case KindHTTP:
|
||||
url := httpURL(t, attrs)
|
||||
@@ -286,9 +292,16 @@ func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef)
|
||||
// Config is derived from the seed, so the seed wins on re-ingest and
|
||||
// attribute changes propagate. `enabled` is deliberately left alone: it
|
||||
// is operational state an operator may have toggled.
|
||||
// last_run_at is seeded to a random point inside the interval so checks
|
||||
// created together do not stay in lockstep. Every check the seed creates
|
||||
// would otherwise come due in the same instant forever: ~165 probes
|
||||
// landing at once each minute rather than spread across it. Deliberately
|
||||
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
||||
// and re-herd everything.
|
||||
tag, err := tx.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1, $2, $3, $4, $5, 30, true)
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1, $2, $3, $4, $5, 30, true,
|
||||
now() - make_interval(secs => random() * $5::int))
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET target_id = EXCLUDED.target_id, kind = EXCLUDED.kind,
|
||||
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
||||
|
||||
Reference in New Issue
Block a user