fix(scheduler): honour check_defs.interval_s, and renumber migrations off main

ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.

- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
  A column rather than scheduler memory because this control plane restarts on
  every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
  failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
  answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
  created by the same seed do not stay in lockstep — otherwise ~165 probes
  land in the same instant each minute instead of spread across it.
  Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.

Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.

Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 14:03:30 +02:00
parent 1dca2cfd7a
commit d7b526a112
11 changed files with 113 additions and 25 deletions

View File

@@ -113,6 +113,13 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
result := executeCheck(ctx, cd)
// Stamp the run before processing the result: due-ness must advance even
// when a check fails, or a permanently failing check would be re-run on
// every pass instead of at its declared interval.
if err := q.MarkCheckRun(ctx, cd.EntityID); err != nil {
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
}
latency := time.Since(start).Milliseconds()
if result.metrics == nil {
@@ -483,8 +490,8 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
if usedPct > float64(cfg.ThresholdPct) {
return checkResult{
health: "degraded", signalKind: "disk",
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
metrics: metrics,
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
metrics: metrics,
}
}
@@ -754,7 +761,6 @@ func sshExec(ctx context.Context, host, port, user, cmd string, timeout time.Dur
return out, nil
}
// metricThreshold defines warn/crit thresholds for a single metric.
type metricThreshold struct {
Warn float64 `json:"warn"`
@@ -790,4 +796,4 @@ func evaluateSeverity(kind string, signalKind string, config []byte, metrics map
return "warning"
}
var _ = uuid.UUID{} // ensure uuid import stays
var _ = uuid.UUID{} // ensure uuid import stays