fix(scheduler): honour check_defs.interval_s, and renumber migrations off main

ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.

- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
  A column rather than scheduler memory because this control plane restarts on
  every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
  failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
  answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
  created by the same seed do not stay in lockstep — otherwise ~165 probes
  land in the same instant each minute instead of spread across it.
  Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.

Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.

Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 14:03:30 +02:00
parent 1dca2cfd7a
commit d7b526a112
11 changed files with 113 additions and 25 deletions

View File

@@ -1,4 +1,4 @@
-- 022_entity_type_monitoring.up.sql
-- 023_entity_type_monitoring.up.sql
-- Declare, per entity type, what monitoring that type warrants.
--
-- Motivation: only 3 of 89 active entities had an enabled check_def, because

View File

@@ -1,4 +1,4 @@
-- 023_executions_created_at_index.up.sql
-- 024_executions_created_at_index.up.sql
-- Support newest-first execution history.
--
-- ListExecutions previously ordered by the target entity's slug, which is

View File

@@ -1,4 +1,4 @@
-- 024_execution_logs.up.sql
-- 025_execution_logs.up.sql
-- Incremental command output for executions.
--
-- Until now `executions.result` was a single JSONB blob written once, at the

View File

@@ -0,0 +1,32 @@
-- 026_check_defs_last_run.up.sql
-- Make check_defs.interval_s actually mean something.
--
-- ListEnabledCheckDefs selected interval_s but never filtered on it, and
-- nothing in the scheduler read it except staleSweep. So every enabled check
-- ran on every 30-second pass and the declared per-check intervals were
-- decorative.
--
-- That went unnoticed at 17 enabled checks (~0.5 SSH/s). Restoring monitoring
-- coverage takes it to ~150, where it would have meant ~126 SSH connections
-- every 30s — roughly 363k/day — and, worst of all, `apt update` on every
-- machine every 30 seconds via updates_check.sh: 14,400 mirror hits a day to
-- answer a question whose answer changes about once a day.
--
-- last_run_at is a column rather than scheduler memory on purpose: an
-- in-memory map resets on restart, and this control plane restarts on every
-- deploy, so every check would fire at once each time — a thundering herd
-- exactly when the stack is least settled.
--
-- NULL means "never run", which is due immediately. Existing rows therefore
-- all fire once on the first pass after this migration, then settle into
-- their declared cadence.
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_run_at TIMESTAMPTZ;
-- The scheduler's hot query: enabled AND due. Partial on enabled since
-- disabled checks are never considered.
CREATE INDEX IF NOT EXISTS idx_check_defs_due
ON check_defs (last_run_at)
WHERE enabled;
COMMENT ON COLUMN check_defs.last_run_at IS 'When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.';