fix(scheduler): honour check_defs.interval_s, and renumber migrations off main

ListEnabledCheckDefs selected interval_s but never filtered on it, so every
enabled check ran on every 30s pass and the declared per-check intervals were
decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126
SSH connections every 30s (~363k/day) and `apt update` on every machine every
30 seconds — 14,400 mirror hits a day to answer a question that changes daily.

- check_defs.last_run_at (migration 026) + a due-ness predicate in the query.
  A column rather than scheduler memory because this control plane restarts on
  every deploy, and an in-memory map would re-fire every check on each restart.
- runCheck stamps last_run_at before processing the result, so a permanently
  failing check backs off to its interval instead of re-running every pass.
- updates and backup-freshness drop to daily. Both answer questions whose
  answers change about once a day; 60s was just the shared ssh-script default.
- last_run_at is seeded to a random offset within the interval so checks
  created by the same seed do not stay in lockstep — otherwise ~165 probes
  land in the same instant each minute instead of spread across it.
  Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them.

Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day
(down from 14,400), with each 60s check landing at its own point in the minute.

Also renumbers 022→023, 023→024, 024→025: origin/main added its own
022_knowledge_revisions, and prod has already applied version 22. Left
colliding, prod would have skipped the monitoring_spec migration entirely and
then failed the seed on a missing column.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
2026-07-28 14:03:30 +02:00
parent 1dca2cfd7a
commit d7b526a112
11 changed files with 113 additions and 25 deletions

View File

@@ -13,15 +13,15 @@ import (
// SeedResult holds counts from a seed ingest operation.
type SeedResult struct {
Lifecycles int
EntityTypes int
Lifecycles int
EntityTypes int
RelationshipTypes int
Entities int
Relationships int
RiskClasses int
ApprovalRules int
AutonomySettings int
Checks int
Entities int
Relationships int
RiskClasses int
ApprovalRules int
AutonomySettings int
Checks int
}
// IngestOntologySeed ingests seeds/ontology.yaml into the DB.
@@ -476,4 +476,3 @@ func keysOf(m map[string]map[string]any) []string {
}
return keys
}