dtoro
d7b526a112
ListEnabledCheckDefs selected interval_s but never filtered on it, so every enabled check ran on every 30s pass and the declared per-check intervals were decorative. Invisible at 17 enabled checks; at ~180 it would have meant ~126 SSH connections every 30s (~363k/day) and `apt update` on every machine every 30 seconds — 14,400 mirror hits a day to answer a question that changes daily. - check_defs.last_run_at (migration 026) + a due-ness predicate in the query. A column rather than scheduler memory because this control plane restarts on every deploy, and an in-memory map would re-fire every check on each restart. - runCheck stamps last_run_at before processing the result, so a permanently failing check backs off to its interval instead of re-running every pass. - updates and backup-freshness drop to daily. Both answer questions whose answers change about once a day; 60s was just the shared ssh-script default. - last_run_at is seeded to a random offset within the interval so checks created by the same seed do not stay in lockstep — otherwise ~165 probes land in the same instant each minute instead of spread across it. Deliberately not in the upsert's DO UPDATE: a re-seed must not re-herd them. Steady state becomes ~180k SSH/day (down from ~363k) and 5 apt runs/day (down from 14,400), with each 60s check landing at its own point in the minute. Also renumbers 022→023, 023→024, 024→025: origin/main added its own 022_knowledge_revisions, and prod has already applied version 22. Left colliding, prod would have skipped the monitoring_spec migration entirely and then failed the seed on a missing column. Co-Authored-By: Claude <noreply@anthropic.com>
44 lines
1.9 KiB
SQL
44 lines
1.9 KiB
SQL
-- 025_execution_logs.up.sql
|
|
-- Incremental command output for executions.
|
|
--
|
|
-- Until now `executions.result` was a single JSONB blob written once, at the
|
|
-- terminal state: {"output": "...everything..."}. Two consequences:
|
|
--
|
|
-- 1. Nothing could be seen while a command ran. A ten-minute apt upgrade
|
|
-- showed an empty row until it finished.
|
|
-- 2. On the sshExecTimeout path the output was discarded entirely — the
|
|
-- code returned "" — so the executions most worth inspecting (the ones
|
|
-- that hung) were the ones that left no trace at all.
|
|
--
|
|
-- Chunks land here as they arrive. `executions.result` still gets the full
|
|
-- output at the end, so existing readers keep working unchanged and this
|
|
-- table is purely additive.
|
|
|
|
CREATE TABLE IF NOT EXISTS execution_logs (
|
|
execution_id UUID NOT NULL,
|
|
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
|
-- Monotonic per execution. ts alone cannot order chunks: several arrive
|
|
-- within the same microsecond on a fast command.
|
|
seq INTEGER NOT NULL,
|
|
-- 'stdout' or 'stderr'. Both are also concatenated into the combined
|
|
-- output, matching what CombinedOutput used to return.
|
|
stream TEXT NOT NULL,
|
|
chunk TEXT NOT NULL,
|
|
PRIMARY KEY (execution_id, seq, ts)
|
|
);
|
|
|
|
SELECT create_hypertable('execution_logs', 'ts',
|
|
chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE);
|
|
|
|
-- The only query that matters: replay one execution's output in order.
|
|
CREATE INDEX IF NOT EXISTS idx_execution_logs_exec_seq
|
|
ON execution_logs (execution_id, seq);
|
|
|
|
-- Matches the events table's 90 days. Command output is bulkier than events,
|
|
-- but keeping it exactly as long as the event stream that references it avoids
|
|
-- dangling 'execution.output' events pointing at rows that no longer exist.
|
|
DO $$ BEGIN
|
|
PERFORM add_retention_policy('execution_logs', INTERVAL '90 days');
|
|
EXCEPTION WHEN OTHERS THEN NULL;
|
|
END $$;
|