Monitoring coverage was 3 of 89 active entities. Three bugs, each hidden by discarded errors in checkdefaults: - writeCheck generated a fresh uuid, inserted the check entity ON CONFLICT (slug) DO NOTHING, then wrote a check_defs row referencing it. On any re-seed the slug already existed, the entity insert no-oped, and the FK violated — aborting the ingest transaction and surfacing as an unrelated failure several entities later. Re-seeding has been broken since; prod's coverage was frozen at its first successful seed. This is what TestSeedIngestIdempotentAndNoDuplicateEdges had been reporting. - shortSlug truncated to the last 8 chars, so all 21 ingress routes collapsed to ".network" and overwrote each other; service:jellyfin collided with lxc:jellyfin. - The ssh-script checker never read the `args` config checkdefaults wrote, so process_check.sh always ran without its unit name and returned "unknown". Coverage is now 75/89. Monitoring is declared per entity type in seeds/ontology.yaml and resolved through the is-a hierarchy, so a type can say it warrants nothing (site, lan, mesh, cluster) and never be reported as a gap. coverageSweep raises an `unmonitored` signal only where a type declares monitoring it lacks — 8 real gaps, no false positives. Also: - entity_types.attribute_schema was never ingested: the seed loader read "attribute_schema" but the YAML says "attributes", so all 60 types stored JSON null. - ListExecutions ignored its declared target/action/correlation_id filters and paginated on a non-unique target slug, dropping and repeating rows. - started_at was captured but only written at terminal state, so a running execution reported NULL for its whole life. The three MCP auto-run copies wrote no timing at all; they are now one autoRun helper. - SSH output was buffered to completion and discarded entirely on timeout. Both sshExec copies now stream through a shared execlog sink into execution_logs, and keep partial output when a command is cancelled. - executions.correlation_id was a random per-execution uuid that correlated nothing; it is now the chat session id, which is what lets the chat tail live output. - reversible_low had no auto-run branch despite policy declaring it unattended. Since computeCommandRisk never returns it, the class only arises when an agent declares it over a read_only command — so gating it penalised candor without adding safety. - backup-target gains a backup-freshness checker (portable find -mmin, since the first target is on macOS), resolving its host by walking backs-up-to backwards. The pre-deploy pg_dump is now a tracked backup target. UI: an Executions section on entity detail with live output tailing, and streamed output under a running `run` call in the chat timeline. Migrations 022-024. Ops.svelte and context.ts exclude execution.output from their refetch triggers, which would otherwise fire once a second per command. Co-Authored-By: Claude <noreply@anthropic.com>
117 lines
3.8 KiB
Go
117 lines
3.8 KiB
Go
package scheduler
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"fmt"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
|
)
|
|
|
|
// checkBackupFreshness reports whether a backup target has a recent artifact.
|
|
//
|
|
// The ontology has carried a `backup-target` type and a `backs-up-to` edge
|
|
// since the first seed, but nothing ever verified that a backup actually
|
|
// happened — a silent backup failure looked exactly like a working one. This
|
|
// makes staleness a Signal like any other, so it flows through the existing
|
|
// dedup, auto-resolve and notifier path rather than needing its own machinery.
|
|
//
|
|
// Config: {"path": "/opt/oikos/backups", "max_age_s": 86400, "host": …}
|
|
//
|
|
// Deliberately uses `find -mmin` rather than `-printf '%T@'` or `stat`:
|
|
// -printf is GNU-only and stat's format flag differs between GNU (-c) and BSD
|
|
// (-f). The first real target for this check is the pre-deploy pg_dump on the
|
|
// mac-mini, which is macOS — so a GNU-only probe would have silently reported
|
|
// "unknown" on the one target that motivated the check.
|
|
func checkBackupFreshness(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
|
cfg := struct {
|
|
Path string `json:"path"`
|
|
MaxAgeS int `json:"max_age_s"`
|
|
Host string `json:"host"`
|
|
User string `json:"user"`
|
|
Port int `json:"port"`
|
|
}{
|
|
MaxAgeS: 86400, // a daily backup that has not run in 24h is stale
|
|
}
|
|
if len(cd.Config) > 0 {
|
|
_ = json.Unmarshal(cd.Config, &cfg)
|
|
}
|
|
if cfg.Path == "" || cfg.Host == "" {
|
|
return checkResult{health: "unknown", signalKind: "backup-misconfigured",
|
|
evidence: "backup check needs both a path and a host"}
|
|
}
|
|
if cfg.Port == 0 {
|
|
cfg.Port = 22
|
|
}
|
|
if cfg.User == "" {
|
|
cfg.User = sshUser
|
|
}
|
|
if cfg.MaxAgeS <= 0 {
|
|
cfg.MaxAgeS = 86400
|
|
}
|
|
|
|
timeout := time.Duration(cd.TimeoutS) * time.Second
|
|
if timeout <= 0 {
|
|
timeout = 30 * time.Second
|
|
}
|
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
|
defer cancel()
|
|
|
|
minutes := cfg.MaxAgeS / 60
|
|
if minutes < 1 {
|
|
minutes = 1
|
|
}
|
|
quoted := shellSingleQuote(cfg.Path)
|
|
|
|
// Two questions in one round trip: is there anything at all, and is any of
|
|
// it recent? "no backups ever" and "backups stopped" are different
|
|
// failures and deserve different severities.
|
|
cmd := fmt.Sprintf(
|
|
`if [ ! -d %s ]; then echo missing; else `+
|
|
`f=$(find %s -type f -mmin -%d 2>/dev/null | head -1); `+
|
|
`a=$(find %s -type f 2>/dev/null | head -1); `+
|
|
`if [ -n "$f" ]; then echo fresh; elif [ -n "$a" ]; then echo stale; else echo empty; fi; fi`,
|
|
quoted, quoted, minutes, quoted)
|
|
|
|
out, err := sshExec(ctx, cfg.Host, strconv.Itoa(cfg.Port), cfg.User, cmd, timeout)
|
|
if err != nil {
|
|
return checkResult{
|
|
health: "unknown", signalKind: "backup-unreachable",
|
|
evidence: fmt.Sprintf("ssh %s: %v", cfg.Host, err),
|
|
err: err,
|
|
}
|
|
}
|
|
|
|
age := time.Duration(cfg.MaxAgeS) * time.Second
|
|
switch strings.TrimSpace(string(out)) {
|
|
case "fresh":
|
|
return checkResult{health: "healthy"}
|
|
case "stale":
|
|
return checkResult{
|
|
health: "degraded", signalKind: "backup-stale",
|
|
evidence: fmt.Sprintf("no backup in %s under %s on %s", age, cfg.Path, cfg.Host),
|
|
}
|
|
case "empty":
|
|
return checkResult{
|
|
health: "down", signalKind: "backup-missing",
|
|
evidence: fmt.Sprintf("%s on %s exists but contains no files", cfg.Path, cfg.Host),
|
|
}
|
|
case "missing":
|
|
return checkResult{
|
|
health: "down", signalKind: "backup-missing",
|
|
evidence: fmt.Sprintf("backup directory %s does not exist on %s", cfg.Path, cfg.Host),
|
|
}
|
|
}
|
|
return checkResult{health: "unknown", signalKind: "backup-unreachable",
|
|
evidence: fmt.Sprintf("unexpected probe output: %q", strings.TrimSpace(string(out)))}
|
|
}
|
|
|
|
// shellSingleQuote makes a path safe to embed in the remote sh command. Paths
|
|
// come from check_defs config, which an operator or the agent can write.
|
|
func shellSingleQuote(s string) string {
|
|
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
|
}
|