oikos/docs/adr/0013-signal-triggers.md

Signal Trigger Architecture

Overview

When Nomos is asked "what are the thermals of hubris?", here is exactly what happens:

sequenceDiagram
    participant N as Nomos (Agent)
    participant A as Oikos API
    participant S as Scheduler (Docker)
    participant H as Hubris (Proxmox)
    participant T as TimescaleDB

    Note over S,H: Every 60s (autonomous loop)
    S->>H: SSH exec /opt/oikos/checks/cpu_check.sh
    H-->>S: {"health":"ok","metrics":{"cpu_pct":2.5,"cpu_temp":48}}
    S->>T: INSERT metric_samples (cpu_pct, cpu_temp)
    S->>T: UPSERT entity_status (health)
    alt unhealthy
        S->>T: UPSERT signal (dedup by target+kind)
    end

    Note over N,T: User asks "what are the thermals of hubris?"
    N->>A: MCP query_metrics(metric=["cpu_pct","cpu_temp"])
    A->>T: SELECT time_bucket(…) FROM metric_samples
    T-->>A: cpu_pct=15%, cpu_temp=48°C
    A-->>N: {avg, min, max} per bucket

Two Paths

Path A — Autonomous Collection (Scheduler)

1. Operator creates a check via REST API: POST /api/v1/checks
2. Scheduler loads enabled checks every 30s from check_defs table
3. For ssh-script checks, scheduler SSHs to target host and runs /opt/oikos/checks/<script>.sh
4. Script returns JSON with health, signalKind, evidence, and metrics
5. Metrics written to TimescaleDB metric_samples table every cycle (healthy or not)
6. If unhealthy: a signal is raised (deduplicated by target_entity_id + kind)
7. If healthy again: the signal is auto-resolved, entity_status health updated
8. All state changes emit SSE events for real-time UI updates

Path B — Query (Nomos via MCP)

1. Nomos calls query_metrics(metric=["cpu_pct","cpu_temp"]) MCP tool
2. API runs time-bucketed aggregation over metric_samples
3. Returns latest readings with avg/min/max per bucket
4. Nomos formats them and presents to the user

Check Kinds

Kind	Where it runs	Protocol	Example
ping	Scheduler container	ICMP (ping binary)	Reachability + latency
http	Scheduler container	HTTP GET	Service endpoint health
tcp	Scheduler container	TCP dial	Port open check
disk	Scheduler container	unix.Statfs	Local disk usage + inodes
cert-expiry	Scheduler container	TLS dial	Certificate days remaining
ssh-script	Remote target via SSH	SSH exec + JSON	Any script in /opt/oikos/checks/

Available Check Scripts

All scripts live in /opt/oikos/checks/ on target hosts. They output JSON:

{"health":"healthy","metrics":{"cpu_pct":2.5,"cpu_temp":48.0}}

or on failure:

{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART failed for Samsung 990"}

Script	Metrics	Signal (on failure)
cpu_check.sh	cpu_pct, cpu_temp	threshold-based
memory_check.sh	mem_pct	threshold-based
load_check.sh	load1, cores	threshold-based
swap_check.sh	swap_pct	threshold-based
disk_usage_check.sh	disk_*_pct, inode_*_pct	threshold-based
disk_smart_check.sh	—	disk-smart-fail
updates_check.sh	security_updates, reboot_required	threshold-based
zfs_check.sh	—	zfs-degraded, zfs-scrub-overdue
process_check.sh	—	<service-name>
uptime_check.sh	uptime_seconds	threshold-based
oom_check.sh	oom_count	oom-kills
journal_check.sh	journal_errors	journal-errors
time_check.sh	clock_drift_s	time-drift
fd_check.sh	fd_pct	threshold-based
docker_health_check.sh	docker_unhealthy, docker_total	docker-unhealthy
caddy_error_rate.sh	caddy_5xx_rate, caddy_requests, caddy_5xx	caddy-errors
backup_freshness.sh	—	backup-stale

Script Deployment

sequenceDiagram
    participant R as Git Repo
    participant T as Sync Timer (5min)
    participant H as Target Host

    Note over R,T: Operator pushes scripts
    R->>T: git pull (homelab-context)
    T->>T: tools/post-pull.sh
    T->>T: → tools/setup-checks.sh
    T->>T: → checks/install.sh
    T->>H: cp *.sh → /opt/oikos/checks/

Defining a Check

curl -X POST http://oikos:8090/api/v1/checks \
  -H 'Content-Type: application/json' \
  -d '{
    "kind": "ssh-script",
    "target": "host:hubris",
    "config": {
      "host": "192.168.8.77",
      "script": "cpu_check.sh",
      "thresholds": {
        "cpu_pct": {"warn": 90, "crit": 95},
        "cpu_temp": {"crit": 85}
      }
    },
    "interval_s": 60
  }'

Signal Lifecycle

stateDiagram-v2
    [*] --> raised
    raised --> acknowledged
    raised --> muted: mute_until set
    raised --> resolved: condition cleared

    acknowledged --> acting: classification exists
    acknowledged --> muted
    acknowledged --> resolved

    acting --> resolved: verification passed
    acting --> raised: retry budget remaining
    acting --> failed

    failed --> acknowledged: operator retry

    muted --> raised: mute_until expired

    resolved --> [*]

Signals deduplicate: one open signal per (target_entity_id, kind). Repeated failures increment occurrence_count instead of creating duplicates.

Threshold Evaluation

Each check config can define per-metric thresholds in the config JSONB:

{
  "thresholds": {
    "cpu_temp": {"crit": 85},
    "cpu_pct": {"warn": 90, "crit": 95}
  }
}

Severity mapping:

• metric >= crit → severity = critical
• metric >= warn → severity = warning
• health == "down" with no thresholds → severity = critical
• Otherwise → severity = warning

Data Flow (DB Tables)

flowchart TD
    CD[check_defs] -->|scheduler reads| EC[executeCheck]
    EC -->|healthy?| RS[resolve signal + upsert entity_status]
    EC -->|unhealthy?| US[UpsertSignal dedup by target+kind]
    EC -->|every cycle| IM[INSERT metric_samples]
    US --> S[signals]
    RS --> ES[entity_status]
    IM --> MS[(metric_samples)]
    MS --> R1H[metric_rollups_1h continuous aggregate]
    MS --> R1D[metric_rollups_1d continuous aggregate]

Prerequisites for SSH Checks

1. Key: SSH private key mounted at /etc/oikos/ssh_key in the scheduler container
2. User: OIKOS_SSH_USER=root (or set "user" in check config)
3. Scripts: Deployed on target host at /opt/oikos/checks/
4. Network: Scheduler container must reach target host (bridge → LAN works)
5. Container: Scheduler needs openssh-client (alpine base) + CAP_NET_RAW (for ping)