oikos/docs/signal-triggers.md

Signal Trigger Architecture

Overview

When Nomos is asked "what are the thermals of hubris?", here is exactly what happens:

┌──────────┐     ┌──────────┐     ┌──────────┐     ┌──────────┐
│  Nomos   │     │  Oikos   │     │Scheduler │     │  Hubris  │
│ (Agent)  │     │   API    │     │ (Docker) │     │(Proxmox) │
└────┬─────┘     └────┬─────┘     └────┬─────┘     └────┬─────┘
     │                 │               │                  │
     │ query_metrics   │               │                  │
     │────────────────>│               │                  │
     │                 │               │                  │
     │  ← cpu_pct=2.5  │ SELECT FROM   │                  │
     │    cpu_temp=48  │ metric_samples│                  │
     │<────────────────│               │                  │
     │                 │               │                  │
     │                 │               │                  │
     ══════ Every 60s (autonomous loop) ══════            │
     │                 │               │                  │
     │                 │               │ SSH exec         │
     │                 │               │─────────────────>│
     │                 │               │ /opt/oikos/      │
     │                 │               │ checks/          │
     │                 │               │ cpu_check.sh     │
     │                 │               │                  │
     │                 │               │ {"health":"ok",  │
     │                 │               │  "metrics":      │
     │                 │               │  {"cpu_pct":2.5, │
     │                 │               │   "cpu_temp":48}}│
     │                 │               │<─────────────────│
     │                 │               │                  │
     │                 │               │ INSERT           │
     │                 │               │ metric_samples   │
     │                 │               │                  │
     │                 │               │ UPSERT signal    │
     │                 │               │ (dedup)         │
     │                 │               │                  │

Two Paths

Path A — Autonomous Collection (Scheduler)

1. Operator creates a check via REST API: POST /api/v1/checks
2. Scheduler loads enabled checks every 30s from check_defs table
3. For ssh-script checks, scheduler SSHs to target host and runs /opt/oikos/checks/<script>.sh
4. Script returns JSON with health, signalKind, evidence, and metrics
5. Metrics written to TimescaleDB metric_samples table every cycle (healthy or not)
6. If unhealthy: a signal is raised (deduplicated by target_entity_id + kind)
7. If healthy again: the signal is auto-resolved, entity_status health updated
8. All state changes emit SSE events for real-time UI updates

Path B — Query (Nomos via MCP)

1. Nomos calls query_metrics(metric=["cpu_pct","cpu_temp"]) MCP tool
2. API runs time-bucketed aggregation over metric_samples
3. Returns latest readings with avg/min/max per bucket
4. Nomos formats them and presents to the user

Check Kinds

Kind	Where it runs	Protocol	Example
ping	Scheduler container	ICMP (ping binary)	Reachability + latency
http	Scheduler container	HTTP GET	Service endpoint health
tcp	Scheduler container	TCP dial	Port open check
disk	Scheduler container	unix.Statfs	Local disk usage + inodes
cert-expiry	Scheduler container	TLS dial	Certificate days remaining
ssh-script	Remote target via SSH	SSH exec + JSON	Any script in /opt/oikos/checks/

Available Check Scripts

All scripts live in /opt/oikos/checks/ on target hosts. They output JSON:

{"health":"healthy","metrics":{"cpu_pct":2.5,"cpu_temp":48.0}}

or on failure:

{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART failed for Samsung 990"}

Script	Metrics	Signal (on failure)
cpu_check.sh	cpu_pct, cpu_temp	threshold-based
memory_check.sh	mem_pct	threshold-based
load_check.sh	load1, cores	threshold-based
swap_check.sh	swap_pct	threshold-based
disk_usage_check.sh	disk_*_pct, inode_*_pct	threshold-based
disk_smart_check.sh	—	disk-smart-fail
updates_check.sh	security_updates, reboot_required	threshold-based
zfs_check.sh	—	zfs-degraded, zfs-scrub-overdue
process_check.sh	—	<service-name>
uptime_check.sh	uptime_seconds	threshold-based
oom_check.sh	oom_count	oom-kills
journal_check.sh	journal_errors	journal-errors
time_check.sh	clock_drift_s	time-drift
fd_check.sh	fd_pct	threshold-based
docker_health_check.sh	docker_unhealthy, docker_total	docker-unhealthy
caddy_error_rate.sh	caddy_5xx_rate, caddy_requests, caddy_5xx	caddy-errors
backup_freshness.sh	—	backup-stale

Script Deployment

  Git Push                    Sync Timer (5min)             Target Host
 ┌────────┐                 ┌────────────────┐            ┌──────────┐
 │ git push│                │ git pull       │            │          │
 │ origin  │───────────────>│ homelab-context│            │          │
 │  main   │                │                │            │          │
 └────────┘                 │ post-pull.sh   │            │          │
                            │  → tools/      │            │          │
                            │    setup-      │            │          │
                            │    checks.sh   │            │          │
                            │    → checks/   │            │          │
                            │      install.sh│            │          │
                            │                │──cp *.sh ─>│ /opt/    │
                            │                │            │ oikos/   │
                            │                │            │ checks/  │
                            └────────────────┘            └──────────┘

Defining a Check

curl -X POST http://oikos:8090/api/v1/checks \
  -H 'Content-Type: application/json' \
  -d '{
    "kind": "ssh-script",
    "target": "host:hubris",
    "config": {
      "host": "192.168.8.77",
      "script": "cpu_check.sh",
      "thresholds": {
        "cpu_pct": {"warn": 90, "crit": 95},
        "cpu_temp": {"crit": 85}
      }
    },
    "interval_s": 60
  }'

Signal Lifecycle

  raised ──> acknowledged ──> acting ──> resolved
     │            │             │
     ├── muted    ├── muted     ├── raised (retry)
     │            │             │
     └── resolved └── resolved  └── failed

Signals deduplicate: one open signal per (target_entity_id, kind). Repeated failures increment occurrence_count instead of creating duplicates.

Threshold Evaluation

Each check config can define per-metric thresholds in the config JSONB:

{
  "thresholds": {
    "cpu_temp": {"crit": 85},
    "cpu_pct": {"warn": 90, "crit": 95}
  }
}

Severity mapping:

• metric >= crit → severity = critical
• metric >= warn → severity = warning
• health == "down" with no thresholds → severity = critical
• Otherwise → severity = warning

Data Flow (DB Tables)

  check_defs ──(scheduler reads)──> executeCheck()
       │                                  │
       │                                  ├── healthy? → resolve signal, upsert entity_status
       │                                  │
       │                                  └── unhealthy? → UpsertSignal(), insert metric_samples
       │
       ▼
  signals ◄──── UpsertSignal (dedup by target+kind)
       │
       ▼
  entity_status ◄── upsert (health, last_check_at)

  metric_samples ◄── INSERT (every cycle, healthy or not)
       │
       ▼
  metric_rollups_1h ◄── continuous aggregate
  metric_rollups_1d ◄── continuous aggregate

Prerequisites for SSH Checks

1. Key: SSH private key mounted at /etc/oikos/ssh_key in the scheduler container
2. User: OIKOS_SSH_USER=root (or set "user" in check config)
3. Scripts: Deployed on target host at /opt/oikos/checks/
4. Network: Scheduler container must reach target host (bridge → LAN works)
5. Container: Scheduler needs openssh-client (alpine base) + CAP_NET_RAW (for ping)