oikos/docs/adr/0013-signal-triggers.md

# Signal Trigger Architecture

## Overview

When Nomos is asked "what are the thermals of hubris?", here is exactly what happens:

```mermaid
sequenceDiagram
    participant N as Nomos (Agent)
    participant A as Oikos API
    participant S as Scheduler (Docker)
    participant H as Hubris (Proxmox)
    participant T as TimescaleDB

    Note over S,H: Every 60s (autonomous loop)
    S->>H: SSH exec /opt/oikos/checks/cpu_check.sh
    H-->>S: {"health":"ok","metrics":{"cpu_pct":2.5,"cpu_temp":48}}
    S->>T: INSERT metric_samples (cpu_pct, cpu_temp)
    S->>T: UPSERT entity_status (health)
    alt unhealthy
        S->>T: UPSERT signal (dedup by target+kind)
    end

    Note over N,T: User asks "what are the thermals of hubris?"
    N->>A: MCP query_metrics(metric=["cpu_pct","cpu_temp"])
    A->>T: SELECT time_bucket(…) FROM metric_samples
    T-->>A: cpu_pct=15%, cpu_temp=48°C
    A-->>N: {avg, min, max} per bucket
```

## Two Paths

### Path A — Autonomous Collection (Scheduler)

1. Operator creates a check via REST API: `POST /api/v1/checks`
2. Scheduler loads enabled checks every 30s from `check_defs` table
3. For `ssh-script` checks, scheduler SSHs to target host and runs `/opt/oikos/checks/<script>.sh`
4. Script returns JSON with `health`, `signalKind`, `evidence`, and `metrics`
5. Metrics written to TimescaleDB `metric_samples` table **every cycle** (healthy or not)
6. If unhealthy: a signal is raised (deduplicated by target_entity_id + kind)
7. If healthy again: the signal is auto-resolved, entity_status health updated
8. All state changes emit SSE events for real-time UI updates

### Path B — Query (Nomos via MCP)

1. Nomos calls `query_metrics(metric=["cpu_pct","cpu_temp"])` MCP tool
2. API runs time-bucketed aggregation over `metric_samples`
3. Returns latest readings with avg/min/max per bucket
4. Nomos formats them and presents to the user

## Check Kinds

| Kind | Where it runs | Protocol | Example |
|------|--------------|----------|---------|
| `ping` | Scheduler container | ICMP (`ping` binary) | Reachability + latency |
| `http` | Scheduler container | HTTP GET | Service endpoint health |
| `tcp` | Scheduler container | TCP dial | Port open check |
| `disk` | Scheduler container | `unix.Statfs` | Local disk usage + inodes |
| `cert-expiry` | Scheduler container | TLS dial | Certificate days remaining |
| `ssh-script` | Remote target via SSH | SSH exec + JSON | Any script in `/opt/oikos/checks/` |

## Available Check Scripts

All scripts live in `/opt/oikos/checks/` on target hosts. They output JSON:

```json
{"health":"healthy","metrics":{"cpu_pct":2.5,"cpu_temp":48.0}}
```
or on failure:
```json
{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART failed for Samsung 990"}
```

| Script | Metrics | Signal (on failure) |
|--------|---------|---------------------|
| `cpu_check.sh` | `cpu_pct`, `cpu_temp` | threshold-based |
| `memory_check.sh` | `mem_pct` | threshold-based |
| `load_check.sh` | `load1`, `cores` | threshold-based |
| `swap_check.sh` | `swap_pct` | threshold-based |
| `disk_usage_check.sh` | `disk_*_pct`, `inode_*_pct` | threshold-based |
| `disk_smart_check.sh` | — | `disk-smart-fail` |
| `updates_check.sh` | `security_updates`, `reboot_required` | threshold-based |
| `zfs_check.sh` | — | `zfs-degraded`, `zfs-scrub-overdue` |
| `process_check.sh` | — | `<service-name>` |
| `uptime_check.sh` | `uptime_seconds` | threshold-based |
| `oom_check.sh` | `oom_count` | `oom-kills` |
| `journal_check.sh` | `journal_errors` | `journal-errors` |
| `time_check.sh` | `clock_drift_s` | `time-drift` |
| `fd_check.sh` | `fd_pct` | threshold-based |
| `docker_health_check.sh` | `docker_unhealthy`, `docker_total` | `docker-unhealthy` |
| `caddy_error_rate.sh` | `caddy_5xx_rate`, `caddy_requests`, `caddy_5xx` | `caddy-errors` |
| `backup_freshness.sh` | — | `backup-stale` |

## Script Deployment

```mermaid
sequenceDiagram
    participant R as Git Repo
    participant T as Sync Timer (5min)
    participant H as Target Host

    Note over R,T: Operator pushes scripts
    R->>T: git pull (homelab-context)
    T->>T: tools/post-pull.sh
    T->>T: → tools/setup-checks.sh
    T->>T: → checks/install.sh
    T->>H: cp *.sh → /opt/oikos/checks/
```

## Defining a Check

```bash
curl -X POST http://oikos:8090/api/v1/checks \
  -H 'Content-Type: application/json' \
  -d '{
    "kind": "ssh-script",
    "target": "host:hubris",
    "config": {
      "host": "192.168.8.77",
      "script": "cpu_check.sh",
      "thresholds": {
        "cpu_pct": {"warn": 90, "crit": 95},
        "cpu_temp": {"crit": 85}
      }
    },
    "interval_s": 60
  }'
```

## Signal Lifecycle

```mermaid
stateDiagram-v2
    [*] --> raised
    raised --> acknowledged
    raised --> muted: mute_until set
    raised --> resolved: condition cleared

    acknowledged --> acting: classification exists
    acknowledged --> muted
    acknowledged --> resolved

    acting --> resolved: verification passed
    acting --> raised: retry budget remaining
    acting --> failed

    failed --> acknowledged: operator retry

    muted --> raised: mute_until expired

    resolved --> [*]
```

Signals deduplicate: **one open signal per (target_entity_id, kind)**.
Repeated failures increment `occurrence_count` instead of creating duplicates.

## Threshold Evaluation

Each check config can define per-metric thresholds in the `config` JSONB:

```json
{
  "thresholds": {
    "cpu_temp": {"crit": 85},
    "cpu_pct": {"warn": 90, "crit": 95}
  }
}
```

Severity mapping:
- metric >= `crit` → severity = `critical`
- metric >= `warn` → severity = `warning`
- `health == "down"` with no thresholds → severity = `critical`
- Otherwise → severity = `warning`

## Data Flow (DB Tables)

```mermaid
flowchart TD
    CD[check_defs] -->|scheduler reads| EC[executeCheck]
    EC -->|healthy?| RS[resolve signal + upsert entity_status]
    EC -->|unhealthy?| US[UpsertSignal dedup by target+kind]
    EC -->|every cycle| IM[INSERT metric_samples]
    US --> S[signals]
    RS --> ES[entity_status]
    IM --> MS[(metric_samples)]
    MS --> R1H[metric_rollups_1h continuous aggregate]
    MS --> R1D[metric_rollups_1d continuous aggregate]
```

## Prerequisites for SSH Checks

1. **Key**: SSH private key mounted at `/etc/oikos/ssh_key` in the scheduler container
2. **User**: `OIKOS_SSH_USER=root` (or set `"user"` in check config)
3. **Scripts**: Deployed on target host at `/opt/oikos/checks/`
4. **Network**: Scheduler container must reach target host (bridge → LAN works)
5. **Container**: Scheduler needs `openssh-client` (alpine base) + `CAP_NET_RAW` (for ping)