# Signal Trigger Architecture

## Overview

When Nomos is asked "what are the thermals of hubris?", here is exactly what happens:

```
┌──────────┐     ┌──────────┐     ┌──────────┐     ┌──────────┐
│  Nomos   │     │  Oikos   │     │Scheduler │     │  Hubris  │
│ (Agent)  │     │   API    │     │ (Docker) │     │(Proxmox) │
└────┬─────┘     └────┬─────┘     └────┬─────┘     └────┬─────┘
     │                 │               │                  │
     │ query_metrics   │               │                  │
     │────────────────>│               │                  │
     │                 │               │                  │
     │  ← cpu_pct=2.5  │ SELECT FROM   │                  │
     │    cpu_temp=48  │ metric_samples│                  │
     │<────────────────│               │                  │
     │                 │               │                  │
     │                 │               │                  │
     ══════ Every 60s (autonomous loop) ══════            │
     │                 │               │                  │
     │                 │               │ SSH exec         │
     │                 │               │─────────────────>│
     │                 │               │ /opt/oikos/      │
     │                 │               │ checks/          │
     │                 │               │ cpu_check.sh     │
     │                 │               │                  │
     │                 │               │ {"health":"ok",  │
     │                 │               │  "metrics":      │
     │                 │               │  {"cpu_pct":2.5, │
     │                 │               │   "cpu_temp":48}}│
     │                 │               │<─────────────────│
     │                 │               │                  │
     │                 │               │ INSERT           │
     │                 │               │ metric_samples   │
     │                 │               │                  │
     │                 │               │ UPSERT signal    │
     │                 │               │ (dedup)         │
     │                 │               │                  │
```

## Two Paths

### Path A — Autonomous Collection (Scheduler)

1. Operator creates a check via REST API: `POST /api/v1/checks`
2. Scheduler loads enabled checks every 30s from `check_defs` table
3. For `ssh-script` checks, scheduler SSHs to target host and runs `/opt/oikos/checks/<script>.sh`
4. Script returns JSON with `health`, `signalKind`, `evidence`, and `metrics`
5. Metrics written to TimescaleDB `metric_samples` table **every cycle** (healthy or not)
6. If unhealthy: a signal is raised (deduplicated by target_entity_id + kind)
7. If healthy again: the signal is auto-resolved, entity_status health updated
8. All state changes emit SSE events for real-time UI updates

### Path B — Query (Nomos via MCP)

1. Nomos calls `query_metrics(metric=["cpu_pct","cpu_temp"])` MCP tool
2. API runs time-bucketed aggregation over `metric_samples`
3. Returns latest readings with avg/min/max per bucket
4. Nomos formats them and presents to the user

## Check Kinds

| Kind | Where it runs | Protocol | Example |
|------|--------------|----------|---------|
| `ping` | Scheduler container | ICMP (`ping` binary) | Reachability + latency |
| `http` | Scheduler container | HTTP GET | Service endpoint health |
| `tcp` | Scheduler container | TCP dial | Port open check |
| `disk` | Scheduler container | `unix.Statfs` | Local disk usage + inodes |
| `cert-expiry` | Scheduler container | TLS dial | Certificate days remaining |
| `ssh-script` | Remote target via SSH | SSH exec + JSON | Any script in `/opt/oikos/checks/` |

## Available Check Scripts

All scripts live in `/opt/oikos/checks/` on target hosts. They output JSON:

```json
{"health":"healthy","metrics":{"cpu_pct":2.5,"cpu_temp":48.0}}
```
or on failure:
```json
{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART failed for Samsung 990"}
```

| Script | Metrics | Signal (on failure) |
|--------|---------|---------------------|
| `cpu_check.sh` | `cpu_pct`, `cpu_temp` | threshold-based |
| `memory_check.sh` | `mem_pct` | threshold-based |
| `load_check.sh` | `load1`, `cores` | threshold-based |
| `swap_check.sh` | `swap_pct` | threshold-based |
| `disk_usage_check.sh` | `disk_*_pct`, `inode_*_pct` | threshold-based |
| `disk_smart_check.sh` | — | `disk-smart-fail` |
| `updates_check.sh` | `security_updates`, `reboot_required` | threshold-based |
| `zfs_check.sh` | — | `zfs-degraded`, `zfs-scrub-overdue` |
| `process_check.sh` | — | `<service-name>` |
| `uptime_check.sh` | `uptime_seconds` | threshold-based |
| `oom_check.sh` | `oom_count` | `oom-kills` |
| `journal_check.sh` | `journal_errors` | `journal-errors` |
| `time_check.sh` | `clock_drift_s` | `time-drift` |
| `fd_check.sh` | `fd_pct` | threshold-based |
| `docker_health_check.sh` | `docker_unhealthy`, `docker_total` | `docker-unhealthy` |
| `caddy_error_rate.sh` | `caddy_5xx_rate`, `caddy_requests`, `caddy_5xx` | `caddy-errors` |
| `backup_freshness.sh` | — | `backup-stale` |

## Script Deployment

```
  Git Push                    Sync Timer (5min)             Target Host
 ┌────────┐                 ┌────────────────┐            ┌──────────┐
 │ git push│                │ git pull       │            │          │
 │ origin  │───────────────>│ homelab-context│            │          │
 │  main   │                │                │            │          │
 └────────┘                 │ post-pull.sh   │            │          │
                            │  → tools/      │            │          │
                            │    setup-      │            │          │
                            │    checks.sh   │            │          │
                            │    → checks/   │            │          │
                            │      install.sh│            │          │
                            │                │──cp *.sh ─>│ /opt/    │
                            │                │            │ oikos/   │
                            │                │            │ checks/  │
                            └────────────────┘            └──────────┘
```

## Defining a Check

```bash
curl -X POST http://oikos:8090/api/v1/checks \
  -H 'Content-Type: application/json' \
  -d '{
    "kind": "ssh-script",
    "target": "host:hubris",
    "config": {
      "host": "192.168.8.77",
      "script": "cpu_check.sh",
      "thresholds": {
        "cpu_pct": {"warn": 90, "crit": 95},
        "cpu_temp": {"crit": 85}
      }
    },
    "interval_s": 60
  }'
```

## Signal Lifecycle

```
  raised ──> acknowledged ──> acting ──> resolved
     │            │             │
     ├── muted    ├── muted     ├── raised (retry)
     │            │             │
     └── resolved └── resolved  └── failed
```

Signals deduplicate: **one open signal per (target_entity_id, kind)**.
Repeated failures increment `occurrence_count` instead of creating duplicates.

## Threshold Evaluation

Each check config can define per-metric thresholds in the `config` JSONB:

```json
{
  "thresholds": {
    "cpu_temp": {"crit": 85},
    "cpu_pct": {"warn": 90, "crit": 95}
  }
}
```

Severity mapping:
- metric >= `crit` → severity = `critical`
- metric >= `warn` → severity = `warning`
- `health == "down"` with no thresholds → severity = `critical`
- Otherwise → severity = `warning`

## Data Flow (DB Tables)

```
  check_defs ──(scheduler reads)──> executeCheck()
       │                                  │
       │                                  ├── healthy? → resolve signal, upsert entity_status
       │                                  │
       │                                  └── unhealthy? → UpsertSignal(), insert metric_samples
       │
       ▼
  signals ◄──── UpsertSignal (dedup by target+kind)
       │
       ▼
  entity_status ◄── upsert (health, last_check_at)

  metric_samples ◄── INSERT (every cycle, healthy or not)
       │
       ▼
  metric_rollups_1h ◄── continuous aggregate
  metric_rollups_1d ◄── continuous aggregate
```

## Prerequisites for SSH Checks

1. **Key**: SSH private key mounted at `/etc/oikos/ssh_key` in the scheduler container
2. **User**: `OIKOS_SSH_USER=root` (or set `"user"` in check config)
3. **Scripts**: Deployed on target host at `/opt/oikos/checks/`
4. **Network**: Scheduler container must reach target host (bridge → LAN works)
5. **Container**: Scheduler needs `openssh-client` (alpine base) + `CAP_NET_RAW` (for ping)