From 81acadec1dcd21bf650cafe399e3eaf04b55f94a Mon Sep 17 00:00:00 2001
From: dtoro <d.toro.v@pm.me>
Date: Wed, 8 Jul 2026 21:37:31 +0200
Subject: [PATCH] docs: signal trigger architecture sequence diagram + full
 explanation
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add docs/signal-triggers.md covering:
- End-to-end sequence diagram (Nomos → API → Scheduler → target host)
- Two paths: autonomous collection (scheduler) + query (MCP)
- All 6 check kinds and 17 available scripts
- Script deployment flow via sync timer
- Signal lifecycle, threshold evaluation, data flow through DB tables
- Prerequisites for SSH checks in Docker
---
 docs/signal-triggers.md | 206 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 206 insertions(+)
 create mode 100644 docs/signal-triggers.md

diff --git a/docs/signal-triggers.md b/docs/signal-triggers.md
new file mode 100644
index 0000000..63fad64
--- /dev/null
+++ b/docs/signal-triggers.md
@@ -0,0 +1,206 @@
+# Signal Trigger Architecture
+
+## Overview
+
+When Nomos is asked "what are the thermals of hubris?", here is exactly what happens:
+
+```
+┌──────────┐     ┌──────────┐     ┌──────────┐     ┌──────────┐
+│  Nomos   │     │  Oikos   │     │Scheduler │     │  Hubris  │
+│ (Agent)  │     │   API    │     │ (Docker) │     │(Proxmox) │
+└────┬─────┘     └────┬─────┘     └────┬─────┘     └────┬─────┘
+     │                 │               │                  │
+     │ query_metrics   │               │                  │
+     │────────────────>│               │                  │
+     │                 │               │                  │
+     │  ← cpu_pct=2.5  │ SELECT FROM   │                  │
+     │    cpu_temp=48  │ metric_samples│                  │
+     │<────────────────│               │                  │
+     │                 │               │                  │
+     │                 │               │                  │
+     ══════ Every 60s (autonomous loop) ══════            │
+     │                 │               │                  │
+     │                 │               │ SSH exec         │
+     │                 │               │─────────────────>│
+     │                 │               │ /opt/oikos/      │
+     │                 │               │ checks/          │
+     │                 │               │ cpu_check.sh     │
+     │                 │               │                  │
+     │                 │               │ {"health":"ok",  │
+     │                 │               │  "metrics":      │
+     │                 │               │  {"cpu_pct":2.5, │
+     │                 │               │   "cpu_temp":48}}│
+     │                 │               │<─────────────────│
+     │                 │               │                  │
+     │                 │               │ INSERT           │
+     │                 │               │ metric_samples   │
+     │                 │               │                  │
+     │                 │               │ UPSERT signal    │
+     │                 │               │ (dedup)         │
+     │                 │               │                  │
+```
+
+## Two Paths
+
+### Path A — Autonomous Collection (Scheduler)
+
+1. Operator creates a check via REST API: `POST /api/v1/checks`
+2. Scheduler loads enabled checks every 30s from `check_defs` table
+3. For `ssh-script` checks, scheduler SSHs to target host and runs `/opt/oikos/checks/<script>.sh`
+4. Script returns JSON with `health`, `signalKind`, `evidence`, and `metrics`
+5. Metrics written to TimescaleDB `metric_samples` table **every cycle** (healthy or not)
+6. If unhealthy: a signal is raised (deduplicated by target_entity_id + kind)
+7. If healthy again: the signal is auto-resolved, entity_status health updated
+8. All state changes emit SSE events for real-time UI updates
+
+### Path B — Query (Nomos via MCP)
+
+1. Nomos calls `query_metrics(metric=["cpu_pct","cpu_temp"])` MCP tool
+2. API runs time-bucketed aggregation over `metric_samples`
+3. Returns latest readings with avg/min/max per bucket
+4. Nomos formats them and presents to the user
+
+## Check Kinds
+
+| Kind | Where it runs | Protocol | Example |
+|------|--------------|----------|---------|
+| `ping` | Scheduler container | ICMP (`ping` binary) | Reachability + latency |
+| `http` | Scheduler container | HTTP GET | Service endpoint health |
+| `tcp` | Scheduler container | TCP dial | Port open check |
+| `disk` | Scheduler container | `unix.Statfs` | Local disk usage + inodes |
+| `cert-expiry` | Scheduler container | TLS dial | Certificate days remaining |
+| `ssh-script` | Remote target via SSH | SSH exec + JSON | Any script in `/opt/oikos/checks/` |
+
+## Available Check Scripts
+
+All scripts live in `/opt/oikos/checks/` on target hosts. They output JSON:
+
+```json
+{"health":"healthy","metrics":{"cpu_pct":2.5,"cpu_temp":48.0}}
+```
+or on failure:
+```json
+{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART failed for Samsung 990"}
+```
+
+| Script | Metrics | Signal (on failure) |
+|--------|---------|---------------------|
+| `cpu_check.sh` | `cpu_pct`, `cpu_temp` | threshold-based |
+| `memory_check.sh` | `mem_pct` | threshold-based |
+| `load_check.sh` | `load1`, `cores` | threshold-based |
+| `swap_check.sh` | `swap_pct` | threshold-based |
+| `disk_usage_check.sh` | `disk_*_pct`, `inode_*_pct` | threshold-based |
+| `disk_smart_check.sh` | — | `disk-smart-fail` |
+| `updates_check.sh` | `security_updates`, `reboot_required` | threshold-based |
+| `zfs_check.sh` | — | `zfs-degraded`, `zfs-scrub-overdue` |
+| `process_check.sh` | — | `<service-name>` |
+| `uptime_check.sh` | `uptime_seconds` | threshold-based |
+| `oom_check.sh` | `oom_count` | `oom-kills` |
+| `journal_check.sh` | `journal_errors` | `journal-errors` |
+| `time_check.sh` | `clock_drift_s` | `time-drift` |
+| `fd_check.sh` | `fd_pct` | threshold-based |
+| `docker_health_check.sh` | `docker_unhealthy`, `docker_total` | `docker-unhealthy` |
+| `caddy_error_rate.sh` | `caddy_5xx_rate`, `caddy_requests`, `caddy_5xx` | `caddy-errors` |
+| `backup_freshness.sh` | — | `backup-stale` |
+
+## Script Deployment
+
+```
+  Git Push                    Sync Timer (5min)             Target Host
+ ┌────────┐                 ┌────────────────┐            ┌──────────┐
+ │ git push│                │ git pull       │            │          │
+ │ origin  │───────────────>│ homelab-context│            │          │
+ │  main   │                │                │            │          │
+ └────────┘                 │ post-pull.sh   │            │          │
+                            │  → tools/      │            │          │
+                            │    setup-      │            │          │
+                            │    checks.sh   │            │          │
+                            │    → checks/   │            │          │
+                            │      install.sh│            │          │
+                            │                │──cp *.sh ─>│ /opt/    │
+                            │                │            │ oikos/   │
+                            │                │            │ checks/  │
+                            └────────────────┘            └──────────┘
+```
+
+## Defining a Check
+
+```bash
+curl -X POST http://oikos:8090/api/v1/checks \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "kind": "ssh-script",
+    "target": "host:hubris",
+    "config": {
+      "host": "192.168.8.77",
+      "script": "cpu_check.sh",
+      "thresholds": {
+        "cpu_pct": {"warn": 90, "crit": 95},
+        "cpu_temp": {"crit": 85}
+      }
+    },
+    "interval_s": 60
+  }'
+```
+
+## Signal Lifecycle
+
+```
+  raised ──> acknowledged ──> acting ──> resolved
+     │            │             │
+     ├── muted    ├── muted     ├── raised (retry)
+     │            │             │
+     └── resolved └── resolved  └── failed
+```
+
+Signals deduplicate: **one open signal per (target_entity_id, kind)**.
+Repeated failures increment `occurrence_count` instead of creating duplicates.
+
+## Threshold Evaluation
+
+Each check config can define per-metric thresholds in the `config` JSONB:
+
+```json
+{
+  "thresholds": {
+    "cpu_temp": {"crit": 85},
+    "cpu_pct": {"warn": 90, "crit": 95}
+  }
+}
+```
+
+Severity mapping:
+- metric >= `crit` → severity = `critical`
+- metric >= `warn` → severity = `warning`
+- `health == "down"` with no thresholds → severity = `critical`
+- Otherwise → severity = `warning`
+
+## Data Flow (DB Tables)
+
+```
+  check_defs ──(scheduler reads)──> executeCheck()
+       │                                  │
+       │                                  ├── healthy? → resolve signal, upsert entity_status
+       │                                  │
+       │                                  └── unhealthy? → UpsertSignal(), insert metric_samples
+       │
+       ▼
+  signals ◄──── UpsertSignal (dedup by target+kind)
+       │
+       ▼
+  entity_status ◄── upsert (health, last_check_at)
+
+  metric_samples ◄── INSERT (every cycle, healthy or not)
+       │
+       ▼
+  metric_rollups_1h ◄── continuous aggregate
+  metric_rollups_1d ◄── continuous aggregate
+```
+
+## Prerequisites for SSH Checks
+
+1. **Key**: SSH private key mounted at `/etc/oikos/ssh_key` in the scheduler container
+2. **User**: `OIKOS_SSH_USER=root` (or set `"user"` in check config)
+3. **Scripts**: Deployed on target host at `/opt/oikos/checks/`
+4. **Network**: Scheduler container must reach target host (bridge → LAN works)
+5. **Container**: Scheduler needs `openssh-client` (alpine base) + `CAP_NET_RAW` (for ping)