checks: deploy scripts + define checks for strong and netbird-vps
Coverage: - host:hubris (192.168.8.77) ✓ cpu, memory, load, disk, ping, cert-expiry - host:strong (192.168.178.181) ✓ cpu, memory, load, disk, ping - host:netbird-vps (82.165.190.79) ✓ cpu, memory, load, disk, ping - ws:mac-mini ~ local disk/ping only (no SSH key on macOS host) - ws:republic-laptop ✗ offline / not reachable Move architecture doc to adr/signal-triggers.md
This commit is contained in:
@@ -1,206 +0,0 @@
|
||||
# Signal Trigger Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
When Nomos is asked "what are the thermals of hubris?", here is exactly what happens:
|
||||
|
||||
```
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Nomos │ │ Oikos │ │Scheduler │ │ Hubris │
|
||||
│ (Agent) │ │ API │ │ (Docker) │ │(Proxmox) │
|
||||
└────┬─────┘ └────┬─────┘ └────┬─────┘ └────┬─────┘
|
||||
│ │ │ │
|
||||
│ query_metrics │ │ │
|
||||
│────────────────>│ │ │
|
||||
│ │ │ │
|
||||
│ ← cpu_pct=2.5 │ SELECT FROM │ │
|
||||
│ cpu_temp=48 │ metric_samples│ │
|
||||
│<────────────────│ │ │
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
══════ Every 60s (autonomous loop) ══════ │
|
||||
│ │ │ │
|
||||
│ │ │ SSH exec │
|
||||
│ │ │─────────────────>│
|
||||
│ │ │ /opt/oikos/ │
|
||||
│ │ │ checks/ │
|
||||
│ │ │ cpu_check.sh │
|
||||
│ │ │ │
|
||||
│ │ │ {"health":"ok", │
|
||||
│ │ │ "metrics": │
|
||||
│ │ │ {"cpu_pct":2.5, │
|
||||
│ │ │ "cpu_temp":48}}│
|
||||
│ │ │<─────────────────│
|
||||
│ │ │ │
|
||||
│ │ │ INSERT │
|
||||
│ │ │ metric_samples │
|
||||
│ │ │ │
|
||||
│ │ │ UPSERT signal │
|
||||
│ │ │ (dedup) │
|
||||
│ │ │ │
|
||||
```
|
||||
|
||||
## Two Paths
|
||||
|
||||
### Path A — Autonomous Collection (Scheduler)
|
||||
|
||||
1. Operator creates a check via REST API: `POST /api/v1/checks`
|
||||
2. Scheduler loads enabled checks every 30s from `check_defs` table
|
||||
3. For `ssh-script` checks, scheduler SSHs to target host and runs `/opt/oikos/checks/<script>.sh`
|
||||
4. Script returns JSON with `health`, `signalKind`, `evidence`, and `metrics`
|
||||
5. Metrics written to TimescaleDB `metric_samples` table **every cycle** (healthy or not)
|
||||
6. If unhealthy: a signal is raised (deduplicated by target_entity_id + kind)
|
||||
7. If healthy again: the signal is auto-resolved, entity_status health updated
|
||||
8. All state changes emit SSE events for real-time UI updates
|
||||
|
||||
### Path B — Query (Nomos via MCP)
|
||||
|
||||
1. Nomos calls `query_metrics(metric=["cpu_pct","cpu_temp"])` MCP tool
|
||||
2. API runs time-bucketed aggregation over `metric_samples`
|
||||
3. Returns latest readings with avg/min/max per bucket
|
||||
4. Nomos formats them and presents to the user
|
||||
|
||||
## Check Kinds
|
||||
|
||||
| Kind | Where it runs | Protocol | Example |
|
||||
|------|--------------|----------|---------|
|
||||
| `ping` | Scheduler container | ICMP (`ping` binary) | Reachability + latency |
|
||||
| `http` | Scheduler container | HTTP GET | Service endpoint health |
|
||||
| `tcp` | Scheduler container | TCP dial | Port open check |
|
||||
| `disk` | Scheduler container | `unix.Statfs` | Local disk usage + inodes |
|
||||
| `cert-expiry` | Scheduler container | TLS dial | Certificate days remaining |
|
||||
| `ssh-script` | Remote target via SSH | SSH exec + JSON | Any script in `/opt/oikos/checks/` |
|
||||
|
||||
## Available Check Scripts
|
||||
|
||||
All scripts live in `/opt/oikos/checks/` on target hosts. They output JSON:
|
||||
|
||||
```json
|
||||
{"health":"healthy","metrics":{"cpu_pct":2.5,"cpu_temp":48.0}}
|
||||
```
|
||||
or on failure:
|
||||
```json
|
||||
{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART failed for Samsung 990"}
|
||||
```
|
||||
|
||||
| Script | Metrics | Signal (on failure) |
|
||||
|--------|---------|---------------------|
|
||||
| `cpu_check.sh` | `cpu_pct`, `cpu_temp` | threshold-based |
|
||||
| `memory_check.sh` | `mem_pct` | threshold-based |
|
||||
| `load_check.sh` | `load1`, `cores` | threshold-based |
|
||||
| `swap_check.sh` | `swap_pct` | threshold-based |
|
||||
| `disk_usage_check.sh` | `disk_*_pct`, `inode_*_pct` | threshold-based |
|
||||
| `disk_smart_check.sh` | — | `disk-smart-fail` |
|
||||
| `updates_check.sh` | `security_updates`, `reboot_required` | threshold-based |
|
||||
| `zfs_check.sh` | — | `zfs-degraded`, `zfs-scrub-overdue` |
|
||||
| `process_check.sh` | — | `<service-name>` |
|
||||
| `uptime_check.sh` | `uptime_seconds` | threshold-based |
|
||||
| `oom_check.sh` | `oom_count` | `oom-kills` |
|
||||
| `journal_check.sh` | `journal_errors` | `journal-errors` |
|
||||
| `time_check.sh` | `clock_drift_s` | `time-drift` |
|
||||
| `fd_check.sh` | `fd_pct` | threshold-based |
|
||||
| `docker_health_check.sh` | `docker_unhealthy`, `docker_total` | `docker-unhealthy` |
|
||||
| `caddy_error_rate.sh` | `caddy_5xx_rate`, `caddy_requests`, `caddy_5xx` | `caddy-errors` |
|
||||
| `backup_freshness.sh` | — | `backup-stale` |
|
||||
|
||||
## Script Deployment
|
||||
|
||||
```
|
||||
Git Push Sync Timer (5min) Target Host
|
||||
┌────────┐ ┌────────────────┐ ┌──────────┐
|
||||
│ git push│ │ git pull │ │ │
|
||||
│ origin │───────────────>│ homelab-context│ │ │
|
||||
│ main │ │ │ │ │
|
||||
└────────┘ │ post-pull.sh │ │ │
|
||||
│ → tools/ │ │ │
|
||||
│ setup- │ │ │
|
||||
│ checks.sh │ │ │
|
||||
│ → checks/ │ │ │
|
||||
│ install.sh│ │ │
|
||||
│ │──cp *.sh ─>│ /opt/ │
|
||||
│ │ │ oikos/ │
|
||||
│ │ │ checks/ │
|
||||
└────────────────┘ └──────────┘
|
||||
```
|
||||
|
||||
## Defining a Check
|
||||
|
||||
```bash
|
||||
curl -X POST http://oikos:8090/api/v1/checks \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{
|
||||
"kind": "ssh-script",
|
||||
"target": "host:hubris",
|
||||
"config": {
|
||||
"host": "192.168.8.77",
|
||||
"script": "cpu_check.sh",
|
||||
"thresholds": {
|
||||
"cpu_pct": {"warn": 90, "crit": 95},
|
||||
"cpu_temp": {"crit": 85}
|
||||
}
|
||||
},
|
||||
"interval_s": 60
|
||||
}'
|
||||
```
|
||||
|
||||
## Signal Lifecycle
|
||||
|
||||
```
|
||||
raised ──> acknowledged ──> acting ──> resolved
|
||||
│ │ │
|
||||
├── muted ├── muted ├── raised (retry)
|
||||
│ │ │
|
||||
└── resolved └── resolved └── failed
|
||||
```
|
||||
|
||||
Signals deduplicate: **one open signal per (target_entity_id, kind)**.
|
||||
Repeated failures increment `occurrence_count` instead of creating duplicates.
|
||||
|
||||
## Threshold Evaluation
|
||||
|
||||
Each check config can define per-metric thresholds in the `config` JSONB:
|
||||
|
||||
```json
|
||||
{
|
||||
"thresholds": {
|
||||
"cpu_temp": {"crit": 85},
|
||||
"cpu_pct": {"warn": 90, "crit": 95}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Severity mapping:
|
||||
- metric >= `crit` → severity = `critical`
|
||||
- metric >= `warn` → severity = `warning`
|
||||
- `health == "down"` with no thresholds → severity = `critical`
|
||||
- Otherwise → severity = `warning`
|
||||
|
||||
## Data Flow (DB Tables)
|
||||
|
||||
```
|
||||
check_defs ──(scheduler reads)──> executeCheck()
|
||||
│ │
|
||||
│ ├── healthy? → resolve signal, upsert entity_status
|
||||
│ │
|
||||
│ └── unhealthy? → UpsertSignal(), insert metric_samples
|
||||
│
|
||||
▼
|
||||
signals ◄──── UpsertSignal (dedup by target+kind)
|
||||
│
|
||||
▼
|
||||
entity_status ◄── upsert (health, last_check_at)
|
||||
|
||||
metric_samples ◄── INSERT (every cycle, healthy or not)
|
||||
│
|
||||
▼
|
||||
metric_rollups_1h ◄── continuous aggregate
|
||||
metric_rollups_1d ◄── continuous aggregate
|
||||
```
|
||||
|
||||
## Prerequisites for SSH Checks
|
||||
|
||||
1. **Key**: SSH private key mounted at `/etc/oikos/ssh_key` in the scheduler container
|
||||
2. **User**: `OIKOS_SSH_USER=root` (or set `"user"` in check config)
|
||||
3. **Scripts**: Deployed on target host at `/opt/oikos/checks/`
|
||||
4. **Network**: Scheduler container must reach target host (bridge → LAN works)
|
||||
5. **Container**: Scheduler needs `openssh-client` (alpine base) + `CAP_NET_RAW` (for ping)
|
||||
Reference in New Issue
Block a user