fix: move completed signal-triggers plan to done/, add missing liveness-drift to index, add plan-consistency lint checks
This commit is contained in:
387
plans/done/2026-07-08-signal-triggers.md
Normal file
387
plans/done/2026-07-08-signal-triggers.md
Normal file
@@ -0,0 +1,387 @@
|
||||
# 2026-07-08 — Signal triggers: host health checks
|
||||
|
||||
**Status:** Done — Phases 1-5 complete
|
||||
|
||||
## Goal
|
||||
|
||||
Add a comprehensive set of signal triggers for host-level monitoring —
|
||||
network reachability, CPU/memory/disk pressure, thermal state, pending
|
||||
updates, disk health, ZFS pool status, and process liveness. Every check
|
||||
raises properly deduped signals through the existing `UpsertSignal` path
|
||||
and feeds the OODA pipeline (observe → classify → decide → act).
|
||||
|
||||
---
|
||||
|
||||
## 1. New check kinds
|
||||
|
||||
| Kind | What it measures | Signal kind | Severity mapping |
|
||||
|------|-----------------|-------------|------------------|
|
||||
| `ping` | ICMP reachability + RTT | `ping-unreachable` | down → critical |
|
||||
| `cpu` | Usage % + thermal (Linux only) | `cpu-pressure`, `cpu-thermal` | >90% → warning, >95% → critical; temp >85°C → critical |
|
||||
| `memory` | RAM usage % | `memory-pressure` | >90% → warning, >95% → critical |
|
||||
| `load` | Load avg / CPU count | `load-pressure` | >CPU×2 → warning, >CPU×4 → critical |
|
||||
| `swap` | Swap usage % | `swap-pressure` | >50% → warning, >80% → critical |
|
||||
| `disk-usage` | Already exists as `disk` — enhance with inode %, per-mountpoint | `disk-full` | >85% → warning, >95% → critical |
|
||||
| `disk-smart` | SMART pre-failure attributes | `disk-smart-fail` | any fail → critical |
|
||||
| `updates` | Pending apt updates (security, critical) | `updates-pending` | security >0 → warning, critical-reboot >0 → critical |
|
||||
| `zfs` | Pool health + scrub status | `zfs-degraded`, `zfs-scrub-overdue` | degraded → critical, scrub >30d → warning |
|
||||
| `process` | Process/service running | `process-down` | not running → critical |
|
||||
| `uptime` | Detect unexpected reboots | `uptime-bounce` | < previous → warning |
|
||||
|
||||
---
|
||||
|
||||
## 2. Execution model
|
||||
|
||||
Two tiers based on where the check runs:
|
||||
|
||||
### Tier A: Local (scheduler host)
|
||||
|
||||
`ping`, `http`, `tcp`, `cert-expiry` — run directly from the scheduler process. Already implemented for `http`/`tcp`/`disk`/`cert-expiry`. Add `ping` here.
|
||||
|
||||
### Tier B: Remote via SSH (`ssh-script`)
|
||||
|
||||
`cpu`, `memory`, `load`, `swap`, `disk-smart`, `updates`, `zfs`, `process`, `uptime` — run on the target host via SSH. The existing `ssh-script` kind (defined in OpenAPI, not implemented in scheduler) is the one generic mechanism.
|
||||
|
||||
### Why one `ssh-script` kind instead of separate kinds per metric
|
||||
|
||||
The check runs a small allowlisted shell snippet on the target host via SSH.
|
||||
The script outputs JSON with `health`, `signalKind`, `evidence`, and optional
|
||||
`metrics` (name→value map). This keeps the scheduler simple — one code path for
|
||||
all remote checks — while the check definition's `config.script` field encodes
|
||||
what to run.
|
||||
|
||||
### SSH config shape
|
||||
|
||||
```json
|
||||
{
|
||||
"host": "192.168.30.10",
|
||||
"port": 22,
|
||||
"user": "root",
|
||||
"script": "cpu_check.sh",
|
||||
"timeout_s": 10
|
||||
}
|
||||
```
|
||||
|
||||
Scripts live in `/opt/oikos/checks/` on each host, deployed by the Homelab sync
|
||||
timer alongside the AGENTS.md context. They are allowlisted — the scheduler only
|
||||
executes scripts whose names match `^[a-z][a-z0-9_-]+\.sh$` and that exist in the
|
||||
check directory.
|
||||
|
||||
---
|
||||
|
||||
## 3. Check scripts (per host, `/opt/oikos/checks/`)
|
||||
|
||||
### `cpu_check.sh`
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
USAGE=$(top -bn1 | awk '/^%Cpu/ {print 100 - $8}')
|
||||
CORES=$(nproc)
|
||||
TEMP=""
|
||||
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
||||
TEMP=$(echo "scale=1; $(cat /sys/class/thermal/thermal_zone0/temp) / 1000" | bc)
|
||||
fi
|
||||
|
||||
echo "{\"health\":\"healthy\",\"metrics\":{\"cpu_pct\":$USAGE,\"cpu_temp\":$TEMP}}"
|
||||
```
|
||||
Signal raised by scheduler logic when `cpu_pct > threshold_pct` or `cpu_temp > threshold_temp`.
|
||||
|
||||
### `memory_check.sh`
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
MEMINFO=$(awk '/MemTotal|MemAvailable|SwapTotal|SwapFree/ {printf "\"%s\":%d,", tolower($1), $2}' /proc/meminfo | sed 's/,$//')
|
||||
USED_PCT=$(python3 -c "print(round((1 - ${memavailable:-0}/${memtotal:-1})*100, 1))")
|
||||
echo "{\"health\":\"healthy\",\"metrics\":{\"mem_pct\":$USED_PCT}}"
|
||||
```
|
||||
Actual implementation would precompute from `/proc/meminfo` in bash directly (avoid python dependency).
|
||||
|
||||
### `load_check.sh`
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
LOAD=$(awk '{print $1}' /proc/loadavg)
|
||||
CORES=$(nproc)
|
||||
echo "{\"health\":\"healthy\",\"metrics\":{\"load1\":$LOAD,\"cores\":$CORES}}"
|
||||
```
|
||||
Scheduler computes `load_pct = load1 / cores` and thresholds on that.
|
||||
|
||||
### `swap_check.sh`
|
||||
Reports swap used / swap total from `/proc/meminfo`.
|
||||
|
||||
### `disk_smart_check.sh`
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
FAIL=0
|
||||
for dev in $(lsblk -ndo NAME,TYPE | awk '$2=="disk"{print "/dev/"$1}'); do
|
||||
smartctl -H "$dev" | grep -q "PASSED" || { FAIL=1; break; }
|
||||
done
|
||||
[ $FAIL -eq 0 ] && echo '{"health":"healthy"}' || echo '{"health":"degraded","signalKind":"disk-smart-fail","evidence":"SMART health check failed"}'
|
||||
```
|
||||
|
||||
### `updates_check.sh`
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
apt update -qq >/dev/null 2>&1
|
||||
SECURITY=$(apt list --upgradable 2>/dev/null | grep -c '\-security' || true)
|
||||
REBOOT=$(test -f /var/run/reboot-required && echo 1 || echo 0)
|
||||
echo "{\"health\":\"healthy\",\"metrics\":{\"security_updates\":$SECURITY,\"reboot_required\":$REBOOT}}"
|
||||
```
|
||||
Scheduler thresholds: `security_updates > 0` → warning signal, `reboot_required > 0` → critical.
|
||||
|
||||
### `zfs_check.sh`
|
||||
```sh
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
STATUS=$(zpool status -x 2>&1)
|
||||
if echo "$STATUS" | grep -q "all pools are healthy"; then
|
||||
echo '{"health":"healthy"}'
|
||||
else
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"zfs-degraded\",\"evidence\":\"$(echo $STATUS | head -1)\"}"
|
||||
fi
|
||||
```
|
||||
|
||||
### `process_check.sh`
|
||||
Runs `systemctl is-active <service>` for a service name passed in config.
|
||||
Signal on `inactive`/`failed`.
|
||||
|
||||
### `uptime_check.sh`
|
||||
Reads current uptime, compares to last stored value (in a temp file or via metric).
|
||||
Signal if uptime went backward (reboot detected).
|
||||
|
||||
---
|
||||
|
||||
## 4. Scheduler changes
|
||||
|
||||
### `internal/scheduler/scheduler.go`
|
||||
|
||||
Add two new cases to `executeCheck()`:
|
||||
|
||||
```go
|
||||
case "ping":
|
||||
return checkPing(ctx, cd)
|
||||
case "ssh-script":
|
||||
return checkSSHScript(ctx, cd)
|
||||
```
|
||||
|
||||
Additional changes:
|
||||
- **Metric recording**: stop hardcoding `probe_latency_ms`. Each check returns
|
||||
a `map[string]float64` of metrics, and the scheduler writes all of them.
|
||||
Change `executeCheck` signature from `(health, signalKind, evidence, err)`
|
||||
to also return `metrics map[string]float64`.
|
||||
- **Threshold evaluation for metric-based checks** (`cpu`, `memory`, `load`,
|
||||
`swap`, `updates`): the scheduler reads `config.thresholds` from check config
|
||||
JSONB and evaluates metrics against them.
|
||||
|
||||
### `checkPing()`
|
||||
|
||||
Uses `golang.org/x/net/icmp` + `golang.org/x/net/ipv4` for non-privileged
|
||||
ICMP echo (or falls back to `net.DialTimeout("ip4:icmp", ...)`).
|
||||
On macOS, `ping -c 1 -t <timeout>` via exec as fallback (ICMP raw sockets
|
||||
require root on macOS). Returns `probe_latency_ms` metric.
|
||||
|
||||
Config:
|
||||
```json
|
||||
{
|
||||
"host": "192.168.30.10",
|
||||
"count": 1,
|
||||
"timeout_s": 5
|
||||
}
|
||||
```
|
||||
|
||||
### `checkSSHScript()`
|
||||
|
||||
Connects via `golang.org/x/crypto/ssh` using agent forwarding or key file.
|
||||
Executes the allowlisted script path, validates JSON output, returns metrics
|
||||
and health.
|
||||
|
||||
Config:
|
||||
```json
|
||||
{
|
||||
"host": "192.168.30.10",
|
||||
"port": 22,
|
||||
"user": "root",
|
||||
"script": "cpu_check.sh",
|
||||
"thresholds": {
|
||||
"cpu_pct": {"warn": 90, "crit": 95},
|
||||
"cpu_temp": {"crit": 85}
|
||||
},
|
||||
"timeout_s": 10
|
||||
}
|
||||
```
|
||||
|
||||
### Threshold config schema
|
||||
|
||||
Each check kind with metric-based thresholds adds a `thresholds` key:
|
||||
|
||||
```json
|
||||
{
|
||||
"thresholds": {
|
||||
"<metric_name>": {"warn": <float>, "crit": <float>}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Missing thresholds → no signal raised; metrics still recorded.
|
||||
|
||||
---
|
||||
|
||||
## 5. DB and migration
|
||||
|
||||
No schema migration needed. `check_defs.config` is JSONB — new check kinds
|
||||
use it with their own config shapes. `signals` table handles any `kind` string.
|
||||
|
||||
One small addition: add the new signal kinds to the OpenAPI `CheckKind` enum
|
||||
and the generated code (`api/openapi.yaml` line 2247, `internal/httpapi/gen/api.gen.go` line 84).
|
||||
|
||||
---
|
||||
|
||||
## 6. Seed data: default checks per host
|
||||
|
||||
Add to `seeds/inventory.yaml` or a new `seeds/checks.yaml` — a set of default
|
||||
checks per entity type:
|
||||
|
||||
- Every `machine`, `workstation`, `proxmox-host`, `lxc` gets:
|
||||
- `cpu` (via ssh-script)
|
||||
- `memory` (via ssh-script)
|
||||
- `load` (via ssh-script)
|
||||
- `swap` (via ssh-script)
|
||||
- `disk-usage` (local or ssh-script for remote)
|
||||
- `updates` (via ssh-script)
|
||||
- `uptime` (via ssh-script)
|
||||
- Every `machine`, `proxmox-host`, `standalone-server` additionally gets:
|
||||
- `disk-smart` (via ssh-script)
|
||||
- `zfs` (via ssh-script) if `storage-pool` edges exist
|
||||
- Every `service` gets:
|
||||
- `process` (via ssh-script, `systemctl is-active`)
|
||||
|
||||
Default intervals:
|
||||
- `cpu`, `memory`, `load`: 60s
|
||||
- `disk-usage`, `swap`: 300s
|
||||
- `ping`: 30s
|
||||
- `updates`: 3600s (hourly)
|
||||
- `disk-smart`, `zfs`: 86400s (daily)
|
||||
- `process`: 30s
|
||||
|
||||
---
|
||||
|
||||
## 7. Policy integration
|
||||
|
||||
New approval rule for `ssh-script` checks:
|
||||
|
||||
```yaml
|
||||
# ssh-script execution is read_only on the target — it only reads metrics
|
||||
- entity_type: machine
|
||||
action: health-check
|
||||
risk_class: read_only
|
||||
autonomy: auto
|
||||
```
|
||||
|
||||
The `ping` kind is `read_only` (no mutation). All new checks are `read_only`
|
||||
— they observe state, no action taken automatically. The *response* to signals
|
||||
(restart service, clear cache, apt upgrade) goes through the existing
|
||||
classification → approval → execution pipeline separately.
|
||||
|
||||
---
|
||||
|
||||
## 8. Other common important signals (per user request)
|
||||
|
||||
Beyond the core checks above, these are worth including:
|
||||
|
||||
| # | Trigger | Why important |
|
||||
|---|---------|---------------|
|
||||
| 11 | **Inode exhaustion** | Filesystem can be "full" with free space but zero inodes (Docker overlay, mail queues). Distinct from disk-usage. |
|
||||
| 12 | **OOM kills** | `dmesg | grep -i 'out of memory'` count since last boot. Indicates memory pressure beyond usage %. |
|
||||
| 13 | **Journal errors** | `journalctl -p err -S -1h --no-pager | wc -l`. Catches kernel panics, segfaults, service failures. |
|
||||
| 14 | **Docker/container health** | `docker ps --filter health=unhealthy`. Catches containers in unhealthy state. |
|
||||
| 15 | **Caddy/nginx error rate** | Parse access logs for 5xx rate over last 5min. Expensive, do at 300s interval. |
|
||||
| 16 | **Time drift** | `chronyc tracking | grep 'System time'`. NTP offset > 1s → warning (affects TLS, auth, DB). |
|
||||
| 17 | **Open file descriptors** | `/proc/sys/fs/file-nr` ratio used/total. >80% → warning (service exhaustion). |
|
||||
| 18 | **Backup freshness** | Check timestamp of last backup file. >schedule+grace → critical. |
|
||||
|
||||
All of these (11–18) are implemented as `ssh-script` checks with
|
||||
corresponding scripts in `/opt/oikos/checks/`.
|
||||
|
||||
---
|
||||
|
||||
## 9. Implementation order
|
||||
|
||||
### Phase 1: Foundation (2-3 days)
|
||||
|
||||
1. **Refactor `executeCheck`** to return metrics map. Update all existing check
|
||||
functions (`http`, `tcp`, `disk`, `cert-expiry`).
|
||||
2. **Add `ping` kind** — ICMP reachability on the scheduler host.
|
||||
3. **Add `ssh-script` kind** — SSH execution engine, script allowlisting,
|
||||
JSON output parsing.
|
||||
4. **Add threshold evaluation** — scheduler reads `config.thresholds` and
|
||||
compares against returned metrics to decide health/signal.
|
||||
|
||||
### Phase 2: Host check scripts (1-2 days)
|
||||
|
||||
5. Write and test each script in `/opt/oikos/checks/`:
|
||||
`cpu_check.sh`, `memory_check.sh`, `load_check.sh`, `swap_check.sh`,
|
||||
`disk_smart_check.sh`, `updates_check.sh`, `zfs_check.sh`,
|
||||
`process_check.sh`, `uptime_check.sh`, `oom_check.sh`, `journal_check.sh`,
|
||||
`time_check.sh`, `fd_check.sh`.
|
||||
6. Add `tools/setup-checks.sh` to auto-deploy scripts via the sync timer
|
||||
(same pattern as `tools/setup-caveman.sh`).
|
||||
|
||||
### Phase 3: Seed data + API (1 day)
|
||||
|
||||
7. Add `seeds/checks.yaml` with default check definitions per entity type.
|
||||
8. Add new check kinds to OpenAPI spec and regenerate Go types.
|
||||
9. Add a `/api/v1/checks/defaults/{entity_type}` endpoint that returns
|
||||
recommended checks for a given entity type (convenience for operators).
|
||||
|
||||
### Phase 4: Policy + observability (1 day)
|
||||
|
||||
10. Add `read_only` policy rules for `ping` and `ssh-script` actions.
|
||||
11. Add per-check-kind metric recording (not just `probe_latency_ms`).
|
||||
12. Wire `cpu_temp`, `mem_pct`, `load_pct`, etc. into `query_metrics` and
|
||||
the MCP `get_trend` tool.
|
||||
|
||||
### Phase 5: Docker + service signals (1 day)
|
||||
|
||||
13. `docker_health_check.sh` — `docker ps --filter health=unhealthy`.
|
||||
14. `caddy_error_rate.sh` — parse Caddy JSON logs for 5xx.
|
||||
15. `backup_freshness.sh` — check last backup timestamp.
|
||||
|
||||
---
|
||||
|
||||
## 10. Dependencies
|
||||
|
||||
| Dependency | For | Risk |
|
||||
|------------|-----|------|
|
||||
| `golang.org/x/crypto/ssh` | SSH client in scheduler | Already in `go.mod` (used by deployer) |
|
||||
| `golang.org/x/net/icmp` + `ipv4` | ICMP ping | New dep; macOS needs root for raw sockets → exec fallback |
|
||||
| `smartmontools` on hosts | `disk_smart_check.sh` | Already installed on Proxmox; add to LXCs |
|
||||
| `zfsutils-linux` on hosts | `zfs_check.sh` | Already on Proxmox; add to LXCs with ZFS |
|
||||
| SSH key on scheduler | `ssh-script` to all hosts | Already deployed (Homelab sync SSH keys) |
|
||||
|
||||
---
|
||||
|
||||
## 11. Risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| `ssh-script` is a remote exec vector | Scripts are allowlisted by name (`^[a-z][a-z0-9_-]+\.sh$`), deployed via git (auditable), and read-only (no mutation). SSH key restricted to a dedicated `oikos-check` user with sudo only for `systemctl is-active`. |
|
||||
| ICMP requires root on macOS | Fallback to `ping` CLI via `os/exec`. Production runs on Linux (strong) where raw sockets work. |
|
||||
| Metrics cardinality explosion | Metrics are per-check-definition, not per-script-output. The script returns a fixed set of known metric names. TimescaleDB handles the volume. |
|
||||
| Check script drift between hosts | Scripts deploy via `tools/setup-checks.sh` in the sync timer — same mechanism that keeps AGENTS.md in sync. Checksum validation before execution. |
|
||||
|
||||
---
|
||||
|
||||
## 12. Verification
|
||||
|
||||
- **Unit tests**: each check function (`checkPing`, `checkSSHScript`) tested
|
||||
with mock SSH server and mock ICMP responses.
|
||||
- **Integration test**: deploy to `strong` (macOS scheduler host), define
|
||||
checks for `hubris` (Proxmox), `dns` (LXC), `caddy` (LXC), run scheduler
|
||||
with `--check-interval 10s`, verify signals appear in `get_signal_history`.
|
||||
- **MCP smoke test**: `search_knowledge("signal triggers")`, `get_signal_history`,
|
||||
`query_metrics(metric=["cpu_pct", "mem_pct"])`, `get_trend`.
|
||||
- **Policy smoke test**: `preflight(service:caddy, restart)` still returns
|
||||
correct risk class despite new check kinds in the DB.
|
||||
Reference in New Issue
Block a user