Compare commits
55 Commits
42751623ea
...
chore/vend
| Author | SHA1 | Date | |
|---|---|---|---|
| 3cd4cf98c3 | |||
| 38c472a118 | |||
| 1d0197da69 | |||
| 0920c4cb6d | |||
| 2254a07baf | |||
| 1b9c761274 | |||
| a126cfa710 | |||
| 86fa57b5cd | |||
| 8e97d589af | |||
| 0dd8c28815 | |||
| 4e294b3630 | |||
| 85f0bb67fa | |||
| c3f478b8f8 | |||
| 1aaedf498a | |||
| 20adb89650 | |||
| 058f1afcdc | |||
| 2b73290994 | |||
| 428f4fe945 | |||
| 195d45a0e9 | |||
| 5b68bdc16c | |||
| 757ef2f34b | |||
| b27e1bf3ec | |||
| 39e9227fdb | |||
| bb05f215c6 | |||
| 467589d78a | |||
| e25e979757 | |||
| bc0ccb4cdc | |||
| c9a00a9532 | |||
| eb16796bf0 | |||
| a3914a1d41 | |||
| 8eb1ca2bac | |||
| e4104eb344 | |||
| 0929c17cbb | |||
| 6487032461 | |||
| fb6b6f9160 | |||
| 62c9fc5c86 | |||
| 6007e922b4 | |||
| 2d8eb91b25 | |||
| 3d88f52988 | |||
| 9016c3a43b | |||
| 04775192c1 | |||
| a104cb4bb4 | |||
| 72f0f46528 | |||
| b87735a111 | |||
| 1540f74342 | |||
| c7729b2ef6 | |||
| b8b4aa2aee | |||
| c10f6920cd | |||
| ad29295c93 | |||
| 6ca6d5b352 | |||
| af450dac2a | |||
| 6ed9dc39e8 | |||
| cc8eae4979 | |||
| 4f706fa65f | |||
| 50e899e5ee |
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: knowledge-graph-audit
|
||||
risk_class: read_only
|
||||
inputs: []
|
||||
verification: "audit_knowledge_graph returns a report with summary.total_findings"
|
||||
docs_update_checklist: []
|
||||
---
|
||||
|
||||
# Knowledge-graph audit
|
||||
|
||||
Goal: validate that the knowledge graph (entities, relationships, checks) and
|
||||
the monitoring built on it reflect live reality — without mutating anything.
|
||||
Read-only. Run this before trusting health, blast-radius, or coverage answers,
|
||||
and whenever something feels off (a healthy host reports `down`, a retired
|
||||
service still alarms, the graph looks thin).
|
||||
|
||||
## 1. Run the drift report
|
||||
|
||||
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
|
||||
ranked list of findings, each with `{category, severity, count, entities,
|
||||
evidence, suggested_runbook}`, plus a `summary` with totals by category.
|
||||
|
||||
The DB-side categories:
|
||||
|
||||
- **orphan_checks** — check entities with truncated/random slugs left by the
|
||||
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
|
||||
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
|
||||
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
|
||||
(the scheduler already skips these, but the rows should be retired).
|
||||
- **down_checks** — enabled probes reporting `down`. Remediation:
|
||||
`service-health-check` (then check whether the failure is real or a
|
||||
probe-config/routing problem — see step 3).
|
||||
- **unknown_checks** — probes that ran but reported `unknown` (usually a
|
||||
misconfigured or not-yet-deployed probe script).
|
||||
- **unmonitored** — active entities whose type declares monitoring but have no
|
||||
enabled `check_def`.
|
||||
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
|
||||
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
|
||||
|
||||
## 2. Triage
|
||||
|
||||
`severity: critical` (down_checks) first. For each finding, read `evidence` and
|
||||
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
|
||||
before acting — the report is a pointer, not a verdict.
|
||||
|
||||
## 3. Common probe-failure causes
|
||||
|
||||
A `down_checks` finding that is NOT a real outage is usually one of:
|
||||
|
||||
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
|
||||
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
|
||||
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
|
||||
exec` automatically when both are present.
|
||||
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
|
||||
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
|
||||
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
|
||||
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
|
||||
|
||||
## 4. What this audit does NOT cover (follow-ups)
|
||||
|
||||
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
|
||||
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
|
||||
proxmox host) against the DB graph and returns:
|
||||
|
||||
- **missing entities** — a guest running in Proxmox with no DB entity.
|
||||
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
|
||||
|
||||
Still manual until that machinery lands:
|
||||
|
||||
- **Misplaced parent** — compare each guest's actual Proxmox host against its
|
||||
`hosts` edge (migrations leave these stale).
|
||||
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
|
||||
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
|
||||
the cert-expiry checks.
|
||||
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
|
||||
runtime-created entities not in version control.
|
||||
|
||||
## 5. Acting on findings
|
||||
|
||||
This skill is read-only — make no changes here. Route each confirmed finding to
|
||||
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
|
||||
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
|
||||
to confirm the finding cleared.
|
||||
|
||||
Docs-update checklist: none — the audit reads state; it changes nothing. If a
|
||||
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
|
||||
in that remediation session.
|
||||
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
||||
# Every docker build in this repo previously sent the whole directory as
|
||||
# build context — including every OTHER git worktree under .claude/worktrees/
|
||||
# (each with its own web/node_modules, ~200-300MB apiece). That's what
|
||||
# starved the mac-mini's disk mid-build on 2026-07-27 (SHA 873b00a): the
|
||||
# context alone crossed 390MB of pure worktree cruft before the host ran out
|
||||
# of space. None of this ever belonged in an image.
|
||||
.claude/worktrees/
|
||||
.git/
|
||||
**/node_modules/
|
||||
**/dist/
|
||||
**/build/
|
||||
*.log
|
||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -24,3 +24,7 @@ cmd/desktop/build/
|
||||
cmd/desktop/Oikos
|
||||
desktop
|
||||
/eval
|
||||
|
||||
# Local tooling artifacts (Playwright MCP session logs, stray screenshots)
|
||||
.playwright-mcp/
|
||||
config-screen.png
|
||||
|
||||
@@ -2284,6 +2284,23 @@ components:
|
||||
type: boolean
|
||||
version:
|
||||
type: integer
|
||||
last_health:
|
||||
type: string
|
||||
description: >-
|
||||
This check's own most recent verdict. An entity's health is the
|
||||
worst of these across its enabled checks, so this is what explains
|
||||
*why* an entity is degraded. Null until the check first runs.
|
||||
nullable: true
|
||||
enum:
|
||||
- healthy
|
||||
- degraded
|
||||
- down
|
||||
- unknown
|
||||
last_run_at:
|
||||
type: string
|
||||
format: date-time
|
||||
description: When this check last executed. Null = never run.
|
||||
nullable: true
|
||||
CheckCreate:
|
||||
type: object
|
||||
required:
|
||||
|
||||
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Oikos check lifecycle — how monitoring works
|
||||
|
||||
This runbook covers how Oikos health checks are derived, created, and wired so
|
||||
an agent (Nomos) doesn't reverse-engineer source when asked to add monitoring to
|
||||
an entity — the problem that stranded session `23da10db` (2026-08-03).
|
||||
|
||||
## Concepts
|
||||
|
||||
- **`check_defs`** (scheduler config, table `check_defs`): the row the scheduler
|
||||
reads to know *what* to probe and *when*. One per check instance.
|
||||
- **`check` entity** (type `check`, slug `check:<kind>:<target>:<n>`): the
|
||||
knowledge-graph entity for that check. It carries attributes
|
||||
(`check_type`, `target`, `port`, …) and `checks` edges to the probed target.
|
||||
- **`monitoring` spec** on an entity type (`entity_types.monitoring_spec`): the
|
||||
default list of check kinds (e.g. `[http, process]` for `service`).
|
||||
- Per-entity override: set `monitoring` in the entity's attributes —
|
||||
`"none"` for zero checks, `["http"]` to replace the type defaults.
|
||||
- **`checkdefaults.Ensure`** (`internal/checkdefaults/defaults.go`): the
|
||||
function that reads the monitoring spec, resolves host/port/URL from
|
||||
attributes + relationships, and writes `check_defs` rows. Idempotent.
|
||||
|
||||
## When checks are derived
|
||||
|
||||
`checkdefaults.Ensure` runs in three situations (as of v0.17.1+):
|
||||
|
||||
1. **Seed/deploy ingest** — `internal/db/seed.go:231`. Every entity gets its
|
||||
default checks once on initial ingest.
|
||||
2. **HTTP `POST /api/v1/entities` (create)** — `ensureDefaultChecks` at
|
||||
`internal/httpapi/impl.go:1012`. Creating an entity via the REST API derives
|
||||
its checks in the same transaction.
|
||||
3. **HTTP `PATCH /api/v1/entities` (patch)** — `ensureDefaultChecks` at
|
||||
`internal/httpapi/impl.go:1280`. Changing an entity's attributes (especially
|
||||
`monitoring`) via the REST API regenerates its checks.
|
||||
4. **MCP `create_entity`** — SAME hook. Creating an entity via the MCP tool
|
||||
derives checks. (Added 2026-08-03; previously MCP had no create.)
|
||||
5. **MCP `update_entity_attributes`** — SAME hook. Changing an entity's
|
||||
`monitoring` attribute via MCP now regenerates checks. (Added 2026-08-03;
|
||||
previously MCP updates silently skipped check derivation — the exact bug
|
||||
that stranded the haos session.)
|
||||
|
||||
## Check slug grammar
|
||||
|
||||
```
|
||||
check:<kind>:<target-type>:<target-name>:<n>
|
||||
```
|
||||
|
||||
Examples: `check:http:service:jellyfin:0`, `check:vm-status:vm:haos:0`,
|
||||
`check:cert-expiry:cert:house.hubris.network:0`.
|
||||
|
||||
## Adding monitoring to an entity
|
||||
|
||||
**If the entity already exists:**
|
||||
|
||||
```
|
||||
update_entity_attributes(slug="service:haos", attributes={"monitoring":["http"]})
|
||||
```
|
||||
|
||||
This regenerates checks via `checkdefaults.Ensure`. The result message tells you
|
||||
how many checks were derived and whether any kinds were skipped (and why).
|
||||
|
||||
**If the entity does not exist yet (a new check, ingress, cert, etc.):**
|
||||
|
||||
```
|
||||
create_entity(type="check", name="HAOS http check",
|
||||
slug="check:http:service:haos:0",
|
||||
attributes={"check_type":"http:service","target":"service:haos","port":"8123"})
|
||||
```
|
||||
|
||||
This creates the entity AND derives its `check_defs`. Same for a new `ingress`
|
||||
(`type=ingress`, monitoring `[http]`) or `cert` (`type=cert`,
|
||||
monitoring `[cert-expiry]`).
|
||||
|
||||
**To remove monitoring:** set `monitoring:["none"]` or transition the entity
|
||||
to a terminal lifecycle state (`set_entity_state` → `deprecated`/`destroyed`).
|
||||
|
||||
## Caveats
|
||||
|
||||
- **A service without a `url` attribute AND without a `probe_unit` gets no
|
||||
process check** (the http check covers liveness; the process check would
|
||||
be redundant without an opt-in `probe_unit`). The skip is logged.
|
||||
- **A service whose address comes from a `hosts` edge** may produce no checks on
|
||||
initial create because the edge doesn't exist yet — the next inventory ingest
|
||||
(or a later `update_entity_attributes` after the edge is created) fills it in.
|
||||
- **A `not found` error from `update_entity_attributes`** means the entity
|
||||
doesn't exist — use `create_entity` instead.
|
||||
- **`check_defs` has target columns** (`target_id`, `target_type`). A check
|
||||
entity needs a `checks` relationship (`create_relationship(source=check:…,
|
||||
target=service:…, type="checks")`) so the scheduler can resolve what to
|
||||
probe. `create_entity` derives the check_def; `create_relationship` links
|
||||
the check entity to its target in the graph.
|
||||
|
||||
## Related files
|
||||
|
||||
- `internal/checkdefaults/defaults.go` — `Ensure`, `Target`, `LogResult`
|
||||
- `internal/httpapi/default_checks.go` — `ensureDefaultChecks` (HTTP hook)
|
||||
- `internal/db/checks.go` — `db.EnsureEntityChecks` (shared hook)
|
||||
- `internal/db/seed.go` — seed-time check derivation
|
||||
- `internal/mcp/tools.go` — `create_entity`, `update_entity_attributes`
|
||||
|
||||
## Revision history
|
||||
|
||||
- **2026-08-03:** Created after session `23da10db` stranded for lack of entity-
|
||||
creation tool and unawareness of check-derivation triggers. Covers the MCP
|
||||
create_entity + update_entity_attributes regen paths added same day.
|
||||
@@ -2,12 +2,27 @@
|
||||
# cpu_check.sh — CPU usage % and thermal temperature.
|
||||
set -euo pipefail
|
||||
|
||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||
if [ -z "$USAGE" ]; then
|
||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||
os=$(uname -s)
|
||||
|
||||
if [ "$os" = "Darwin" ]; then
|
||||
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
|
||||
# Usage is 100 minus the idle figure that precedes the literal `idle`.
|
||||
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
|
||||
/^CPU usage/ {
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
|
||||
}
|
||||
}' || true)
|
||||
else
|
||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||
if [ -z "$USAGE" ]; then
|
||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||
fi
|
||||
fi
|
||||
|
||||
[ -z "$USAGE" ] && USAGE=0
|
||||
|
||||
TEMP=""
|
||||
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
||||
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
||||
|
||||
22
checks/disk_usage_check.sh
Normal file → Executable file
22
checks/disk_usage_check.sh
Normal file → Executable file
@@ -2,18 +2,34 @@
|
||||
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
||||
set -euo pipefail
|
||||
|
||||
MOUNTS=$(df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||
# `timeout` caps each df so a single hung/stale mountpoint (a stale NFS
|
||||
# export, a wedged ZFS pool) can't stall the whole check — that hung the
|
||||
# scheduler's 30s budget on hubris. Available on Linux (coreutils); absent on
|
||||
# Darwin, whose local mounts don't hang, so it degrades to an empty prefix.
|
||||
TO=""
|
||||
if command -v timeout >/dev/null 2>&1; then TO="timeout 8"; fi
|
||||
|
||||
# Build the mount list WITHOUT statting anything: reading /proc/mounts never
|
||||
# blocks the way `df` does on a stuck filesystem, so the enumeration itself
|
||||
# can't hang. Fall back to `df` on hosts without /proc/mounts (macOS).
|
||||
if [ -r /proc/mounts ]; then
|
||||
MOUNTS=$(awk '$1 ~ /^\// && $2 !~ /^\/(snap|dev|proc|sys|run|private)/ {print $2}' /proc/mounts || true)
|
||||
else
|
||||
MOUNTS=$($TO df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||
fi
|
||||
FIRST=1
|
||||
|
||||
echo -n '{"health":"healthy","metrics":{'
|
||||
for m in $MOUNTS; do
|
||||
LINE=$(df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||
# Each df is bounded: a stuck mount times out and is skipped (LINE empty)
|
||||
# rather than hanging the probe.
|
||||
LINE=$($TO df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||
if [ -z "$LINE" ]; then continue; fi
|
||||
USED=$(echo "$LINE" | awk '{print $1}')
|
||||
FREE=$(echo "$LINE" | awk '{print $2}')
|
||||
PCT=$(echo "$LINE" | awk '{print $3}')
|
||||
|
||||
INODE_LINE=$(df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||
INODE_LINE=$($TO df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
||||
|
||||
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
||||
|
||||
@@ -1,5 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# process_check.sh — systemd service liveness.
|
||||
# process_check.sh — service liveness.
|
||||
#
|
||||
# A service entity's name is a logical label, rarely the literal systemd unit
|
||||
# or container name. matrix = matrix-synapse.service + element-web/mautrix-*
|
||||
# containers; authentik = authentik-server/-worker containers. So checking
|
||||
# `systemctl is-active matrix` reports "inactive" for a healthy service.
|
||||
#
|
||||
# Resolution order, any hit = healthy:
|
||||
# 1. exact systemd unit `systemctl is-active <name>`
|
||||
# 2. a systemd unit with the name as prefix `<name>*.service`
|
||||
# 3. a running docker container whose name contains <name>
|
||||
# An explicit probe target overrides the label — see checkdefaults, which
|
||||
# passes a `probe_unit`/`container`/`systemd_unit` attribute as $1 when set.
|
||||
set -euo pipefail
|
||||
|
||||
SERVICE="${1:-}"
|
||||
@@ -8,15 +20,31 @@ if [ -z "$SERVICE" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! command -v systemctl >/dev/null 2>&1; then
|
||||
echo '{"health":"unknown","signalKind":"process-check","evidence":"systemctl not found"}'
|
||||
exit 0
|
||||
ok() { echo "{\"health\":\"healthy\"}"; exit 0; }
|
||||
|
||||
# 1. exact systemd unit
|
||||
if command -v systemctl >/dev/null 2>&1; then
|
||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null | head -1 || true)
|
||||
[ "$STATE" = "active" ] && ok
|
||||
|
||||
# 2. prefix match: matrix -> matrix-synapse.service, house -> house.service, etc.
|
||||
# --no-legend strips the header/footer so grep can see the unit rows; the
|
||||
# pattern is a systemd unit glob.
|
||||
if systemctl list-units --type=service --state=active --no-legend "$SERVICE*.service" 2>/dev/null \
|
||||
| grep -q '\.service'; then
|
||||
ok
|
||||
fi
|
||||
fi
|
||||
|
||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null || echo "unknown")
|
||||
|
||||
if [ "$STATE" = "active" ]; then
|
||||
echo "{\"health\":\"healthy\"}"
|
||||
else
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"$SERVICE\",\"evidence\":\"$SERVICE is $STATE\"}"
|
||||
# 3. a running docker container whose name contains the label.
|
||||
if command -v docker >/dev/null 2>&1; then
|
||||
if docker ps --filter "status=running" --filter "name=$SERVICE" --format '{{.Names}}' 2>/dev/null \
|
||||
| grep -q .; then
|
||||
ok
|
||||
fi
|
||||
fi
|
||||
|
||||
STATE=${STATE:-inactive}
|
||||
STATE=${STATE//\"/}
|
||||
SAFE_SERVICE=${SERVICE//\"/}
|
||||
echo "{\"health\":\"degraded\",\"signalKind\":\"process\",\"evidence\":\"$SAFE_SERVICE is $STATE (no active unit/container matched)\"}"
|
||||
|
||||
@@ -57,6 +57,13 @@ type agent struct {
|
||||
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
||||
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
||||
httpClient *http.Client
|
||||
// gate serializes turns per session (at most one in-flight turn per
|
||||
// sessionID). See turngate.go and plan 2026-08-03 F1.
|
||||
gate *turnGate
|
||||
// queue holds operator messages that arrived while a turn was already
|
||||
// running; they are auto-run when the gate frees (plan 2026-08-03 F2).
|
||||
// See messagequeue.go.
|
||||
queue *messageQueue
|
||||
}
|
||||
|
||||
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
||||
@@ -117,6 +124,8 @@ func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug
|
||||
apiBase: apiBase,
|
||||
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
||||
httpClient: &http.Client{Timeout: 15 * time.Second},
|
||||
gate: newTurnGate(),
|
||||
queue: newMessageQueue(),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -172,6 +181,11 @@ type agentEvent struct {
|
||||
Data any `json:"data,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Iteration int `json:"iteration,omitempty"`
|
||||
// IsThinking marks text/text_delta events that carry the model's internal
|
||||
// reasoning (text produced before tool calls in the same iteration), as
|
||||
// distinct from the final response text. The frontend renders these as
|
||||
// collapsible thinking blocks separated from the response.
|
||||
IsThinking bool `json:"is_thinking,omitempty"`
|
||||
}
|
||||
|
||||
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
||||
@@ -368,7 +382,12 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
var msg openai.ChatCompletionMessage
|
||||
var acc openai.ChatCompletionAccumulator
|
||||
|
||||
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
||||
// Capture token usage from this LLM response for activity logging.
|
||||
// Previously always NULL — every agent_activity row had no token
|
||||
// count. Now each tool call in this iteration gets the same total.
|
||||
totalTokens := 0
|
||||
|
||||
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
||||
acc = openai.ChatCompletionAccumulator{}
|
||||
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
|
||||
for stream.Next() {
|
||||
@@ -400,14 +419,19 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
msg = acc.Choices[0].Message
|
||||
finishReason := acc.Choices[0].FinishReason
|
||||
|
||||
// Capture token usage from this iteration.
|
||||
if acc.Usage.TotalTokens > 0 {
|
||||
totalTokens = int(acc.Usage.TotalTokens)
|
||||
}
|
||||
|
||||
if len(msg.ToolCalls) == 0 {
|
||||
if isRefusalOrEmpty(msg.Content) {
|
||||
if attempt < maxLLMRetries {
|
||||
slog.Warn("nomos: empty or refusal response, retrying",
|
||||
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
||||
"content_len", len(msg.Content), "finish_reason", finishReason)
|
||||
continue
|
||||
}
|
||||
if attempt < maxLLMRetries {
|
||||
slog.Warn("nomos: empty or refusal response, retrying",
|
||||
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
||||
"content_len", len(msg.Content), "finish_reason", finishReason)
|
||||
continue
|
||||
}
|
||||
// B.4: surface the real error context (finish_reason +
|
||||
// refusal text) instead of a generic "empty response" —
|
||||
// the operator can tell "content_filter — rephrase" from
|
||||
@@ -456,7 +480,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
// led to each step. Emitting it lets the persist layer accumulate
|
||||
// per-iteration reasoning into the row's text field.
|
||||
if strings.TrimSpace(msg.Content) != "" {
|
||||
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID})
|
||||
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID, IsThinking: true})
|
||||
}
|
||||
|
||||
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
|
||||
@@ -491,7 +515,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
slog.Warn("nomos: run retry cap hit — refusing dispatch",
|
||||
"target", t, "failures", n, "session", sessionID)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
|
||||
tc.Function.Arguments, directive, 0, false, correlationID)
|
||||
tc.Function.Arguments, directive, 0, false, correlationID, totalTokens)
|
||||
emit(agentEvent{
|
||||
Type: "tool_result",
|
||||
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
|
||||
@@ -542,7 +566,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
inputStr := string(inputJSON)
|
||||
|
||||
if callErr != nil {
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID, totalTokens)
|
||||
|
||||
// Retry cap: dispatch errors (e.g. MCP client timeout)
|
||||
// count toward the cap too. A command that keeps timing
|
||||
@@ -572,7 +596,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
||||
}
|
||||
|
||||
resultJSON, _ := json.Marshal(result)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID)
|
||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID, totalTokens)
|
||||
|
||||
// Link any execution this tool queued/started back to this
|
||||
// session, so the auto-continuation worker can feed its result
|
||||
|
||||
@@ -76,16 +76,20 @@ func (a *agent) processIdleSweep(ctx context.Context) {
|
||||
s := s
|
||||
if s.CompletionNudges == 0 {
|
||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||
return
|
||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||
s.Goal, idleTaskThreshold)
|
||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||
// P1: only count the nudge if it actually delivered. resumeSession
|
||||
// skips (returns false) when a turn is already active; bumping the
|
||||
// counter anyway would make the next sweep auto-close a merely-busy
|
||||
// session as "unanswered."
|
||||
if a.resumeSession(ctx, s.ID, note) {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||
}
|
||||
}
|
||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||
s.Goal, idleTaskThreshold)
|
||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||
a.resumeSession(ctx, s.ID, note)
|
||||
})
|
||||
continue
|
||||
}
|
||||
@@ -163,7 +167,9 @@ func (a *agent) processContinuations(ctx context.Context) {
|
||||
continue
|
||||
}
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
|
||||
// markContinued now happens inside continueSession, AFTER resumeSession
|
||||
// actually runs (P0). Pre-marking here consumed the item even when
|
||||
// resumeSession skipped on a busy session, losing the result.
|
||||
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
||||
}
|
||||
}
|
||||
@@ -179,7 +185,18 @@ func (a *agent) processContinuations(ctx context.Context) {
|
||||
// something new to poll for.
|
||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||
a.resumeSession(ctx, p.SessionID, buildContinuationNote(p))
|
||||
// P0 (plans/2026-08-03-nomos-chat-changes-review.md): mark the execution
|
||||
// continued ONLY after the turn actually ran. resumeSession skips (returns
|
||||
// false) when another turn is already active for this session; marking
|
||||
// before that — as the old code did — consumed the item (continued_at set,
|
||||
// never re-queued by pendingContinuations) and silently lost the result.
|
||||
// On a skip, leave it pending so the next worker tick retries once the
|
||||
// active turn frees the permit.
|
||||
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||
return
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID)
|
||||
}
|
||||
|
||||
// resumeSession re-invokes the agent for a session with a system-injected note —
|
||||
@@ -187,7 +204,32 @@ func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
||||
// in place as each tool call lands) so the frontend poller sees each step,
|
||||
// instead of total silence until the whole resume concludes.
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
//
|
||||
// F1 (plan 2026-08-03): this is the single entry point for EVERY background
|
||||
// turn — the continuation worker, idle sweep, answer-question, /resume, and the
|
||||
// empty-message reconnect all funnel through here. It acquires the session's
|
||||
// turn permit non-blocking and SKIPS if a turn is already running. A duplicate
|
||||
// resume while a turn (live or background) is active is exactly the
|
||||
// interleaving that corrupted the activity panel and made tasks feel stuck.
|
||||
//
|
||||
// Returns whether the turn actually ran. Callers that mutate state before
|
||||
// resuming (the continuation worker's markContinued, the idle sweep's nudge
|
||||
// bump) MUST gate that mutation on a true return — otherwise a busy-skip leaves
|
||||
// the state changed but the work undone (lost continuation / false auto-close).
|
||||
// See plans/2026-08-03-nomos-chat-changes-review.md P0/P1.
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||
if !a.gate.acquire(sessionID, 0) {
|
||||
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||
return false
|
||||
}
|
||||
// Release the gate, then drain any operator message that was queued while
|
||||
// this background turn ran (plan 2026-08-03 F2). Queued messages are run as
|
||||
// real user turns server-side; resumeSession itself never enqueues.
|
||||
defer func() {
|
||||
a.gate.release(sessionID)
|
||||
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||
}()
|
||||
|
||||
placeholder, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": "",
|
||||
@@ -200,6 +242,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
|
||||
var toolCalls []map[string]any
|
||||
var finalText, errText string
|
||||
var finalThinking string
|
||||
|
||||
persist := func() {
|
||||
if msgID == uuid.Nil {
|
||||
@@ -212,6 +255,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": text,
|
||||
"thinking": finalThinking,
|
||||
"tool_calls": toolCalls,
|
||||
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
||||
})
|
||||
@@ -242,15 +286,19 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
if attempt > 0 {
|
||||
select {
|
||||
case <-cctx.Done():
|
||||
return
|
||||
return true // a turn ran on an earlier attempt; consume, don't re-loop
|
||||
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
||||
}
|
||||
}
|
||||
toolCalls, finalText, errText = nil, "", ""
|
||||
finalThinking = ""
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting
|
||||
// (same fix as main.go's chat handler). Without this, a resumed
|
||||
// turn's intermediate thinking is lost on reload.
|
||||
// (same fix as main.go's chat handler). Without this, a resumed
|
||||
// turn's intermediate thinking is lost on reload.
|
||||
var textParts []string
|
||||
var thinkingParts []string
|
||||
emit := func(ev agentEvent) {
|
||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||
if m, ok := ev.Data.(map[string]any); ok {
|
||||
@@ -277,8 +325,13 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
}
|
||||
if ev.Type == "text" {
|
||||
if t, ok := ev.Data.(string); ok && t != "" {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
if ev.IsThinking {
|
||||
thinkingParts = append(thinkingParts, t)
|
||||
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||
} else {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
}
|
||||
persist()
|
||||
}
|
||||
}
|
||||
@@ -314,9 +367,10 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
||||
// No placeholder was inserted (rare), save directly.
|
||||
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
||||
}
|
||||
return // do not call persist() again — already persisted above
|
||||
return true // do not call persist() again — already persisted above
|
||||
}
|
||||
persist() // final state — same row, updated one last time with the concluding text
|
||||
return true
|
||||
}
|
||||
|
||||
// buildContinuationNote frames the finished execution for the model: what
|
||||
|
||||
@@ -1,6 +1,11 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestExtractExecutionIDs(t *testing.T) {
|
||||
// Real tool-result phrasings that should yield an execution id.
|
||||
@@ -37,3 +42,37 @@ func TestExtractExecutionIDs(t *testing.T) {
|
||||
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResumeSession_SkipsWhenBusy guards the P0 fix
|
||||
// (plans/2026-08-03-nomos-chat-changes-review.md): resumeSession must skip —
|
||||
// return false, body never executed — when a turn is already active for the
|
||||
// session. continueSession relies on this so it only marks a continuation
|
||||
// "continued" after a turn really ran (otherwise the result is lost: marked
|
||||
// continued, never re-queued by pendingContinuations).
|
||||
//
|
||||
// A minimal agent with only a gate is enough: if the body ever ran, chatWith
|
||||
// would dereference the nil provider and panic. Returning false cleanly proves
|
||||
// the body was skipped.
|
||||
func TestResumeSession_SkipsWhenBusy(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate()}
|
||||
if !a.gate.acquire("sess", 0) {
|
||||
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||
}
|
||||
ran := a.resumeSession(context.Background(), "sess", "note")
|
||||
if ran {
|
||||
t.Fatal("resumeSession must return false (skip) while a turn is active for the session")
|
||||
}
|
||||
}
|
||||
|
||||
// TestContinueSession_DefersWhenBusy guards the other half of P0: when the
|
||||
// session is busy, continueSession defers (leaves the execution pending for the
|
||||
// next worker tick) instead of running or marking it. It must return cleanly
|
||||
// without reaching resumeSession's body (nil provider → panic) or markContinued.
|
||||
func TestContinueSession_DefersWhenBusy(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate()}
|
||||
if !a.gate.acquire("sess", 0) {
|
||||
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||
}
|
||||
p := pendingContinuation{ExecID: uuid.New(), SessionID: "sess", Status: "completed"}
|
||||
a.continueSession(context.Background(), p) // must not panic; must not run/mark
|
||||
}
|
||||
|
||||
@@ -319,8 +319,10 @@ func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sess
|
||||
}
|
||||
// Fetch the plan (steps with generation numbers) for the
|
||||
// plan_generations assertion. A 404 or empty response is fine — a
|
||||
// pure-DB Q&A with no propose_plan has no plan.
|
||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan"); perr == nil {
|
||||
// pure-DB Q&A with no propose_plan has no plan. ?all=true returns every
|
||||
// generation so the assertion can count them (the default view returns
|
||||
// only the current generation).
|
||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan?all=true"); perr == nil {
|
||||
if planResp.StatusCode == 200 {
|
||||
pb, _ := io.ReadAll(planResp.Body)
|
||||
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
||||
|
||||
@@ -167,6 +167,147 @@ func sseEvent(w http.ResponseWriter, flusher http.Flusher, event agentEvent) {
|
||||
flusher.Flush()
|
||||
}
|
||||
|
||||
// runChatTurn is the shared core of an operator-initiated turn: insert an
|
||||
// assistant placeholder, run a.chat with incremental persistence (so whatever
|
||||
// happened before an abort is never lost), finalize the row, and derive a
|
||||
// title. It is agnostic to the transport: `sink` receives every agent event
|
||||
// for delivery (SSE for a live handleChat, a no-op for a queued turn that has
|
||||
// no client attached — the frontend learns about those via the poller + the
|
||||
// status-driven "working" signal). The caller MUST already hold the session's
|
||||
// turn-gate permit.
|
||||
func (a *agent) runChatTurn(pctx, ctx context.Context, sessionID, message string, sink func(agentEvent)) {
|
||||
toolCalls := []map[string]any{}
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting with the
|
||||
// final `text` event (see the original inline comment in handleChat).
|
||||
var textParts []string
|
||||
var thinkingParts []string
|
||||
var finalText string
|
||||
var finalThinking string
|
||||
|
||||
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
||||
msgID, err := a.store.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
||||
if err != nil {
|
||||
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
||||
}
|
||||
persist := func() {
|
||||
if msgID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": finalText,
|
||||
"thinking": finalThinking,
|
||||
"tool_calls": toolCalls,
|
||||
})
|
||||
a.store.updateMessage(pctx, msgID, body)
|
||||
}
|
||||
|
||||
a.chat(ctx, sessionID, message, func(ev agentEvent) {
|
||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||
if m, ok := ev.Data.(map[string]any); ok {
|
||||
m["type"] = ev.Type
|
||||
// One entry per tool call: tool_use creates it, tool_result
|
||||
// merges the result into the same entry (matched by id).
|
||||
id, _ := m["id"].(string)
|
||||
if id != "" && ev.Type == "tool_result" {
|
||||
for _, existing := range toolCalls {
|
||||
if eID, _ := existing["id"].(string); eID == id {
|
||||
for k, v := range m {
|
||||
existing[k] = v
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
toolCalls = append(toolCalls, m)
|
||||
}
|
||||
}
|
||||
persist() // live: survives even if the client disconnects right after
|
||||
}
|
||||
if ev.Type == "text" {
|
||||
if t, ok := ev.Data.(string); ok && t != "" {
|
||||
if ev.IsThinking {
|
||||
thinkingParts = append(thinkingParts, t)
|
||||
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||
} else {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
}
|
||||
persist()
|
||||
}
|
||||
}
|
||||
sink(ev)
|
||||
})
|
||||
|
||||
// B.6: if the turn ended with no text and no tool calls (the model
|
||||
// empty-response'd and all retries failed), delete the placeholder row
|
||||
// instead of persisting an empty bubble.
|
||||
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
||||
a.store.deleteMessage(pctx, msgID)
|
||||
} else {
|
||||
persist() // final state — same row, updated one last time
|
||||
}
|
||||
|
||||
// Title: prefer the goal once set; else the first assistant answer.
|
||||
if finalText != "" && sessionID != "ephemeral" {
|
||||
var goalTitle string
|
||||
if sess, gerr := a.store.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
||||
goalTitle = truncate(sess.Goal, 120)
|
||||
}
|
||||
title := goalTitle
|
||||
if title == "" {
|
||||
title = truncate(finalText, 80)
|
||||
}
|
||||
if title != "" {
|
||||
a.store.updateSessionTitle(pctx, sessionID, title)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// drainAcquireWait is how long drainQueued blocks for a busy gate before
|
||||
// re-queuing and deferring to the holder's own release-drain. A package var so
|
||||
// tests can shorten it; in production it just needs to outlast the brief
|
||||
// release→drain handoff window.
|
||||
var drainAcquireWait = 5 * time.Second
|
||||
|
||||
// drainQueued runs every queued operator message for a session as its own turn,
|
||||
// one at a time, under the turn gate. Called (in a goroutine) whenever a turn
|
||||
// releases the gate — from handleChat (live) and resumeSession (background) —
|
||||
// so a message queued while the agent was busy is acted on as soon as it's
|
||||
// free, without the operator re-sending. See messagequeue.go (plan 2026-08-03
|
||||
// F2).
|
||||
//
|
||||
// Each queued turn is persisted incrementally and has no SSE client (the
|
||||
// browser detached after receiving the `queued` event); the frontend sees the
|
||||
// result via the 3s poller and the status-driven "working" indicator.
|
||||
func (a *agent) drainQueued(ctx context.Context, sessionID string) {
|
||||
for {
|
||||
msg, ok := a.queue.dequeue(sessionID)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
// Block briefly for the gate. If a live turn grabbed it first, put the
|
||||
// message back — that turn's release will drain it again. Never stack.
|
||||
if !a.gate.acquire(sessionID, drainAcquireWait) {
|
||||
a.queue.requeueFront(sessionID, msg)
|
||||
return
|
||||
}
|
||||
slog.Info("nomos: running queued operator message", "session", sessionID)
|
||||
pctx := context.Background()
|
||||
// Run the turn inside a per-iteration closure so the gate release is
|
||||
// deferred to the end of THIS turn (and runs even if runChatTurn
|
||||
// panics — safego recovers the panic at the goroutine boundary, so a
|
||||
// non-deferred release would be skipped and the session's permit held
|
||||
// forever, deadlocking all future turns). A bare `defer release` in
|
||||
// the loop would be wrong too: Go defers run at function exit, not
|
||||
// iteration exit, so the gate would stay held across iterations.
|
||||
func() {
|
||||
defer a.gate.release(sessionID)
|
||||
a.runChatTurn(pctx, ctx, sessionID, msg, func(agentEvent) {})
|
||||
}()
|
||||
}
|
||||
}
|
||||
|
||||
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
if r.Method != http.MethodPost {
|
||||
http.Error(w, "method not allowed", 405)
|
||||
@@ -186,12 +327,22 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
return
|
||||
}
|
||||
|
||||
// Empty message with an existing session = reconnect/resume. The
|
||||
// frontend sends this after a dropped SSE stream to re-establish the
|
||||
// connection and catch up on any auto-continuation work that happened
|
||||
// while disconnected. Route into resumeSession so the agent sees a
|
||||
// system note and reports current state.
|
||||
// Empty message with an existing session = reconnect/resume. This path is
|
||||
// defensive now — the frontend (post F2) recovers a dropped SSE via the
|
||||
// poller + terminal task.status clearing, and no longer POSTs empty
|
||||
// messages. If a client ever does, route into resumeSession so the agent
|
||||
// reports current state — but SKIP a terminal session (done/failed/
|
||||
// abandoned): there's nothing to resume, and running a "report state"
|
||||
// turn there is just a spare turn the operator never asked for (P2.1).
|
||||
if req.Message == "" && req.SessionID != "" {
|
||||
if sess, err := st.getSession(context.Background(), req.SessionID); err == nil {
|
||||
switch sess.Status {
|
||||
case "done", "failed", "abandoned":
|
||||
slog.Info("nomos: reconnect skipped — session already terminal", "session", req.SessionID, "status", sess.Status)
|
||||
w.WriteHeader(202)
|
||||
return
|
||||
}
|
||||
}
|
||||
slog.Info("nomos: reconnect", "session", req.SessionID)
|
||||
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
||||
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
||||
@@ -200,7 +351,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
})
|
||||
// Return 202 so the frontend doesn't try to consume an SSE stream
|
||||
// from this POST — resumeSession writes to the DB directly and
|
||||
// the poller (already running from handleDisconnect) picks it up.
|
||||
// the poller picks it up.
|
||||
w.WriteHeader(202)
|
||||
return
|
||||
}
|
||||
@@ -217,6 +368,17 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
|
||||
w.WriteHeader(200)
|
||||
|
||||
// All writes to w (events + the keepalive comment below) go through one
|
||||
// mutex: http.ResponseWriter is NOT safe for concurrent use, and the
|
||||
// keepalive ticker runs alongside the turn's event sink (plan 2026-08-03
|
||||
// F3). Without this, interleaved writes corrupt the SSE stream.
|
||||
var writeMu sync.Mutex
|
||||
writeEvent := func(ev agentEvent) {
|
||||
writeMu.Lock()
|
||||
defer writeMu.Unlock()
|
||||
sseEvent(w, flusher, ev)
|
||||
}
|
||||
|
||||
ctx := r.Context()
|
||||
sessionID := req.SessionID
|
||||
|
||||
@@ -268,111 +430,65 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||
st.answerQuestion(pctx, sessionID, qid, req.Message)
|
||||
}
|
||||
|
||||
sseEvent(w, flusher, agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||
writeEvent(agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||
|
||||
toolCalls := []map[string]any{}
|
||||
// P3: accumulate per-iteration reasoning instead of overwriting with
|
||||
// the final `text` event. The agent loop emits a `text` event for each
|
||||
// LLM iteration that produced text (intermediate reasoning before tool
|
||||
// calls + the final answer). Without accumulation, only the last `text`
|
||||
// survives in the persisted row — a reload shows the final summary but
|
||||
// not the thinking that led to each tool call.
|
||||
var textParts []string
|
||||
var finalText string
|
||||
|
||||
// Incremental persistence, mirroring resumeSession's existing
|
||||
// placeholder+update pattern (continue.go): insert a placeholder now,
|
||||
// update the SAME row after every tool call, so whatever happened before
|
||||
// an abort is never lost — only what hadn't happened yet is.
|
||||
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
||||
msgID, err := st.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
||||
if err != nil {
|
||||
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
||||
}
|
||||
persist := func() {
|
||||
if msgID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{
|
||||
"role": "assistant",
|
||||
"text": finalText,
|
||||
"tool_calls": toolCalls,
|
||||
})
|
||||
st.updateMessage(pctx, msgID, body)
|
||||
// F1/F2 (plan 2026-08-03): serialize turns per session. The user message is
|
||||
// already persisted above, so it is never lost. Wait briefly for a finishing
|
||||
// background turn; if one is still running after that, QUEUE this message
|
||||
// (don't reject it) and tell the client so it shows a "queued" state. The
|
||||
// in-flight turn's release drains the queue (drainQueued) and runs it as a
|
||||
// real turn server-side. This never stacks concurrent turns — the gate still
|
||||
// guarantees one in-flight turn per session.
|
||||
const turnWait = 5 * time.Second
|
||||
if !a.gate.acquire(sessionID, turnWait) {
|
||||
a.queue.enqueue(sessionID, req.Message)
|
||||
slog.Info("nomos: turn already active, queued operator message", "session", sessionID)
|
||||
writeEvent(agentEvent{Type: "queued", Data: sessionID, SessionID: sessionID})
|
||||
writeEvent(agentEvent{Type: "done", Data: map[string]any{
|
||||
"session_id": sessionID,
|
||||
"queued": true,
|
||||
}, SessionID: sessionID})
|
||||
return
|
||||
}
|
||||
defer func() {
|
||||
a.gate.release(sessionID)
|
||||
// Run any message that was queued while this turn held the gate. In a
|
||||
// goroutine so the HTTP response finishes without waiting on the next
|
||||
// turn; the queued turn has no SSE client of its own.
|
||||
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||
}()
|
||||
|
||||
a.chat(ctx, sessionID, req.Message, func(ev agentEvent) {
|
||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||
if m, ok := ev.Data.(map[string]any); ok {
|
||||
m["type"] = ev.Type
|
||||
// One entry per tool call: tool_use creates it, tool_result
|
||||
// merges the result into the same entry (matched by id).
|
||||
// Before this fix, both events appended separate entries,
|
||||
// doubling every tool call in the persisted transcript
|
||||
// (confirmed pre-existing in d9cdcee1, v0.3.x era).
|
||||
id, _ := m["id"].(string)
|
||||
if id != "" && ev.Type == "tool_result" {
|
||||
for _, existing := range toolCalls {
|
||||
if eID, _ := existing["id"].(string); eID == id {
|
||||
for k, v := range m {
|
||||
existing[k] = v
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
} else {
|
||||
toolCalls = append(toolCalls, m)
|
||||
}
|
||||
}
|
||||
persist() // live: survives even if the client disconnects right after
|
||||
}
|
||||
if ev.Type == "text" {
|
||||
// P3: accumulate. Each `text` event is one iteration's reasoning
|
||||
// (or the final answer). Join with newlines so the persisted row
|
||||
// reads as the full transcript of what the agent said, not just
|
||||
// the last thing.
|
||||
if t, ok := ev.Data.(string); ok && t != "" {
|
||||
textParts = append(textParts, t)
|
||||
finalText = strings.Join(textParts, "\n\n")
|
||||
persist()
|
||||
// F3 (plan 2026-08-03): keep the SSE alive during long turns. A turn can
|
||||
// run for many minutes (provisioning chains, deep research); the model
|
||||
// often takes 20-40s between tool iterations, and with nothing flushed in
|
||||
// that gap a proxy/browser idle timeout silently closes the stream. The
|
||||
// client then sees streaming=false while the server keeps working — the
|
||||
// "I can't tell it's working" desync. An SSE comment line (":keepalive") is
|
||||
// ignored by EventSource but resets idle timers.
|
||||
keepDone := make(chan struct{})
|
||||
go func() {
|
||||
t := time.NewTicker(12 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-keepDone:
|
||||
return
|
||||
case <-t.C:
|
||||
writeMu.Lock()
|
||||
fmt.Fprintf(w, ":keepalive\n\n")
|
||||
flusher.Flush()
|
||||
writeMu.Unlock()
|
||||
}
|
||||
}
|
||||
sseEvent(w, flusher, ev)
|
||||
}()
|
||||
// Defer the close (not a statement after runChatTurn) so the goroutine
|
||||
// exits even if runChatTurn panics — net/http recovers handler panics, so
|
||||
// a non-deferred close would be skipped and the ticker would keep writing
|
||||
// to a dead ResponseWriter forever.
|
||||
defer close(keepDone)
|
||||
a.runChatTurn(pctx, ctx, sessionID, req.Message, func(ev agentEvent) {
|
||||
writeEvent(ev)
|
||||
})
|
||||
|
||||
// B.6: if the turn ended with no text and no tool calls (the model
|
||||
// empty-response'd and all retries failed), delete the placeholder row
|
||||
// instead of persisting an empty bubble. The error event was already
|
||||
// streamed to the frontend via the 'done with error=true' event, so the
|
||||
// operator sees the error inline — an empty assistant bubble in the
|
||||
// transcript adds nothing and looks like the agent is broken.
|
||||
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
||||
st.deleteMessage(pctx, msgID)
|
||||
} else {
|
||||
persist() // final state — same row, updated one last time with the concluding text
|
||||
}
|
||||
|
||||
// Generate a meaningful title from the assistant's first answer
|
||||
// instead of reusing the raw user message for every session.
|
||||
// P2.9 (2026-07-20): prefer the goal as the title when one is set —
|
||||
// the first assistant text is often a greeting or narrative that
|
||||
// doesn't describe the task ("Hey! 👋 Nomos here, running on
|
||||
// mac-mini:8092..."). The goal is the operator's actual intent.
|
||||
// Sessions that never call set_goal (pure Q&A) fall back to the
|
||||
// assistant text, which is still better than the raw user message.
|
||||
if finalText != "" && sessionID != "ephemeral" {
|
||||
var goalTitle string
|
||||
if sess, gerr := st.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
||||
goalTitle = truncate(sess.Goal, 120)
|
||||
}
|
||||
title := goalTitle
|
||||
if title == "" {
|
||||
title = truncate(finalText, 80)
|
||||
}
|
||||
if title != "" {
|
||||
st.updateSessionTitle(pctx, sessionID, title)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
||||
@@ -483,7 +599,8 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
||||
if len(parts) == 2 && r.Method == http.MethodGet {
|
||||
switch parts[1] {
|
||||
case "plan":
|
||||
steps, err := st.getPlanSteps(r.Context(), id)
|
||||
all := r.URL.Query().Has("all") && r.URL.Query().Get("all") != "0" && r.URL.Query().Get("all") != "false"
|
||||
steps, err := st.getPlanSteps(r.Context(), id, all)
|
||||
if err != nil {
|
||||
http.Error(w, err.Error(), 500)
|
||||
return
|
||||
@@ -701,7 +818,7 @@ func newMCPClient(baseURL, token string) (*mcpClient, error) {
|
||||
c := &mcpClient{
|
||||
baseURL: baseURL,
|
||||
token: token,
|
||||
http: &http.Client{Timeout: 30 * time.Second},
|
||||
http: &http.Client{Timeout: 120 * time.Second},
|
||||
}
|
||||
|
||||
resp, err := c.doRequest("initialize", map[string]any{
|
||||
|
||||
82
cmd/nomos/messagequeue.go
Normal file
82
cmd/nomos/messagequeue.go
Normal file
@@ -0,0 +1,82 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
"sync"
|
||||
)
|
||||
|
||||
// maxQueuedPerSession caps a session's queue. A held turn plus unbounded
|
||||
// enqueues would grow memory without limit; an operator nudging a long
|
||||
// autonomous turn realistically queues only a handful, so a generous cap is
|
||||
// pure insurance. Overflow drops the newest enqueue and logs (the message is
|
||||
// already persisted in the DB by handleChat before enqueue, so it isn't lost
|
||||
// from the transcript — it just won't auto-run).
|
||||
const maxQueuedPerSession = 20
|
||||
|
||||
// messageQueue holds operator messages that arrived while a turn was already
|
||||
// running for a session. Plan 2026-08-03 (F2): instead of rejecting the
|
||||
// operator's message with "Nomos is still finishing a previous step… send it
|
||||
// again", the message is queued and auto-run when the in-flight turn releases
|
||||
// the session's turn-gate permit.
|
||||
//
|
||||
// The queue only schedules WHEN a turn runs, not WHETHER the message is stored
|
||||
// — handleChat persists the user message before acquiring the gate, so a queued
|
||||
// message is already in the transcript; this just makes sure a turn eventually
|
||||
// acts on it.
|
||||
//
|
||||
// Draining is strictly one-at-a-time under the turn gate (see drainQueued in
|
||||
// main.go), so this cannot stack concurrent turns — the exact hazard the gate
|
||||
// itself exists to prevent. Background resumeSession callers never touch this
|
||||
// queue; they keep their non-blocking skip.
|
||||
type messageQueue struct {
|
||||
mu sync.Mutex
|
||||
queue map[string][]string
|
||||
}
|
||||
|
||||
func newMessageQueue() *messageQueue {
|
||||
return &messageQueue{queue: map[string][]string{}}
|
||||
}
|
||||
|
||||
// enqueue appends a message to the back of the session's FIFO. Returns false
|
||||
// (and logs) if the session is already at maxQueuedPerSession — the caller's
|
||||
// message is already persisted in the DB, so this only skips auto-running it.
|
||||
func (q *messageQueue) enqueue(sessionID, msg string) bool {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
if len(q.queue[sessionID]) >= maxQueuedPerSession {
|
||||
slog.Warn("nomos: message queue full; dropping auto-run for operator message", "session", sessionID, "cap", maxQueuedPerSession)
|
||||
return false
|
||||
}
|
||||
q.queue[sessionID] = append(q.queue[sessionID], msg)
|
||||
return true
|
||||
}
|
||||
|
||||
// dequeue pops the next message from the front of the session's FIFO. Returns
|
||||
// ok=false when empty.
|
||||
func (q *messageQueue) dequeue(sessionID string) (string, bool) {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
xs := q.queue[sessionID]
|
||||
if len(xs) == 0 {
|
||||
return "", false
|
||||
}
|
||||
m := xs[0]
|
||||
q.queue[sessionID] = xs[1:]
|
||||
return m, true
|
||||
}
|
||||
|
||||
// requeueFront pushes a message back to the front — used when a drainer popped
|
||||
// a message but lost the race for the gate to a live turn; that turn's own
|
||||
// release will drain it again.
|
||||
func (q *messageQueue) requeueFront(sessionID, msg string) {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
q.queue[sessionID] = append([]string{msg}, q.queue[sessionID]...)
|
||||
}
|
||||
|
||||
// peek reports the queued depth for a session (test/diagnostic helper).
|
||||
func (q *messageQueue) peek(sessionID string) int {
|
||||
q.mu.Lock()
|
||||
defer q.mu.Unlock()
|
||||
return len(q.queue[sessionID])
|
||||
}
|
||||
142
cmd/nomos/messagequeue_test.go
Normal file
142
cmd/nomos/messagequeue_test.go
Normal file
@@ -0,0 +1,142 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestMessageQueue_FIFO(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
q.enqueue("s", "first")
|
||||
q.enqueue("s", "second")
|
||||
q.enqueue("s", "third")
|
||||
|
||||
want := []string{"first", "second", "third"}
|
||||
for _, w := range want {
|
||||
got, ok := q.dequeue("s")
|
||||
if !ok || got != w {
|
||||
t.Fatalf("dequeue = %q,%v want %q,true", got, ok, w)
|
||||
}
|
||||
}
|
||||
if _, ok := q.dequeue("s"); ok {
|
||||
t.Fatal("dequeue on drained queue should return ok=false")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_RequeueFront(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
q.enqueue("s", "a")
|
||||
q.enqueue("s", "b")
|
||||
// Pop "a", then push it back to the front; "a" must come out before "b".
|
||||
a, _ := q.dequeue("s")
|
||||
q.requeueFront("s", a)
|
||||
got, _ := q.dequeue("s")
|
||||
if got != "a" {
|
||||
t.Fatalf("after requeueFront, dequeue = %q want %q", got, "a")
|
||||
}
|
||||
got2, _ := q.dequeue("s")
|
||||
if got2 != "b" {
|
||||
t.Fatalf("next dequeue = %q want %q", got2, "b")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_IsolatedPerSession(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
q.enqueue("s1", "one")
|
||||
q.enqueue("s2", "two")
|
||||
if got, _ := q.dequeue("s1"); got != "one" {
|
||||
t.Fatalf("s1 = %q want one", got)
|
||||
}
|
||||
if got, _ := q.dequeue("s2"); got != "two" {
|
||||
t.Fatalf("s2 = %q want two", got)
|
||||
}
|
||||
if q.peek("s1") != 0 || q.peek("s2") != 0 {
|
||||
t.Fatal("both sessions should be drained")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_Concurrent(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
const n = maxQueuedPerSession // stay under the cap so every enqueue lands
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < n; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
q.enqueue("s", "m")
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
if q.peek("s") != n {
|
||||
t.Fatalf("peek = %d want %d (all enqueues must be counted)", q.peek("s"), n)
|
||||
}
|
||||
seen := 0
|
||||
for {
|
||||
if _, ok := q.dequeue("s"); !ok {
|
||||
break
|
||||
}
|
||||
seen++
|
||||
}
|
||||
if seen != n {
|
||||
t.Fatalf("drained %d want %d", seen, n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageQueue_CapsOverflow(t *testing.T) {
|
||||
q := newMessageQueue()
|
||||
for i := 0; i < maxQueuedPerSession; i++ {
|
||||
if !q.enqueue("s", "m") {
|
||||
t.Fatalf("enqueue #%d within cap should succeed", i)
|
||||
}
|
||||
}
|
||||
if q.enqueue("s", "overflow") {
|
||||
t.Fatal("enqueue past the cap should return false (dropped)")
|
||||
}
|
||||
if got := q.peek("s"); got != maxQueuedPerSession {
|
||||
t.Fatalf("peek = %d want %d (overflow must not append)", got, maxQueuedPerSession)
|
||||
}
|
||||
}
|
||||
|
||||
// drainQueued on an empty queue must be a no-op: it returns immediately and
|
||||
// never touches the gate (so the session stays free for the next turn).
|
||||
func TestDrainQueued_NoOpOnEmpty(t *testing.T) {
|
||||
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||
a.drainQueued(context.Background(), "s")
|
||||
if !a.gate.acquire("s", 0) {
|
||||
t.Fatal("gate should be free after a no-op drain (drain must not hold it)")
|
||||
}
|
||||
a.gate.release("s")
|
||||
}
|
||||
|
||||
// With a queued message but the gate held by another turn, drainQueued must
|
||||
// re-queue the message and return WITHOUT running a turn (no store/provider → a
|
||||
// real run would panic). This is the "never stack" property: a busy gate
|
||||
// defers to the holder's own release-drain.
|
||||
func TestDrainQueued_RequeuesWhenBusy(t *testing.T) {
|
||||
prev := drainAcquireWait
|
||||
drainAcquireWait = 10 * time.Millisecond
|
||||
t.Cleanup(func() { drainAcquireWait = prev })
|
||||
|
||||
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||
if !a.gate.acquire("s", 0) {
|
||||
t.Fatal("precondition: hold the gate")
|
||||
}
|
||||
a.queue.enqueue("s", "queued-msg")
|
||||
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
a.drainQueued(context.Background(), "s") // must not panic; must requeue
|
||||
close(done)
|
||||
}()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("drainQueued did not return promptly while the gate was busy")
|
||||
}
|
||||
if got := a.queue.peek("s"); got != 1 {
|
||||
t.Fatalf("message should be re-queued while busy; peek = %d want 1", got)
|
||||
}
|
||||
a.gate.release("s")
|
||||
}
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
@@ -25,6 +26,13 @@ const maxToolResultSize = 4096
|
||||
// The caller translates this into a directive tool result.
|
||||
var errPlanInFlight = errors.New("plan already in flight")
|
||||
|
||||
// errPlanStepNotFound is returned by updatePlanStep when no step matches the
|
||||
// given seq in the CURRENT (MAX) generation — either the seq is out of range,
|
||||
// or (after a re-plan) the model addressed a stale 1-based number. seq is
|
||||
// generation-relative, so this never resurrects a superseded generation's row.
|
||||
// The caller translates it into a directive tool result (P0.1).
|
||||
var errPlanStepNotFound = errors.New("plan step not found in current generation")
|
||||
|
||||
type store struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
@@ -799,10 +807,11 @@ func (s *store) setGoal(ctx context.Context, sessionID, goal string) error {
|
||||
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
|
||||
// The rows are kept for the generation counter + audit trail; proposePlan
|
||||
// excludes `replaced` from its in-flight check, so the next propose_plan
|
||||
// takes the fresh-generation path.
|
||||
// takes the fresh-generation path. replaced_reason records the cause
|
||||
// (2026-08-04 plan-step integrity audit).
|
||||
s.pool.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID)
|
||||
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID, "goal superseded")
|
||||
if _, err := s.pool.Exec(ctx,
|
||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
|
||||
sessionID, goal); err != nil {
|
||||
@@ -842,6 +851,14 @@ func (s *store) reopenSession(ctx context.Context, sessionID string) bool {
|
||||
s.pool.Exec(ctx,
|
||||
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
|
||||
sessionID)
|
||||
// Mark the prior plan's steps as replaced so the P1 plan-first gate in
|
||||
// classifyAndGate forces a fresh propose_plan before any run. Without
|
||||
// this, the agent could resume a session and call run against the old
|
||||
// (completed) plan — exactly what caused the ZimaOS continuation to
|
||||
// have 81 ad-hoc tool calls with zero plan structure (2026-08-04).
|
||||
s.pool.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID, "session reopened — awaiting new plan")
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
|
||||
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
|
||||
return true
|
||||
@@ -879,16 +896,15 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
|
||||
var startSeq int
|
||||
var anyStarted bool
|
||||
// `replaced` steps (from a prior plan generation superseded by a
|
||||
// follow-up sub-task — see reopenSession) are excluded: they prove a
|
||||
// prior plan was completed and superseded, not that a plan is in flight.
|
||||
// Without this exclusion, reopenSession's `replaced` marking would be
|
||||
// follow-up sub-task — see setGoal/reopenSession) are excluded: they
|
||||
// prove a prior plan was completed and superseded, not that a plan is in
|
||||
// flight. Without this exclusion, setGoal's `replaced` marking would be
|
||||
// useless — propose_plan would still refuse on the follow-up.
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT COALESCE(max(seq), 0), COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&startSeq, &anyStarted); err != nil {
|
||||
SELECT COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&anyStarted); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if anyStarted {
|
||||
@@ -898,35 +914,29 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
return nil, errPlanInFlight
|
||||
}
|
||||
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
||||
// This preserves the rows for the generation counter (MAX(generation)+1
|
||||
// below) and the plan_generations eval assertion. Without this, a first
|
||||
// plan that was proposed but never executed (all pending) would be
|
||||
// wiped, resetting the counter to 1 — making a follow-up's plan look
|
||||
// like generation 1 instead of 2. `replaced` steps are excluded from
|
||||
// the anyStarted check above, so they don't block the fresh proposal.
|
||||
// The rows are kept for the generation counter (MAX(generation)+1 below)
|
||||
// and the plan_generations eval assertion. `replaced` steps are excluded
|
||||
// from the anyStarted check above, so they don't block this proposal.
|
||||
// replaced_reason records the cause — required by the plan-step integrity
|
||||
// gate (2026-08-04 session audit).
|
||||
if _, err := tx.Exec(ctx,
|
||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||
sessionID); err != nil {
|
||||
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||
sessionID, "superseded by new plan generation"); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// startSeq keeps the max(seq) from the query above: if prior steps
|
||||
// exist (replaced or done), the new generation's steps start after them
|
||||
// (no seq collisions across generations). If no rows exist (first plan),
|
||||
// startSeq is 0 and the first step is seq 1.
|
||||
|
||||
// Resolve the generation number for this plan. Generation 1 is the
|
||||
// initial plan; a genuine revise (which currently goes through the same
|
||||
// fresh-start path above because all steps were pending) resets to 1
|
||||
// since the DELETE wiped the prior rows. The column is wired here so a
|
||||
// future explicit mid-flight revise path can increment it.
|
||||
// nextGen: generation 1 for the first plan, MAX(generation)+1 for every
|
||||
// revise/follow-up (prior rows were marked `replaced` above, not deleted,
|
||||
// so the counter survives). seq is generation-relative — it resets to
|
||||
// 1..N for this generation, so (session_id, generation, seq) is the
|
||||
// addressing key and the model's 1-based update_plan_step always maps to
|
||||
// the CURRENT plan after a re-plan (P0.1).
|
||||
var nextGen int
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT COALESCE(MAX(generation), 0) + 1
|
||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// After the DELETE above, no rows remain, so MAX(generation) is NULL →
|
||||
// nextGen = 1. (Keep the query for the future revise path; it's cheap.)
|
||||
|
||||
out := make([]map[string]any, 0, len(steps))
|
||||
for i, st := range steps {
|
||||
@@ -934,7 +944,7 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
if st.TargetSlug != "" {
|
||||
targetSlug = &st.TargetSlug
|
||||
}
|
||||
seq := startSeq + i + 1
|
||||
seq := i + 1
|
||||
var id uuid.UUID
|
||||
if err := tx.QueryRow(ctx, `
|
||||
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
||||
@@ -973,10 +983,27 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
||||
// be marked complete while an earlier step is still pending, preventing the
|
||||
// agent from marking step 5 done before step 4 (observed in production: the
|
||||
// agent rushed to close all steps in a final turn, in reverse order).
|
||||
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID string) error {
|
||||
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID, replacedReason string) error {
|
||||
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
||||
return nil
|
||||
}
|
||||
// Resolve the CURRENT generation: seq is generation-relative (1-based
|
||||
// within the plan the model is working), so (session_id, MAX(generation),
|
||||
// seq) is the addressing key. A re-plan's superseded generations have
|
||||
// their own seq space and must never be touched by a follow-up's
|
||||
// update_plan_step — that was the root cause of "the plan was off"
|
||||
// (gen-1 `replaced` rows resurrected as `done` while gen-2 work went
|
||||
// unrecorded). The MAX(generation) step is by construction the active
|
||||
// plan, never `replaced`, so this can't resurrect a superseded row (P0.1).
|
||||
var curGen int
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(MAX(generation), 0) FROM session_plan_steps WHERE session_id = $1`,
|
||||
sessionID).Scan(&curGen); err != nil {
|
||||
return err
|
||||
}
|
||||
if curGen == 0 {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
stamp := ""
|
||||
switch status {
|
||||
case "running":
|
||||
@@ -984,16 +1011,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
case "done", "failed", "skipped", "blocked", "replaced":
|
||||
stamp = ", finished_at = now()"
|
||||
}
|
||||
// Completion ordering: for terminal states, check that no earlier step
|
||||
// is still pending. Running steps can start out of order (the agent
|
||||
// may dispatch parallel work), but completion must be sequential.
|
||||
// Completion ordering, scoped to the CURRENT generation: for terminal
|
||||
// states, no earlier step in THIS plan may still be pending. Running
|
||||
// steps can start out of order (the agent may dispatch parallel work),
|
||||
// but completion must be sequential. Earlier generations are superseded
|
||||
// and irrelevant.
|
||||
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
||||
var blockedBy int
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
SELECT COALESCE(MIN(seq), 0)
|
||||
FROM session_plan_steps
|
||||
WHERE session_id = $1 AND seq < $2 AND status = 'pending'`,
|
||||
sessionID, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||
WHERE session_id = $1 AND generation = $2 AND seq < $3 AND status = 'pending'`,
|
||||
sessionID, curGen, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
||||
}
|
||||
}
|
||||
@@ -1004,12 +1033,33 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
||||
var stepID uuid.UUID
|
||||
var targetSlug *string
|
||||
// stamp is a fixed literal from the switch above — never user input.
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $3, execution_id = COALESCE($4, execution_id)`+stamp+`
|
||||
WHERE session_id = $1 AND seq = $2
|
||||
RETURNING id, target_slug`, sessionID, seq, status, execPtr).Scan(&stepID, &targetSlug); err != nil {
|
||||
return err
|
||||
// status <> 'replaced' is defense-in-depth: MAX(generation) can't hold a
|
||||
// replaced row, but if it ever could, this refuses the write instead of
|
||||
// resurrecting it. No matching row → errPlanStepNotFound (stale/out-of-range seq).
|
||||
if status == "replaced" && replacedReason != "" {
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $4, execution_id = COALESCE($5, execution_id), replaced_reason = $6`+stamp+`
|
||||
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr, replacedReason).Scan(&stepID, &targetSlug)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
} else {
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $4, execution_id = COALESCE($5, execution_id)`+stamp+`
|
||||
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr).Scan(&stepID, &targetSlug)
|
||||
if err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return errPlanStepNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
// Anchor the event to the step's target entity when it has one, else the task.
|
||||
entPtr := s.taskEntityPtr(ctx, sessionID)
|
||||
@@ -1098,13 +1148,58 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
if outcome != "success" {
|
||||
closeStatus = "skipped"
|
||||
}
|
||||
// Auto-close only the CURRENT generation's in-flight steps — superseded
|
||||
// generations were already resolved when their plan was replaced. Stamp
|
||||
// started_at so no `done` step is left with a NULL start time (P0.1 fix
|
||||
// 5), and emit a plan.step.finished event per closed step so the panel
|
||||
// converges instead of freezing on "running" after the task completes
|
||||
// (P1.1: no bulk plan-step status write without a corresponding event).
|
||||
type closingStep struct {
|
||||
id uuid.UUID
|
||||
seq int
|
||||
targetSlug *string
|
||||
}
|
||||
var toClose []closingStep
|
||||
if rows, qerr := s.pool.Query(ctx, `
|
||||
SELECT id, seq, target_slug FROM session_plan_steps
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status IN ('pending', 'running')`, sessionID); qerr == nil {
|
||||
for rows.Next() {
|
||||
var cs closingStep
|
||||
if err := rows.Scan(&cs.id, &cs.seq, &cs.targetSlug); err == nil {
|
||||
toClose = append(toClose, cs)
|
||||
}
|
||||
}
|
||||
rows.Close()
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
UPDATE session_plan_steps
|
||||
SET status = $2, finished_at = COALESCE(finished_at, now())
|
||||
WHERE session_id = $1 AND status IN ('pending', 'running')`,
|
||||
SET status = $2,
|
||||
started_at = COALESCE(started_at, now()),
|
||||
finished_at = COALESCE(finished_at, now())
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status IN ('pending', 'running')`,
|
||||
sessionID, closeStatus); err != nil {
|
||||
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
|
||||
}
|
||||
// Emit one plan.step.finished per closed step so the live panel advances
|
||||
// (mirrors updatePlanStep's event). A bulk UPDATE that skips the event
|
||||
// bus guarantees a stale panel — the rule is: no plan-step status change
|
||||
// without a corresponding event.
|
||||
taskEnt := s.taskEntityPtr(ctx, sessionID)
|
||||
for _, cs := range toClose {
|
||||
evEnt := taskEnt
|
||||
if cs.targetSlug != nil && *cs.targetSlug != "" {
|
||||
var tid uuid.UUID
|
||||
if s.pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, *cs.targetSlug).Scan(&tid) == nil {
|
||||
evEnt = &tid
|
||||
}
|
||||
}
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "plan.step.finished", evEnt, "info", "nomos", sessionID,
|
||||
map[string]any{"step_id": cs.id.String(), "seq": cs.seq, "status": closeStatus})
|
||||
}
|
||||
|
||||
// Clean up assent and destructive window keys from autonomy_settings.
|
||||
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
||||
@@ -1148,9 +1243,141 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
||||
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
||||
"cancelled_executions": cancelledCount, "blocker": blocker})
|
||||
// Auto-persist knowledge so the graph learns from this session regardless
|
||||
// of whether the agent remembered to call upsert_knowledge (2026-08-04
|
||||
// session audit: only 2.4% of sessions called upsert_knowledge manually).
|
||||
if outcome == "success" || outcome == "partial" {
|
||||
autoUpsertKnowledge(ctx, s, sessionID, outcome, summary)
|
||||
}
|
||||
// Plan quality metric: compute step completion rate for the session's
|
||||
// current plan generation. Tracked as a task attribute so the trend
|
||||
// can be monitored over time (2026-08-04 session audit: 38% baseline).
|
||||
writePlanCompletionRate(ctx, s, sessionID)
|
||||
// Auto-feedback: create a feedback entry linking the session's outcome
|
||||
// to its last execution, feeding the pattern-extraction pipeline that
|
||||
// has been empty since launch (2026-08-04 session audit: 0 feedback rows).
|
||||
if outcome == "success" || outcome == "partial" {
|
||||
autoFeedback(ctx, s, sessionID, outcome, summary)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// autoUpsertKnowledge creates a knowledge entry for a completed session,
|
||||
// capturing what was done and linking it to the entities involved. Called
|
||||
// automatically from completeTask so every session leaves a trace, even if
|
||||
// the agent forgot to call upsert_knowledge. Only fired for success/partial
|
||||
// outcomes (failures don't have actionable discoveries).
|
||||
func autoUpsertKnowledge(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||
var goal string
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||
sessionID).Scan(&goal); err != nil || goal == "" {
|
||||
return
|
||||
}
|
||||
title := "Session " + sessionID[:8] + ": " + goal
|
||||
if len(title) > 200 {
|
||||
title = title[:200]
|
||||
}
|
||||
content := "## Outcome\n" + outcome + "\n\n## Summary\n" + summary
|
||||
kind := "investigation"
|
||||
slug := "investigation:nomos/" + sessionID
|
||||
tags := []string{"nomos-session", "auto-generated"}
|
||||
|
||||
// Upsert the knowledge entity.
|
||||
docID, _ := uuid.NewV7()
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
INSERT INTO entities (id, slug, type, name, attributes)
|
||||
VALUES ($1, $2, $3, $4, '{}')
|
||||
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||
slog.Warn("nomos: autoUpsertKnowledge entity insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Upsert the knowledge content.
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO knowledge_entities (entity_id, title, content, source, tags, updated_at)
|
||||
VALUES ($1, $2, $3, 'nomos-agent', $4, now())
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||
tags = EXCLUDED.tags, updated_at = now()`,
|
||||
docID, title, content, tags); err != nil {
|
||||
slog.Warn("nomos: autoUpsertKnowledge content insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
// Link to the task entity.
|
||||
var taskEntID uuid.UUID
|
||||
if s.pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`,
|
||||
sessionID).Scan(&taskEntID) == nil && taskEntID != uuid.Nil {
|
||||
s.pool.Exec(ctx, `
|
||||
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||
SELECT $1, $2, 'involves', '{"by":"nomos","auto":true}'::jsonb, now()
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM relationships
|
||||
WHERE source_id = $1 AND target_id = $2 AND type = 'involves' AND valid_to IS NULL)`,
|
||||
taskEntID, docID)
|
||||
}
|
||||
|
||||
slog.Info("nomos: auto-upserted knowledge for session",
|
||||
"session", sessionID, "outcome", outcome, "slug", slug)
|
||||
}
|
||||
|
||||
// writePlanCompletionRate computes the step completion rate for the current
|
||||
// plan generation and writes it as a task entity attribute so the trend can
|
||||
// be tracked. Baseline from 2026-08-04 audit: 38% (15/39 steps reached done).
|
||||
func writePlanCompletionRate(ctx context.Context, s *store, sessionID string) {
|
||||
var total, completed int
|
||||
s.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*), COALESCE(SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END), 0)
|
||||
FROM session_plan_steps
|
||||
WHERE session_id = $1
|
||||
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||
AND status <> 'replaced'`, sessionID).Scan(&total, &completed)
|
||||
if total > 0 {
|
||||
rate := float64(completed) / float64(total)
|
||||
attrs, _ := json.Marshal(map[string]any{"plan_completion_rate": rate, "plan_steps_total": total, "plan_steps_completed": completed})
|
||||
s.pool.Exec(ctx, `
|
||||
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
|
||||
WHERE id = (SELECT entity_id FROM agent_sessions WHERE id = $1)`,
|
||||
sessionID, string(attrs))
|
||||
slog.Info("nomos: plan completion rate", "session", sessionID, "rate", fmt.Sprintf("%.0f%%", rate*100),
|
||||
"completed", completed, "total", total)
|
||||
}
|
||||
}
|
||||
|
||||
// autoFeedback creates a feedback entry linking the session's outcome to its
|
||||
// last execution, feeding the pattern-extraction pipeline that has been empty
|
||||
// since launch. Only created for success/partial outcomes (failures don't
|
||||
// have a specific execution to tie to).
|
||||
func autoFeedback(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||
// Find the last execution linked to this session.
|
||||
var execID uuid.UUID
|
||||
if err := s.pool.QueryRow(ctx, `
|
||||
SELECT pe.execution_id FROM nomos_plan_executions pe
|
||||
WHERE pe.session_id = $1::uuid
|
||||
ORDER BY pe.created_at DESC LIMIT 1`, sessionID).Scan(&execID); err != nil || execID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
fbID, _ := uuid.NewV7()
|
||||
slug := "feedback:" + fbID.String()
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'feedback', $3, '{}')`,
|
||||
fbID, slug, "feedback for "+sessionID[:8]); err != nil {
|
||||
slog.Warn("nomos: autoFeedback entity insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO feedback (entity_id, execution_id, outcome, observation, lesson, tags, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, now())`,
|
||||
fbID, execID, outcome, summary, summary, []string{"nomos-session", "auto-generated", "session:" + sessionID[:8]})
|
||||
if err != nil {
|
||||
slog.Warn("nomos: autoFeedback insert", "session", sessionID, "error", err)
|
||||
return
|
||||
}
|
||||
slog.Info("nomos: auto-feedback created for session", "session", sessionID, "outcome", outcome)
|
||||
}
|
||||
|
||||
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
|
||||
// reason. Order matters — earlier patterns take precedence. These are the
|
||||
// recurring failure signatures from the 2026-07-20 session audit. A
|
||||
@@ -1240,6 +1467,53 @@ func (s *store) hadDiscovery(ctx context.Context, sessionID string) bool {
|
||||
return count > 0
|
||||
}
|
||||
|
||||
// sessionGoal returns the session's goal text, empty string if not found.
|
||||
// Used by complete_task to check whether the goal involved a reachability
|
||||
// verification before marking success.
|
||||
func (s *store) sessionGoal(ctx context.Context, sessionID string) string {
|
||||
if s == nil || sessionID == "" {
|
||||
return ""
|
||||
}
|
||||
var goal string
|
||||
s.pool.QueryRow(ctx,
|
||||
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||
sessionID).Scan(&goal)
|
||||
return goal
|
||||
}
|
||||
|
||||
// hadRecentVerification checks whether the session successfully verified
|
||||
// reachability in recent turns — ping_service, or a run with curl/wget that
|
||||
// returned successfully. Used by complete_task as a soft warning when the
|
||||
// goal involved a reachability check but no recent verification occurred.
|
||||
func (s *store) hadRecentVerification(ctx context.Context, sessionID string) bool {
|
||||
if s == nil || sessionID == "" {
|
||||
return true // fail safe: don't warn when we can't check
|
||||
}
|
||||
// Check for ping_service calls in the last 5 activity entries for this session.
|
||||
var pingCount int
|
||||
s.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT 1 FROM agent_activity
|
||||
WHERE session_id = $1 AND tool_name = 'ping_service' AND success = true
|
||||
ORDER BY ts DESC LIMIT 5
|
||||
) sub`, sessionID).Scan(&pingCount)
|
||||
if pingCount > 0 {
|
||||
return true
|
||||
}
|
||||
// Check for run calls with curl/wget that returned successfully.
|
||||
var curlCount int
|
||||
s.pool.QueryRow(ctx, `
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT 1 FROM agent_activity
|
||||
WHERE session_id = $1
|
||||
AND tool_name = 'run'
|
||||
AND success = true
|
||||
AND (input_summary LIKE '%curl%' OR input_summary LIKE '%wget%')
|
||||
ORDER BY ts DESC LIMIT 10
|
||||
) sub`, sessionID).Scan(&curlCount)
|
||||
return curlCount > 0
|
||||
}
|
||||
|
||||
// staleGoalSession is a goal-bearing task that's gone idle without reaching
|
||||
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
|
||||
// plans/2026-07-11-task-completion-safety-net.md).
|
||||
@@ -1347,15 +1621,23 @@ type planStep struct {
|
||||
|
||||
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
||||
// panel when it first opens a task (live events only carry deltas from then on).
|
||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string) ([]planStep, error) {
|
||||
// By default only the CURRENT (MAX) generation is returned — the panel shows the
|
||||
// live plan, not an archaeological record of every superseded generation. Pass
|
||||
// all=true for the audit/eval view that needs every generation (the
|
||||
// plan_generations assertion counts distinct generations across the full set).
|
||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string, all bool) ([]planStep, error) {
|
||||
if s == nil {
|
||||
return nil, nil
|
||||
}
|
||||
genFilter := ""
|
||||
if !all {
|
||||
genFilter = "AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)"
|
||||
}
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id::text, seq, title, detail, status,
|
||||
execution_id::text, target_slug,
|
||||
started_at::text, finished_at::text, generation
|
||||
FROM session_plan_steps WHERE session_id = $1 ORDER BY seq`, sessionID)
|
||||
FROM session_plan_steps WHERE session_id = $1 `+genFilter+` ORDER BY generation, seq`, sessionID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -1835,7 +2117,7 @@ func (s *store) resolveArgEntityID(ctx context.Context, args map[string]any) uui
|
||||
// The (nullable) session_id column carries the conversation id. args is the
|
||||
// tool call's own arguments, used to best-effort tag the row with the
|
||||
// entity it acted on (see resolveArgEntityID).
|
||||
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string) {
|
||||
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string, tokenCount int) {
|
||||
if s == nil || agentID == uuid.Nil {
|
||||
return
|
||||
}
|
||||
@@ -1847,8 +2129,8 @@ func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, t
|
||||
s.pool.Exec(ctx, `
|
||||
INSERT INTO agent_activity
|
||||
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
|
||||
duration_ms, success, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
|
||||
duration_ms, success, correlation_id, token_count)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
|
||||
durationMs, success, correlationID)
|
||||
durationMs, success, correlationID, tokenCount)
|
||||
}
|
||||
|
||||
@@ -192,7 +192,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// Mark step 1 as started.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep: %v", err)
|
||||
}
|
||||
|
||||
@@ -205,7 +205,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// The original step 1 must be untouched — not erased, not appended to.
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID)
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID, false)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
@@ -217,7 +217,8 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
}
|
||||
|
||||
// Third call BEFORE anything runs on a fresh session: every step is
|
||||
// still pending, so this must REPLACE, not refuse.
|
||||
// still pending, so this must REPLACE (mark the prior plan `replaced`),
|
||||
// not refuse. The new plan becomes generation 2.
|
||||
sess2, err := s.createSession(ctx, "plan replace test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
@@ -228,15 +229,146 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
||||
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
||||
t.Fatalf("proposePlan (revise before execution): %v", err)
|
||||
}
|
||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID)
|
||||
// Default (current generation) view: only the revised step.
|
||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID, false)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
||||
t.Fatalf("got %+v, want a single 'Revised' step (pre-execution revise must replace, not refuse)", revisedSteps)
|
||||
t.Fatalf("got %+v, want a single 'Revised' step (current-generation view)", revisedSteps)
|
||||
}
|
||||
if revisedSteps[0].Generation != 1 {
|
||||
t.Fatalf("revised step generation = %d, want 1 (fresh-start after DELETE resets generation)", revisedSteps[0].Generation)
|
||||
if revisedSteps[0].Seq != 1 {
|
||||
t.Fatalf("revised step seq = %d, want 1 (seq is generation-relative, resets to 1..N)", revisedSteps[0].Seq)
|
||||
}
|
||||
if revisedSteps[0].Generation != 2 {
|
||||
t.Fatalf("revised step generation = %d, want 2 (prior pending plan is replaced, not deleted, so the counter increments)", revisedSteps[0].Generation)
|
||||
}
|
||||
// all=true audit view: both generations, the original marked `replaced`.
|
||||
allSteps, err := s.getPlanSteps(ctx, sess2.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps(all): %v", err)
|
||||
}
|
||||
if len(allSteps) != 2 {
|
||||
t.Fatalf("all=true got %d steps, want 2 (Original replaced gen1 + Revised gen2)", len(allSteps))
|
||||
}
|
||||
if allSteps[0].Title != "Original" || allSteps[0].Status != "replaced" || allSteps[0].Generation != 1 {
|
||||
t.Errorf("gen1 step = %+v, want Original/replaced/gen1", allSteps[0])
|
||||
}
|
||||
if allSteps[1].Title != "Revised" || allSteps[1].Generation != 2 || allSteps[1].Seq != 1 {
|
||||
t.Errorf("gen2 step = %+v, want Revised/gen2/seq1", allSteps[1])
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdatePlanStep_GenerationRelative is the P0.1 regression proof: after a
|
||||
// re-plan, update_plan_step(seq=N) — using the 1-based number the model
|
||||
// naturally carries — must address the CURRENT generation and never resurrect
|
||||
// a superseded generation's `replaced` row. Before the fix, seq was globally
|
||||
// increasing across generations, so seq=1 after a re-plan flipped the gen-1
|
||||
// `replaced` step back to `running`/`done` while the real gen-2 work went
|
||||
// unrecorded.
|
||||
func TestUpdatePlanStep_GenerationRelative(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
sess, err := s.createSession(ctx, "gen-relative seq test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
// Generation 1: two steps.
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||
t.Fatalf("proposePlan #1: %v", err)
|
||||
}
|
||||
// Re-plan: setGoal marks the gen-1 plan `replaced`, proposePlan starts gen 2.
|
||||
if err := s.setGoal(ctx, sess.ID, "follow-up sub-task"); err != nil {
|
||||
t.Fatalf("setGoal: %v", err)
|
||||
}
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "C"}, {Title: "D"}}); err != nil {
|
||||
t.Fatalf("proposePlan #2: %v", err)
|
||||
}
|
||||
|
||||
// The model addresses the new plan with 1-based seq. seq=1 must hit
|
||||
// gen-2 "C", leaving gen-1 "A" (replaced) untouched.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(seq=1, running): %v", err)
|
||||
}
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "done", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(seq=1, done): %v", err)
|
||||
}
|
||||
|
||||
all, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps(all): %v", err)
|
||||
}
|
||||
byTitle := map[string]planStep{}
|
||||
for _, st := range all {
|
||||
byTitle[st.Title] = st
|
||||
}
|
||||
// gen-1 steps stay `replaced` — NOT resurrected to running/done.
|
||||
if byTitle["A"].Status != "replaced" || byTitle["A"].Generation != 1 {
|
||||
t.Errorf("A = %+v, want replaced/gen1 (a superseded row must never be touched)", byTitle["A"])
|
||||
}
|
||||
if byTitle["B"].Status != "replaced" || byTitle["B"].Generation != 1 {
|
||||
t.Errorf("B = %+v, want replaced/gen1", byTitle["B"])
|
||||
}
|
||||
// gen-2 seq=1 advanced; seq=2 untouched.
|
||||
if byTitle["C"].Status != "done" || byTitle["C"].Generation != 2 || byTitle["C"].Seq != 1 {
|
||||
t.Errorf("C = %+v, want done/gen2/seq1 (the 1-based update must address the current generation)", byTitle["C"])
|
||||
}
|
||||
if byTitle["D"].Status != "pending" || byTitle["D"].Seq != 2 {
|
||||
t.Errorf("D = %+v, want pending/seq2", byTitle["D"])
|
||||
}
|
||||
|
||||
// Out-of-range seq must be refused (no current-gen step there).
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 99, "running", "", ""); !errors.Is(err, errPlanStepNotFound) {
|
||||
t.Fatalf("updatePlanStep(seq=99) err = %v, want errPlanStepNotFound", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCompleteTask_AutoCloseEmitsEvents is the P1.1 regression proof:
|
||||
// completeTask's bulk auto-close of in-flight steps must emit one
|
||||
// plan.step.finished event per closed step (so the live panel converges
|
||||
// instead of freezing on "running" after the task completes) and must stamp
|
||||
// started_at so no closed step is left un-timestamped (P0.1 fix 5).
|
||||
func TestCompleteTask_AutoCloseEmitsEvents(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
ctx := context.Background()
|
||||
|
||||
sess, err := s.createSession(ctx, "auto-close events test")
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||
t.Fatalf("proposePlan: %v", err)
|
||||
}
|
||||
// A is running, B still pending at completion time.
|
||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||
t.Fatalf("updatePlanStep(1, running): %v", err)
|
||||
}
|
||||
if err := s.completeTask(ctx, sess.ID, "success", "done"); err != nil {
|
||||
t.Fatalf("completeTask: %v", err)
|
||||
}
|
||||
|
||||
// Every auto-closed step should now carry both a started_at and a
|
||||
// finished_at (no NULL-started `done` step).
|
||||
steps, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||
if err != nil {
|
||||
t.Fatalf("getPlanSteps: %v", err)
|
||||
}
|
||||
for _, st := range steps {
|
||||
if st.Status == "done" && st.StartedAt == nil {
|
||||
t.Errorf("step %q done but started_at is NULL (P0.1 fix 5: stamp it)", st.Title)
|
||||
}
|
||||
}
|
||||
|
||||
// Exactly two plan.step.finished events — one per closed step (A and B).
|
||||
var finished int
|
||||
if err := s.pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM events WHERE type = 'plan.step.finished' AND correlation_id = $1`,
|
||||
sess.ID).Scan(&finished); err != nil {
|
||||
t.Fatalf("count events: %v", err)
|
||||
}
|
||||
if finished != 2 {
|
||||
t.Fatalf("plan.step.finished events = %d, want 2 (one per auto-closed step)", finished)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -265,7 +397,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
|
||||
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
|
||||
agentID := uuid.New()
|
||||
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1")
|
||||
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1", 0)
|
||||
if !s.hadDiscovery(ctx, sess.ID) {
|
||||
t.Fatal("hadDiscovery = false after a successful run call, want true")
|
||||
}
|
||||
@@ -278,7 +410,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2")
|
||||
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2", 0)
|
||||
if s.hadDiscovery(ctx, sess2.ID) {
|
||||
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
|
||||
}
|
||||
@@ -288,7 +420,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3")
|
||||
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3", 0)
|
||||
if s.hadDiscovery(ctx, sess3.ID) {
|
||||
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
|
||||
}
|
||||
@@ -298,12 +430,12 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("createSession: %v", err)
|
||||
}
|
||||
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4")
|
||||
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4", 0)
|
||||
if !s.hadEntityWriteback(ctx, sess4.ID) {
|
||||
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
|
||||
}
|
||||
// And the discovery+writeback combination (the conv3 scenario).
|
||||
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5")
|
||||
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5", 0)
|
||||
if !s.hadDiscovery(ctx, sess4.ID) {
|
||||
t.Fatal("hadDiscovery = false after run+writeback, want true")
|
||||
}
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -245,13 +246,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// the seq-order enforcement (5.6) require it to be completed last,
|
||||
// and D.1's complete_task gate enforces the actual calls. Together
|
||||
// they close the loop structurally — neither relies on the agent
|
||||
// reading SOUL.md.
|
||||
// reading SOUL.md. The match is broadened past the literal tool
|
||||
// names so a natural-language step ("Write back: update entity
|
||||
// attributes…") isn't doubled by an auto-appended duplicate (P1.2).
|
||||
hasWritebackStep := false
|
||||
for _, st := range steps {
|
||||
if strings.Contains(st.Title, "update_entity_attributes") ||
|
||||
strings.Contains(st.Title, "create_relationship") ||
|
||||
strings.Contains(st.Detail, "update_entity_attributes") ||
|
||||
strings.Contains(st.Detail, "create_relationship") {
|
||||
t := strings.ToLower(st.Title + " " + st.Detail)
|
||||
if strings.Contains(t, "update_entity_attributes") ||
|
||||
strings.Contains(t, "create_relationship") ||
|
||||
strings.Contains(t, "upsert_knowledge") ||
|
||||
strings.Contains(t, "write back") ||
|
||||
strings.Contains(t, "writeback") {
|
||||
hasWritebackStep = true
|
||||
break
|
||||
}
|
||||
@@ -280,8 +285,19 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
// The writeback step is now always present (D.2 auto-appends it if
|
||||
// the agent forgot), so the old advisory nudge is replaced by the
|
||||
// structural gate: D.1 refuses complete_task without the actual
|
||||
// update_entity_attributes/create_relationship calls.
|
||||
result := fmt.Sprintf("Plan set (%d steps)%s. If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), appendedNote)
|
||||
// update_entity_attributes/create_relationship calls. Enumerate the
|
||||
// step seqs so the model knows exactly which numbers to address with
|
||||
// update_plan_step (seq is 1-based within this plan — the addressing
|
||||
// key, not a global counter).
|
||||
var seqs strings.Builder
|
||||
for i, p := range persisted {
|
||||
if i > 0 {
|
||||
seqs.WriteString("; ")
|
||||
}
|
||||
title := fmt.Sprint(p["title"])
|
||||
fmt.Fprintf(&seqs, "%v=%s", p["seq"], title)
|
||||
}
|
||||
result := fmt.Sprintf("Plan set (%d steps): %s.%s Address them with update_plan_step(seq=N). If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), seqs.String(), appendedNote)
|
||||
return result, true
|
||||
|
||||
case "update_plan_step":
|
||||
@@ -291,7 +307,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
if seq <= 0 || status == "" {
|
||||
return "error: update_plan_step needs seq (>=1) and status", true
|
||||
}
|
||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID); err != nil {
|
||||
reason, _ := args["replaced_reason"].(string)
|
||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID, reason); err != nil {
|
||||
if errors.Is(err, errPlanStepNotFound) {
|
||||
// The seq doesn't address a step in the CURRENT plan — most
|
||||
// often a stale 1-based number the model carried across a
|
||||
// re-plan, or an out-of-range seq. seq is generation-relative
|
||||
// (1..N within the latest propose_plan), so a superseded
|
||||
// generation's row is never touched (P0.1 fix 3). Direct the
|
||||
// model instead of silently no-op'ing.
|
||||
return fmt.Sprintf("Step %d is not in the current plan. seq is 1-based within your latest propose_plan (a re-plan resets it to 1..N, so an old step number no longer applies). The plan was not changed. Re-address with the correct 1-based seq, or if you've lost track, re-read the plan.", seq), true
|
||||
}
|
||||
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
||||
}
|
||||
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
||||
@@ -349,6 +375,18 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
|
||||
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
|
||||
}
|
||||
// D.2: refuse success when the goal mentions a reachability/uptime
|
||||
// check but no verification was done. The agent can't claim "X is
|
||||
// reachable" based on a shell command alone — the proxy (Caddy) can
|
||||
// return 200 for a terminal page (ttyd) or fallback while the actual
|
||||
// dashboard is still down. Must call ping_service or run a successful
|
||||
// curl before claiming success.
|
||||
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) {
|
||||
goal := a.store.sessionGoal(ctx, sessionID)
|
||||
if mentionsReachability(goal) && !a.store.hadRecentVerification(ctx, sessionID) {
|
||||
return "Refused: the goal involves a reachability or uptime check (\"make X reachable\", \"get X up\", etc.), but no ping_service call or successful curl/HTTP request against the target was detected. Caddy can return 200 for a terminal or fallback page while the actual service is still down — you must verify the service itself, not just the proxy. Call ping_service(target) or run a curl against the actual service URL, then call complete_task again. Outcome held until verified.", true
|
||||
}
|
||||
}
|
||||
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
|
||||
if errors.Is(err, errTaskAlreadyComplete) {
|
||||
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
|
||||
@@ -365,6 +403,28 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
||||
}
|
||||
}
|
||||
|
||||
// reachabilityPatterns matches goal text that involves making something
|
||||
// reachable/accessible/working. Used by complete_task to surface a soft
|
||||
// warning when the session goal was about reachability but no verification
|
||||
// occurred before marking success.
|
||||
var reachabilityPatterns = []*regexp.Regexp{
|
||||
regexp.MustCompile(`(?i)https?://[^\s]+`),
|
||||
regexp.MustCompile(`(?i)\.hubris\.net\w+`),
|
||||
regexp.MustCompile(`(?i)(un)?reachable`),
|
||||
regexp.MustCompile(`(?i)(not?\s+)?(accessible|reachable|responding|resolving)`),
|
||||
regexp.MustCompile(`(?i)diagnose\s+why`),
|
||||
regexp.MustCompile(`(?i)(fix|restore|bring\s+back).*(accessible|reachable|online)`),
|
||||
}
|
||||
|
||||
func mentionsReachability(goal string) bool {
|
||||
for _, p := range reachabilityPatterns {
|
||||
if p.MatchString(goal) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// autoCompleteTrivialTask is the case-1 fix from
|
||||
// plans/2026-07-11-task-completion-safety-net.md: a session that never
|
||||
// called set_goal never framed itself as a structured task, so a turn that
|
||||
|
||||
90
cmd/nomos/turngate.go
Normal file
90
cmd/nomos/turngate.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// turnGate enforces at most one in-flight agent turn per session.
|
||||
//
|
||||
// Why this exists (plan 2026-08-03, F1): handleChat runs a turn in the HTTP
|
||||
// request goroutine, and every "resume" path (the empty-message reconnect,
|
||||
// the auto-continuation worker, the idle sweep, answer-question, the /resume
|
||||
// endpoint) launches ANOTHER goroutine running a full turn. Nothing prevented
|
||||
// two turns for the SAME session at once, so a network blip that triggered a
|
||||
// reconnect would spawn a duplicate resumeSession while the original turn was
|
||||
// still alive — their tool calls interleaved on the wire and in the persisted
|
||||
// transcript, which is the root cause behind the "parallel/nesting/sequence
|
||||
// is off" and "task didn't end / flaky" reports.
|
||||
//
|
||||
// Model: one permit (buffered-1 channel seeded with a single token) per
|
||||
// session id. Acquiring consumes the token; releasing puts it back.
|
||||
// - Background/best-effort callers (resumeSession and everything it backs)
|
||||
// use a non-blocking acquire and SKIP when busy — a duplicate nudge while a
|
||||
// turn is already running adds nothing, and the continuation/idle tickers
|
||||
// will retry on their own.
|
||||
// - The live chat path (an operator message) waits briefly for a finishing
|
||||
// background turn, then bails with an actionable error if still busy — see
|
||||
// handleChat.
|
||||
//
|
||||
// The permits map grows one entry per session id seen. For this single-agent
|
||||
// homelab process that set is small and bounded by real sessions; cleanup is
|
||||
// intentionally omitted (a sweep would race with acquire/release and the
|
||||
// memory is negligible).
|
||||
type turnGate struct {
|
||||
mu sync.Mutex
|
||||
permits map[string]chan struct{}
|
||||
}
|
||||
|
||||
func newTurnGate() *turnGate {
|
||||
return &turnGate{permits: make(map[string]chan struct{})}
|
||||
}
|
||||
|
||||
// permit returns the single token-channel for sessionID, creating and seeding
|
||||
// it on first use. Creation is guarded so two concurrent first-callers for the
|
||||
// same id share one channel.
|
||||
func (g *turnGate) permit(sessionID string) chan struct{} {
|
||||
g.mu.Lock()
|
||||
defer g.mu.Unlock()
|
||||
ch, ok := g.permits[sessionID]
|
||||
if !ok {
|
||||
ch = make(chan struct{}, 1)
|
||||
ch <- struct{}{}
|
||||
g.permits[sessionID] = ch
|
||||
}
|
||||
return ch
|
||||
}
|
||||
|
||||
// acquire takes the session's permit. With wait <= 0 it is non-blocking
|
||||
// (returns false immediately if a turn is active). With wait > 0 it blocks up
|
||||
// to wait for the permit, returning false on timeout. Every true return MUST
|
||||
// be paired with exactly one release.
|
||||
func (g *turnGate) acquire(sessionID string, wait time.Duration) bool {
|
||||
ch := g.permit(sessionID)
|
||||
if wait <= 0 {
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
t := time.NewTimer(wait)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ch:
|
||||
return true
|
||||
case <-t.C:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
// release returns the session's permit. Idempotent: a release with no matching
|
||||
// acquire (or a double release) is a no-op rather than a blocking send.
|
||||
func (g *turnGate) release(sessionID string) {
|
||||
ch := g.permit(sessionID)
|
||||
select {
|
||||
case ch <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
}
|
||||
114
cmd/nomos/turngate_test.go
Normal file
114
cmd/nomos/turngate_test.go
Normal file
@@ -0,0 +1,114 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTurnGate_NonBlockingSkipsWhenBusy(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("first non-blocking acquire should succeed on a free session")
|
||||
}
|
||||
// A second non-blocking acquire (a background resume) must skip, not queue.
|
||||
if g.acquire("s1", 0) {
|
||||
t.Fatal("second non-blocking acquire should fail while a turn is active")
|
||||
}
|
||||
// A different session is independent.
|
||||
if !g.acquire("s2", 0) {
|
||||
t.Fatal("acquire on a different session should succeed")
|
||||
}
|
||||
g.release("s2")
|
||||
g.release("s1")
|
||||
// After release, the session is free again.
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("acquire should succeed again after release")
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
func TestTurnGate_BlockingAcquireWaitsForRelease(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
if !g.acquire("s1", 0) {
|
||||
t.Fatal("first acquire should succeed")
|
||||
}
|
||||
|
||||
got := make(chan bool, 1)
|
||||
go func() { got <- g.acquire("s1", 2*time.Second) }()
|
||||
|
||||
select {
|
||||
case <-got:
|
||||
t.Fatal("blocking acquire should wait, not return before release")
|
||||
case <-time.After(50 * time.Millisecond):
|
||||
// expected: still waiting
|
||||
}
|
||||
|
||||
g.release("s1")
|
||||
select {
|
||||
case ok := <-got:
|
||||
if !ok {
|
||||
t.Fatal("blocking acquire should succeed after release")
|
||||
}
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("blocking acquire did not return after release")
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
func TestTurnGate_BlockingAcquireTimesOut(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
g.acquire("s1", 0) // hold the permit
|
||||
|
||||
start := time.Now()
|
||||
if g.acquire("s1", 60*time.Millisecond) {
|
||||
t.Fatal("acquire should time out while permit is held")
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed < 50*time.Millisecond {
|
||||
t.Fatalf("acquire returned too fast (%v); expected to wait ~60ms", elapsed)
|
||||
}
|
||||
g.release("s1")
|
||||
}
|
||||
|
||||
// TestTurnGate_SingleFlightConcurrent is the core F1 guarantee: many concurrent
|
||||
// background acquirers on the SAME session, exactly one runs at a time. This is
|
||||
// the property that prevents two turns interleaving tool calls.
|
||||
func TestTurnGate_SingleFlightConcurrent(t *testing.T) {
|
||||
g := newTurnGate()
|
||||
const n = 50
|
||||
var inFlight, maxInFlight int64
|
||||
var runs int64
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(n)
|
||||
start := make(chan struct{})
|
||||
for i := 0; i < n; i++ {
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
<-start
|
||||
if !g.acquire("shared", 0) { // background-style: skip if busy
|
||||
return
|
||||
}
|
||||
defer g.release("shared")
|
||||
cur := atomic.AddInt64(&inFlight, 1)
|
||||
for {
|
||||
m := atomic.LoadInt64(&maxInFlight)
|
||||
if cur <= m || atomic.CompareAndSwapInt64(&maxInFlight, m, cur) {
|
||||
break
|
||||
}
|
||||
}
|
||||
atomic.AddInt64(&runs, 1)
|
||||
time.Sleep(2 * time.Millisecond)
|
||||
atomic.AddInt64(&inFlight, -1)
|
||||
}()
|
||||
}
|
||||
close(start)
|
||||
wg.Wait()
|
||||
|
||||
if maxInFlight != 1 {
|
||||
t.Fatalf("max in-flight turns = %d, want 1 (turns must not overlap)", maxInFlight)
|
||||
}
|
||||
if runs == 0 {
|
||||
t.Fatal("expected at least one turn to run")
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,27 @@
|
||||
:80 {
|
||||
root * /srv
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
|
||||
# /wails/runtime.js is injected by the Wails desktop wrapper, which serves
|
||||
# the same dist/ from its own asset handler. In a browser it does not
|
||||
# exist, and the SPA fallback below answered it with index.html — so the
|
||||
# browser parsed "<!doctype html>" as JavaScript and threw
|
||||
# "SyntaxError: expected expression, got '<'" on every page load.
|
||||
# Return a real 404 instead: the tag fails quietly, and the desktop app is
|
||||
# unaffected because it never reaches this server.
|
||||
handle /wails/* {
|
||||
error 404
|
||||
}
|
||||
|
||||
# Same reasoning for any other asset: a missing .js/.css/.map answered with
|
||||
# HTML is always a confusing parse error rather than an honest 404. Only
|
||||
# real routes should fall through to the SPA.
|
||||
@asset path_regexp \.(js|mjs|css|map|json|png|jpg|svg|ico|woff2?)$
|
||||
handle @asset {
|
||||
file_server
|
||||
}
|
||||
|
||||
handle {
|
||||
file_server
|
||||
try_files {path} /index.html
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,7 +8,8 @@ FROM node:22-alpine AS builder
|
||||
|
||||
WORKDIR /build/web
|
||||
COPY web/package.json web/package-lock.json ./
|
||||
RUN npm ci
|
||||
COPY web/vendor /build/vendor
|
||||
RUN npm install --no-audit --no-fund
|
||||
COPY VERSION ./
|
||||
COPY web/ ./
|
||||
RUN npm run build
|
||||
|
||||
@@ -83,6 +83,18 @@ services:
|
||||
command: ["api"]
|
||||
stop_signal: SIGTERM
|
||||
stop_grace_period: 30s
|
||||
# Exists so nomos can wait for the API to actually answer rather than just
|
||||
# for its container to exist — see nomos's depends_on below. wget is
|
||||
# BusyBox's, already in the alpine runtime image, so this adds no
|
||||
# dependency. /healthz pings the DB, so "healthy" means genuinely ready.
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8090/healthz"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
# Migrations and seed run before this container, but the first bind can
|
||||
# still take a moment; failures inside the start period don't count.
|
||||
start_period: 10s
|
||||
|
||||
# Scheduler (Phase 3) — observe loop
|
||||
scheduler:
|
||||
@@ -139,7 +151,12 @@ services:
|
||||
profiles: ["full"]
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_started
|
||||
# service_started only waits for the container to exist, so nomos came
|
||||
# up while the API was still binding :8090, failed its MCP initialize,
|
||||
# exited 1, and crash-looped for ~25s on every single deploy. It always
|
||||
# recovered, which is exactly why it went unnoticed. service_healthy
|
||||
# waits for the API to actually answer.
|
||||
condition: service_healthy
|
||||
environment:
|
||||
NOMOS_MCP_URL: http://api:8090/mcp
|
||||
NOMOS_AGENT_SLUG: agent:nomos
|
||||
|
||||
@@ -234,9 +234,33 @@ sequenceDiagram
|
||||
|
||||
---
|
||||
|
||||
**2026-07-08 — renamed to Nomos.** The Hermes agent gateway was renamed to
|
||||
**2026-07-08 — renamed to Nomos.**
|
||||
Nomos (from *oikonomos*, the steward of the oikos) under the
|
||||
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
|
||||
|
||||
### Hermes MCP client setup
|
||||
|
||||
To connect a Hermes Agent instance to oikos as a native MCP client, add to
|
||||
`~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
oikos:
|
||||
url: "https://mcp.hubris.network/mcp"
|
||||
headers:
|
||||
Authorization: "Bearer <OIKOS_MCP_BEARER_TOKEN>"
|
||||
timeout: 180
|
||||
```
|
||||
|
||||
Run `/reload-mcp` in-session or restart Hermes. Tools appear as
|
||||
`mcp__oikos__*`.
|
||||
|
||||
**Caveat:** Hermes stores the bearer token in plaintext in `config.yaml` —
|
||||
it does not support `${VAR}` interpolation in MCP server headers. Ensure
|
||||
`security.redact_secrets: true` (default) so the token value is stripped
|
||||
from tool output and logs. File an upstream feature request at
|
||||
https://github.com/NousResearch/hermes-agent/issues for env-var
|
||||
interpolation support.
|
||||
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
|
||||
(`agent:nomos`), and all referencing docs were updated. All architectural
|
||||
principles in this ADR remain unchanged.
|
||||
133
internal/audit/audit.go
Normal file
133
internal/audit/audit.go
Normal file
@@ -0,0 +1,133 @@
|
||||
// Package audit produces read-only drift reports over the knowledge graph and
|
||||
// monitoring state. It is the shared engine behind the
|
||||
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
|
||||
//
|
||||
// It surfaces the structural gaps an operator otherwise discovers only by
|
||||
// accident: orphan check entities, checks targeting retired entities, probes
|
||||
// stuck down/unknown, unmonitored declared types, and live edges pointing at
|
||||
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
|
||||
// follow-up that needs host-hop execution; these categories are pure DB
|
||||
// queries, so the report is cheap, safe to run unattended, and testable.
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
)
|
||||
|
||||
// Finding is one drift item the operator should look at.
|
||||
type Finding struct {
|
||||
Category string `json:"category"`
|
||||
Severity string `json:"severity"` // info | warning | critical
|
||||
Count int `json:"count"`
|
||||
Entities []string `json:"entities"`
|
||||
Evidence string `json:"evidence"`
|
||||
SuggestedRunbook string `json:"suggested_runbook"`
|
||||
}
|
||||
|
||||
// Summary tallies findings by category.
|
||||
type Summary struct {
|
||||
TotalFindings int `json:"total_findings"`
|
||||
ByCategory map[string]int `json:"by_category"`
|
||||
}
|
||||
|
||||
// Report runs every drift check and returns the findings plus a summary.
|
||||
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
|
||||
specs := []struct {
|
||||
finding Finding
|
||||
query string
|
||||
}{
|
||||
{
|
||||
Finding{Category: "orphan_checks", Severity: "warning",
|
||||
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
|
||||
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
|
||||
`SELECT e.slug FROM entities e
|
||||
WHERE e.type = 'check'
|
||||
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "dead_checks", Severity: "warning",
|
||||
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
|
||||
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
|
||||
`SELECT e.slug FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "down_checks", Severity: "critical",
|
||||
Evidence: "enabled checks reporting health=down",
|
||||
SuggestedRunbook: "service-health-check"},
|
||||
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled AND cd.last_health = 'down'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "unknown_checks", Severity: "warning",
|
||||
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||
WHERE cd.enabled AND cd.last_health = 'unknown'`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "unmonitored", Severity: "warning",
|
||||
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT DISTINCT e.slug FROM signals sg
|
||||
JOIN entities e ON e.id = sg.target_entity_id
|
||||
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "dangling_edges", Severity: "warning",
|
||||
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
|
||||
SuggestedRunbook: "lifecycle-destroy-node"},
|
||||
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
|
||||
JOIN entities src ON src.id = r.source_id
|
||||
JOIN entities tgt ON tgt.id = r.target_id
|
||||
WHERE r.valid_to IS NULL
|
||||
AND src.state NOT IN ('destroyed','deprecated')
|
||||
AND tgt.state IN ('destroyed','deprecated')`,
|
||||
},
|
||||
{
|
||||
Finding{Category: "polluted_attrs", Severity: "warning",
|
||||
Evidence: "routing-critical attributes carrying prose (breaks resolution) — e.g. host='hubris (confirmed via pct…')",
|
||||
SuggestedRunbook: "knowledge-graph-audit"},
|
||||
`SELECT slug || ': host=' || (attributes->>'host') FROM entities
|
||||
WHERE attributes->>'host' IS NOT NULL
|
||||
AND (attributes->>'host') ~ '[ (]'`,
|
||||
},
|
||||
}
|
||||
|
||||
findings := make([]Finding, 0, len(specs))
|
||||
summary := Summary{ByCategory: map[string]int{}}
|
||||
for _, sp := range specs {
|
||||
f := runFinding(ctx, pool, sp.finding, sp.query)
|
||||
findings = append(findings, f)
|
||||
summary.TotalFindings += f.Count
|
||||
summary.ByCategory[f.Category] = f.Count
|
||||
}
|
||||
return findings, summary
|
||||
}
|
||||
|
||||
const entityCap = 50
|
||||
|
||||
// runFinding runs a single-column slug query and folds the rows into a Finding.
|
||||
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
|
||||
rows, err := pool.Query(ctx, query)
|
||||
if err != nil {
|
||||
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
|
||||
return f
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var slug string
|
||||
if err := rows.Scan(&slug); err != nil {
|
||||
continue
|
||||
}
|
||||
f.Count++
|
||||
if len(f.Entities) < entityCap {
|
||||
f.Entities = append(f.Entities, slug)
|
||||
}
|
||||
}
|
||||
return f
|
||||
}
|
||||
69
internal/audit/audit_test.go
Normal file
69
internal/audit/audit_test.go
Normal file
@@ -0,0 +1,69 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// Integration tests against a real Postgres, guarded by
|
||||
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
|
||||
|
||||
func newAuditPool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||
if base == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
return createTestDB(t, base)
|
||||
}
|
||||
|
||||
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
|
||||
pool := newAuditPool(t)
|
||||
ctx := context.Background()
|
||||
|
||||
// An orphan check entity (truncated random slug, the legacy bug shape).
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
|
||||
|
||||
// An active entity + a check_def on it stuck down.
|
||||
target := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
|
||||
checkE := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
|
||||
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
|
||||
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
|
||||
|
||||
// A deprecated entity still carrying an enabled check (dead_checks).
|
||||
dep := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
|
||||
depCheck := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
|
||||
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
|
||||
|
||||
findings, summary := Report(ctx, pool)
|
||||
|
||||
byCat := map[string]int{}
|
||||
for _, f := range findings {
|
||||
byCat[f.Category] = f.Count
|
||||
}
|
||||
if byCat["orphan_checks"] < 1 {
|
||||
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
|
||||
}
|
||||
if byCat["down_checks"] < 1 {
|
||||
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
|
||||
}
|
||||
if byCat["dead_checks"] < 1 {
|
||||
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
|
||||
}
|
||||
if summary.TotalFindings < 3 {
|
||||
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
|
||||
}
|
||||
}
|
||||
67
internal/audit/testutil_test.go
Normal file
67
internal/audit/testutil_test.go
Normal file
@@ -0,0 +1,67 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// createTestDB provisions a throwaway migrated database, same convention as
|
||||
// internal/scheduler/coverage_test.go.
|
||||
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
func getenvOrDefault(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatalf("exec %s: %v", q, err)
|
||||
}
|
||||
}
|
||||
@@ -24,13 +24,16 @@ import (
|
||||
// check_defs.kind values — one semantic kind can expand to several concrete
|
||||
// checks (`resource` becomes four ssh-script rows).
|
||||
const (
|
||||
KindPing = "ping"
|
||||
KindResource = "resource"
|
||||
KindUpdates = "updates"
|
||||
KindProcess = "process"
|
||||
KindHTTP = "http"
|
||||
KindCapacity = "capacity"
|
||||
KindBackup = "backup-freshness"
|
||||
KindPing = "ping"
|
||||
KindResource = "resource"
|
||||
KindUpdates = "updates"
|
||||
KindProcess = "process"
|
||||
KindHTTP = "http"
|
||||
KindCapacity = "capacity"
|
||||
KindBackup = "backup-freshness"
|
||||
KindCertExpiry = "cert-expiry"
|
||||
KindVMStatus = "vm-status"
|
||||
KindDNS = "dns"
|
||||
)
|
||||
|
||||
// defaultBackupMaxAge is how long a backup target may go without a new
|
||||
@@ -108,6 +111,18 @@ func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (
|
||||
attrs = map[string]any{}
|
||||
}
|
||||
|
||||
// Per-entity override: an explicit `monitoring` attribute wins over the
|
||||
// type declaration. A single entity can opt out (monitoring: none) or pick
|
||||
// different kinds without introducing a new type — e.g. service:haos opts
|
||||
// out because its VM is already covered by a vm-status check and the
|
||||
// service can't be SSH-probed (haos blocks SSH).
|
||||
if mo, ok := attrs["monitoring"]; ok {
|
||||
mon = resolveMonitoringAttr(mo, mon)
|
||||
if mon.None() {
|
||||
return res, nil
|
||||
}
|
||||
}
|
||||
|
||||
// A service has no address of its own — it lives on the container that
|
||||
// provides it. Fall back to the graph before giving up.
|
||||
host := resolveHost(attrs)
|
||||
@@ -146,6 +161,27 @@ func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (
|
||||
return res, nil
|
||||
}
|
||||
|
||||
// resolveMonitoringAttr turns an entity's `monitoring` attribute into a
|
||||
// MonitoringResolution that overrides the type's declaration. Accepts the
|
||||
// scalar "none" (or empty) to opt out, or a list of kind strings to override.
|
||||
func resolveMonitoringAttr(v any, fallback ontology.MonitoringResolution) ontology.MonitoringResolution {
|
||||
switch vv := v.(type) {
|
||||
case string:
|
||||
if vv == "none" || vv == "" {
|
||||
return ontology.MonitoringResolution{Declared: true, Source: "attribute"}
|
||||
}
|
||||
case []any:
|
||||
kinds := make([]string, 0, len(vv))
|
||||
for _, k := range vv {
|
||||
if s, ok := k.(string); ok && s != "" {
|
||||
kinds = append(kinds, s)
|
||||
}
|
||||
}
|
||||
return ontology.MonitoringResolution{Declared: true, Kinds: kinds, Source: "attribute"}
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
// buildKind turns one declared semantic kind into concrete check_defs, or
|
||||
// returns the reason it could not.
|
||||
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
|
||||
@@ -201,12 +237,36 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
|
||||
if host == "" {
|
||||
return nil, "no address on the entity or its host"
|
||||
}
|
||||
if t.Name == "" {
|
||||
// A service's name is a logical label, not usually its systemd unit
|
||||
// or container name (matrix = matrix-synapse.service + containers).
|
||||
// Prefer an explicit probe target when declared; process_check.sh also
|
||||
// matches a unit prefix or a docker container as a fallback.
|
||||
unit := ""
|
||||
for _, key := range []string{"probe_unit", "systemd_unit", "container"} {
|
||||
if v, _ := attrs[key].(string); v != "" {
|
||||
unit = v
|
||||
break
|
||||
}
|
||||
}
|
||||
// Ontology intent: "http when it has a url, else a process check." A
|
||||
// url-fronted service is already liveness-probed via http (the real
|
||||
// endpoint, through the TLS terminator); the process check is redundant
|
||||
// and fragile (needs host access + the exact unit/container name), and
|
||||
// under worst-of aggregation it lets a broken supplementary probe veto
|
||||
// a working service. Emit it only for services WITHOUT a url, or when
|
||||
// an explicit probe_unit opts into binary-level depth.
|
||||
if unit == "" {
|
||||
if httpURL(t, attrs) != "" {
|
||||
return nil, "url present and no probe_unit; http check covers liveness"
|
||||
}
|
||||
unit = t.Name
|
||||
}
|
||||
if unit == "" {
|
||||
return nil, "no name to check a process for"
|
||||
}
|
||||
// process_check.sh takes the unit name as $1 and reports "unknown"
|
||||
// without it.
|
||||
return []checkDef{ssh("process_check.sh", t.Name)}, ""
|
||||
// process_check.sh takes the unit/container name as $1 and reports
|
||||
// "unknown" without it.
|
||||
return []checkDef{ssh("process_check.sh", unit)}, ""
|
||||
|
||||
case KindBackup:
|
||||
// A backup target is checked from the machine that writes to it, so it
|
||||
@@ -246,11 +306,79 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
|
||||
config: map[string]any{"url": url, "max_status": 500},
|
||||
interval: 60,
|
||||
}}, ""
|
||||
|
||||
case KindDNS:
|
||||
// Resolve the entity's name via DNS to verify the zone is reachable.
|
||||
// Uses the entity name (zone apex) or falls back to the slug.
|
||||
name := t.Name
|
||||
if name == "" {
|
||||
name = strings.TrimPrefix(t.Slug, "zone:")
|
||||
}
|
||||
if name == "" {
|
||||
return nil, "no name to resolve"
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "dns",
|
||||
config: map[string]any{"name": name},
|
||||
interval: 300, // 5 min — DNS changes are rare; the cost of a miss
|
||||
// is a stale IP, not a service outage.
|
||||
}}, ""
|
||||
|
||||
case KindCertExpiry:
|
||||
// The host whose cert to read (SNI / cert CN). Prefer an explicit
|
||||
// `hostname` attribute, then `cn`, then a dotted name. Hourly: expiry
|
||||
// changes once a day, but a renewal or mis-issued cert is worth
|
||||
// noticing within the hour.
|
||||
host := certHost(t, attrs)
|
||||
if host == "" {
|
||||
return nil, "no hostname / cn / dotted name to dial for the cert"
|
||||
}
|
||||
// `dial` is the TLS terminator's address to connect to (Caddy's lab
|
||||
// IP), used when the hostname doesn't resolve/reach from the scheduler.
|
||||
// Without it the probe can't reach *.hubris.network from a container
|
||||
// with no mesh / split-horizon DNS.
|
||||
dial, _ := attrs["dial"].(string)
|
||||
config := map[string]any{"host": host, "warn_days": 30, "crit_days": 7}
|
||||
if dial != "" {
|
||||
config["dial"] = dial
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "cert-expiry",
|
||||
config: config,
|
||||
interval: 3600,
|
||||
}}, ""
|
||||
|
||||
case KindVMStatus:
|
||||
// "Is the VM powered on" via `qm status` on its Proxmox host — the
|
||||
// right reachability probe for a VM, since many block ICMP and lack a
|
||||
// guest agent. checkVMStatus re-reads pve_id + host at runtime.
|
||||
if _, ok := attrs["pve_id"]; !ok {
|
||||
return nil, "no pve_id to run qm status"
|
||||
}
|
||||
return []checkDef{{
|
||||
kind: "vm-status",
|
||||
config: map[string]any{},
|
||||
interval: 60,
|
||||
}}, ""
|
||||
}
|
||||
|
||||
return nil, "no builder for this kind yet"
|
||||
}
|
||||
|
||||
// certHost works out the hostname to TLS-dial for a certificate's expiry.
|
||||
func certHost(t Target, attrs map[string]any) string {
|
||||
for _, key := range []string{"hostname", "cn", "san"} {
|
||||
if v, ok := attrs[key].(string); ok && v != "" {
|
||||
return v
|
||||
}
|
||||
}
|
||||
// A dotted name is a hostname (hubris.network, media.hubris.network).
|
||||
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||
return t.Name
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// writeCheck upserts one check_def and its backing check entity.
|
||||
//
|
||||
// The entity upsert MUST return the row's id. The previous version generated
|
||||
@@ -299,14 +427,15 @@ func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef)
|
||||
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
||||
// and re-herd everything.
|
||||
tag, err := tx.Exec(ctx,
|
||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1, $2, $3, $4, $5, 30, true,
|
||||
`INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||
VALUES ($1, $2, $6, $3, $4, $5, 30, true,
|
||||
now() - make_interval(secs => random() * $5::int))
|
||||
ON CONFLICT (entity_id) DO UPDATE
|
||||
SET target_id = EXCLUDED.target_id, kind = EXCLUDED.kind,
|
||||
SET target_id = EXCLUDED.target_id, target_type = EXCLUDED.target_type,
|
||||
kind = EXCLUDED.kind,
|
||||
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
||||
updated_at = now()`,
|
||||
checkID, t.ID, def.kind, configJSON, def.interval)
|
||||
checkID, t.ID, def.kind, configJSON, def.interval, t.Type)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
|
||||
}
|
||||
@@ -380,6 +509,12 @@ func resolveHost(attrs map[string]any) string {
|
||||
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
// public_ipv4 before mesh: the scheduler container has no mesh interface,
|
||||
// so a standalone-server reachable only by mesh IP (netbird-vps) is
|
||||
// unprobeable even though a public IPv4 is available.
|
||||
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
|
||||
return ip
|
||||
}
|
||||
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
||||
@@ -409,6 +544,13 @@ func resolveSSHUser(attrs map[string]any) string {
|
||||
return u
|
||||
}
|
||||
}
|
||||
// Workstations carry their login as a top-level `user` attribute
|
||||
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
|
||||
// no explicit ssh.user was set, so a host that genuinely wants root still
|
||||
// gets root.
|
||||
if u, ok := attrs["user"].(string); ok && u != "" {
|
||||
return u
|
||||
}
|
||||
return "root"
|
||||
}
|
||||
|
||||
|
||||
34
internal/db/checks.go
Normal file
34
internal/db/checks.go
Normal file
@@ -0,0 +1,34 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// EnsureEntityChecks derives an entity's default check_defs from the
|
||||
// monitoring spec of its type (resolving per-entity `monitoring` overrides).
|
||||
//
|
||||
// This is the single shared hook that keeps the check graph in sync with
|
||||
// entity mutations. Both the HTTP create/patch handlers and the MCP
|
||||
// entity-mutation tools (create_entity, update_entity_attributes) call it so
|
||||
// that flipping an entity's `monitoring` attribute regenerates checks
|
||||
// regardless of which surface made the change — previously only the HTTP
|
||||
// path ran check derivation, so entities mutated via MCP silently produced no
|
||||
// checks (see plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md, A2).
|
||||
func EnsureEntityChecks(ctx context.Context, tx pgx.Tx, id uuid.UUID, slug, entityType, name string, attrs []byte) (checkdefaults.Result, error) {
|
||||
tree, err := LoadTypeTree(ctx, tx)
|
||||
if err != nil {
|
||||
return checkdefaults.Result{}, err
|
||||
}
|
||||
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
||||
ID: id, Slug: slug, Type: entityType, Name: name, Attrs: attrs,
|
||||
})
|
||||
if err != nil {
|
||||
return res, err
|
||||
}
|
||||
checkdefaults.LogResult(slug, entityType, res)
|
||||
return res, nil
|
||||
}
|
||||
@@ -306,7 +306,17 @@ func TestBlastRadiusTerminatesOnCycles(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
seedAll(t, pool, seedsDir())
|
||||
|
||||
// Build a dependency cycle: gitea → caddy → authentik → gitea
|
||||
// Build a dependency cycle: gitea → caddy → authentik → gitea.
|
||||
//
|
||||
// `depends-on` is declared blast_direction: backward — "A depends-on B"
|
||||
// means B failing breaks A — so the blast radius of gitea walks the edges
|
||||
// BACKWARDS: whoever depends on gitea is affected first. That is authentik
|
||||
// (1 hop), then caddy which depends on authentik (2 hops).
|
||||
//
|
||||
// This test previously asserted caddy=1, authentik=2, which is the same
|
||||
// cycle walked the wrong way round: blast_radius used to follow every edge
|
||||
// source→target regardless of what the edge means, so it answered "what
|
||||
// does gitea depend on" while being named for the opposite question.
|
||||
cycle := []byte(`
|
||||
version: 1
|
||||
relationships:
|
||||
@@ -342,14 +352,24 @@ relationships:
|
||||
}
|
||||
got[slug] = depth
|
||||
}
|
||||
want := map[string]int{"service:gitea": 0, "service:caddy": 1, "service:authentik": 2}
|
||||
want := map[string]int{"service:gitea": 0, "service:authentik": 1, "service:caddy": 2}
|
||||
for slug, depth := range want {
|
||||
if got[slug] != depth {
|
||||
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
||||
}
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Errorf("blast_radius returned %d nodes, want %d: %v", len(got), len(want), got)
|
||||
// Deliberately not an exact node count. Walking the right way round also
|
||||
// surfaces the real seed's own dependents of gitea (homelab-mcp and what
|
||||
// depends on it), which are correct answers — the old exact-count
|
||||
// assertion only held because the forward walk found nothing real.
|
||||
// What matters here is that the cycle terminates rather than recursing.
|
||||
if len(got) > 20 {
|
||||
t.Errorf("blast_radius did not terminate sensibly: %d nodes: %v", len(got), got)
|
||||
}
|
||||
for slug, depth := range got {
|
||||
if depth > 5 {
|
||||
t.Errorf("blast_radius[%s] = %d, beyond the max_depth bound", slug, depth)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
146
internal/db/lifecycle.go
Normal file
146
internal/db/lifecycle.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package db
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// ErrTransitionInvalid is a sentinel returned by ValidateTransition when the
|
||||
// from→to pair is not a declared lifecycle transition or a precondition fails.
|
||||
// Callers test with errors.Is to distinguish semantic validation failures
|
||||
// (→ HTTP 409) from infrastructure errors (→ HTTP 500).
|
||||
var ErrTransitionInvalid = errors.New("invalid lifecycle transition")
|
||||
|
||||
// ValidateTransition enforces an entity type's lifecycle: fromState → toState
|
||||
// must be a declared transition, and every precondition it lists must hold. A
|
||||
// type with no lifecycle defined allows any state. A no-op (fromState ==
|
||||
// toState) passes immediately.
|
||||
//
|
||||
// Shared by the HTTP PATCH path and the MCP set_entity_state tool so both
|
||||
// surfaces apply identical lifecycle rules — previously only the HTTP path
|
||||
// validated transitions, so an agent changing state via MCP could skip the
|
||||
// graph's retire/deprecate guardrails entirely.
|
||||
func ValidateTransition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, fromState, toState string) error {
|
||||
if toState == fromState {
|
||||
return nil
|
||||
}
|
||||
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, entityType)
|
||||
if err != nil {
|
||||
if err == pgx.ErrNoRows {
|
||||
return nil // no lifecycle defined → any state allowed
|
||||
}
|
||||
return err
|
||||
}
|
||||
var transitions map[string]map[string]json.RawMessage
|
||||
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
||||
return fmt.Errorf("parse lifecycle transitions: %w", err)
|
||||
}
|
||||
tos, ok := transitions[fromState]
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: no transitions defined from %q", ErrTransitionInvalid, fromState)
|
||||
}
|
||||
trans, ok := tos[toState]
|
||||
if !ok {
|
||||
return fmt.Errorf("%w: %s → %s is not a declared lifecycle transition", ErrTransitionInvalid, fromState, toState)
|
||||
}
|
||||
var gate struct {
|
||||
Requires []string `json:"requires"`
|
||||
}
|
||||
if err := json.Unmarshal(trans, &gate); err == nil {
|
||||
for _, check := range gate.Requires {
|
||||
if err := checkPrecondition(ctx, tx, entityID, entityType, check); err != nil {
|
||||
return fmt.Errorf("%w: precondition %q not met: %w", ErrTransitionInvalid, check, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// checkPrecondition evaluates one mechanical precondition named by a lifecycle
|
||||
// transition's `requires` list. Soft/operator-confirmed checks pass; unknown
|
||||
// checks are skipped (operator intent overrides). Moved here from httpapi so
|
||||
// both surfaces share one implementation.
|
||||
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
||||
switch check {
|
||||
case "no-inbound-edges":
|
||||
var count int
|
||||
if err := tx.QueryRow(ctx,
|
||||
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
||||
}
|
||||
case "backups-verified", "secrets-revoked", "ingress-dns-removed":
|
||||
var attrs string
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||
return err
|
||||
}
|
||||
want := map[string]string{
|
||||
"backups-verified": "backups_verified",
|
||||
"secrets-revoked": "secrets_revoked",
|
||||
"ingress-dns-removed": "ingress_dns_removed",
|
||||
}[check]
|
||||
if !strings.Contains(attrs, want) {
|
||||
return fmt.Errorf("%s not recorded in entity attributes", want)
|
||||
}
|
||||
case "age-key-enrolled-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "age_pubkey") {
|
||||
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
||||
}
|
||||
}
|
||||
case "mesh-joined-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "mesh_ip") {
|
||||
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
||||
}
|
||||
}
|
||||
case "health-check-answering":
|
||||
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
||||
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
||||
h := "unknown"
|
||||
if err == nil {
|
||||
h = st.Health
|
||||
}
|
||||
return fmt.Errorf("health check not answering (status: %s)", h)
|
||||
}
|
||||
case "doc-page-complete":
|
||||
var count int
|
||||
if err := tx.QueryRow(ctx, `
|
||||
SELECT count(*) FROM relationships r
|
||||
JOIN entities ke ON ke.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
||||
entityID).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count == 0 {
|
||||
return fmt.Errorf("no documentation linked to entity")
|
||||
}
|
||||
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
||||
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
||||
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
||||
"ingress-live-if-public", "doc-page-stub", "un-deprecate-note", "write-off-note":
|
||||
// Soft checks — always pass. Operator-confirmed via the transition
|
||||
// request itself, or not mechanically enforceable.
|
||||
default:
|
||||
// Unknown preconditions are skipped (operator intent overrides).
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -25,8 +25,8 @@ ON CONFLICT (actor, key) DO NOTHING;
|
||||
|
||||
-- name: InsertAuditEntry :exec
|
||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||
status_code, detail, source_ip, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
|
||||
status_code, detail, source_ip, correlation_id, session_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);
|
||||
|
||||
-- name: InsertEvent :one
|
||||
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
|
||||
@@ -63,12 +63,28 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||
e.slug AS entity_slug
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled = true
|
||||
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||
AND (cd.last_run_at IS NULL
|
||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
|
||||
|
||||
-- name: MarkCheckRun :exec
|
||||
UPDATE check_defs SET last_run_at = now() WHERE entity_id = $1;
|
||||
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1;
|
||||
|
||||
-- name: WorstHealthForTarget :one
|
||||
-- An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||
-- yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||
-- newly added check does not drag a known-good entity down before it has
|
||||
-- produced a verdict.
|
||||
SELECT COALESCE(
|
||||
(SELECT last_health FROM check_defs
|
||||
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||
ORDER BY CASE last_health
|
||||
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||
LIMIT 1),
|
||||
'unknown')::text AS health;
|
||||
|
||||
-- name: GetCheckDef :one
|
||||
SELECT * FROM check_defs WHERE entity_id = $1;
|
||||
|
||||
@@ -67,12 +67,20 @@ func IngestOntologySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*S
|
||||
targetType, _ := rtMap["target"].(string)
|
||||
cardinality, _ := rtMap["cardinality"].(string)
|
||||
desc, _ := rtMap["description"].(string)
|
||||
// Which end of the edge depends on the other; drives blast_radius().
|
||||
// Absent means 'none' — an undeclared edge contributes nothing rather
|
||||
// than silently producing a wrong dependency answer.
|
||||
blastDirection, _ := rtMap["blast_direction"].(string)
|
||||
if blastDirection == "" {
|
||||
blastDirection = "none"
|
||||
}
|
||||
_, err := tx.Exec(ctx,
|
||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)
|
||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description, blast_direction)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
||||
target_type = $4, cardinality = $5, description = $6`,
|
||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc)
|
||||
target_type = $4, cardinality = $5, description = $6,
|
||||
blast_direction = $7`,
|
||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc, blastDirection)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
||||
}
|
||||
|
||||
@@ -114,6 +114,8 @@ type CheckDef struct {
|
||||
UpdatedAt time.Time
|
||||
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
||||
LastRunAt *time.Time
|
||||
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
type Classification struct {
|
||||
@@ -255,6 +257,20 @@ type KnowledgeEntity struct {
|
||||
UpdatedAt time.Time
|
||||
ContentHash *string
|
||||
Search interface{}
|
||||
EditedBy string
|
||||
DeletedAt *time.Time
|
||||
}
|
||||
|
||||
type KnowledgeRevision struct {
|
||||
ID int64
|
||||
EntityID uuid.UUID
|
||||
Title string
|
||||
Content string
|
||||
Source *string
|
||||
Tags []string
|
||||
EditedBy string
|
||||
VersionAt time.Time
|
||||
RevisedAt time.Time
|
||||
}
|
||||
|
||||
type Ledger struct {
|
||||
@@ -364,6 +380,8 @@ type RelationshipType struct {
|
||||
Cardinality string
|
||||
Description *string
|
||||
CreatedAt time.Time
|
||||
// Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().
|
||||
BlastDirection string
|
||||
}
|
||||
|
||||
type RiskClass struct {
|
||||
@@ -380,18 +398,19 @@ type SeedVersion struct {
|
||||
}
|
||||
|
||||
type SessionPlanStep struct {
|
||||
ID uuid.UUID
|
||||
SessionID uuid.UUID
|
||||
Seq int32
|
||||
Title string
|
||||
Detail string
|
||||
Status string
|
||||
ExecutionID *uuid.UUID
|
||||
TargetSlug *string
|
||||
StartedAt *time.Time
|
||||
FinishedAt *time.Time
|
||||
CreatedAt time.Time
|
||||
Generation int32
|
||||
ID uuid.UUID
|
||||
SessionID uuid.UUID
|
||||
Seq int32
|
||||
Title string
|
||||
Detail string
|
||||
Status string
|
||||
ExecutionID *uuid.UUID
|
||||
TargetSlug *string
|
||||
StartedAt *time.Time
|
||||
FinishedAt *time.Time
|
||||
CreatedAt time.Time
|
||||
Generation int32
|
||||
ReplacedReason *string
|
||||
}
|
||||
|
||||
type SessionQuestion struct {
|
||||
|
||||
@@ -99,7 +99,7 @@ func (q *Queries) ListLifecycleDefs(ctx context.Context) ([]LifecycleDef, error)
|
||||
}
|
||||
|
||||
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
|
||||
SELECT name, inverse, source_type, target_type, cardinality, description, created_at FROM relationship_types ORDER BY name
|
||||
SELECT name, inverse, source_type, target_type, cardinality, description, created_at, blast_direction FROM relationship_types ORDER BY name
|
||||
`
|
||||
|
||||
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
|
||||
@@ -119,6 +119,7 @@ func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType
|
||||
&i.Cardinality,
|
||||
&i.Description,
|
||||
&i.CreatedAt,
|
||||
&i.BlastDirection,
|
||||
); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -51,7 +51,7 @@ func (q *Queries) GetAutonomySetting(ctx context.Context, key string) (string, e
|
||||
}
|
||||
|
||||
const getCheckDef = `-- name: GetCheckDef :one
|
||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at FROM check_defs WHERE entity_id = $1
|
||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at, last_health FROM check_defs WHERE entity_id = $1
|
||||
`
|
||||
|
||||
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
||||
@@ -69,6 +69,7 @@ func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef
|
||||
&i.Enabled,
|
||||
&i.UpdatedAt,
|
||||
&i.LastRunAt,
|
||||
&i.LastHealth,
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
@@ -352,8 +353,8 @@ func (q *Queries) InsertApproval(ctx context.Context, arg InsertApprovalParams)
|
||||
|
||||
const insertAuditEntry = `-- name: InsertAuditEntry :exec
|
||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||
status_code, detail, source_ip, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
|
||||
status_code, detail, source_ip, correlation_id, session_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
|
||||
`
|
||||
|
||||
type InsertAuditEntryParams struct {
|
||||
@@ -367,6 +368,7 @@ type InsertAuditEntryParams struct {
|
||||
Detail []byte
|
||||
SourceIp *string
|
||||
CorrelationID *string
|
||||
SessionID *uuid.UUID
|
||||
}
|
||||
|
||||
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
|
||||
@@ -381,6 +383,7 @@ func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryPara
|
||||
arg.Detail,
|
||||
arg.SourceIp,
|
||||
arg.CorrelationID,
|
||||
arg.SessionID,
|
||||
)
|
||||
return err
|
||||
}
|
||||
@@ -695,7 +698,9 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
||||
e.slug AS entity_slug
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||
WHERE cd.enabled = true
|
||||
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||
AND (cd.last_run_at IS NULL
|
||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
|
||||
`
|
||||
@@ -1133,11 +1138,16 @@ func (q *Queries) ListSkills(ctx context.Context, status *string) ([]Skill, erro
|
||||
}
|
||||
|
||||
const markCheckRun = `-- name: MarkCheckRun :exec
|
||||
UPDATE check_defs SET last_run_at = now() WHERE entity_id = $1
|
||||
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1
|
||||
`
|
||||
|
||||
func (q *Queries) MarkCheckRun(ctx context.Context, entityID uuid.UUID) error {
|
||||
_, err := q.db.Exec(ctx, markCheckRun, entityID)
|
||||
type MarkCheckRunParams struct {
|
||||
EntityID uuid.UUID
|
||||
LastHealth *string
|
||||
}
|
||||
|
||||
func (q *Queries) MarkCheckRun(ctx context.Context, arg MarkCheckRunParams) error {
|
||||
_, err := q.db.Exec(ctx, markCheckRun, arg.EntityID, arg.LastHealth)
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -1462,3 +1472,25 @@ func (q *Queries) UpsertSignal(ctx context.Context, arg UpsertSignalParams) (Sig
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
|
||||
const worstHealthForTarget = `-- name: WorstHealthForTarget :one
|
||||
SELECT COALESCE(
|
||||
(SELECT last_health FROM check_defs
|
||||
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||
ORDER BY CASE last_health
|
||||
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||
LIMIT 1),
|
||||
'unknown')::text AS health
|
||||
`
|
||||
|
||||
// An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||
// yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||
// newly added check does not drag a known-good entity down before it has
|
||||
// produced a verdict.
|
||||
func (q *Queries) WorstHealthForTarget(ctx context.Context, targetID *uuid.UUID) (string, error) {
|
||||
row := q.db.QueryRow(ctx, worstHealthForTarget, targetID)
|
||||
var health string
|
||||
err := row.Scan(&health)
|
||||
return health, err
|
||||
}
|
||||
|
||||
@@ -89,6 +89,7 @@ func (s *Server) CreateApprovalRule(ctx context.Context, req gen.CreateApprovalR
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||
&id, "POST", "/api/v1/policy/approval-rules", "",
|
||||
nil,
|
||||
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -148,6 +149,7 @@ func (s *Server) PatchApprovalRule(ctx context.Context, req gen.PatchApprovalRul
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"action": req.Body.Action}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -154,6 +154,7 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
||||
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
|
||||
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
|
||||
nil,
|
||||
map[string]any{"decision": status}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
20
internal/httpapi/audit.go
Normal file
20
internal/httpapi/audit.go
Normal file
@@ -0,0 +1,20 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/dtoro/oikos/internal/audit"
|
||||
)
|
||||
|
||||
// serveAuditDrift returns a read-only DB-side drift report: orphan check
|
||||
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
|
||||
// declared types, and dangling edges. Companion to the knowledge-graph-audit
|
||||
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
|
||||
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
|
||||
findings, summary := audit.Report(req.Context(), s.pool)
|
||||
writeJSON(w, map[string]any{
|
||||
"findings": findings,
|
||||
"summary": summary,
|
||||
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
|
||||
})
|
||||
}
|
||||
@@ -81,6 +81,7 @@ func (s *Server) PatchAutonomySettings(ctx context.Context, req gen.PatchAutonom
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
nil, "PATCH", "/api/v1/policy/autonomy", "",
|
||||
nil,
|
||||
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
||||
SELECT cd.entity_id, e.slug, cd.kind,
|
||||
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
||||
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
||||
e.version
|
||||
e.version, cd.last_health, cd.last_run_at
|
||||
FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.entity_id
|
||||
LEFT JOIN entities te ON te.id = cd.target_id
|
||||
@@ -43,10 +43,19 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
||||
var c gen.Check
|
||||
var targetSlug string
|
||||
var configBytes []byte
|
||||
// last_health is what turns a check list from configuration into an
|
||||
// explanation: an entity's health is the worst of these, so this is
|
||||
// the field that says which probe is responsible.
|
||||
var lastHealth *string
|
||||
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version); err != nil {
|
||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version,
|
||||
&lastHealth, &c.LastRunAt); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if lastHealth != nil {
|
||||
h := gen.CheckLastHealth(*lastHealth)
|
||||
c.LastHealth = &h
|
||||
}
|
||||
if targetSlug != "" {
|
||||
c.Target = &targetSlug
|
||||
}
|
||||
@@ -173,6 +182,7 @@ func (s *Server) CreateCheck(ctx context.Context, req gen.CreateCheckRequestObje
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||
&id, "POST", "/api/v1/checks", "",
|
||||
nil,
|
||||
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -260,6 +270,7 @@ func (s *Server) PatchCheck(ctx context.Context, req gen.PatchCheckRequestObject
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -285,6 +296,15 @@ func checkDefToGen(cd sqlcgen.CheckDef) gen.Check {
|
||||
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
||||
c.Config = &config
|
||||
}
|
||||
// Carried through so toggling a check does not blank its verdict in the
|
||||
// UI — the entity window renders last_health to explain which probe is
|
||||
// responsible for an entity's health, and a patch response missing it
|
||||
// would erase that until the next poll.
|
||||
c.LastRunAt = cd.LastRunAt
|
||||
if cd.LastHealth != nil {
|
||||
h := gen.CheckLastHealth(*cd.LastHealth)
|
||||
c.LastHealth = &h
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
|
||||
@@ -3,31 +3,22 @@ package httpapi
|
||||
import (
|
||||
"context"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// ensureDefaultChecks derives an entity's default checks from the monitoring
|
||||
// kinds its type declares.
|
||||
// kinds its type declares. Thin wrapper over the shared db.EnsureEntityChecks
|
||||
// hook so the HTTP create/patch paths and the MCP entity-mutation tools stay
|
||||
// in lockstep.
|
||||
//
|
||||
// Note the ordering caveat: an entity created through the API usually has no
|
||||
// edges yet, so a type whose address comes from its host (a service) will
|
||||
// produce no checks on this pass. That gap is real and deliberately visible —
|
||||
// coverageSweep reports it, and the next inventory ingest fills it in once
|
||||
// the hosting edge exists.
|
||||
// Note the ordering caveat (carried from db.LoadTypeTree / checkdefaults.Ensure):
|
||||
// an entity created through the API usually has no edges yet, so a type whose
|
||||
// address comes from its host (a service) will produce no checks on this pass.
|
||||
// That gap is real and deliberately visible — coverageSweep reports it, and
|
||||
// the next inventory ingest fills it in once the hosting edge exists.
|
||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
|
||||
tree, err := db.LoadTypeTree(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
||||
ID: entityID, Slug: slug, Type: entityType, Name: name, Attrs: attrsJSON,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
checkdefaults.LogResult(slug, entityType, res)
|
||||
return nil
|
||||
_, err := db.EnsureEntityChecks(ctx, tx, entityID, slug, entityType, name, attrsJSON)
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -67,6 +67,7 @@ func (s *Server) CreateEntityType(ctx context.Context, req gen.CreateEntityTypeR
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||
nil, "POST", "/api/v1/ontology/entity-types", "",
|
||||
nil,
|
||||
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -148,6 +149,7 @@ func (s *Server) PatchEntityType(ctx context.Context, req gen.PatchEntityTypeReq
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
|
||||
nil,
|
||||
map[string]any{"status": req.Body.Status}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -240,6 +240,7 @@ func (s *Server) RequestExecution(ctx context.Context, req gen.RequestExecutionR
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||
&id, "POST", "/api/v1/executions", "",
|
||||
nil,
|
||||
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -307,6 +308,7 @@ func (s *Server) CancelExecution(ctx context.Context, req gen.CancelExecutionReq
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
|
||||
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
|
||||
nil,
|
||||
map[string]any{"status": "cancelled"}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -86,6 +86,14 @@ const (
|
||||
CheckKindTcp CheckKind = "tcp"
|
||||
)
|
||||
|
||||
// Defines values for CheckLastHealth.
|
||||
const (
|
||||
CheckLastHealthDegraded CheckLastHealth = "degraded"
|
||||
CheckLastHealthDown CheckLastHealth = "down"
|
||||
CheckLastHealthHealthy CheckLastHealth = "healthy"
|
||||
CheckLastHealthUnknown CheckLastHealth = "unknown"
|
||||
)
|
||||
|
||||
// Defines values for CheckCreateKind.
|
||||
const (
|
||||
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
||||
@@ -273,10 +281,10 @@ const (
|
||||
|
||||
// Defines values for TrendDirection.
|
||||
const (
|
||||
Degrading TrendDirection = "degrading"
|
||||
Improving TrendDirection = "improving"
|
||||
Stable TrendDirection = "stable"
|
||||
Unknown TrendDirection = "unknown"
|
||||
TrendDirectionDegrading TrendDirection = "degrading"
|
||||
TrendDirectionImproving TrendDirection = "improving"
|
||||
TrendDirectionStable TrendDirection = "stable"
|
||||
TrendDirectionUnknown TrendDirection = "unknown"
|
||||
)
|
||||
|
||||
// Defines values for ListApprovalsParamsStatus.
|
||||
@@ -462,7 +470,13 @@ type Check struct {
|
||||
Id openapi_types.UUID `json:"id"`
|
||||
IntervalS int `json:"interval_s"`
|
||||
Kind CheckKind `json:"kind"`
|
||||
Slug string `json:"slug"`
|
||||
|
||||
// LastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||
LastHealth *CheckLastHealth `json:"last_health"`
|
||||
|
||||
// LastRunAt When this check last executed. Null = never run.
|
||||
LastRunAt *time.Time `json:"last_run_at"`
|
||||
Slug string `json:"slug"`
|
||||
|
||||
// Target Entity slug (instance-scoped)
|
||||
Target *string `json:"target"`
|
||||
@@ -477,6 +491,9 @@ type Check struct {
|
||||
// CheckKind defines model for Check.Kind.
|
||||
type CheckKind string
|
||||
|
||||
// CheckLastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||
type CheckLastHealth string
|
||||
|
||||
// CheckCreate defines model for CheckCreate.
|
||||
type CheckCreate struct {
|
||||
Config *map[string]interface{} `json:"config,omitempty"`
|
||||
@@ -8208,175 +8225,177 @@ func (sh *strictHandler) GetTrends(w http.ResponseWriter, r *http.Request, entit
|
||||
// Base64 encoded, gzipped, json marshaled Swagger object
|
||||
var swaggerSpec = []string{
|
||||
|
||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpCn6MpPsyLU/FFljO7FjraXJt6mRiwK7D0mM0EAPgKbEuFyV",
|
||||
"X/sAW3nCPMlXuHY3iSabF1nOVP7YkhqNBs4FOPfzqZfyvOAMmJK9k0+9GeAMhPnx/ApP9f8ZyFSQQhHO",
|
||||
"eie9DyB5KVJAcxCScIYmXKA3k8E7rNJZL+nJdAY51u+pRQG9k55UgrBp7/Pnz0mvwALnoNwHzkohuVj9",
|
||||
"xPsC/1oCSs1jNBE8RxgVAuaElxIJkAVnEr6RiMG9GtlhvaRH9Lu/liAWvaTHcK4/Hh62LyvpnTNF1OJN",
|
||||
"trqSn3568xJxgSQtp6gPx9NjdDPjUp3MyrEg8ubIf7bAalZ9lWS9pCfg15IIyHonSpTQZQWXtIwA3D5r",
|
||||
"LOFOnuQ4HeSEkZsE3dD79CTFWbZoW49+d8sV/Sh4fkX025+igNVYaYB1wkWOVe+kl2EFA6VfTSLzvskg",
|
||||
"L7gCli7+DIvV3Z5RAkwNpsBAYAUZuoXFCySgoHgh0R1RM8LQs+9mSIAqBUNqBogLMiUM00AaHgqWmKtF",
|
||||
"1z4+0F+vrz/H92+BTdWsd/L02f+KLn1iaXwVQ1d4WpEp4QK9Or96gb57+gxxFvgkJzJ3PBJfXMVD2yDq",
|
||||
"LcmJasMSNQ/rE2QwwSVVvZPvnyR6zyQv897Jsyf6N8Lsb0/D7glTMAVhPnTF19GD4ttTw2e9U4sxcx5c",
|
||||
"AMsIm54WheBzTPWfUs4UMLM/XBSUpFjDfPiL1ID/VPvg/xQw6Z30/sewOs6G9qkchgnNJ5fobYbZFJBU",
|
||||
"eArZC4RRDgoPsHsD3WGJUgGGEvtZielAr0hwetT7nPQuBB9TyNcstLAjfrfdgv28kfWeC8EF6n/48Qz9",
|
||||
"8N33fzDLuCRThulPhYZ1djCo2Vlja3BfQtKP8Jg3WDydAlOnqSJzogyDF4IXIBSxSMbuychSw6ceME1z",
|
||||
"P/cU53SUYkoNA2DJmaYS/e2UaAbqJb08LUae7kCmmJp99T6ukFbSw3oVI5JFmCbppVwIsC+7IaykFI8p",
|
||||
"eI5beSUrhR2fyzXjA78kPTDHdtfpGwutzUJYUaqRLPMci0WnmXiptn1FgpRbgEKWaQpyHRjGnFPATA9W",
|
||||
"/BbYKOWlJcfNcDNkYA+VDmuxQkvHu6c6Vn+2V7SSvRqlJEu0WZEVH/8CqdLfq59Nq3Rt2WuV3OwBMsKq",
|
||||
"62It1Wfr39lMs26OcTcygPuCCJBbLdOSTBhblhauy8NuCcvqvA73kJbKMnXBKUkXg9QcxPp3rBQINjDI",
|
||||
"aGfwAi8oxxGZzYhIGjdcQobs7Cgjk0k7yCr8CiJvRynFlrxXSd9JaKsPFFalrG+xsJeZpipDM5CZs4wR",
|
||||
"84OFtRUT5/wWsugeZWkXtlYm1BJQuK9SzlIQTG4mjxg/ODnRkXIDGg6HYacNcmmQ+Dq2+VBS2Ip1cKk4",
|
||||
"4/liRGEOtA5f/aS6Bgw/wBxEFI7uLPY3ThOW7/ACjQHhsVQCpwr1CZuBIEqijN+xoy6M1pELNhFXygsY",
|
||||
"2bWuR7lWuQoQAzsW8TkIQTKQXdbqxNHYdRMjiTgtLKGlmnUT8s8MnTw+CeyLm/XM1AloUVCVGVHnTInF",
|
||||
"diBKFRddr287eFn6MrdgL+npL2JlVeaFVOCVvKykLZDdRZgChQmtbaWCwGHEphzUjHebwmjKncQeY/YY",
|
||||
"kaLbaHNMjlKeQUe55wCSTIXZwLhxKrOEeAlK6RlXSO3WauZwj/NCr7k3pXyM6bGm4BFOVexwK61SsJX0",
|
||||
"MMe0jLNj91Pq1ujxdqb159DZDNLb1c2mnE1IxO7yV0yJ1XP0Uetuvwi9arTWybAm/Ha8F/TWxBzTkYxT",
|
||||
"87L0NFOq0POk+t+MyFt9AYNQA3Mla3Bkgkw0lgorgUg5G9itxcWMNqlGYTGFDeJHnzCpMEthYM7IrNOF",
|
||||
"aSduuZDd7Poh6ut/t5qZ5MC1/hMH5Rq6Snp/56yL1rFGcnJUUkNofUUVtXQg1LabsiLXdbQYzDxtelmT",
|
||||
"5sLw3z95knydFLiJhtZTQtjg0+j+PObXY7qO5Fa8XXgb4S5o24SnyLWxjt4/xxappREycTah3SSx1J+k",
|
||||
"K0PGFEs1EjgjVhsiCvK4ROX+gIXAi7gYcRA9uuMR7JTOkUFTBixddxCwMh9b6Fd2qhhiBaQ8z4EZPT6A",
|
||||
"dV8dVPBSwbIYPLC3ck0UnnHaolQaq11nY88toZ0HV9y6wxEaF5rtbpsWwSVS2ah9Wp/CGWcK7lWE4o0B",
|
||||
"aEIoyJG1QkSsChdYzSTiE2RGI33pidKsGJk3kZphhfzryRaEL4mjtuYHr0gOUuG8MNqeVvIZ3CtUcEqR",
|
||||
"hh1IjfBuTCB5IS1pT9t3eCVKQGSCjvXo4wXO9XdSUmjYydrOYjY+TruAzowbfnssQZXFsZztDrPaNb6k",
|
||||
"zHPGFWckRalFd3C/OKZNNsmTay9mQ0iXkAqw4vqK2CxXl/SGTYgkKaZImheRHoawsaGSMQWkOFIzIlFq",
|
||||
"Zt8CDquSsIwu+yWWszHHIrusjMJLLOBUdDnyZqvoZWOUMwJyNF6MtIJjyBZnGdFbxfSiMefq60t2Oivm",
|
||||
"GdOw1ECBDI0XyM6btGiG7uP+0j/wt53qtPrpuT4hhNvw0lT6mZ9pXKa3oOxk3w9ywkoFyF/hCcq5VJqp",
|
||||
"9Bv6oqzjuokQO1H3ay4Y2DdQt5vXvxCjlmVeC7baA6HdTxfHfGIAg559N4shYgaYqoh0lcFU4AxarAEZ",
|
||||
"v2uR+O18i/hDqTCNINysj4+lxqnZB6cZ6BMaW2+0EY6+kQjuC0g1LaTYChQxwbNkt6xldUuY80tNqs26",
|
||||
"nVWzxNBpr3uLO5iDcAasXdHHC2CozzgbCJCcziE7cn7AVXz6z62samlrK5wdO2kC8uNbSiJnWJx2G+wc",
|
||||
"g9g5E5zSD+6OXaG1GZfK+6easDlNVYkp8gPMlTcDBGY+wqYox+mMsCgD5yBnzrbUnPTSRtvo5+jNhREG",
|
||||
"tIBqzq+5NVFYsalVqdoUTnInTxjcDSguFC+ONtqbzLTr4OZiMGJy1qgQZI4VjG5jsR+nr84Hl+dnH86v",
|
||||
"Bn8+/9vg+PjYbJdyfXlmkIpF0bZXM3c5piSNT42n8FTPZ8doGjVTX76/uKxJOXHjjLu+R/Z+drJw2x3/",
|
||||
"EyNagsD0tFQzd6WjNy87zWzlg61nd6/FiMrS28gTzMh4Y9d9wL1RkZiVU5B9cRNpLGEhWUF5HJztoIiT",
|
||||
"mYoHFiglyLhUjXOsem0X5bG6blqvAMiQHZWguxkwx/AGdESinDOiuHX0BdtJh4PcXz4fD+d8Moq5uZsc",
|
||||
"CJpbMou3SzNBL40ttikaG9eWY32RMMxSGJVMWZv/blP5I7f1mKus1bUIvV6Lab5jdEGbbWknY/d2jjdn",
|
||||
"bHL3odl9NUeDnBvLaeeYVtdbg2/abOB4irXOa+hbf+AbicKLIxdQFPn0Rqy1Y6e5kpfWgCettgSIkgmk",
|
||||
"i5TqhTjj3mhJdVjF45KuWEplPL9IizPB/wuV+bnbXdhEUjsCLuLRg6cKUTDcxoLIMCFAM4lyt8JCgASm",
|
||||
"jnvJFrh7B8KEtM1XcNgFcV+Ec+OovjJWpArDVjFAfSUwk0ZorfYUF1daEHDlyKAFhqN6lGR9QX+6fP8X",
|
||||
"dOlBtdF+13g5su2Ma+BGHxE58nQYNwdTvABRN/7loLC9QAW2JqlSaKxM+RyEQZ9R9qaMtEbSBEB3NfO1",
|
||||
"IrTAQl/ent02GxcNTEdrnTKrkTUmMAjM/VkISE3Q48e9Dlx3ujrEeCg30RFW8nEtfW08ZUcrwbwPQTnB",
|
||||
"3zHBVEYdQCuUdEgS2plk1h+3cTytR0iLP+Yg+NiVNqNH1NxF7C77jbaPrsAKP2BsRd2KUKMd3kt6d1h4",
|
||||
"E70gSsvzcQ+EUWnjBk7ZXaAKISxB8LOGgWOBiYRsi8AJd3/XjAluiVHSCrGLW/nOavHIm0N0nCmj6/i0",
|
||||
"4dPr/BbXYFN7Bpk+kN9u63jrzo4+gfOItHR5SyhF9inqr8pM9ok7LI5iEpMAaU7c/T18D+ihs4NrN+Nm",
|
||||
"wK6T1MW+1BOJm3WRu83AWRt5tDaQ1tn6zPGj+XiyqP1sB08wsdEXennZiJfGEq5vOGr/Lrj+YTTG6a37",
|
||||
"Tf84cu993MflWVtIRLLbOho3hOFu6wwNx1ergbM6xaqTVYDB9nqOirAElm1XZ53Gl1jRmpONczDnJggc",
|
||||
"Musi6/PCGq2PesnW4Ur1ZL6Nl4Oba20c3SuBi9lfCdytwhCyKTQDINal2nxwCJQzUsRcMJUdqs1sv6Nx",
|
||||
"KRKZGXGTkQwNrssnT55DsHU5jdTavAhLaZnB/7Y0acxHzkMN0Zg5xrMtgOPsfRGwKFGy1Oc8xZ3Z+lMo",
|
||||
"xYVZ1IyoiP96WcbkNpDaYjCG9tcGBq2+VO+6aGxwyYUQ8HkovK1Y93Y8jlvjsAQ0I8BIrs9in6Gll+wC",
|
||||
"vZT5QMMntaudba2xxcGwiwdTtqHqP27DFrfhMuQdACu/XBTuf2b8jmq+eU0i10pXOzVht5CNoly0MS5E",
|
||||
"YHbbKXCrXahhpChil8hrMp1RMp1p1Jg8Xh9h0omvbOx451hzRRSFNTuu+DDjaZnbsBFRsjHnt8YaNAep",
|
||||
"yLQte2qzvdkuIIbkt17Vf6mP7FWOqptio4aKrN0WuCW2FYicaClip5eDNTGiDXyaCJ6foE+Kn6BPDlTy",
|
||||
"BP3McA7ZwLBqgo6Pjz9+/vx5o3ub+LQpc6+sGKtr64jB+x0oQdJLEA7Ckctm0aZ45ebdlihCSsuiTkkC",
|
||||
"3/WS3tOZ/qclctBIg+suNjyfdmK/LfJBc3zfacqcsE7jtrEwhPyEpRIY+A5ZWCCfd7Dhs8vCpex0b4VL",
|
||||
"d514dGUGRUMqFlYhcFQQcF4hMraICxsJu51xoygoAdkei90Mq21C878IlZwhyu9AoDEvWZZoia2wQSQw",
|
||||
"t+/ZFOLh970ISptj4reyVuFKsXbINo7UYCDYS9wqKlivPPu1xAIzRVhbZPgWuaizRcHVDCT5u8098Iv3",
|
||||
"Kc9LBktzgYQxH9tTwNdBczd3Z4OSarqvh1SDlFYwX1OL14Vx1so2LN9eS3lptfxBIbiI3BQ/EqDZwGT0",
|
||||
"1cJxkB2O+t89e3bUHuVn3Hzx47lNc16CnZ0hjO9yqvh8nQ20E0s12CSUBK9DD495qU7GVMtjteCBUpDN",
|
||||
"mrf5zFp3y4VWPeRaG8Y6p/a7Ny8TlHIBMkEC56N8nKCMyNvRdJwgUiRIQV5QE4yYm5i2BGmxnaQgo0GJ",
|
||||
"XEbkxUtaTr0790Lw+5zfm8gwF3RVi1GI2jLiEWavyxyzgQCc6XMFOX9Ix8gv8116n578ApQuJoRt7yqn",
|
||||
"92mC5nmCuEAZT29BmHoomLB6aHV3Z7kD3gYctwWUVem43ewHIRowanYKhjH05qWJMhA4vUWFXwZhU/3L",
|
||||
"VICxv224JqLXcW9pCWu3fRk4cWnT+mSJ1Myag8CU2oMHkUlz4cHwubsFoMVZf1YKASxETbSGYEgFxTrJ",
|
||||
"0ax7lIOUeNrNeTwhjMjZ/vbnh7BhhwBUUTLnETOKWcCDvNVqZsvlqqDoYA3Ro9aekmuTBRwzenxZ9MRm",
|
||||
"adgmN5yzGx0flecvfli60nKBXTqbeMNpa2NH1kzQJqKay3tkypptox+QbKT4rrSzjBMLnaQyPruzsra2",
|
||||
"TSjqFuXVGTHrbebt+Nj4XjeLXxwgm2AQj/NJscgIw3TJdc0ZDBQfcBOW7X7JMdPEo/+rnvnfzMONtvOY",
|
||||
"5YNpoRT2C7FxlqROpUx622Zeb3w/Ho9RX1PzC0kD6lG8EXl75j2h8aSkUfXJCm3MIawqV+F/tIluIm85",
|
||||
"XUMSK6Za5WxRrjbhsgU/cfisbiSyjBhwXAm2VVI2hv6OLl+vI3W8VIVUIwnAtvLWTygu1imDM06zUcbv",
|
||||
"2L6xhNvWm6pCQ6wAP3Cm77hWv/W+KbkFuhiluOzI13mp9o6n5GlqhK71Bo8DhOlsFAUr06ELuMHprfcB",
|
||||
"eOOC+Y7etlVTbc5QJQp93OWSr2LsbeLkIYpe+fJWtRggJxutwHuZTZaoJ8rJt4TSvWxqmwNxTB7tqLIc",
|
||||
"dHyjc324bexjpdxTqF4TMGgz9Um2pcG/EDyFrBQx8dNHPWbICMJDGz8y9AEgQx8XVFDM0Ifngx+OVkKx",
|
||||
"vd9uFCKX2qq5MH0OentkvuIL37yReihSPO7Crbuzj92Q56XWKGLu0xXVbeepLFwPMVdEG5Ih8CeeCNnd",
|
||||
"XpoJPHGxCz6IIdhJBUyMTbamzG2IQ/bGUrE+7aSyZ+9mOl1JEQnG02AUrVig9Yy6dMrnchRqnmMWMZq8",
|
||||
"4sFYZsrQuTi5Xry8pKtluMw4RIU6WTF3esZLPcCYmWRc6OqedyIgXlxN6W2o1hRYihfdS6xopb8ZWy3l",
|
||||
"rJf4qjkmH5y1XLptd98WgI4Xvvn9k42lD9y6k4DuGJVceafUEgAZzzFdtEjTREAVVLZDBMmqwMk1x0Xt",
|
||||
"rsQ45ihhgAUqBP/FfjpBf8iQjfjb7Ets95tKymOa01v7uQlRqACBMrzY2ikY3HQVtKKBGRLSUksoJiHF",
|
||||
"VQsALECclrFcxfdOLRrOCdyBOEF6mJaebtH7Ny/P0J/+66oe70rY4PTiDfrXP/6JznCWLa7ZhIs7LLIB",
|
||||
"LtUMEZNtBUzCgLBBBoWaJYhxmxfmrDdaQhOlmh0dXzNTDPrEmAVJiuw6bTKpLZheZZ72TZEvdGMCpW/0",
|
||||
"u76guCEm82ZF7YaVTGlqI9O6mtcu+cGVJM8UF3w1qO3MFvAe6LsckN6sL7DyntxyiWY8B4rH6P3lMbrS",
|
||||
"4uWEUNAb10O+/TZs8pqZXX77LeqbmuA4VQMjFx6doFfceAxAIKnKsURYAKpK2t8RNUMcF2Sgj70psOSa",
|
||||
"2bxXifr+82dv3yRoUmqpBP30Rh5ZeBkw4xyQLCA9vmbX7IyzuUYnZzX55PnRyTUboHPrhdJf9wXD0U1b",
|
||||
"efKbY/3KWyKVRKUEdPPJ3NFJvcvC5xu7eNeaocBTwqzDq+8OGmRKzqPvnyQox/fo2ZMnR2ben5jEE0AX",
|
||||
"7y+vbPGTQqGbpXr8N6hvK/sXFC/QHWEZv7NvvyvNoYCEaz4hUYqFWKAbd9vdvECvzq9cTwCJbs6v8PQm",
|
||||
"QRenV2evkY/fQDe+xP4N6rvi/L4ov/1MqLlTwez58+c/oJ+uzszzcxeUZJ7iLBMgpVnXuBldivrNLhEG",
|
||||
"UVczQO/OLmw5kAlOAfWlEoBzM8Prq6uLBPHJhKQEU01Aly//fGRziEtmItEVuhnmaXFzzTirCGFMGBYL",
|
||||
"hFmmB/NSGTuq4SVL15plXZDQC0RMBiWnEt0JXFyzip6sfoxMSg3ChtqlVrOyghOmpOVHSlJwrhjHZBc2",
|
||||
"u1sf14I6xpQnw6HTvI+dJ3nossBrfsSeZbfTizc1qeWk9/T4yfETo+YWwHBBeie958dPjp9bJ/DMnHdD",
|
||||
"c0gMcK3IvLs0rRGIcPYm6530/k8JYtGsR9/sQfJzvJlBrST4mtYLLe82aojvMEE9dGPtyzHJudrcMHTw",
|
||||
"6DDW9XboMNL1bOkw0jam+PxxqcnDsydPtmpRsBRE6NWGTvpDE/URfaTeQGb7qmVmCZE7euXK8UtAwJSJ",
|
||||
"4/qcVIJZfAsBZrVmELVIVttlAY1hhufE1MgwZnY8lcambcNMx8SbXe8HfuG2lmbvpGfFATPrMJROaeUk",
|
||||
"fS2chlGdmChoHRUuD1cbvY15vBVn5ZN7V5z/zfFGaIryeGwRCGp/ftAEinCNQj0vVCWAtmGE4SeSfR6G",
|
||||
"1iMa1k2K34Dg0FJK47hwASJNlnppujMENCRbfmGpkZKlJRMN80eeLfYgo/qmQ1qrZVPLpYvAmVHVjPGW",
|
||||
"yN+W4jOv352eVQ0MrG7Ql4RNKQxKCQnyeVNOpB5IksHmMkVhG3FKbLZY+rwnI+7afeilWyQSkHKRQXaI",
|
||||
"m8HiyoToAFvUYemg2wrQriwTvG6WacrMxvuvkcHMkG6yV72I+S7Sl62E/5CCV5vYx9kub64k8P1H6Nvv",
|
||||
"Yqt6KDzi1aYX4cU91FcSvTy/PDs6BHubmbeX95o8azzI68W9MzukE9OuiF0daT/EdezArL6S+sqrtcy+",
|
||||
"3xZl2wYGj0fUjiIOJKwZEkQZTAhz6S8VQbsKjxsktjbJysZAWWg9oli1EZUuVquTPPL0sJ+Ootcmjh8A",
|
||||
"v3YmhB2O+/a0GWA5yLDCCfJmyj8cdcZ57PgyQvq+srkvD9MkIVM1ZkcKcm1CH5R0bFWbLyzJtlKObz25",
|
||||
"P+XYmYzsSqxp1RHRroTSqIyy4cJbGtvNyhHqGHxpidNXxl+1dWzVEOA3d0k2+1s84m25RE4HOFbdjKA1",
|
||||
"O6s4atlyBshFE/JSDvwTZNQypAQm9GhXe4jzSg1tCWODmmiyi68TKX2d4iS4uyTCDOEpoFtYFJiIxPXT",
|
||||
"NX9vLzybGI9GLTm2EfXFG+kNx+gMUwrC1kvEVADOFmiG56C/4YtYMHPtMMj06dLIjjCBXtbB0TwVbEXj",
|
||||
"M1+X/yFO82ax6S98oC9VbI61GzYjcmAqNNe2HkDbxIBlAWEHoG/7MYQRgztf3Phf//gnIlKW4GnI00+N",
|
||||
"dsISKip3hNtC4rbZXYPCP0laTj8P06pJSDQM44PzMN7NSDpzvUBM/4/EutUs2Zqy0rbfhm9vgUybD0PE",
|
||||
"UzIHhpT3NRovM0PeAWwafBivHWFSAc4Qn6ApUagoKY0R6StQzQYnK/dWbAuIM7pwi5NhcURW67JNpp8/",
|
||||
"f/7DUUtzfdu5ZOu23x8fUkRpQCJ2Kru2IBlQhQ9As69AOTJI6zM7iOIKnNsSZ7KTVHtJy2nv88cIZcuq",
|
||||
"a8l0bZHxgQkwMdZB84Z1Jme+8K6d9hu5cmK3E6ZvmPLgePcfiuD9slPvlQOpth5y65q8fFliyHwLmGGt",
|
||||
"GE5UEn4FaqVfzAMibuVbMSu5H4P84vfH03sGA8FLlg2UIIUJqdOCT4gFcm3+keA8NyFBqMBT2MPHWi9o",
|
||||
"06qC+ACTTWf4j4QqEKY+Qr1ZoivEJZEeDSzDTMm2w3tXC3vIGNz2xVC1des3fTnetS8usXs5tg9drR7O",
|
||||
"TGDO0AXJxr7y656m938rLam9qNoX0o5cTT5K5MFOXaiYJyg7tXJVOxsSz/319lVaEhvF/b+wKdFTUbst",
|
||||
"0dRKy8BWzji/wtO2Kd2woRnjJjyIDZKhFeVgA1U0LUh+8DCojO1q8JnTbOtNQCq10/W3mgddOeVMav3c",
|
||||
"xH3aqhRac05xgVOjAvuI76MEeUOWm926G6vqtCaiJaIzNxVlfQz63QWXe0ynCFUIvm7aX6kD8oXpf7VG",
|
||||
"RftJ58qyJk2E/FpC+ahsEraAMNKvlSqQbv/t/z1L0F/fJSjU+DhCZqAp2rEvP3njfZsUGkjvAc0fbceX",
|
||||
"w5mrB/R42HkVahkshxjvdMkdwE2ynPbge3TUTx0sINJ1pNY5pmoXkMHkxTUjlMIU08YkNpYbfffkBy3X",
|
||||
"mukG1fOjY3Rho/im+iPXzB6IWitdVK8+R31/ygW4HEXPO729Xc+6B/b31JvHfHH7YBuDOI9Pdbc+Foc4",
|
||||
"h1FV3kIzSK1TzFIXmYOcWkPT5XpQdbluO8L+qMd9sMM6eZNMQk1DDwngeW5KIZK8zHsn30dSuR5am1gO",
|
||||
"EixsslFLk9jOVZnaCiXZD2xd1maLCJ3JxNbZ9ai1du2pwMUMZcTVSDuEUdvnjPgPkok1BbmDfYIJlV/0",
|
||||
"OF8laB+AJjdfyKHQSjeKFkB3DuarEuGiHNEbc8MsoZKLSe4zJgbz5OPhTc/7qNzri7zvTMf1aQ/hZHxp",
|
||||
"gI5EfVrT9zx40voauiggRx49IvFa23YQqYdVLncbFS8XWHvA63P5UxHsXTT9kFC4Y8jt4wDyPac0XsTO",
|
||||
"mDqXhf4vhcqaadp045Xr44fP5649a4cT58HzrqJW0VrZkP8EAD+ybXPufCaPZdqc20zeAwb9viZScWGc",
|
||||
"3eBZYWdHxNxCy+SetroDL21qwCUwheyGjtE5Tmf2+99IdEOyG58VbXvgC36HSIb6AmSZwzUzB9nNWy0q",
|
||||
"mxkGb17eHCXoxoxeelcDNUE3GVY4PPnT5fu/XDPzKrLQPkavAQs1Bqz0uZUbOGvOW6Cn38tj9EeQagCT",
|
||||
"CRfGDUvMk3/945/XzNSVhgwVIAayHOudjkGgcTmZgEhQJngx4DQDqVwS9cXvj16YNOhX51fIweyaKY7G",
|
||||
"OL2dkLgr/tLAtO2wanXhBAigQsCE3O/rsbFKVvViAwVrZ9jMtgrulQXHoKKg9glX/bCX58i9eAiz/9wT",
|
||||
"kJ0T9S8vz4/2YY4qOmqtn64atmsu5INHyH8lGSn/XldHyBJ9xOujoq1DOcbq1Lp1HGDS4uy4mgGaYZZR",
|
||||
"EMveiX6I4TM0eJTYGF7p/BRDX/wwuWaYZQiImoFAwIw13F0LoRpz34azukThI8RFLYLwmoXMQWd7Mz4Q",
|
||||
"XwiiORNh6Ma3l7sJUX+nVHIE9+avPsjFRvQITsEEoNlwLDvd+7+8/Ru6wws7Ruotxq4C55E4r6cdf5Xu",
|
||||
"w+V2cF/ahVhx3BpW8N4T1M9tiVKXQR6cWIcQsj4EAqpTnyPtBfrX//v/VZqqTWzQf3JUu1WIbS3+sBq7",
|
||||
"2SFSo6WHM/l2wsch7GIBxK5t3O+Q66C52xm1lz2hiYShbQn5IFnfZ2bqx0flWeh6eQBXu5kLYeQP12Ho",
|
||||
"xYnqdRd2zC/WZ7NoTzA+N48vAbK9jTlLooMprMRtrFwTen87ffcW1VpvrdZoZYpTPt3lVXtHbv3ikrwR",
|
||||
"FpDU9hEm7yKHaIiicakv+IMcrj4hAEk9sd6NtDWtUtdC4OUffaf/lx/QELmSQD4Sr5EqtpAK8k7EY+z5",
|
||||
"605V08Rzk7J2qbAInth+3Q979ALxnChjTLubaYHBehD6toVRW/Sd4HwnoX6Ng+jZBgdRYmqUUlNo0Uqs",
|
||||
"ne313WuTSrUwtZ0mXOS91bC8pU6hv3DCvB9k5P6mxbeCFyU1El5osXrs+j0mXTbhPhPfQ6jJuNw5oeuu",
|
||||
"HjJ+vWopG+FI8xDNzdO9GfKyHFtK1ZQ7J7LElPzd1XIzPVDR75DpgbqDeV8zXtXjtI3zfqQA6rVH64OB",
|
||||
"tNmuNQJWO+CAscVmY65Vrp/WmvVNRzdEWKa3wsU+ZrxQaHsoAYu0HdKX5nHozdnNYPFrb1kJ2M8K8BXo",
|
||||
"9o3upIfzv70m6hCK+o8lpQOTP2LRaYu8BiRXXuq+FwFkglzHzwaLhle2oqFPwfvRISarTku/NXS+Ne1n",
|
||||
"K8AfomIHpUFuk0OPM2Qb3SLFozGqXdEY52bTaDbq2erO1EYzskVpN3jn3rlBnU6Wzm62Dvd8KJl7EFEl",
|
||||
"Lqf5lpmR+ANcKl6LP2g2ULUtPnZLuN7FifeYR2ujR+3heNFOiyQcqGaiIVaTkTewc6I8UO6u13Bd0Ws7",
|
||||
"Nd9XutgBUeQYSUNx27wP0yApwhIhWK77hI120LGO3LU4ki3XutLSaRNlNSAS/XRji11IMKDuMDc88uRi",
|
||||
"zMq13C19mdeW6/9WW22NRP2aulOnjZhZDAIC4ib9ekgtwtKlMI/yUjnFIBjeNefgQTCH3s2AoSrCdsUa",
|
||||
"Xk+kubLa5VecTKNX+JgJNZbY1xboefbkWQc6tEbyeqXPvY22SiswagYVJRvFxubs1wi6O7027TVRih1+",
|
||||
"0tdxrNRPRNpxKX5bCDqt4e2vschQBhSUKQDPuEKyLAouTBX3makL77rpSgT3RNp6BaEfSEjhtzEFL59H",
|
||||
"WKMWer4bZ3yR8HO9tEcMQW/jiFrhoUfiiFrBooD1KlRyH05wVYnXByJc+EHbyN57FNfcMaKgW/zDbymS",
|
||||
"wLfRf7w4gkAaB4oiKCpS8/RMwXWS2yyJ+LcPWrMt3ukESTwBtUBzTOfgjt7LV384OkanocC3Ps6LurSz",
|
||||
"Iupcftd2WF+EXvRf/qRukmRrqeVfSywwU6ZRVbQjz2rPq6rhf63ZVa2zlfEjhTExtfZxyywHhvsab4mr",
|
||||
"KiEJe0ZCfVdvHtAQVbBFw+omOerOa0t3h4uzCclvJYVuxf0/lBRk7yuoS68Xcsg0CbOvA9eZR6JsqmaV",
|
||||
"e3WHgKqzoFw1GhSEr70wcrf+JHLNIW0RJzxRIFbzuv3J97xVH2uA+ivVyOpr3EYnexQ2v7DxD04balAJ",
|
||||
"6mclpoOIM3stzXRg64eugroflTywdvJvSh6GIhx7H5IwXFzlOmPkqRtzCUoRNn3cs765lgMe92F3hyi4",
|
||||
"bheJpJsT9W8JpQN5R1Q6SxCDOYiBr7lqKtoc7XAlxGXaD5hIE+foF0EkqtMLhQz1nz15hn5XhUIeo7f8",
|
||||
"DkzxI6Js+oNbOrqZUj7G9FhPN8KpOkHXPT6ZXPdutAaLMxtTabc08oPQLbgsCn/tkDyHjGAFdKG//uTo",
|
||||
"xFxNNbDYUpxmHnSHXXwMZuuLjpjTJkaeu50bejv6EaYXDRpt8ww9nNz6dfLIqcGmTdhRgthQ7UcUksPx",
|
||||
"6GndV6dsnJAvGmT2/scfNUsEgtzv/BRE3g5MwO8GafkDkbdnbtxjphT7ZRxSUCbyFnkYHEheFvU5tzwa",
|
||||
"NXoaycj2kKRgVd/lgr1ZI826W3KOaSm5bcDL2mSdvWeqheFtZcheIsXvYhVsa24mYIfpOXTOMi3V1Kfu",
|
||||
"S1DSloEZKW51FxPJQiS6hcLeCDOT17g42qUsR5sade5aVlofWqQQzapfEPVnBAQW6WwxwHdYwNELlGKR",
|
||||
"EYapbds34SKFrE2RWk9zX4ciVV/j4zi3mgUQvkj/iQZFuoilneq/+J4D6y6FSzemc0YgHCzTPPTUZhPe",
|
||||
"S3p3zlSU9FJBFEmjvcYfJA++SyOg35KZ3+L8Ea38nugOVbk40PB2rXhqPGJTaHB6+yD5M6fprYN5HOvr",
|
||||
"d25fPVy7ktO0itDEDng7dippQC8vrXRzcPC9KxXU4HcIJ4Re66hkitCuFeBbe0Qud8SvZt6jieOXpQgN",
|
||||
"4EAKrsTK1dXbQxCFAMnp/GHo4oOd+8Ck0Y7mFWR+FchzUKjwl2NWYkoXu6JPq6obhAY7pFtrTGt+GZmo",
|
||||
"0/947x/yWtdYecxb3VLFoS51Mxvqm5QqFRLrChD20dFuLn077UO7HywqvmJfe0EYg2zkoBqvibjqbteY",
|
||||
"aPW1f33edccQX71v3dCkafBE9K8eKTu60WsUPnRTdTjN/+pHfn0H2M4HUtjT/vhyU3nbj6ktaPG2/TG0",
|
||||
"Z96/SceTXZOfrszorY+if7dkDrPNA5KOA9shGB1YhjDDdCGJK19Iqc/hMJXJI4lU2yR0PGQyld4KpKUx",
|
||||
"3eipx4AFiNNSzXonP3/UGLfd2O2HS0F7J70hLshw/tTQg9vPatsml9zv8s5DXoEpOWtK4dRt581t2LSa",
|
||||
"lTgU23kNQuu3pOptRaSts0w4S3ybo1rhKNfLaHXO8+1SHdx8vMq++BS3e5gtuuJCfYtq4wNqdOeMLijU",
|
||||
"oKh6K7hOjUnwUkrUzyAlGQxxqmrTQr1E06eWuEuztCB66fOsNkM431bfrztgkqVQoyQ4x6qpnBdldaKQ",
|
||||
"IelIw+UJV7a6Wo7jp2jqlUxsxrL5bkZU4qoPJihk43tMNbgsBu6CC7X6nqvk8Pnj5/8OAAD//2XM3suJ",
|
||||
"7gAA",
|
||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpClq7JlkI9f3Q5E1thM71lqafJuKXBTYfUhihAZ6ADQlxuWq",
|
||||
"/NoH2MoT5km2cO1uEk02L7KcqfyxJTUaDZwLcO7nUy/lecEZMCV7p596M8AZCPPjxTWe6v8zkKkghSKc",
|
||||
"9U57H0DyUqSA5iAk4QxNuEBvJoN3WKWzXtKT6QxyrN9TiwJ6pz2pBGHT3ufPn5NegQXOQbkPnJdCcrH6",
|
||||
"ifcF/qUElJrHaCJ4jjAqBMwJLyUSIAvOJHwjEYMHNbLDekmP6Hd/KUEsekmP4Vx/PDxsX1bSu2CKqMWb",
|
||||
"bHUlP/305iXiAklaTlEfjqfH6HbGpTqdlWNB5O2R/2yB1az6Ksl6SU/ALyURkPVOlSihywquaBkBuH3W",
|
||||
"WMK9PM1xOsgJI7cJuqUP6WmKs2zRth797pYr+lHw/Jrotz9FAaux0gDrhIscq95pL8MKBkq/mkTmfZNB",
|
||||
"XnAFLF38CRaruz2nBJgaTIGBwAoydAeLF0hAQfFConuiZoShZ9/PkABVCobUDBAXZEoYpoE0PBQsMVeL",
|
||||
"rn18oL9eX3+OH94Cm6pZ7/S7Z/8ruvSJpfFVDF3jaUWmhAv06uL6Bfr+u2eIs8AnOZG545H44ioe2gZR",
|
||||
"b0lOVBuWqHlYnyCDCS6p6p3+cJLoPZO8zHunz070b4TZ374LuydMwRSE+dA1X0cPim9PDZ/1Ti3GzHlw",
|
||||
"CSwjbHpWFILPMdV/SjlTwMz+cFFQkmIN8+HPUgP+U+2D/1PApHfa+x/D6jgb2qdyGCY0n1yitxlmU0BS",
|
||||
"4SlkLxBGOSg8wO4NdI8lSgUYSuxnJaYDvSLB6VHvc9K7FHxMIV+z0MKO+M12C/bzRtZ7IQQXqP/hx3P0",
|
||||
"++9/+J1ZxhWZMkx/KjSss4NBzc4aW4P7EpJ+hMe8weLZFJg6SxWZE2UYvBC8AKGIRTJ2T0aWGj71gGma",
|
||||
"+1tPcU5HKabUMACWnGkq0d9OiWagXtLL02Lk6Q5kiqnZV+/jCmklPaxXMSJZhGmSXsqFAPuyG8JKSvGY",
|
||||
"gue4lVeyUtjxuVwzPvBL0gNzbHedvrHQ2iyEFaUayTLPsVh0momXattXJEi5BShkmaYg14FhzDkFzPRg",
|
||||
"xe+AjVJeWnLcDDdDBvZQ6bAWK7R0vHuqY/Vv9opWslejlGSJNiuy4uOfIVX6e/WzaZWuLXutkps9QEZY",
|
||||
"dV2spfps/TubadbNMe5GBvBQEAFyq2Vakgljy9LCdXnYHWFZndfhAdJSWaYuOCXpYpCag1j/jpUCwQYG",
|
||||
"Ge0MXuAF5TgisxkRSeOGS8iQnR1lZDJpB1mFX0Hk3Sil2JL3Kuk7CW31gcKqlPUtFvYy01RlaAYyc5Yx",
|
||||
"Yn6wsLZi4pzfQRbdoyztwtbKhFoCCvdVylkKgsnN5BHjBycnOlJuQMPhMOy0QS4NEl/HNh9KCluxDi4V",
|
||||
"ZzxfjCjMgdbhq59U14DhB5iDiMLRncX+xmnC8h1eoDEgPJZK4FShPmEzEERJlPF7dtSF0TpywSbiSnkB",
|
||||
"I7vW9SjXKlcBYmDHIj4HIUgGsstanTgau25iJBGnhSW0VLNuQv65oZOnJ4F9cbOemToBLQqqMiPqgimx",
|
||||
"2A5EqeKi6/VtBy9LX+YW7CU9/UWsrMq8kAq8kpeVtAWyuwhToDChta1UEDiM2JSDmvFuUxhNuZPYY8we",
|
||||
"I1J0G22OyVHKM+go9xxAkqkwGxg3TmWWEK9AKT3jCqndWc0cHnBe6DX3ppSPMT3WFDzCqYodbqVVCraS",
|
||||
"HuaYlnF27H5K3Rk93s60/hw6n0F6t7rZlLMJidhd/oIpsXqOPmrd7RehV43WOhnWhN+O94LemphjOpJx",
|
||||
"al6WnmZKFXqeVP+bEXmnL2AQamCuZA2OTJCJxlJhJRApZwO7tSgHUyzVaAaYqohx43pGJEo16L6RiN8z",
|
||||
"lHOpkIAUmEJzEBlJ1TE6Y8hy7jcS2ZkQkUY0uedCKsQn+hcJCKeCS4n09epAZyeXCZIcKf0xItH9DCsE",
|
||||
"DwXFhEn07f1s8S3C/hN6QAZTgTPIjtGfS0pRyRSh5nNmMjQh+qOiZPJYXxAebmZhBj7udf0jv9cndcnu",
|
||||
"mP7pY4cr1MBLlMzRehNe/z0DZvdhl6IHIyvvhuX+FzIXlV6gXt9u0n2rLKqwmMIGobFPmFSYpTAwN1vW",
|
||||
"ScyxE7eIUW52/RD19b9bzUxy4FprjTPAmtMg6f2dsy664hp51/F2jQ3rK6p4vMPx0ibfVIfMuhMkGOfa",
|
||||
"tOnmSRGG//bkJHmCc6MDBW6iofWUEDb4XXR/HvPrMV1HciveLr1ldxe0bcJT5LJfR++fY4vUMiSZOEve",
|
||||
"bvJz6u+/lSFje6LhjFgdlijI43Kw+wMWAi/iwt9BrB8dL05nKhgZNGXA0nUHASvzsYV+ZV2MIVZAyvMc",
|
||||
"mLG+BLDuazkQvFSwrLwMrCxVU2BmnLaYAoyttbOJ7o7QzoMrbt3hCI2rOna3TTvuEqlstBlYT9A5Zwoe",
|
||||
"VITijdluQijIkbUdRWxBl1jNpBY+zGikLz1RmhUj8yZSWtDwrydbEL4kjtqWpCWSg1Q4L4yOrgUSBg8K",
|
||||
"FZxSpGEHUrVd+BGFo5CWtKftO7wWJSAyQcd69PEC5/o7KSk07GRtZzHLLKddQGfGDb89lqDK4ljOdodZ",
|
||||
"7RpfMsFwxhVnJEWpRXdwmjmmTTZpAWsvZkNIV5AKsErWirIjV5f0hk2IJCmmSJoXkR6GsLF8kzEFpJy0",
|
||||
"mprZt4DDqv4io8t+ieVszLHIripT/hILOMOKHHljY/SyMVIzATkaL0ZaLTVki7OM6K1ietmYc/X1Jeuq",
|
||||
"FfOMQV9qoECGxgtk501a9Hn3cX/pH/jbTuFd/fRcnxDCbXhpKv3MzzQu0ztQdrIfBjlhpQLkr/CkofDo",
|
||||
"i7KO6yZC7ETdr7ngFtlA3W5e/0KMWpZ5LVjYD4R2P10c84lVc559P4sholItm+AKWlh8AVodiz7xelz0",
|
||||
"oVSYRhBu1sfHUuPU7IPTDPQJjVmlN34jtdIJqaaFFFuBIiZ4emVxM+Y6qpwrILPXvcWdVhWd2XFX9PEC",
|
||||
"GOozzgYCJKdzyI6c93YVn/5zK6ta2toKZ8dOmoD8+JaSyBkWp90GO8cgdsEEp/SDu2NXaG3GpfJexSZs",
|
||||
"zlJVYor8AGeqQGDmI2yKcpzOCIsycA5y5iyCzUmvbIyUfo7eXBphQAuo5vyaW8OSFZtalapNQUD38pTB",
|
||||
"/YDiQvHiaKOV0Ey7Dm4uciYmZ40KQeZYweguFrFz9upicHVx/uHievCni78Ojo+PzXYp15dnBqlYFG17",
|
||||
"NXOXY0rS+NR4Ct/p+ewYTaNm6qv3l1c1KSduUnPX98jez04Wbrvjf2JESxCYnpVq5q509OZlp5mtfLD1",
|
||||
"7O61GFFZeht5ghkZH/q6D7g3KhKzcgqyL24ijSUsJCsoj4OzHRRxMlPxcBClBBmXqnGOVa/tojy2WTJr",
|
||||
"VwBkzkqZoHtrr4OacTHnjChu3bPb2A795fPxcC5Do5ibuylqbDSLd/bWeyxRY4s7WxZzrC8ShlkKI2NZ",
|
||||
"3T0EwR+5rcdc5WOoxVX2WhwqHWNC2mxLO7kotnOXOmOTuw/N7qs5GuTcWE47x7Q6TBt80+a5wFOsdV5D",
|
||||
"3/oD30gUXhy5MLDIpzdirR07zZW8tAY8abUlQJRMIF2kVC/EGfdGS6rDKh6XdMVSKuOvR1qcCV57qMzP",
|
||||
"3e7CJpLaEXAZj/k8U4iC4TYWRIYJAZpJlLsVFgIkMHXcS7bA3TsQJhBxvoLDLoj7IpwbR/W1sSJVGLaK",
|
||||
"AeorgZk0Qmu1p7i40oKAa0cGLTAc1WNb6wv649X7P6MrD6qN9rvGy5FtZ1wDN/qIyJGnw7g5mOIFiLrx",
|
||||
"LweF7QUqsDVJlUJjZcrnIAz6jLI3ZaQ1/ikAuquZrxWhBRb68vbsttm4aGA6WuuUWY2HMuFcYO7PQkBq",
|
||||
"QlU/7nXgutPVIcZDuYmOsJKPa+lr4yk7WgnBfgzKCf6OCaYy6gBaoaRDktDOJLP+uI3jaT1CWvwxB8HH",
|
||||
"rrQZPaLmLs562W+0fUwMVvgRI2LqVoQa7fBe0rvHwpvoBVFano97IIxKGzdwyu4CVQg8CoKfNQwcC0wk",
|
||||
"ZFuEu7j7u2ZMcEuMklaION3Kd1aLIt8cWOVMGV3Hpw2fXue3uAab2jM0+JH8dltHyXd29AmcR6SlqztC",
|
||||
"KbJPUX9VZrJP3GFxFJOYBEhz4u7v4XtED50dXLsZNwN2naQu9qWeSLSzi7duhjvbeLG14c/O1meOH83H",
|
||||
"k0XtZzt4gomNvtDLy0a8NJZwfcNR+3fB9Q+jMU7v3G/6x5F77+M+Ls/aQiKS3dYx1CF4eltnaDi+Wg2c",
|
||||
"1SlWnawCDLbXc1SEJbBsuzrrNL7EitacbJyDOTeh+5BZF1mfF9ZofdRLtg5Xqqdgbrwc3Fxrox9fCVzM",
|
||||
"/kLgfhWGkE2hGQCxLkHqg0OgnJEi5oKp7FBtZvsdjUuReNqIm4xkaHBTnpw8h2DrchqptXkRltIyg/+y",
|
||||
"NGnMR85DDdFIR8azLYDj7H0RsChRstRnqsWd2fpTKMWFWdSMqIj/elnG5Db83WIwhvbXBgatvlTvumhs",
|
||||
"cMmFEPB5KLytWPcOHQkooBkBRnJ9Fvu8Or1kF+ilzAe2C4PsJv03jS0Ohl08mLINVf9xG7a4DZch7wBY",
|
||||
"+eWicP8T4/dU881rErlWutqpCbuDbBTloo1xIQKzu06BW+1CDSNFEbtEXpPpjJLpTKPGZF/7CJNOfGUj",
|
||||
"/jtnCCiiKKzZccWHGU/L3IaNiJKNOb8z1qA5SEWmbTlvm+3NdgExJL/1qv5LfWSvclTdFBs1VGTttsAt",
|
||||
"sa1A5ERLETu9HKyJEW3g00Tw/BR9UvwUfXKgkqfobwznkA0Mqybo+Pj44+fPnze6t4lPdjP3yoqxuraO",
|
||||
"GLzfgRIkvQLhIBy5bBZtildu3m2JIqS0LOqUJPB9L+l9N9P/tEQOGmlw3cWG59NO7LdFFm+OHzpNmRPW",
|
||||
"adw2FoaQVbJUuATfIwsL5LNFNnx2WbiUne6tcOmuE4+uzaBoSMXCKgSOCgLOK0TGFnFpI2G3M24UBSUg",
|
||||
"22Oxm2G1S2kOhErOEOX3INCYlyxLtMRW2CASmNv3bOL38IdeBKXNMfFbWatwpVg7ZBtHajAQ7CVuFRWs",
|
||||
"V579UmKBmSKsLTJ8iwzi2aLgagaS/N3mHvjF+0T1JYOluUDCmI/tifvroLmbu7NBSTXd10OqQUormK+p",
|
||||
"xevCOGvFNpZvr6VswlrWpxBcRG6KHwnQbGDyMGvhOMgOR/3vnz07ao/yM26++PHcpjkvwc7OEMZ3OVV8",
|
||||
"vs4G2omlGmwSSoLXoYfHvFSnY6rlsVrwQCnIZs3bfGatu+VSqx5yrQ1jnVP73ZuXCUq5AJkggfNRPk5Q",
|
||||
"RuTdaDpOECkSpCAvqAlGzE1MW4K02E5SkNGgRC4j8uIVLafenXsp+EPOH0xkmAu6qsUoRG0Z8Qiz12WO",
|
||||
"2UAAzvS5gpw/pGPkl/kufUhPfwZKFxPCtneV04c0QfM8QVygjKd3IEwVG0xYPbS6u7PcAW8DjtsCyqok",
|
||||
"6m72gxANGDU7BcMYevPSRBkInN6hwi+DsKn+ZSrA2N82XBPR67i3tIS1274KnLi0aX2yRCqdzUFgSu3B",
|
||||
"g8ikufBg+NzdAtDirD8vhQAWoiZaQzCkgmKd5GjWPcpBSjzt5jyeEEbkbH/782PYsEMAqiiZ84gZxSzg",
|
||||
"Qd5pNbPlclVQdLCG6FFrT8m1yQKOGT2+LHpiszRskxvO2Y2Oj8rzFz8sXUHAwC6dTbzhtLWxI2smaBNR",
|
||||
"zeU9MsXottEPSDZSfFfaWcaJhU5SGZ/dWVlb2yYUdYvy6oyY9TbzdnxsfK+bxS8OkE0wiMf5pFhkhGG6",
|
||||
"5LrmDAaKD7gJy3a/5Jhp4tH/Vc/8b+bhRtt5zPLBtFAK+4XYOEtSpwI0vW0zrze+H4/HqK+p+YWkAfUo",
|
||||
"3oi8O/ee0HhS0qj6ZIU25hBWFRnxP9pEN5G3nK4hiRVTrXK2KFebcNmCnzh8VjcSWUYMOK5w3iopG0N/",
|
||||
"R5ev15E6XqpCqpEEYFt56ycUF+uUwRmn2Sjj92zfWMJtq4RVoSFWgB8403dcq99635TcAV2MUlx25Ou8",
|
||||
"VHvHU/I0NULXeoPHAcJ0NoqClenQBdzg9M77ALxxwXxHb9uqqTZnqBKFPu5yyVcx9jZx8hClynxRsloM",
|
||||
"kJONVuC9zCZL1BPl5DtC6V42tc2BOCaPdlRZDjq+0bmq3zb2sVLuKVSvCRi0mfok29LgXwieQlaKmPjp",
|
||||
"ox4zZAThoY0fGfoAkKGPCyooZujD88Hvj1ZCsb3fbhQil9pq8DB9Dnp7ZL7iC9+8kXooUjzuwq27s4/d",
|
||||
"kOeV1ihi7tMV1W3nqSxcDzFXRBuSIfAnngjZ3V6aCTxxsQs+iCHYSQVMjE22psxtiEP2xlKxPu2ksmfv",
|
||||
"ZjpdSREJxtNgFK1YoPWMunLK53IUap5jFjGavOLBWGaKB7o4uV68KKirQLnMOESF6mYxd3rGSz3AmJlk",
|
||||
"XOjqnnciIF4ST+ltqNYUWIoX3UusaKW/GVst5ayX+Ko5Jh+ctVy6bXffFoCOF7757cnG0gdu3UlAd4xK",
|
||||
"rr1TagmAjOeYLlqkaSKgCirbIYJkVeDkmuOidldiHHOUMMACFYL/bD+doN9lyEb8bfYltvtNJeUxzemt",
|
||||
"/dyEKFSAQBlebO0UDG66ClrRwAwJaaklFJOQ4qoFABYgzspYruJ7pxYN5wTuQZwiPUxLT3fo/ZuX5+iP",
|
||||
"/31dj3clbHB2+Qb96x//ROc4yxY3bMLFPRbZAJe2FFsGE2ASBoQNMijULEGM27wwZ73REpoo1ezo+IaZ",
|
||||
"Et6nxixIUmTXaZNJbZn7KvO0b4p8oVsTKH2r3/Vl4A0xmTcrajesZAqKG5nWVSp3yQ+ukHymuOCrQW3n",
|
||||
"tuz6QN/lgPRmfYGV9+SOSzTjOVA8Ru+vjpEpWTchFHwNum+/DZu8YWaX336L+qaSO07VwMiFR6foFTce",
|
||||
"AxBIqnIsERaAqkYE90TNEMcFGehjbwosuWE271Wivv/8+ds3CZqUWipBP72RRxZeBsw4ByQLSI9v2A07",
|
||||
"52yu0clZTT55fnR6wwbownqh9Nd9mXd021ZU/vZYv/KWSCVRKQHdfjJ3dFLvjfH51i7eNdQo8JQw6/Dq",
|
||||
"u4MGmUYB6IeTBOX4AT07OTky8/7EJJ4Aunx/dW2LnxQK3S51UbhFfduPoaB4ge4Jy/i9fftdaQ4FJFzL",
|
||||
"EIlSLMQC3brb7vYFenVx7To5SHR7cY2ntwm6PLs+f418/Aa69Y0RblHftVTwrRTsZ0LNnQpmz58//z36",
|
||||
"6frcPL9wQUnmKc4yAVKadY2b0aWo3+ztYRB1PQP07vzSlgOZ4BRQXyoBODczvL6+vkwQn0xISjDVBHT1",
|
||||
"8k9HNoe4ZCYSXaHbYZ4WtzeMs4oQxoRhsUCYZXowL02FRMtLlq41y7ogoRemWqIpw4PuBS5uWEVPVj9G",
|
||||
"JqUGYemKLALLCk6YkpYfKUnBuWIck13a7G59XAvqGFOeDodO8z52nuShywKv+RF7lt3OLt/UpJbT3nfH",
|
||||
"J8cnRs0tgOGC9E57z49Pjp9bJ/DMnHdDc0gMcK01gLs0rRGIcPYm6532/ncJYtHsItDsHPO3eAuKWiH3",
|
||||
"NQ0zWt5tVH7fYYJ66Mbal2OSc7W5Yei70mGs68jRYaTrtNNhpG0n8vnjUmuOZycnWzWWWAoi9GpDJ/2h",
|
||||
"ifqIPlJv+7N91TKzhMgdvXLl+CUgYMrEcX1OKsEsvoUAs1oLj1okq+2NgcYww3NiamQYMzueSmPTtmGm",
|
||||
"Y+LNrg8Dv3BbS7N32rPigJl1GEqntHKSvhbOwqhOTBS0jgqXh6to38Y83oqz8sm9+wT86ngjtLJ5OrYI",
|
||||
"BLU/P2gCRbhGoZ4XqhJA2zDC8BPJPg9DwxgN6ybFb0BwaASmcVy4AJEmS700PTUCGpItv7DU/srSkomG",
|
||||
"+QPPFnuQUX3TIa3Vsqnl0kXgzKhqxnhL5G9L8ZnX787Oq7YTVjfoS8KmFAalhAT5vCknUg8kyWBzmaKw",
|
||||
"jTglNhtjfd6TEXftGfXSLRIJSLnIIDvEzWBxZUJ0gC3qsHTQbQVoV5YJXjfLNGVm4/3XyGBmSDfZq156",
|
||||
"fhfpy/YveEzBq03s42yXN1cS+P4j9O13sVWdL57watOL8OIe6iuJXl5cnR8dgr3NzNvLe02etZXy14p7",
|
||||
"53ZIJ6ZdEbs60n6I69iBWX0l9ZVXa5l9vy7Ktm0nno6oHUUcSFizTQ0ymBDm0l8qgnYVHjdIbG2SlY2B",
|
||||
"stB6QrFqIypdrFYneeS7w346il6bOH4A/NqZEHY47tvTZoDlIMMKJ8ibKX931BnnsePLCOn7yua+PEyT",
|
||||
"hEzVmB0pyDV3fVTSsVVtvrAk20o5vmHo/pRjZzKyK7GmVUdEuxJKozLKhgtvaWw3K0eoY/ClJU5fGX/V",
|
||||
"1rFVQ4Bf3SXZ7G/xhLflEjkd4Fh1M4LW7KziqGXLGSAXTchLOfBPkFHLkBKY0KNd7SHOKzW0JYwNaqLJ",
|
||||
"Lr5OpPR1ipPg7pIIM4SngO5gUWAiEtcF2fy9vfBsYjwateTYRtQXb6Q3HKNzTCkIWy8RUwE4W6AZnkO9",
|
||||
"1RMz1w6DTJ8ujewIE+hlHRzNU8FWND73dfkf4zRvFpv+wgf6UsXmWJNoMyIHpkJLdOsBtE0MWBYQdgD6",
|
||||
"th9DGDG498WN//WPfyIiZQmehjz91GgnLKGicke4LSRuWxQ2KPyTpOX08zCtmoREwzA+OA/j/YykM9cL",
|
||||
"xPT/SKxbzZKtKStt+2349hbItPkwRDwlc2BIeV+j8TIz5B3ApsGH7V7GpAKcIT5BU6JQUVIaI9JXoJoN",
|
||||
"TlburdgWEGd04RYnw+KIrNZlW4M/f/7cBLvF7z6bhblls/aPjymiNCARO5VdW5AMqMIHoNlXoBwZpPWZ",
|
||||
"HURxBc5tiTPZSaq9ouW09/ljhLJl1bVkurbI+MAEmBjroHnDOpMzX3jXTvuNXDmx2wnTN0x5dLz7D0Xw",
|
||||
"ftWp98qBVFsPuXVNXr4sMWS+BcywVgwnKgm/ArXSL+YREbfyrZiV3I9BfvH74+k9g4HgJcsGSpDChNRp",
|
||||
"wSfEAqU2SggJznMTEoQKPIU9fKz1gjatKogPMNl0hv9IqAJh6iPUmyW6QlwS6dHAMsyUbDu8d7Wwh4zB",
|
||||
"bV8MVVu3ftOX41374hK7l2P70NXq4cwE5gxdkGzsK7/saXr/t9KS2ouqfSHtyNXko0Qe7NSFinmCslMr",
|
||||
"V7WzIfHCX29fpSWxUdz/C5sSPRW12xJNrbQMbOWMi2s8bZvSDRuaMW7Cg9ggGVpRDjZQRdOC5AcPg8rY",
|
||||
"rgafO8223gSkUjtdf6t50JVTzqTWz03cp61KoTXnFBc4NSqwj/g+SpA3ZLnZrbuxqk5rIloiOnNTUdbH",
|
||||
"oN9dcLnHdIpQheDrpv2VOiBfmP5Xa1S0n3SuLGvSRMgvJZRPyiZhCwgj/VqpAun23/6f8wT95V2CQo2P",
|
||||
"I2QGmqId+/KTN963SaGB9B7R/NF2fDmcuXpAT4edV6GWwXKI8U6X3AHcJMtpD75HR/3UwQIiXUdqnWOq",
|
||||
"dgEZTF7cMEIpTDFtTGJjudH3J7/Xcq2ZblA9PzpGlzaKb6o/csPsgai10kX16nPU96dcgMtR9LzT29v1",
|
||||
"rHtkf0+9ecwXtw+2MYjz+FR361NxiHMYVeUtNIPUOsUsdZE5yKk1NF2uB1WX67Yj7A963Ac7rJM3ySTU",
|
||||
"NPSQAJ7nphQiycu8d/pDJJXrsbWJ5SDBwiYbtTSJ7VyVqa1Qkv3A1mVttojQmUxsnV2PWmvXngpczFBG",
|
||||
"XI20Qxi1fc6I/yCZWFOQO9gnmFD5RY/zVYL2AWhy84UcCq10o2gBdOdgvioRLsoRvTE3zBIquZjkPmNi",
|
||||
"ME8+Ht70vI/Kvb7I+850XJ/2EE7GlwboSNSnNX3Pgyetr6GLAnLk0RMSr7VtB5F6WOVyt1HxcoG1R7w+",
|
||||
"lz8Vwd5l0w8JhTuG3D4OIN9zSuNF7Iypc1no/1KorJmmTTdeuT5++GLu2rN2OHEePe8qahWtlQ35TwDw",
|
||||
"E9s2585n8lSmzbnN5D1g0O9rIhUXxtkNnhV2dkTMLbRM7mmrO/DKpgZcAVPIbugYXeB0Zr//jUS3JLv1",
|
||||
"WdG2B77g94hkqC9AljncMHOQ3b7VorKZYfDm5e1Rgm7N6KV3NVATdJthhcOTP169//MNM68iC+1j9Bqw",
|
||||
"UGPASp9buYGz5rwF+u4HeYz+AFINYDLhwrhhiXnyr3/884aZutKQoQLEQJZjvdMxCDQuJxMQCcoELwac",
|
||||
"ZiCVS6K+/O3RC5MG/eriGjmY3TDF0RindxMSd8VfGZi2HVatLpwAAVQImJCHfT02VsmqXmygYO0Mm9lW",
|
||||
"wYOy4BhUFNQ+4aof9uoCuRcPYfafewKyc6L+1dXF0T7MUUVHrfXTVcN2zYV89Aj5ryQj5d/r6ghZok94",
|
||||
"fVS0dSjHWJ1at44DTFqcHdczQDPMMgpi2TvRDzF8hgaPEhvDK52fYuiLHyY3DLMMAVEzEAiYsYa7ayFU",
|
||||
"Y+7bcFaXKHyEuKhFEN6wkDnobG/GB+ILQTRnIgzd+vZytyHq74xKjuDB/NUHudiIHsEpmAA0G45lp3v/",
|
||||
"57d/Rfd4YcdIvcXYVeA8Ehf1tOOv0n243A7uS7sQK45bwwree4L6uS1R6jLIgxPrEELWh0BAdepzpL1A",
|
||||
"//q//69KU7WJDfpPjmq3CrGtxR9WYzc7RGq09Hgm3074OIRdLIDYtY37DXIdNHc7o/ayJzSRMLQtIR8l",
|
||||
"6/vcTP30qDwPXS8P4Go3cyGM/OE6DL04Ub3uwo75xfpsFu0Jxhfm8RVAtrcxZ0l0MIWVuI2Va0Lvr2fv",
|
||||
"3qJa663VGq1Mccqnu7xq78itX1ySN8ICkto+wuRd5BANUTQu9QV/kMPVJwQgqSfWu5G2plXqWgi8/IPv",
|
||||
"9P/yAxoiVxLIR+I1UsUWUkHeiXiMPX/dqWqaeG5S1q4UFsET26/7YY9eIJ4TZYxp9zMtMFgPQt+2MGqL",
|
||||
"vhOc7yTUr3EQPdvgIEpMjVJqCi1aibWzvb57bVKpFqa204SLvLcalrfUKfRnTpj3g4zc37T4VvCipEbC",
|
||||
"Cy1Wj12/x6TLJtxn4nsINRmXOyd03dVjxq9XLWUjHGkeorl5ujdDXpVjS6macudElpiSv7tabqYHKvoN",
|
||||
"Mj1QdzDva8arepy2cd6PFEC99mh9NJA227VGwGoHHDC22GzMtcr101qzvunohgjL9Fa42MeMFwptDyVg",
|
||||
"kbZD+so8Dr05uxksfuktKwH7WQG+At2+0Z30cP6310QdQlH/saR0YPJHLDptkdeA5MpL3fcigEyQ6/jZ",
|
||||
"YNHwylY09Cl4PzrEZNVp6deGzrem/WwF+ENU7KA0yG1y6HGGbKNbpHg0RrUrGuPcbBrNRj1b3ZnaaEa2",
|
||||
"KO0G79w7N6jTydLZzdbhng8lcw8iqsTlNN8yMxJ/gEvFa/EHzQaqtsXHbgnXuzjxnvJobfSoPRwv2mmR",
|
||||
"hAPVTDTEajLyBnZOlAfK3fUarit6bafm+0oXOyCKHCNpKG6b92EaJEVYIgTLdZ+w0Q461pG7Fkey5VpX",
|
||||
"WjptoqwGRKKfbmyxCwkG1B3mhkeeXIxZuZa7pS/z2nL932qrrZGoX1N36rQRM4tBQEDcpF8PqUVYuhTm",
|
||||
"UV4qpxgEw7vmHDwI5tD7GTBURdiuWMPriTTXVrv8ipNp9AqfMqHGEvvaAj3PTp51oENrJK9X+tzbaKu0",
|
||||
"AqNmUFGyUWxszn6NoLvTa9NeE6XY4Sd9HcdK/USkHZfit4Wg0xre/hqLDGVAQZkC8IwrJMui4MJUcZ+Z",
|
||||
"uvCum65E8ECkrVcQ+oGEFH4bU/DyeYQ1aqHnu3HGFwk/10t7whD0No6oFR56Io6oFSwKWK9CJffhBFeV",
|
||||
"eH0gwqUftI3svUdxzR0jCrrFP/yaIgl8G/2niyMIpHGgKIKiIjVPzxRcJ7nNkoh/+6A12+KdTpDEE1AL",
|
||||
"NMd0Du7ovXr1u6NjdBYKfOvjvKhLOyuiztX3bYf1ZehF/+VP6iZJtpZa/qXEAjNlGlVFO/Ks9ryqGv7X",
|
||||
"ml3VOlsZP1IYE1Nrn7bMcmC4r/GWuK4SkrBnJNR39eYBDVEFWzSsbpKj7ry2dHe4OJuQ/FZS6Fbc/0NJ",
|
||||
"Qfa+grr0eiGHTJMw+zpwnXkkyqZqVrlXdwioOg/KVaNBQfjaCyN3608i1xzSFnHCEwViNa/bn3zPW/Wx",
|
||||
"Bqi/Uo2svsZtdLInYfNLG//gtKEGlaB+VmI6iDiz19JMB7Z+7Cqo+1HJI2sn/6bkYSjCsfchCcPFVa4z",
|
||||
"Rp65MVegFGHTpz3rm2s54HEfdneIgut2kUi6OVH/jlA6kPdEpbMEMZiDGPiaq6aizdEOV0Jcpv2AiTRx",
|
||||
"jn4RRKI6vVDIUP/ZyTP0myoU8hi95fdgih8RZdMf3NLR7ZTyMabHeroRTtUpuunxyeSmd6s1WJzZmEq7",
|
||||
"pZEfhO7AZVH4a4fkOWQEK6AL/fWTo1NzNdXAYktxmnnQPXbxMZitLzpiTpsYee52bujt6EeYXjZotM0z",
|
||||
"9Hhy69fJI2cGmzZhRwliQ7WfUEgOx6OndV+dsnFCvmiQ2fsff9QsEQhyv/NTEHk3MAG/G6TlD0Tenbtx",
|
||||
"T5lS7JdxSEGZyDvkYXAgeVnU59zyaNToaSQj20OSglV9lwv2Zo00627JOaal5LYBL2uTdfaeqRaGt5Uh",
|
||||
"e4kUv49VsK25mYAdpufQBcu0VFOfui9BSVsGZqS41V1MJAuR6A4KeyPMTF7j4miXshxtatSFa1lpfWiR",
|
||||
"QjSrfkHUnxEQWKSzxQDfYwFHL1CKRUYYprZt34SLFLI2RWo9zX0dilR9jU/j3GoWQPgi/ScaFOkilnaq",
|
||||
"/+J7Dqy7FK7cmM4ZgXCwTPPQU5tNeC/p3TtTUdJLBVEkjfYaf5Q8+C6NgH5NZn6L8ye08nuiO1Tl4kDD",
|
||||
"27XiqfGITaHB6d2j5M+cpXcO5nGsr9+5ffVw7UrO0ipCEzvg7dippAG9vLTSzcHB965UUIPfIZwQeq2j",
|
||||
"kilCu1aAb+0RudwRv5p5jyaOX5YiNIADKbgSK9fXbw9BFAIkp/PHoYsPdu4Dk0Y7mleQ+VUgz0Ghwl+O",
|
||||
"WYkpXeyKPq2qbhAa7JBurTGt+WVkok7/471/zGtdY+Upb3VLFYe61M1sqG9SqlRIrCtA2EdHu7n07bSP",
|
||||
"7X6wqPiKfe0FYQyykYNqvCbiqrtdY6LV1/71edcdQ3z1vnVDk6bBE9G/eqTs6EavUfjQTdXhNP+LH/n1",
|
||||
"HWA7H0hhT/vjy03lbT+mtqDF2/bH0J55/yYdT3ZNfro2o7c+iv7dkjnMNg9IOg5sh2B0YBnCDNOFJK58",
|
||||
"IaU+h8NUJo8kUm2T0PGYyVR6K5CWxnSjpx4DFiDOSjXrnf7to8a47cZuP1wK2jvtDXFBhvPvDD24/ay2",
|
||||
"bXLJ/S7vPOQVmJKzphRO3Xbe3IZNq1mJQ7Gd1yC0fkuq3lZE2jrLhLPEtzmqFY5yvYxW57zYLtXBzcer",
|
||||
"7ItPcbuH2aIrLtS3qDY+oEZ3zuiCQg2KqreC69SYBC+lRP0MUpLBEKeqNi3USzR9aom7NEsLopc+z2oz",
|
||||
"hPNt9f26AyZZCjVKgnOsmsp5UVYnChmSjjRcnnBlq6vlOH6Kpl7JxGYsm+9mRCWu+mCCQja+x1SDy2Lg",
|
||||
"LrhQq++5Sg6fP37+/wEAAP//VvsxiT/wAAA=",
|
||||
}
|
||||
|
||||
// GetSwagger returns the content of the embedded swagger specification file
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/big"
|
||||
"strconv"
|
||||
@@ -24,7 +25,12 @@ import (
|
||||
const (
|
||||
defaultLimit = 50
|
||||
maxLimit = 200
|
||||
graphNodeCap = 500
|
||||
// graphNodeCap bounds the whole-graph view. The cognition transactional
|
||||
// types (execution, task) are audit records, not topology, and previously
|
||||
// crowded out every host/lxc/service; the default whole-graph view below
|
||||
// excludes them so the cap is spent on the actual fleet graph. Operators
|
||||
// still reach executions/tasks via list_entities.
|
||||
graphNodeCap = 2000
|
||||
)
|
||||
|
||||
// actorInfo returns the caller's (type, label) from the request context,
|
||||
@@ -307,14 +313,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
|
||||
// alphabetically. Without this the cap fills with exec:* rows and
|
||||
// drops every host/lxc/service/vm — and every edge those entities
|
||||
// connect — because edges require both endpoints in the node set.
|
||||
// Exclude the cognition transactional types (execution/task): they
|
||||
// are audit records rather than topology, and at ~380 rows they
|
||||
// consumed most of the old 500-node cap.
|
||||
nodes, err = s.queryEntities(ctx, `
|
||||
SELECT `+entityCols+`
|
||||
FROM entities e
|
||||
LEFT JOIN entity_status st ON st.entity_id = e.id
|
||||
WHERE e.id IN (
|
||||
WHERE e.type NOT IN ('execution','task')
|
||||
AND e.id IN (
|
||||
SELECT e2.id FROM entities e2
|
||||
LEFT JOIN relationships r ON r.valid_to IS NULL
|
||||
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
||||
WHERE e2.type NOT IN ('execution','task')
|
||||
GROUP BY e2.id
|
||||
ORDER BY count(r.type) DESC, e2.slug
|
||||
LIMIT $1
|
||||
@@ -807,7 +818,7 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
|
||||
method, path, status_code, detail, source_ip, correlation_id
|
||||
method, path, status_code, detail, source_ip, correlation_id, session_id::text
|
||||
FROM audit_log
|
||||
WHERE ($1::text IS NULL OR actor_type = $1)
|
||||
AND ($2::text IS NULL OR actor_id::text = $2)
|
||||
@@ -828,10 +839,10 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
||||
for rows.Next() {
|
||||
var a gen.AuditEntry
|
||||
var detailBytes []byte
|
||||
var actID, entID, method, path, sourceIP, corrID *string
|
||||
var actID, entID, method, path, sourceIP, corrID, sessionID *string
|
||||
var statusCode *int
|
||||
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
|
||||
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID); err != nil {
|
||||
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID, &sessionID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
a.ActorId = actID
|
||||
@@ -989,6 +1000,7 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
||||
entityID := inserted.ID
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||
&entityID, "POST", "/api/v1/entities", "",
|
||||
nil,
|
||||
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -1052,49 +1064,15 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
||||
|
||||
// Validate lifecycle transition if state is being changed.
|
||||
if req.Body.State != nil && *req.Body.State != "" {
|
||||
// Get lifecycle def for the entity's type.
|
||||
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, current.Type)
|
||||
if err != nil {
|
||||
if err == pgx.ErrNoRows {
|
||||
// No lifecycle defined — any state is allowed.
|
||||
} else {
|
||||
return nil, err
|
||||
}
|
||||
} else {
|
||||
var transitions map[string]map[string]json.RawMessage
|
||||
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
||||
return nil, fmt.Errorf("parse lifecycle transitions: %w", err)
|
||||
}
|
||||
|
||||
fromState := ""
|
||||
if current.State != nil {
|
||||
fromState = *current.State
|
||||
}
|
||||
toState := *req.Body.State
|
||||
|
||||
if toState != fromState {
|
||||
tos, ok := transitions[fromState]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%w: no transitions from %q", domain.ErrInvalidTransition, fromState)
|
||||
}
|
||||
trans, ok := tos[toState]
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("%w: %s → %s", domain.ErrInvalidTransition, fromState, toState)
|
||||
}
|
||||
|
||||
// Parse preconditions: {"requires": ["check-name", ...]}
|
||||
var gate struct {
|
||||
Requires []string `json:"requires"`
|
||||
}
|
||||
if err := json.Unmarshal(trans, &gate); err == nil && len(gate.Requires) > 0 {
|
||||
for _, check := range gate.Requires {
|
||||
if err := checkPrecondition(ctx, tx, id, current.Type, check); err != nil {
|
||||
return nil, fmt.Errorf("%w: precondition %q not met: %v",
|
||||
domain.ErrInvalidTransition, check, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
fromState := ""
|
||||
if current.State != nil {
|
||||
fromState = *current.State
|
||||
}
|
||||
if err := db.ValidateTransition(ctx, tx, id, current.Type, fromState, *req.Body.State); err != nil {
|
||||
if errors.Is(err, db.ErrTransitionInvalid) {
|
||||
return nil, fmt.Errorf("%w: %v", domain.ErrInvalidTransition, err)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1133,6 +1111,7 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
||||
patchActorType, patchActor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
|
||||
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"version": expectedVersion}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -1262,6 +1241,7 @@ func (s *Server) EnrollClient(ctx context.Context, req gen.EnrollClientRequestOb
|
||||
entityID := id
|
||||
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
|
||||
&entityID, "POST", "/api/v1/clients/enroll", "",
|
||||
nil,
|
||||
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
|
||||
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
|
||||
"info", "oikos-api", "",
|
||||
@@ -1456,6 +1436,7 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
|
||||
_, actor := actorInfo(ctx)
|
||||
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
||||
&entityID, "POST", "/api/v1/entities/provision", "",
|
||||
nil,
|
||||
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
|
||||
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
|
||||
"info", "oikos-api", "",
|
||||
@@ -1546,97 +1527,5 @@ func generateAgeKeypair() (pubKey, privKey string, err error) {
|
||||
return pub, priv, nil
|
||||
}
|
||||
|
||||
// checkPrecondition validates a named lifecycle transition precondition.
|
||||
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
||||
switch check {
|
||||
case "no-inbound-edges":
|
||||
var count int
|
||||
err := tx.QueryRow(ctx,
|
||||
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count > 0 {
|
||||
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
||||
}
|
||||
case "backups-verified":
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "backups_verified") {
|
||||
return fmt.Errorf("backup verification not recorded in entity attributes")
|
||||
}
|
||||
case "secrets-revoked":
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "secrets_revoked") {
|
||||
return fmt.Errorf("secret revocation not recorded in entity attributes")
|
||||
}
|
||||
case "ingress-dns-removed":
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "ingress_dns_removed") {
|
||||
return fmt.Errorf("ingress/DNS removal not recorded in entity attributes")
|
||||
}
|
||||
case "age-key-enrolled-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "age_pubkey") {
|
||||
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
||||
}
|
||||
}
|
||||
case "mesh-joined-if-needed":
|
||||
if entityType == "workstation" {
|
||||
var attrs string
|
||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !strings.Contains(attrs, "mesh_ip") {
|
||||
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
||||
}
|
||||
}
|
||||
case "health-check-answering":
|
||||
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
||||
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
||||
return fmt.Errorf("health check not answering (status: %s)", st.Health)
|
||||
}
|
||||
case "doc-page-complete":
|
||||
var count int
|
||||
err := tx.QueryRow(ctx, `
|
||||
SELECT count(*) FROM relationships r
|
||||
JOIN entities ke ON ke.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
||||
entityID).Scan(&count)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count == 0 {
|
||||
return fmt.Errorf("no documentation linked to entity")
|
||||
}
|
||||
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
||||
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
||||
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
||||
"ingress-live-if-public", "doc-page-stub":
|
||||
// Soft checks — always pass. These are operator-confirmed via the
|
||||
// transition request itself, or are not mechanically enforceable.
|
||||
default:
|
||||
// Unknown preconditions are skipped (operator intent overrides).
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ─── Helpers ───────────────────────────────────────────────────────────
|
||||
|
||||
@@ -155,14 +155,14 @@ func (s *Server) GetTrends(ctx context.Context, req gen.GetTrendsRequestObject)
|
||||
f, _ := slopeNum.Float64Value()
|
||||
t.Slope = float32Ptr(float32(f.Float64))
|
||||
if f.Float64 > 0.01 {
|
||||
t.Direction = gen.Improving
|
||||
t.Direction = gen.TrendDirectionImproving
|
||||
} else if f.Float64 < -0.01 {
|
||||
t.Direction = gen.Degrading
|
||||
t.Direction = gen.TrendDirectionDegrading
|
||||
} else {
|
||||
t.Direction = gen.Stable
|
||||
t.Direction = gen.TrendDirectionStable
|
||||
}
|
||||
} else {
|
||||
t.Direction = gen.Unknown
|
||||
t.Direction = gen.TrendDirectionUnknown
|
||||
}
|
||||
items = append(items, t)
|
||||
}
|
||||
|
||||
@@ -111,6 +111,7 @@ func (s *Server) PatchPattern(ctx context.Context, req gen.PatchPatternRequestOb
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -65,6 +65,7 @@ func (s *Server) CreateRelationship(ctx context.Context, req gen.CreateRelations
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||
nil, "POST", "/api/v1/relationships", "",
|
||||
nil,
|
||||
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
@@ -108,6 +109,7 @@ func (s *Server) EndRelationship(ctx context.Context, req gen.EndRelationshipReq
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
|
||||
nil, "DELETE", "/api/v1/relationships", "",
|
||||
nil,
|
||||
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
@@ -237,6 +237,11 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
|
||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
|
||||
|
||||
// Drift audit: read-only DB-side report of orphan checks, checks on
|
||||
// retired targets, stuck down/unknown probes, unmonitored declared types,
|
||||
// and dangling edges. Companion to the knowledge-graph-audit skill.
|
||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
|
||||
|
||||
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
||||
// unlike ListExecutions which sorts by target for pagination) and the
|
||||
// per-session "what did this session do" digest.
|
||||
|
||||
@@ -136,6 +136,7 @@ func (s *Server) PatchSkill(ctx context.Context, req gen.PatchSkillRequestObject
|
||||
actorType, actor := actorInfo(ctx)
|
||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
|
||||
nil,
|
||||
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
|
||||
return nil, auditErr
|
||||
}
|
||||
|
||||
251
internal/mcp/create_entity_test.go
Normal file
251
internal/mcp/create_entity_test.go
Normal file
@@ -0,0 +1,251 @@
|
||||
package mcp
|
||||
|
||||
// Integration tests for the entity-mutation MCP tools (create_entity,
|
||||
// update_entity_attributes), focused on the capability gap that stranded
|
||||
// session 23da10db: entities mutated via MCP must derive/regenerate checks the
|
||||
// same way the HTTP create/patch paths do. Guarded by OIKOS_TEST_DATABASE_URL
|
||||
// (see internal/db/integration_test.go); run via `make test-db`.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
)
|
||||
|
||||
// newTestPool mirrors internal/httpapi/api_test.go: a throwaway database,
|
||||
// migrated and seeded with ontology/inventory/policy so create_entity's type
|
||||
// validation and checkdefaults derivation have a real type tree to work
|
||||
// against.
|
||||
func newTestPool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||
if baseURL == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_mcp_test_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
qi := strings.Index(baseURL, "?")
|
||||
base, params := baseURL, ""
|
||||
if qi >= 0 {
|
||||
base, params = baseURL[:qi], baseURL[qi:]
|
||||
}
|
||||
testURL := base[:strings.LastIndex(base, "/")+1] + dbName + params
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, e := pgx.Connect(ctx, baseURL); e == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||
content, err := os.ReadFile("../../seeds/" + f)
|
||||
if err != nil {
|
||||
t.Fatalf("read seed %s: %v", f, err)
|
||||
}
|
||||
name := f
|
||||
if err := pool.SeedIngest(ctx, name, content,
|
||||
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||
var err error
|
||||
switch name {
|
||||
case "ontology.yaml":
|
||||
_, err = db.IngestOntologySeed(ctx, tx, data)
|
||||
case "inventory.yaml":
|
||||
_, err = db.IngestInventorySeed(ctx, tx, data)
|
||||
case "policy.yaml":
|
||||
_, err = db.IngestPolicySeed(ctx, tx, data)
|
||||
}
|
||||
return err
|
||||
}); err != nil {
|
||||
t.Fatalf("ingest %s: %v", f, err)
|
||||
}
|
||||
}
|
||||
return pool
|
||||
}
|
||||
|
||||
// callTool invokes a registered tool's handler in-process and returns its
|
||||
// concatenated text result.
|
||||
func callTool(t *testing.T, pool *db.Pool, name string, args map[string]any) string {
|
||||
t.Helper()
|
||||
var handler toolHandler
|
||||
for _, r := range allTools(pool, uuid.Nil) {
|
||||
if r.tool.Name == name {
|
||||
handler = r.handler
|
||||
break
|
||||
}
|
||||
}
|
||||
if handler == nil {
|
||||
t.Fatalf("tool %q not registered", name)
|
||||
}
|
||||
argsJSON, _ := json.Marshal(args)
|
||||
res, err := handler(context.Background(), &mcp.CallToolRequest{Params: &mcp.CallToolParamsRaw{
|
||||
Name: name,
|
||||
Arguments: argsJSON,
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("tool %s returned error: %v", name, err)
|
||||
}
|
||||
var sb strings.Builder
|
||||
for _, c := range res.Content {
|
||||
if tc, ok := c.(*mcp.TextContent); ok {
|
||||
sb.WriteString(tc.Text)
|
||||
}
|
||||
}
|
||||
return sb.String()
|
||||
}
|
||||
|
||||
// checkCountFor returns the number of derived check_defs targeting slug.
|
||||
func checkCountFor(t *testing.T, pool *db.Pool, slug string) int {
|
||||
t.Helper()
|
||||
var n int
|
||||
err := pool.QueryRow(context.Background(),
|
||||
`SELECT count(*) FROM check_defs cd
|
||||
JOIN entities e ON e.id = cd.target_id
|
||||
WHERE e.slug = $1`, slug).Scan(&n)
|
||||
if err != nil {
|
||||
t.Fatalf("count check_defs for %s: %v", slug, err)
|
||||
}
|
||||
return n
|
||||
}
|
||||
|
||||
// TestCreateEntity_DerivesChecks proves create_entity inserts an entity AND
|
||||
// derives its default checks in one call (the HTTP create path did this; the
|
||||
// MCP path previously could not create at all).
|
||||
func TestCreateEntity_DerivesChecks(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
slug := "service:mcp-create-test"
|
||||
|
||||
out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service",
|
||||
"slug": slug,
|
||||
"name": "mcp-create-test",
|
||||
"attributes": `{"url":"https://mcp-create-test.example"}`,
|
||||
})
|
||||
if !strings.Contains(out, "Created "+slug) {
|
||||
t.Fatalf("create_entity result = %q, want Created %s", out, slug)
|
||||
}
|
||||
if !strings.Contains(out, "Derived") {
|
||||
t.Errorf("create_entity result = %q, want a Derived check summary", out)
|
||||
}
|
||||
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||
t.Errorf("check_defs targeting %s = %d, want >=1 (create did not derive checks)", slug, got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCreateEntity_DuplicateAndInvalid covers the guard rails: a repeat create
|
||||
// is reported as "already exists" (not an error), and an unknown type is
|
||||
// rejected with a clear message.
|
||||
func TestCreateEntity_DuplicateAndInvalid(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||
}); !strings.Contains(out, "Created service:mcp-dup") {
|
||||
t.Fatalf("first create = %q", out)
|
||||
}
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||
}); !strings.Contains(out, "already exists") {
|
||||
t.Errorf("duplicate create = %q, want 'already exists'", out)
|
||||
}
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "no-such-type", "slug": "no-such-type:x", "name": "x",
|
||||
}); !strings.Contains(out, "not found in ontology") {
|
||||
t.Errorf("unknown type = %q, want 'not found in ontology'", out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdateEntityAttributes_RegeneratesChecks is the regression guard for the
|
||||
// haos session: setting an entity's `monitoring` attribute via MCP must
|
||||
// regenerate checks. Before this fix the MCP update path skipped
|
||||
// ensureDefaultChecks, so flipping monitoring produced nothing.
|
||||
func TestUpdateEntityAttributes_RegeneratesChecks(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
slug := "service:mcp-regen-test"
|
||||
|
||||
// Create with monitoring:none — no checks derived.
|
||||
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||
"type": "service", "slug": slug, "name": "mcp-regen-test",
|
||||
"attributes": `{"monitoring":"none","url":"https://mcp-regen.example"}`,
|
||||
}); !strings.Contains(out, "Created "+slug) {
|
||||
t.Fatalf("create = %q", out)
|
||||
}
|
||||
if got := checkCountFor(t, pool, slug); got != 0 {
|
||||
t.Fatalf("check_defs with monitoring:none = %d, want 0", got)
|
||||
}
|
||||
|
||||
// Flip monitoring to [http] via update_entity_attributes — checks must
|
||||
// regenerate. This is exactly what failed for service:haos.
|
||||
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||
"slug": slug,
|
||||
"attributes": `{"monitoring":["http"]}`,
|
||||
})
|
||||
if !strings.Contains(out, "Updated "+slug) {
|
||||
t.Fatalf("update result = %q, want Updated %s", out, slug)
|
||||
}
|
||||
if !strings.Contains(out, "Derived") {
|
||||
t.Errorf("update result = %q, want a Derived check summary (regeneration)", out)
|
||||
}
|
||||
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||
t.Errorf("check_defs after monitoring:[http] = %d, want >=1 (MCP update did not regenerate checks)", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestUpdateEntityAttributes_NotFound keeps the existing error contract.
|
||||
func TestUpdateEntityAttributes_NotFound(t *testing.T) {
|
||||
pool := newTestPool(t)
|
||||
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||
"slug": "service:does-not-exist",
|
||||
"attributes": `{"x":1}`,
|
||||
})
|
||||
if !strings.Contains(out, "not found") {
|
||||
t.Errorf("update missing entity = %q, want 'not found'", out)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFormatCheckResult is a pure unit test for the result-message helper, so
|
||||
// the formatting contract holds even when the DB is unavailable.
|
||||
func TestFormatCheckResult(t *testing.T) {
|
||||
if got := formatCheckResult(checkdefaults.Result{Created: 2}); !strings.Contains(got, "Derived 2 check") {
|
||||
t.Errorf("created-only = %q, want Derived 2", got)
|
||||
}
|
||||
got := formatCheckResult(checkdefaults.Result{Created: 1, Skipped: []checkdefaults.Skip{{Kind: "process", Reason: "no host"}}})
|
||||
if !strings.Contains(got, "Derived 1 check") || !strings.Contains(got, "Skipped process") || !strings.Contains(got, "no host") {
|
||||
t.Errorf("created+skipped = %q", got)
|
||||
}
|
||||
if got := formatCheckResult(checkdefaults.Result{Undeclared: true}); !strings.Contains(got, "no monitoring") {
|
||||
t.Errorf("undeclared = %q, want no-monitoring hint", got)
|
||||
}
|
||||
if formatCreateResult("a", "b", checkdefaults.Result{Created: 0}) != "Created a (b)." {
|
||||
t.Error("create result with no checks should have no suffix")
|
||||
}
|
||||
}
|
||||
119
internal/mcp/discover.go
Normal file
119
internal/mcp/discover.go
Normal file
@@ -0,0 +1,119 @@
|
||||
package mcp
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
)
|
||||
|
||||
// discoverInfraDrift compares the live Proxmox guests (pct/qm list on every
|
||||
// proxmox host) against the DB graph, surfacing drift the DB-only audit
|
||||
// cannot see: guests running with no entity (missing), and entities whose
|
||||
// pve_id is no longer live (ghost). This is the auto-discover/validate half
|
||||
// of the knowledge-graph audit skill — read-only, reaches hosts over the same
|
||||
// SSH/pct path the checks use.
|
||||
func discoverInfraDrift(ctx context.Context, pool *db.Pool) any {
|
||||
// 1. proxmox hosts to query.
|
||||
hostRows, err := pool.Query(ctx,
|
||||
`SELECT slug FROM entities WHERE type='proxmox-host' AND COALESCE(state,'active')='active'`)
|
||||
if err != nil {
|
||||
return map[string]any{"error": "query hosts: " + err.Error()}
|
||||
}
|
||||
var hosts []string
|
||||
for hostRows.Next() {
|
||||
var s string
|
||||
if hostRows.Scan(&s) == nil {
|
||||
hosts = append(hosts, s)
|
||||
}
|
||||
}
|
||||
hostRows.Close()
|
||||
|
||||
// 2. DB guests keyed by pve_id.
|
||||
type dbGuest struct {
|
||||
Slug string `json:"slug"`
|
||||
Type string `json:"type"`
|
||||
PveID string `json:"pve_id"`
|
||||
Host string `json:"host"`
|
||||
}
|
||||
dbGuests := map[string]dbGuest{}
|
||||
gr, err := pool.Query(ctx,
|
||||
`SELECT slug, type, COALESCE(attributes->>'pve_id',''), COALESCE(attributes->>'host','')
|
||||
FROM entities WHERE type IN ('lxc','vm')`)
|
||||
if err != nil {
|
||||
return map[string]any{"error": "query guests: " + err.Error()}
|
||||
}
|
||||
for gr.Next() {
|
||||
var g dbGuest
|
||||
if gr.Scan(&g.Slug, &g.Type, &g.PveID, &g.Host) == nil && g.PveID != "" {
|
||||
dbGuests[g.PveID] = g
|
||||
}
|
||||
}
|
||||
gr.Close()
|
||||
|
||||
// 3. enumerate live guests from every host.
|
||||
live := map[string]string{} // pve_id -> "host:name"
|
||||
hostErrors := map[string]string{}
|
||||
for _, hs := range hosts {
|
||||
et, rerr := remote.ResolveExecTarget(ctx, pool, hs, sshUser)
|
||||
if rerr != nil {
|
||||
hostErrors[hs] = "resolve: " + rerr.Error()
|
||||
continue
|
||||
}
|
||||
for _, cmd := range []string{"pct list", "qm list"} {
|
||||
out, eerr := sshExecStream(ctx, et.Host, et.User, et.Wrap(cmd),
|
||||
execlog.Sink(func(string, []byte) {}))
|
||||
if eerr != nil {
|
||||
hostErrors[hs+" "+cmd] = eerr.Error()
|
||||
continue
|
||||
}
|
||||
scanIDs(out, hs, live)
|
||||
}
|
||||
}
|
||||
|
||||
// 4. diff.
|
||||
var missing, ghost []string
|
||||
for id, hn := range live {
|
||||
if _, ok := dbGuests[id]; !ok {
|
||||
missing = append(missing, id+" on "+hn)
|
||||
}
|
||||
}
|
||||
for id, g := range dbGuests {
|
||||
if _, ok := live[id]; !ok {
|
||||
ghost = append(ghost, g.Slug+" (pve_id="+id+")")
|
||||
}
|
||||
}
|
||||
|
||||
return map[string]any{
|
||||
"hosts_queried": len(hosts),
|
||||
"live_guests": len(live),
|
||||
"db_guests": len(dbGuests),
|
||||
"missing_entities": missing, // in Proxmox, no DB entity
|
||||
"ghost_entities": ghost, // in DB, not live in Proxmox
|
||||
"host_errors": hostErrors,
|
||||
}
|
||||
}
|
||||
|
||||
// scanIDs parses `pct list` / `qm list` output (VMID ... Name) into the live map.
|
||||
func scanIDs(output, hostSlug string, live map[string]string) {
|
||||
sc := bufio.NewScanner(strings.NewReader(output))
|
||||
for sc.Scan() {
|
||||
line := strings.TrimSpace(sc.Text())
|
||||
if line == "" || strings.HasPrefix(strings.ToLower(line), "vmid") {
|
||||
continue
|
||||
}
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) < 2 {
|
||||
continue
|
||||
}
|
||||
id := fields[0]
|
||||
name := ""
|
||||
if len(fields) >= 4 {
|
||||
name = fields[len(fields)-1] // pct: last col is name; qm: name near end
|
||||
}
|
||||
live[id] = hostSlug + ":" + name
|
||||
}
|
||||
}
|
||||
@@ -5,7 +5,6 @@ package mcp
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"html"
|
||||
@@ -25,6 +24,7 @@ import (
|
||||
"github.com/dtoro/oikos/internal/execlog"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/policy"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
"github.com/google/jsonschema-go/jsonschema"
|
||||
"github.com/google/uuid"
|
||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||
@@ -72,9 +72,121 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
|
||||
for _, t := range allTools(pool, agentID) {
|
||||
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
|
||||
}
|
||||
|
||||
// Resource templates: let MCP clients browse and attach entities,
|
||||
// knowledge entries, and executions as conversation resources.
|
||||
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||
URITemplate: "oikos://entity/{slug}",
|
||||
Name: "Entity",
|
||||
Description: "Oikos entity by slug (e.g. host:hubris, lxc:jellyfin)",
|
||||
MIMEType: "application/json",
|
||||
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||
slug := matches["slug"]
|
||||
var id uuid.UUID
|
||||
if u, err := uuid.Parse(slug); err == nil {
|
||||
id = u
|
||||
} else {
|
||||
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", slug).Scan(&id)
|
||||
}
|
||||
if id == uuid.Nil {
|
||||
return "", fmt.Errorf("entity not found: %s", slug)
|
||||
}
|
||||
result := queryEntity(ctx, pool, slug)
|
||||
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||
}))
|
||||
|
||||
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||
URITemplate: "oikos://knowledge/{id}",
|
||||
Name: "Knowledge",
|
||||
Description: "Knowledge entry by entity slug or UUID",
|
||||
MIMEType: "application/json",
|
||||
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||
idOrSlug := matches["id"]
|
||||
var entityID uuid.UUID
|
||||
if u, err := uuid.Parse(idOrSlug); err == nil {
|
||||
entityID = u
|
||||
} else {
|
||||
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", idOrSlug).Scan(&entityID)
|
||||
}
|
||||
if entityID == uuid.Nil {
|
||||
return "", fmt.Errorf("knowledge not found: %s", idOrSlug)
|
||||
}
|
||||
result := queryRows(ctx, pool, `
|
||||
SELECT ke.title, ke.content, ke.tags::text, e.slug, e.type AS kind,
|
||||
ke.updated_at::text
|
||||
FROM knowledge_entities ke
|
||||
JOIN entities e ON e.id = ke.entity_id
|
||||
WHERE ke.entity_id = $1`, entityID)
|
||||
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||
}))
|
||||
|
||||
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||
URITemplate: "oikos://execution/{id}",
|
||||
Name: "Execution",
|
||||
Description: "Execution by UUID (returns status, result, timing)",
|
||||
MIMEType: "application/json",
|
||||
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||
result := queryRows(ctx, pool, `
|
||||
SELECT e.entity_id, te.slug AS target, e.action, e.risk_class,
|
||||
e.status, e.result::text, e.duration_ms,
|
||||
e.started_at::text, e.completed_at::text
|
||||
FROM executions e
|
||||
JOIN entities te ON te.id = e.target_entity_id
|
||||
WHERE e.entity_id = $1`, matches["id"])
|
||||
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||
}))
|
||||
|
||||
return s
|
||||
}
|
||||
|
||||
// resourceHandler adapts a simple func(ctx, params) → (string, error) into
|
||||
// an MCP ResourceHandler, reading the URI matched by a ResourceTemplate.
|
||||
func resourceHandler(pool *db.Pool, fn func(ctx context.Context, matches map[string]string) (string, error)) mcp.ResourceHandler {
|
||||
return func(ctx context.Context, req *mcp.ReadResourceRequest) (*mcp.ReadResourceResult, error) {
|
||||
uri := req.Params.URI
|
||||
matches := matchURITemplate(uri)
|
||||
if matches == nil {
|
||||
return nil, mcp.ResourceNotFoundError(uri)
|
||||
}
|
||||
|
||||
text, err := fn(ctx, matches)
|
||||
if err != nil {
|
||||
return nil, mcp.ResourceNotFoundError(uri)
|
||||
}
|
||||
|
||||
result, err := json.MarshalIndent(json.RawMessage(text), "", " ")
|
||||
if err != nil {
|
||||
result = []byte(text)
|
||||
}
|
||||
|
||||
return &mcp.ReadResourceResult{
|
||||
Contents: []*mcp.ResourceContents{{
|
||||
URI: uri,
|
||||
MIMEType: "application/json",
|
||||
Text: string(result),
|
||||
}},
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// matchURITemplate extracts parameters from a URI that matches one of the
|
||||
// oikos:// resource templates. Returns nil if the URI doesn't match.
|
||||
func matchURITemplate(uri string) map[string]string {
|
||||
// oikos://entity/{slug}
|
||||
if rest, ok := strings.CutPrefix(uri, "oikos://entity/"); ok && rest != "" {
|
||||
return map[string]string{"slug": rest}
|
||||
}
|
||||
// oikos://knowledge/{id}
|
||||
if rest, ok := strings.CutPrefix(uri, "oikos://knowledge/"); ok && rest != "" {
|
||||
return map[string]string{"id": rest}
|
||||
}
|
||||
// oikos://execution/{id}
|
||||
if rest, ok := strings.CutPrefix(uri, "oikos://execution/"); ok && rest != "" {
|
||||
return map[string]string{"id": rest}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// withActivityLogging wraps a tool handler to record agent_activity rows.
|
||||
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
|
||||
if agentID == uuid.Nil {
|
||||
@@ -461,31 +573,12 @@ func sshExecStream(ctx context.Context, host, user, command string, sink execlog
|
||||
}
|
||||
}
|
||||
|
||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
|
||||
var attrs string
|
||||
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
|
||||
}
|
||||
|
||||
var m map[string]interface{}
|
||||
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||
}
|
||||
|
||||
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
|
||||
return ip, sshUser, nil
|
||||
}
|
||||
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
|
||||
for _, proto := range []string{"netbird", "tailscale"} {
|
||||
if p, ok := mesh[proto].(map[string]interface{}); ok {
|
||||
if ip, ok := p["ip"].(string); ok && ip != "" {
|
||||
return ip, sshUser, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
|
||||
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
|
||||
// wrapper over the shared resolver (internal/remote), kept so slug-based
|
||||
// callers keep working; the shared resolver also prefers public_ipv4 over
|
||||
// mesh and honors a per-entity ssh.user.
|
||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
|
||||
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
|
||||
}
|
||||
|
||||
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
||||
@@ -564,109 +657,28 @@ func isPrivateHost(host string) bool {
|
||||
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
||||
// `qm guest exec <pve_id> -- ...` for a VM.
|
||||
//
|
||||
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
|
||||
// "strong", not "host:strong") — see pct_create's entity registration. The
|
||||
// pre-existing pct_exec handler queried resolveHost with that bare value
|
||||
// directly, which can never match a "host:*" slug and always fails; this
|
||||
// prefixes it correctly.
|
||||
//
|
||||
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
|
||||
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
|
||||
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
|
||||
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
|
||||
// which broke on nested shell quoting and forced manual escaping workarounds
|
||||
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
|
||||
// `host` attribute is optional: if absent, fall back to looking up the
|
||||
// `hosts` relationship on the VM entity, then to hubris (the documented
|
||||
// default Proxmox host) — same fallback chain as LXCs.
|
||||
// Delegates to the shared resolver (internal/remote), the single path used by
|
||||
// both the MCP `run` tool and the scheduler's checks. The historical notes
|
||||
// (host attr without prefix, vm host-resolution chain, nested-quoting
|
||||
// handling via base64) all still hold — they now live in remote.guestWrap.
|
||||
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
||||
if strings.HasPrefix(targetSlug, "host:") {
|
||||
host, user, err = resolveHost(ctx, pool, targetSlug)
|
||||
return host, user, func(cmd string) string { return cmd }, err
|
||||
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
|
||||
if err != nil {
|
||||
return "", "", nil, err
|
||||
}
|
||||
if strings.HasPrefix(targetSlug, "lxc:") {
|
||||
var pveID, hostAttr string
|
||||
// COALESCE the host column: many older LXC entities (seeded from
|
||||
// inventory, not provisioned by pct_create) have pve_id but no host
|
||||
// attribute at all. Scanning a SQL NULL into a plain string errors
|
||||
// the whole row, wrongly reporting "missing pve_id" even when it was
|
||||
// present — COALESCE avoids the NULL, "" is handled below.
|
||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
||||
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
||||
id := pveID
|
||||
return host, user, func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
|
||||
}, err
|
||||
}
|
||||
if strings.HasPrefix(targetSlug, "vm:") {
|
||||
// VMs: same host-resolution chain as LXCs (attributes.host →
|
||||
// `hosts` relationship → hubris default), but reached via
|
||||
// `qm guest exec` instead of `pct exec`. Requires the QEMU
|
||||
// guest agent running inside the VM (the standard Proxmox
|
||||
// setup; ZimaOS/HAOS in this fleet already have it).
|
||||
var pveID, hostAttr string
|
||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
||||
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
||||
id := pveID
|
||||
return host, user, func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
|
||||
// default; pipe through `jq -r .out` if available, else cat.
|
||||
// The base64 round-trip mirrors the LXC path so nested quoting
|
||||
// (the original VM-target pain point — session 55927f0a) is
|
||||
// handled identically to LXC dispatch.
|
||||
return fmt.Sprintf(
|
||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||
id, b64, id, b64)
|
||||
}, err
|
||||
}
|
||||
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||
return et.Host, et.User, et.Wrap, nil
|
||||
}
|
||||
|
||||
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
||||
// LXC/VM target. Resolution order:
|
||||
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
|
||||
// "host:" prefix in inventory.yaml and pct_create).
|
||||
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
|
||||
// looked up in the relationships table — the canonical graph source.
|
||||
// 3. "hubris" as a documented default Proxmox host fallback.
|
||||
//
|
||||
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
|
||||
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
|
||||
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
|
||||
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
|
||||
// This slug-based wrapper looks up the entity id so slug callers keep working;
|
||||
// the shared resolver takes an id directly.
|
||||
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
||||
hostSlug := strings.TrimSpace(hostAttr)
|
||||
if hostSlug == "" {
|
||||
// Fall back to the `hosts` relationship — the graph edge from
|
||||
// the Proxmox host to this LXC/VM. This is the canonical source
|
||||
// for "who owns this VM" in inventory.yaml; the `host` attribute
|
||||
// is a denormalized shortcut that not every entity has.
|
||||
var relHostSlug string
|
||||
// hosts relationship: source=host, target=lxc/vm. Look up the
|
||||
// source slug given the target.
|
||||
if err := pool.QueryRow(ctx, `
|
||||
SELECT e.slug FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
|
||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||
hostSlug = relHostSlug
|
||||
}
|
||||
var id uuid.UUID
|
||||
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
|
||||
id = uuid.Nil
|
||||
}
|
||||
if hostSlug == "" {
|
||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||
}
|
||||
if !strings.HasPrefix(hostSlug, "host:") {
|
||||
hostSlug = "host:" + hostSlug
|
||||
}
|
||||
return hostSlug
|
||||
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||
}
|
||||
|
||||
// classifyAndGate is the shared classify→execute-or-queue path for every
|
||||
@@ -728,8 +740,106 @@ func autoRun(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, comma
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// autoRunAsync starts a command in a goroutine, marking it running and returning
|
||||
// immediately. The caller gets an execution_id to poll with get_execution_status.
|
||||
// Used for commands containing sleep/wait/poll loops that would exceed the MCP
|
||||
// client timeout (120s) — the execution continues server-side.
|
||||
func autoRunAsync(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) {
|
||||
startedAt := time.Now()
|
||||
if _, err := pool.Exec(ctx,
|
||||
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||
id, startedAt); err != nil {
|
||||
slog.Error("mcp: mark execution running (async)", "error", err, "execution_id", id)
|
||||
}
|
||||
|
||||
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||
if err != nil {
|
||||
pool.Exec(ctx,
|
||||
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonErr("%s", err.Error()), int(time.Since(startedAt).Milliseconds()))
|
||||
slog.Error("mcp: async run resolve target", "error", err, "execution_id", id, "target", targetSlug)
|
||||
return
|
||||
}
|
||||
|
||||
var correlationID string
|
||||
if qerr := pool.QueryRow(ctx,
|
||||
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||
correlationID = ""
|
||||
}
|
||||
|
||||
go func() {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
slog.Error("mcp: async run panic", "panic", r, "execution_id", id)
|
||||
pool.Exec(context.Background(),
|
||||
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonErr("panic: %v", r), int(time.Since(startedAt).Milliseconds()))
|
||||
}
|
||||
}()
|
||||
|
||||
sink, flush := execlog.New(context.Background(), pool, id, correlationID)
|
||||
out, execErr := sshExecStream(context.Background(), host, user, wrap(command), sink)
|
||||
flush()
|
||||
if execErr != nil {
|
||||
pool.Exec(context.Background(),
|
||||
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonErr("%s: %s", execErr.Error(), out), int(time.Since(startedAt).Milliseconds()))
|
||||
slog.Error("mcp: async run failed", "error", execErr, "execution_id", id, "output", out)
|
||||
} else {
|
||||
pool.Exec(context.Background(),
|
||||
`UPDATE executions SET status='completed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||
id, jsonOut(out), int(time.Since(startedAt).Milliseconds()))
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// isLongRunningCommand detects shell commands containing sleep, wait, or poll
|
||||
// loops that indicate the command will exceed the MCP client timeout (120s).
|
||||
// These commands should use autoRunAsync to avoid the client timing out while
|
||||
// the command continues server-side.
|
||||
func isLongRunningCommand(cmd string) bool {
|
||||
cmd = strings.TrimSpace(cmd)
|
||||
|
||||
// sleep with duration — `sleep 30`, `sleep 1m`, etc.
|
||||
if sleepRe.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
|
||||
// while/shell poll loops with sleep: `while ...; do ... sleep; done`
|
||||
if pollRe.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
|
||||
// standalone wait command
|
||||
if waitRe.MatchString(cmd) {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
var (
|
||||
sleepRe = regexp.MustCompile(`\bsleep\s+\d`)
|
||||
pollRe = regexp.MustCompile(`\bwhile\b.*\bsleep\b`)
|
||||
waitRe = regexp.MustCompile(`\bwait\s+\d|[&;]\s*wait\b`)
|
||||
)
|
||||
|
||||
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
||||
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
||||
|
||||
// Transport-aware escalation: read-only commands on LXC targets that
|
||||
// touch config paths (/opt/, /etc/) escalate to config_mutation.
|
||||
// The classifier only scores the command text, not the transport layer
|
||||
// — SSH-ing into a container to read /opt/ is riskier than running
|
||||
// the same command locally on the Proxmox host via pct exec.
|
||||
// Caught live: "cat /etc/hostname" on lxc:dns queued as config_mutation
|
||||
// while "pct exec 107 -- cat /etc/hostname" on host:hubris auto-ran.
|
||||
if riskClass == policy.RiskReadOnly && strings.HasPrefix(targetSlug, "lxc:") {
|
||||
if strings.Contains(command, "/opt/") || strings.Contains(command, "/etc/") || strings.Contains(command, "/var/lib/") {
|
||||
riskClass = policy.RiskConfigMutation
|
||||
}
|
||||
}
|
||||
|
||||
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
||||
actionCol := "run:" + string(runParams)
|
||||
|
||||
@@ -747,6 +857,57 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
|
||||
}
|
||||
|
||||
// Target validation: host-only commands (qm, pct, pvesh, iptables) must
|
||||
// not be dispatched against lxc:/vm: targets — those aren't Proxmox hosts
|
||||
// and don't have these tools. Caught live 2026-08-04: the agent ran
|
||||
// `qm stop 100` against lxc:dns, wasting a turn.
|
||||
|
||||
// Command syntax validation: catch LLM-generated bash bugs before they
|
||||
// hit the shell. The model sometimes inserts literal \n between commands
|
||||
// or puts spaces inside flags — these always fail, so reject early.
|
||||
if syntaxErr := validateCommandSyntax(command); syntaxErr != "" {
|
||||
return textResult(syntaxErr)
|
||||
}
|
||||
|
||||
if cmdPrefix, hostOnly := hostOnlyCommand(command); hostOnly && !strings.HasPrefix(targetSlug, "host:") {
|
||||
hostSuggestion := resolveProxmoxHostSlug(ctx, pool, targetSlug, "")
|
||||
if hostSuggestion == "" {
|
||||
hostSuggestion = "host:hubris or host:strong"
|
||||
}
|
||||
return textResult(fmt.Sprintf("Cannot run %q on %s — %s is a Proxmox host command. Use target %s instead.",
|
||||
cmdPrefix, targetSlug, cmdPrefix, hostSuggestion))
|
||||
}
|
||||
|
||||
// systemctl and docker work on hosts and LXCs, but not VMs.
|
||||
if cmdPrefix, hostLxc := hostLxcCommand(command); hostLxc {
|
||||
if !strings.HasPrefix(targetSlug, "host:") && !strings.HasPrefix(targetSlug, "lxc:") {
|
||||
return textResult(fmt.Sprintf("Cannot run %q on %s — %s only works on host:* or lxc:* targets.",
|
||||
cmdPrefix, targetSlug, cmdPrefix))
|
||||
}
|
||||
}
|
||||
|
||||
// VM transport pre-flight: qm guest exec requires the QEMU guest agent
|
||||
// to be running inside the VM. If it's not, the execution would queue
|
||||
// for approval and never execute — the agent has no way to learn it's
|
||||
// stuck (spotted live 2026-08-05: vm:zimaos had qemu_guest_agent=not_running,
|
||||
// the run queued forever, and the agent fell back to unsafe raw SSH).
|
||||
if strings.HasPrefix(targetSlug, "vm:") {
|
||||
var rawAttrs []byte
|
||||
if err := pool.QueryRow(ctx, `SELECT attributes FROM entities WHERE id = $1`, targetID).Scan(&rawAttrs); err == nil {
|
||||
var attrs map[string]any
|
||||
if json.Unmarshal(rawAttrs, &attrs) == nil {
|
||||
if qga, ok := attrs["qemu_guest_agent"]; ok {
|
||||
qgaStr, _ := qga.(string)
|
||||
if qgaStr == "not_running" || qgaStr == "" {
|
||||
return textResult(fmt.Sprintf(
|
||||
"run on %s blocked: QEMU guest agent is not running (%s). qm guest exec cannot reach this VM. Start the agent inside the guest first (e.g. via SSH/systemctl start qemu-guest-agent), then re-run. If the agent is running but the entity attribute is stale, update it with update_entity_attributes(slug=%s, attributes={\"qemu_guest_agent\":\"running\"}).",
|
||||
targetSlug, qgaStr, targetSlug))
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Dedup: an identical pending command (same target, command, and
|
||||
// purpose) blocks a re-request — stops a tool-calling loop from queuing
|
||||
// the same approval repeatedly.
|
||||
@@ -821,8 +982,51 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
SELECT 1 FROM relationships
|
||||
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
|
||||
id, "task:"+sessionID)
|
||||
// Link execution to session for auto-continuation (nomos_plan_executions
|
||||
// was always empty — executions were never traceable back to sessions).
|
||||
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||
pool.Exec(ctx, `
|
||||
INSERT INTO nomos_plan_executions (execution_id, session_id)
|
||||
VALUES ($1, $2) ON CONFLICT (execution_id) DO NOTHING`, id, sid)
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-classify: write the classification decision to the classifications
|
||||
// table (was always empty — 0 rows despite 1,884 executions). The route
|
||||
// matches the auto-run vs queue-for-approval decision below.
|
||||
classRoute := "escalate"
|
||||
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||
classRoute = "auto-act"
|
||||
} else if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||
classRoute = "auto-act"
|
||||
} else if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||
classRoute = "auto-act"
|
||||
}
|
||||
classReason, _ := json.Marshal(map[string]string{
|
||||
"command": command, "purpose": purpose, "target": targetSlug, "declared_risk": declaredRisk,
|
||||
})
|
||||
classID, _ := uuid.NewV7()
|
||||
pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'classification', $3, '{}')`,
|
||||
classID, "classification:"+classID.String(), "classification for "+execSlug)
|
||||
pool.Exec(ctx, `INSERT INTO classifications (entity_id, action, risk_class, route, reasoning, correlation_id)
|
||||
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||
classID, actionCol, riskClass, classRoute, classReason, correlationID)
|
||||
// Link classification to execution.
|
||||
pool.Exec(ctx, `UPDATE executions SET classification_id = $2 WHERE entity_id = $1`, id, classID)
|
||||
|
||||
// Audit: record the execution creation with session_id for traceability.
|
||||
// Every run call, whether auto-run or queued-for-approval, gets an audit
|
||||
// entry so the agent's activity is traceable back to the originating session.
|
||||
var auditSessionID *uuid.UUID
|
||||
if sessionID != "" && sessionID != "ephemeral" {
|
||||
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||
auditSessionID = &sid
|
||||
}
|
||||
}
|
||||
_ = observability.Audit(ctx, sqlcgen.New(pool), "agent", "nomos", "run",
|
||||
&id, "POST", "/mcp", correlationID, auditSessionID,
|
||||
map[string]any{"command": command, "target": targetSlug, "risk_class": riskClass, "purpose": purpose})
|
||||
|
||||
// read_only and reversible_low both run unattended, as seeds/policy.yaml
|
||||
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
|
||||
// sync pull. Unattended + ledger.").
|
||||
@@ -841,6 +1045,11 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// agent still cannot talk a command DOWN: declaring reversible_low on
|
||||
// something computed as config_mutation keeps config_mutation.
|
||||
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||
if isLongRunningCommand(command) {
|
||||
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||
return textResult(fmt.Sprintf("run on %s (%s, async): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||
targetSlug, riskClass, id, id))
|
||||
}
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
@@ -859,6 +1068,12 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// consent. The assent window, opened only on operator approval, is the
|
||||
// sole gate for config_mutation auto-run.)
|
||||
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||
if isLongRunningCommand(command) {
|
||||
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||
slog.Info("mcp: run async via assent window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (config_mutation, async via assent window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||
targetSlug, id, id))
|
||||
}
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
@@ -874,6 +1089,12 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
||||
// operator isn't asked to re-type "I confirm" for every single command
|
||||
// against the thing they just confirmed.
|
||||
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||
if isLongRunningCommand(command) {
|
||||
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||
slog.Info("mcp: run async via destructive window", "target", targetSlug, "execution_id", id)
|
||||
return textResult(fmt.Sprintf("run on %s (destructive, async via confirmed-target window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||
targetSlug, id, id))
|
||||
}
|
||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||
if xerr != nil {
|
||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||
@@ -940,22 +1161,112 @@ func executeApprovedViaAPI(ctx context.Context, execID uuid.UUID, targetSlug, ac
|
||||
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
|
||||
// check used by the P1 plan-first gate.
|
||||
|
||||
// hostOnlyCommands maps command prefixes that are only valid on Proxmox host
|
||||
// targets (not LXCs or VMs). Running these against an lxc: or vm: target
|
||||
// always fails with "command not found" and wastes a turn.
|
||||
var hostOnlyCommands = map[string]bool{
|
||||
"qm": true,
|
||||
"pct": true,
|
||||
"pvesh": true,
|
||||
"iptables": true,
|
||||
}
|
||||
|
||||
// hostLxcCommands maps command prefixes valid on host:* and lxc:* but not vm:*.
|
||||
var hostLxcCommands = map[string]bool{
|
||||
"systemctl": true,
|
||||
"docker": true,
|
||||
}
|
||||
|
||||
// hostOnlyCommand checks whether the leading word of cmd is a host-only
|
||||
// command. Returns the command word and true if the command can only run on
|
||||
// a host: target.
|
||||
func hostOnlyCommand(cmd string) (string, bool) {
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
parts := strings.Fields(trimmed)
|
||||
if len(parts) == 0 {
|
||||
return "", false
|
||||
}
|
||||
first := parts[0]
|
||||
// Check for shell wrappers: bash -c 'actual_cmd', sh -c 'actual_cmd'
|
||||
if (first == "bash" || first == "sh") && len(parts) >= 3 && parts[1] == "-c" {
|
||||
// The actual command is inside the -c argument; extract the first word.
|
||||
// This handles `bash -c 'qm stop 100'` but not deeply nested wrappers.
|
||||
actual := strings.Trim(strings.Join(parts[2:], " "), "'\"")
|
||||
if inner := strings.Fields(actual); len(inner) > 0 {
|
||||
first = inner[0]
|
||||
}
|
||||
}
|
||||
// Strip path: /usr/sbin/qm → qm
|
||||
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||
first = first[idx+1:]
|
||||
}
|
||||
return first, hostOnlyCommands[first]
|
||||
}
|
||||
|
||||
// hostLxcCommand checks whether the leading word of cmd is a command valid on
|
||||
// host:* and lxc:* targets but not vm:*. Returns the command word and true if
|
||||
// the command is restricted to host/lxc.
|
||||
func hostLxcCommand(cmd string) (string, bool) {
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
parts := strings.Fields(trimmed)
|
||||
if len(parts) == 0 {
|
||||
return "", false
|
||||
}
|
||||
first := parts[0]
|
||||
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||
first = first[idx+1:]
|
||||
}
|
||||
return first, hostLxcCommands[first]
|
||||
}
|
||||
|
||||
// validateCommandSyntax checks for common LLM-generated bash errors that always
|
||||
// fail at the shell. Returns an error message or "" if the command looks valid.
|
||||
func validateCommandSyntax(cmd string) string {
|
||||
// Reject literal \n (the LLM sometimes writes `echo "---" && \n curl ...`
|
||||
// — the \n is literal in the command string, not an actual newline).
|
||||
if strings.Contains(cmd, "\\n") {
|
||||
return fmt.Sprintf("Command contains literal '\\n' — use ';' or '&&' between commands, not a literal backslash-n. Command: %q", cmd)
|
||||
}
|
||||
|
||||
// Reject `&& \n` patterns (the LLM writes `cmd1 && \n cmd2` — the \n is
|
||||
// a literal newline that bash interprets as a command separator, but the
|
||||
// leading backslash makes it a syntax error).
|
||||
if andBackslashRe.MatchString(cmd) {
|
||||
return fmt.Sprintf("Command contains '&&' followed by a literal backslash-newline — remove the backslash or use ';' instead. Command: %q", cmd)
|
||||
}
|
||||
|
||||
// Reject `\` at end of command with no continuation (last line ends with
|
||||
// backslash but there's nothing after it).
|
||||
trimmed := strings.TrimSpace(cmd)
|
||||
if strings.HasSuffix(trimmed, "\\") {
|
||||
return fmt.Sprintf("Command ends with a backslash but has nothing after it to continue. Remove the trailing '\\'. Command: %q", cmd)
|
||||
}
|
||||
|
||||
// Warn on common flag typos: `head - n`, `grep - i`, `tail - n`, etc.
|
||||
// These are space-between-flag-and-value errors the LLM produces.
|
||||
if flagSpaceRe.MatchString(cmd) {
|
||||
return fmt.Sprintf("Command has a space between a flag and its value (e.g. 'head - n' instead of 'head -n'). Remove the space. Command: %q", cmd)
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
var andBackslashRe = regexp.MustCompile(`&&\s*\\\s*\n`)
|
||||
var flagSpaceRe = regexp.MustCompile(`\b(head|tail|grep|sed|awk|sort|uniq|wc)\s+(-\w)\s+\w`)
|
||||
|
||||
// sessionHasPlan reports whether this nomos session has any plan step on
|
||||
// record (any generation, any status). Used by the P1 plan-first gate in
|
||||
// classifyAndGate to refuse `run` before `propose_plan` has been called.
|
||||
// A `replaced` step (from a prior plan generation that was superseded by a
|
||||
// follow-up sub-task — see store.reopenSession) still counts: it proves the
|
||||
// agent once framed a plan for this session, and the reopen path guarantees a
|
||||
// fresh `propose_plan` will run before the next `run` anyway. Fails closed
|
||||
// (returns true) when the query errors so a transient DB issue doesn't block
|
||||
// an otherwise-valid run.
|
||||
// record that isn't `replaced`. Replaced steps (from session reopen via
|
||||
// store.reopenSession) don't count — the agent must propose fresh plan before
|
||||
// any `run`. Fails closed (returns true) when the query errors so a transient
|
||||
// DB issue doesn't block an otherwise-valid run.
|
||||
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
|
||||
if sessionID == "" {
|
||||
return true // no session → no gate (direct MCP call from a script)
|
||||
}
|
||||
var count int
|
||||
if err := pool.QueryRow(ctx,
|
||||
`SELECT COUNT(*) FROM session_plan_steps WHERE session_id = $1`,
|
||||
`SELECT COUNT(*) FROM session_plan_steps
|
||||
WHERE session_id = $1 AND status <> 'replaced'`,
|
||||
sessionID).Scan(&count); err != nil {
|
||||
return true // fail open on DB error — don't block work over a flake
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -16,7 +16,7 @@ import (
|
||||
// starts being populated when OIDC identity resolution lands.
|
||||
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
||||
action string, entityID *uuid.UUID, method, path, correlationID string,
|
||||
detail map[string]any) error {
|
||||
sessionID *uuid.UUID, detail map[string]any) error {
|
||||
|
||||
if detail == nil {
|
||||
detail = map[string]any{}
|
||||
@@ -36,6 +36,7 @@ func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
||||
Path: &path,
|
||||
Detail: detailJSON,
|
||||
CorrelationID: corr,
|
||||
SessionID: sessionID,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -105,6 +105,15 @@ var curlLeadRe = regexp.MustCompile(`(?i)^curl\b`)
|
||||
// When any of these appears, the curl command is no longer read-only.
|
||||
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
|
||||
|
||||
// curlDevNullOutRe matches curl output redirected to /dev/null in any of curl's
|
||||
// argument forms (space, =, or attached). /dev/null is a no-op sink, so a GET
|
||||
// that discards its body — the canonical reachability idiom
|
||||
// `curl -o /dev/null -w '%{http_code}' URL` — is read-only. Output to any real
|
||||
// path (-o /tmp/x) stays a potential mutation. Stripped before curlMutateRe so
|
||||
// the remaining flags (-X, -d, ...) still classify correctly: a
|
||||
// `curl -o /dev/null -X POST` stays config_mutation.
|
||||
var curlDevNullOutRe = regexp.MustCompile(`(?i)(^|\s)-o\s*/dev/null(\s|$)|(^|\s)--output[=\s]\s*/dev/null(\s|$)`)
|
||||
|
||||
// redirectOutRe matches shell output redirection to a file (> or >> followed
|
||||
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
|
||||
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
|
||||
@@ -308,6 +317,10 @@ func curlIsReadOnly(curlCmd string) bool {
|
||||
if !curlLeadRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
// -o /dev/null is a no-op sink: strip it before flag detection so the
|
||||
// canonical GET-and-discard reachability probe stays read-only.
|
||||
// A `curl -o /dev/null -X POST` still fails curlMutateRe after stripping.
|
||||
curlCmd = curlDevNullOutRe.ReplaceAllString(curlCmd, " ")
|
||||
if curlMutateRe.MatchString(curlCmd) {
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -102,6 +102,33 @@ func TestClassifyCommand_CurlPipeSh_ConfigMutation(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyCommand_CurlDevNull_ReadOnly(t *testing.T) {
|
||||
// -o /dev/null is a no-op sink — the canonical GET-and-discard
|
||||
// reachability idiom must stay read_only. Output to real paths stays
|
||||
// config_mutation. POST/data flags after stripping still gate.
|
||||
cases := []struct {
|
||||
cmd string
|
||||
cls string
|
||||
}{
|
||||
// read_only: GET with body discarded to /dev/null
|
||||
{`curl -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.8.101:8123`, RiskReadOnly},
|
||||
{`curl -sS -o /dev/null https://home.hubris.network`, RiskReadOnly},
|
||||
{`curl --output /dev/null https://example.com`, RiskReadOnly},
|
||||
{`curl -o /dev/null https://example.com`, RiskReadOnly},
|
||||
{`curl -o/dev/null -w '%{http_code}' https://example.com`, RiskReadOnly},
|
||||
// config_mutation: POST/data still caught after stripping devnull
|
||||
{`curl -o /dev/null -X POST https://example.com`, RiskConfigMutation},
|
||||
{`curl -o /dev/null -d '{"x":1}' https://example.com`, RiskConfigMutation},
|
||||
// config_mutation: -o to real path stays config_mutation
|
||||
{`curl -o /etc/caddy/Caddyfile http://example.com`, RiskConfigMutation},
|
||||
}
|
||||
for _, c := range cases {
|
||||
if got := ClassifyCommand(c.cmd, ""); got != c.cls {
|
||||
t.Errorf("ClassifyCommand(%q) = %q, want %q", c.cmd, got, c.cls)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
||||
cases := []string{
|
||||
"apt-get install -y nginx",
|
||||
|
||||
303
internal/remote/remote.go
Normal file
303
internal/remote/remote.go
Normal file
@@ -0,0 +1,303 @@
|
||||
// Package remote resolves how to execute a command on a target entity and
|
||||
// turns a plain shell command into whatever must be sent over the SSH
|
||||
// connection that reaches it.
|
||||
//
|
||||
// The canonical access model: a host or workstation is reached by direct SSH
|
||||
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
|
||||
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
|
||||
// credential per host (the host's root key), no per-guest keys, sshd, or
|
||||
// lan_ip required for execution. Network probes (http/ping) still hit a
|
||||
// guest's lan_ip directly; only command execution host-hops.
|
||||
//
|
||||
// This is the single resolver shared by the scheduler's check execution and
|
||||
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
|
||||
// directly (broken for headless/keyless/mesh-only guests), while MCP
|
||||
// host-hopped (working). Keeping one path keeps them in lockstep.
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// DefaultUser is the SSH user when an entity declares no ssh.user. The
|
||||
// Proxmox hosts and their guests are all administered as root.
|
||||
const DefaultUser = "root"
|
||||
|
||||
// ExecTarget is a resolved execution endpoint: the SSH address and user to
|
||||
// connect to, plus Wrap, which rewrites a plain command for transport.
|
||||
type ExecTarget struct {
|
||||
Host string
|
||||
User string
|
||||
// Wrap turns a plain shell command into the form that must be sent over
|
||||
// the SSH connection to this target: the identity function for a host,
|
||||
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
|
||||
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
|
||||
// nested quoting identical across both guest kinds.
|
||||
Wrap func(cmd string) string
|
||||
}
|
||||
|
||||
// IsGuest reports whether an entity type is reached via pct/qm exec through a
|
||||
// Proxmox host rather than by direct SSH. docker-container is reached via its
|
||||
// host's docker socket, not pct, so it is not a guest here.
|
||||
func IsGuest(entityType string) bool {
|
||||
return entityType == "lxc" || entityType == "vm"
|
||||
}
|
||||
|
||||
// ResolveHost resolves a `host:<slug>` to its reachable network address and
|
||||
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
|
||||
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
|
||||
// container has no mesh interface — a standalone-server with only a mesh IP
|
||||
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
|
||||
//
|
||||
// fallbackUser is used when the entity declares no ssh.user; callers pass
|
||||
// their configured default (the scheduler uses "root", the MCP run tool uses
|
||||
// its configured OIKOS_SSH_USER).
|
||||
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
|
||||
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
|
||||
}
|
||||
var m map[string]any
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
|
||||
}
|
||||
|
||||
if v, ok := m["lan_ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||
addr = v
|
||||
}
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
|
||||
}
|
||||
|
||||
user = fallbackUser
|
||||
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||
user = u
|
||||
}
|
||||
}
|
||||
return addr, user, nil
|
||||
}
|
||||
|
||||
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
|
||||
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
|
||||
// stored without the "host:" prefix), the `hosts` relationship on the guest
|
||||
// (the canonical graph edge), then "hubris" as the documented default.
|
||||
//
|
||||
// entityID is the guest's entity id; the relationship lookup uses it
|
||||
// directly rather than a slug subquery.
|
||||
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
|
||||
hostSlug := strings.TrimSpace(hostAttr)
|
||||
// Only trust a clean token as a host name. The attribute is operator/
|
||||
// agent-writable and has been polluted with prose before ("hubris
|
||||
// (confirmed via pct config…)") — using that verbatim produces a slug that
|
||||
// never resolves. Treat anything with whitespace or parens as invalid and
|
||||
// fall back to the canonical `hosts` edge below.
|
||||
if strings.ContainsAny(hostSlug, " \t()") {
|
||||
hostSlug = ""
|
||||
}
|
||||
if hostSlug == "" {
|
||||
var relHostSlug string
|
||||
if err := pool.QueryRow(ctx, `
|
||||
SELECT e.slug FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1
|
||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||
hostSlug = relHostSlug
|
||||
}
|
||||
}
|
||||
if hostSlug == "" {
|
||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||
}
|
||||
if !strings.HasPrefix(hostSlug, "host:") {
|
||||
hostSlug = "host:" + hostSlug
|
||||
}
|
||||
return hostSlug
|
||||
}
|
||||
|
||||
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
|
||||
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
|
||||
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
|
||||
switch {
|
||||
case strings.HasPrefix(targetSlug, "host:"):
|
||||
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
|
||||
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
|
||||
var (
|
||||
id uuid.UUID
|
||||
pveID string
|
||||
typ string
|
||||
hostAttr string
|
||||
)
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
|
||||
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
|
||||
}
|
||||
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
|
||||
}
|
||||
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||
}
|
||||
|
||||
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
|
||||
// target's id and type — the data the scheduler has at check-execution time
|
||||
// (check_defs carry target_id + target_type, not a slug). Guests route via
|
||||
// pct/qm exec; everything else (hosts, workstations, services resolved to
|
||||
// their hosting machine) is reached by direct SSH to the entity's own address.
|
||||
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
|
||||
if IsGuest(targetType) {
|
||||
return resolveGuest(ctx, pool, targetID, targetType, fallbackUser)
|
||||
}
|
||||
|
||||
// A service (or other non-compute target) has no address of its own — it
|
||||
// runs on whatever compute entity provides/hosts it. Resolve that host and
|
||||
// route through it: pct if the host is a guest, direct SSH (with the
|
||||
// host's correct user) if it's a machine. Previously a service check baked
|
||||
// its hosting LXC's lan_ip and SSHed it directly as root, which fails
|
||||
// because the scheduler key isn't in each LXC — only on the Proxmox hosts.
|
||||
if hostID, hostType, ok := hostingCompute(ctx, pool, targetID); ok {
|
||||
if IsGuest(hostType) {
|
||||
return resolveGuest(ctx, pool, hostID, hostType, fallbackUser)
|
||||
}
|
||||
addr, user, err := resolveHostByID(ctx, pool, hostID, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
}
|
||||
|
||||
// No hosting entity found: reach the target directly at its own address
|
||||
// (a host/workstation, or a service whose host wasn't resolvable).
|
||||
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||
}
|
||||
|
||||
// hostingCompute walks the provides/runs-on/hosts edges backward from a target
|
||||
// to the compute entity that runs it (a service's LXC, an LXC's Proxmox host).
|
||||
// Returns the host's id, type, and whether one was found. Most-specific edge
|
||||
// first: provides names the runtime container directly.
|
||||
func hostingCompute(ctx context.Context, pool *db.Pool, targetID uuid.UUID) (uuid.UUID, string, bool) {
|
||||
var hid uuid.UUID
|
||||
var htype string
|
||||
err := pool.QueryRow(ctx, `
|
||||
SELECT e.id, e.type FROM relationships r
|
||||
JOIN entities e ON e.id = r.source_id
|
||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||
AND r.type IN ('provides','runs-on','hosts')
|
||||
ORDER BY CASE r.type WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1 ELSE 2 END
|
||||
LIMIT 1`, targetID).Scan(&hid, &htype)
|
||||
if err != nil {
|
||||
return uuid.Nil, "", false
|
||||
}
|
||||
return hid, htype, true
|
||||
}
|
||||
|
||||
// resolveGuest resolves a guest's execution endpoint: the owning Proxmox host
|
||||
// (SSH'd directly) with a pct/qm exec wrapper around the command.
|
||||
func resolveGuest(ctx context.Context, pool *db.Pool, guestID uuid.UUID, guestType, fallbackUser string) (ExecTarget, error) {
|
||||
var pveID, hostAttr string
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||
guestID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", guestID)
|
||||
}
|
||||
hostSlug := ResolveProxmoxHostSlug(ctx, pool, guestID, hostAttr)
|
||||
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||
if err != nil {
|
||||
return ExecTarget{}, err
|
||||
}
|
||||
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(guestType, pveID)}, nil
|
||||
}
|
||||
|
||||
// resolveHostByID is ResolveHost keyed by entity id.
|
||||
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
|
||||
var raw string
|
||||
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
|
||||
return "", "", fmt.Errorf("entity %s not found", id)
|
||||
}
|
||||
var m map[string]any
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||
}
|
||||
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
|
||||
if v, ok := m[key].(string); ok && v != "" {
|
||||
addr = v
|
||||
break
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||
addr = v
|
||||
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||
addr = v
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if addr == "" {
|
||||
return "", "", fmt.Errorf("no IP found for entity %s", id)
|
||||
}
|
||||
user = fallbackUser
|
||||
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||
user = u
|
||||
}
|
||||
}
|
||||
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
|
||||
// Workstations carry their login as a top-level `user` attribute
|
||||
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
|
||||
// explicit ssh.user was set, so a host that genuinely wants root still
|
||||
// gets root.
|
||||
user = u
|
||||
}
|
||||
return addr, user, nil
|
||||
}
|
||||
|
||||
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
|
||||
func guestWrap(entityType, pveID string) func(cmd string) string {
|
||||
if entityType == "vm" {
|
||||
return func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
|
||||
// falling back to the raw form. Mirrors the LXC base64 round-trip.
|
||||
return fmt.Sprintf(
|
||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||
pveID, b64, pveID, b64)
|
||||
}
|
||||
}
|
||||
return func(cmd string) string {
|
||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
|
||||
}
|
||||
}
|
||||
203
internal/remote/remote_test.go
Normal file
203
internal/remote/remote_test.go
Normal file
@@ -0,0 +1,203 @@
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
|
||||
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
|
||||
// same convention as internal/scheduler/coverage_test.go.
|
||||
|
||||
func TestIsGuest(t *testing.T) {
|
||||
cases := map[string]bool{
|
||||
"lxc": true, "vm": true,
|
||||
"proxmox-host": false, "workstation": false,
|
||||
"service": false, "docker-container": false,
|
||||
}
|
||||
for typ, want := range cases {
|
||||
if got := IsGuest(typ); got != want {
|
||||
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestWrapLXC(t *testing.T) {
|
||||
w := guestWrap("lxc", "132")
|
||||
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
|
||||
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
|
||||
t.Fatalf("lxc wrap must use pct exec: %q", out)
|
||||
}
|
||||
if strings.Contains(out, "qm guest exec") {
|
||||
t.Fatalf("lxc wrap must not use qm: %q", out)
|
||||
}
|
||||
// The base64 payload must round-trip to the original command.
|
||||
i := strings.Index(out, "echo ")
|
||||
j := strings.LastIndex(out, " | base64 -d | bash")
|
||||
if i < 0 || j < 0 || j <= i {
|
||||
t.Fatalf("cannot locate base64 payload in %q", out)
|
||||
}
|
||||
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
|
||||
if err != nil {
|
||||
t.Fatalf("decode payload: %v", err)
|
||||
}
|
||||
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
|
||||
t.Fatalf("round-trip mismatch: %q", string(dec))
|
||||
}
|
||||
}
|
||||
|
||||
func TestGuestWrapVM(t *testing.T) {
|
||||
w := guestWrap("vm", "108")
|
||||
out := w("uname -a")
|
||||
if !strings.Contains(out, "qm guest exec 108") {
|
||||
t.Fatalf("vm wrap must use qm guest exec: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetUnsupported(t *testing.T) {
|
||||
// No DB needed: an unsupported slug prefix errors before any query.
|
||||
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
|
||||
t.Fatal("expected error for unsupported target prefix")
|
||||
}
|
||||
}
|
||||
|
||||
// --- integration tests (require a real Postgres) ---
|
||||
|
||||
func newRemotePool(t *testing.T) *db.Pool {
|
||||
t.Helper()
|
||||
base := testDatabaseURL(t)
|
||||
return createTestDB(t, base)
|
||||
}
|
||||
|
||||
func testDatabaseURL(t *testing.T) string {
|
||||
t.Helper()
|
||||
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||
if u == "" {
|
||||
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||
}
|
||||
return u
|
||||
}
|
||||
|
||||
func TestResolveHostPrefersLAN(t *testing.T) {
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
|
||||
uuid.New())
|
||||
|
||||
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveHost: %v", err)
|
||||
}
|
||||
if addr != "10.0.0.1" {
|
||||
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
|
||||
}
|
||||
if user != "root" {
|
||||
t.Errorf("user = %q, want root", user)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
|
||||
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
|
||||
// public_ipv4 — the scheduler container has no mesh interface.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
|
||||
uuid.New())
|
||||
|
||||
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveHost: %v", err)
|
||||
}
|
||||
if addr != "82.165.190.79" {
|
||||
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
|
||||
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
|
||||
// HOST's address (the host-hop target), wrapped as `pct exec`.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hostID := uuid.New()
|
||||
guestID := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
|
||||
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||
}
|
||||
if et.Host != "192.168.8.77" {
|
||||
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
|
||||
}
|
||||
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
|
||||
if !strings.Contains(out, "pct exec 132") {
|
||||
t.Errorf("guest wrap must use pct exec 132, got %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
|
||||
// A host-like target resolves to its own address with identity wrap.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hid := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||
}
|
||||
if et.Host != "192.168.178.182" {
|
||||
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
|
||||
}
|
||||
// Workstation's top-level `user` must be honored (the mac-mini fix).
|
||||
if et.User != "dtoro" {
|
||||
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
|
||||
}
|
||||
if cmd := et.Wrap("uptime"); cmd != "uptime" {
|
||||
t.Errorf("host wrap must be identity, got %q", cmd)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveExecTargetForCheckServiceRoutesViaHostingGuest(t *testing.T) {
|
||||
// A service has no address of its own; it must route through its hosting
|
||||
// LXC via the provides edge, host-hopping through the LXC's proxmox host.
|
||||
pool := newRemotePool(t)
|
||||
ctx := context.Background()
|
||||
hostID := uuid.New()
|
||||
guestID := uuid.New()
|
||||
svcID := uuid.New()
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'lxc:gitea','lxc','gitea','active','{"pve_id":"104","lan_ip":"192.168.8.121"}'::jsonb,1,now(),now())`, guestID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||
VALUES ($1,'service:gitea','service','gitea','active','{}'::jsonb,1,now(),now())`, svcID)
|
||||
// provides: lxc -> service; hosts: proxmox-host -> lxc
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'provides',now(),now())`, guestID, svcID)
|
||||
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||
|
||||
et, err := ResolveExecTargetForCheck(ctx, pool, svcID, "service", DefaultUser)
|
||||
if err != nil {
|
||||
t.Fatalf("ResolveExecTargetForCheck for service: %v", err)
|
||||
}
|
||||
// Reaches the proxmox host (host-hop), wrapped as pct exec into the guest.
|
||||
if et.Host != "192.168.8.77" {
|
||||
t.Errorf("Host = %q, want proxmox host 192.168.8.77 (via provides->hosts)", et.Host)
|
||||
}
|
||||
if out := et.Wrap("p"); !strings.Contains(out, "pct exec 104") {
|
||||
t.Errorf("service check must wrap as pct exec 104, got %q", out)
|
||||
}
|
||||
}
|
||||
68
internal/remote/testutil_test.go
Normal file
68
internal/remote/testutil_test.go
Normal file
@@ -0,0 +1,68 @@
|
||||
package remote
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// createTestDB provisions a throwaway migrated database off baseURL, the same
|
||||
// convention as internal/scheduler/coverage_test.go. The base URL must point
|
||||
// at a Postgres superuser-capable connection.
|
||||
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
|
||||
admin, err := pgx.Connect(ctx, baseURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect admin: %v", err)
|
||||
}
|
||||
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
|
||||
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||
admin.Close(ctx)
|
||||
t.Fatalf("create test db: %v", err)
|
||||
}
|
||||
admin.Close(ctx)
|
||||
|
||||
at := strings.LastIndex(baseURL, "/")
|
||||
testURL := baseURL[:at+1] + dbName
|
||||
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||
testURL += baseURL[at+q:]
|
||||
}
|
||||
|
||||
pool, err := db.New(ctx, testURL)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test db: %v", err)
|
||||
}
|
||||
if err := pool.Migrate(ctx); err != nil {
|
||||
t.Fatalf("migrate: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
pool.Close()
|
||||
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||
admin.Close(ctx)
|
||||
}
|
||||
})
|
||||
return pool
|
||||
}
|
||||
|
||||
func getenvOrDefault(key, def string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||
t.Fatalf("exec %s: %v", q, err)
|
||||
}
|
||||
}
|
||||
@@ -85,7 +85,16 @@ func coverageSweep(ctx context.Context, pool *db.Pool) {
|
||||
case !mon.Declared:
|
||||
undeclared++
|
||||
case mon.None():
|
||||
// Explicitly unmonitorable. Nothing to say.
|
||||
// Explicitly unmonitorable. Nothing to raise — but a type that
|
||||
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
|
||||
// open `unmonitored` signals from before the change. They are no
|
||||
// longer a gap, so close them; otherwise they linger forever,
|
||||
// because resolveCoverageSignal only runs from the hasCheck path
|
||||
// and a None() entity never gains a check.
|
||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||
resolved++
|
||||
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
|
||||
}
|
||||
case e.hasCheck:
|
||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||
resolved++
|
||||
|
||||
25
internal/scheduler/reachability_test.go
Normal file
25
internal/scheduler/reachability_test.go
Normal file
@@ -0,0 +1,25 @@
|
||||
package scheduler
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// The scheduler runs in Docker on macOS, whose VM does not route ICMP to the
|
||||
// LAN — every ping check reported "down" for hosts that were demonstrably up.
|
||||
// tcpReachable is the fallback that keeps "is it reachable" answerable.
|
||||
func TestTCPReachableAnswersWhenICMPCannot(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
// localhost:22 is open on this machine (sshd), and port 1 is not.
|
||||
if !tcpReachable(ctx, "127.0.0.1", 22, 3*time.Second) {
|
||||
t.Skip("no sshd on localhost — cannot exercise the positive case")
|
||||
}
|
||||
if tcpReachable(ctx, "127.0.0.1", 1, 1*time.Second) {
|
||||
t.Error("port 1 should not be reachable")
|
||||
}
|
||||
// Defaults to 22 when unset, which is what checkdefaults' ping configs use.
|
||||
if !tcpReachable(ctx, "127.0.0.1", 0, 3*time.Second) {
|
||||
t.Error("port 0 should default to 22")
|
||||
}
|
||||
}
|
||||
@@ -23,6 +23,7 @@ import (
|
||||
"github.com/dtoro/oikos/internal/db"
|
||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||
"github.com/dtoro/oikos/internal/observability"
|
||||
"github.com/dtoro/oikos/internal/remote"
|
||||
"github.com/google/uuid"
|
||||
"golang.org/x/sync/errgroup"
|
||||
"golang.org/x/sys/unix"
|
||||
@@ -111,12 +112,20 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
q := sqlcgen.New(pool)
|
||||
start := time.Now()
|
||||
|
||||
result := executeCheck(ctx, cd)
|
||||
result := executeCheck(ctx, pool, cd)
|
||||
|
||||
// Stamp the run before processing the result: due-ness must advance even
|
||||
// when a check fails, or a permanently failing check would be re-run on
|
||||
// every pass instead of at its declared interval.
|
||||
if err := q.MarkCheckRun(ctx, cd.EntityID); err != nil {
|
||||
// every pass instead of at its declared interval. last_health records THIS
|
||||
// check's own verdict, which is what makes the aggregation below possible.
|
||||
checkHealth := result.health
|
||||
if checkHealth == "" {
|
||||
checkHealth = "healthy"
|
||||
}
|
||||
if err := q.MarkCheckRun(ctx, sqlcgen.MarkCheckRunParams{
|
||||
EntityID: cd.EntityID,
|
||||
LastHealth: &checkHealth,
|
||||
}); err != nil {
|
||||
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
|
||||
}
|
||||
|
||||
@@ -150,16 +159,12 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
|
||||
if result.signalKind == "" || result.health == "healthy" {
|
||||
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: "healthy",
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
if prevHealth != "" && prevHealth != "healthy" {
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, "info",
|
||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": "healthy"})
|
||||
}
|
||||
// NOT unconditionally "healthy": this check passing says nothing about
|
||||
// the entity's other checks. Writing healthy here is what let one
|
||||
// passing probe erase a genuine failure reported by another — and,
|
||||
// alternating with a failing probe, produced 226 health flips an hour
|
||||
// on a host that was fine throughout.
|
||||
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||
return
|
||||
}
|
||||
|
||||
@@ -180,22 +185,52 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
||||
return
|
||||
}
|
||||
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: result.health,
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
_ = sig
|
||||
|
||||
if prevHealth == "" || prevHealth == "healthy" {
|
||||
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
||||
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
||||
}
|
||||
if prevHealth != result.health {
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": result.health})
|
||||
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||
}
|
||||
|
||||
// applyAggregateHealth sets the target's health to the worst verdict across
|
||||
// all of its enabled checks, and emits health.changed only when that aggregate
|
||||
// actually moves.
|
||||
//
|
||||
// Health is a property of the entity, but each check only ever observes one
|
||||
// facet of it — reachability, disk, a systemd unit. Letting whichever check
|
||||
// finished last overwrite the entity's health meant a host with six checks
|
||||
// reported whichever facet was sampled most recently, so one failing probe and
|
||||
// five passing ones oscillated forever instead of settling on "degraded".
|
||||
func applyAggregateHealth(ctx context.Context, pool *db.Pool, q *sqlcgen.Queries,
|
||||
targetID uuid.UUID, checkSlug, prevHealth string) {
|
||||
|
||||
health, err := q.WorstHealthForTarget(ctx, &targetID)
|
||||
if err != nil {
|
||||
slog.Error("scheduler: aggregate health", "entity", checkSlug, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: health,
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
|
||||
if prevHealth == health {
|
||||
return
|
||||
}
|
||||
severity := "info"
|
||||
switch health {
|
||||
case "down":
|
||||
severity = "critical"
|
||||
case "degraded", "stale":
|
||||
severity = "warning"
|
||||
}
|
||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||
map[string]any{"slug": checkSlug, "from": prevHealth, "to": health})
|
||||
}
|
||||
|
||||
// currentHealth reads the last recorded health for an entity, or "" if none.
|
||||
@@ -217,7 +252,6 @@ func emitSchedulerEvent(ctx context.Context, pool *db.Pool, eventType string, en
|
||||
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
||||
// entity id); targetID is the observed entity whose status this affects.
|
||||
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
||||
q := sqlcgen.New(pool)
|
||||
// Check if there's an open signal on this entity
|
||||
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
||||
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
||||
@@ -227,14 +261,12 @@ func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UU
|
||||
if tag.RowsAffected() > 0 {
|
||||
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
||||
map[string]any{"slug": slug})
|
||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||
}
|
||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||
EntityID: targetID,
|
||||
Health: "healthy",
|
||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||
Details: []byte(`{}`),
|
||||
})
|
||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||
// Deliberately does NOT write health. Resolving THIS check's signal says
|
||||
// nothing about the target's other checks; the caller re-derives health
|
||||
// from all of them. Forcing "healthy" here was a second path by which one
|
||||
// passing probe erased another probe's genuine failure.
|
||||
}
|
||||
|
||||
// checkResult bundles the outcome of a single check execution.
|
||||
@@ -246,8 +278,10 @@ type checkResult struct {
|
||||
err error
|
||||
}
|
||||
|
||||
// executeCheck dispatches to the appropriate checker by kind.
|
||||
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
// executeCheck dispatches to the appropriate checker by kind. pool is needed
|
||||
// by the ssh-script path, which resolves the target's execution endpoint
|
||||
// (guests route through their Proxmox host; see internal/remote).
|
||||
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
switch cd.Kind {
|
||||
case "http":
|
||||
return checkHTTP(ctx, cd)
|
||||
@@ -257,12 +291,16 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
|
||||
return checkDisk(ctx, cd)
|
||||
case "cert-expiry":
|
||||
return checkCertExpiry(ctx, cd)
|
||||
case "vm-status":
|
||||
return checkVMStatus(ctx, pool, cd)
|
||||
case "ping":
|
||||
return checkPing(ctx, cd)
|
||||
case "ssh-script":
|
||||
return checkSSHScript(ctx, cd)
|
||||
return checkSSHScript(ctx, pool, cd)
|
||||
case "backup-freshness":
|
||||
return checkBackupFreshness(ctx, cd)
|
||||
case "dns":
|
||||
return checkDNS(ctx, cd)
|
||||
default:
|
||||
return checkResult{health: "unknown"}
|
||||
}
|
||||
@@ -448,6 +486,53 @@ func checkTCP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResu
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
|
||||
// checkDNS verifies a DNS name resolves, catching a stale or unreachable
|
||||
// zone. It looks up NS records first (a zone always has NS), falling back to
|
||||
// an A/AAAA lookup for hostnames. Uses the system resolver; for split-horizon
|
||||
// correctness reserve an explicit `server` in the config.
|
||||
func checkDNS(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Name string `json:"name"`
|
||||
Server string `json:"server"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
}
|
||||
if cfg.Name == "" {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
|
||||
// Resolve via an explicit server when supplied (split-horizon), else the
|
||||
// system default resolver.
|
||||
lookup := func(q string) (int, error) {
|
||||
r := &net.Resolver{}
|
||||
if cfg.Server != "" {
|
||||
r = &net.Resolver{PreferGo: true, Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
|
||||
d := net.Dialer{Timeout: 5 * time.Second}
|
||||
return d.DialContext(ctx, network, net.JoinHostPort(cfg.Server, "53"))
|
||||
}}
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
ns, err := r.LookupNS(ctx, q)
|
||||
if err == nil && len(ns) > 0 {
|
||||
return len(ns), nil
|
||||
}
|
||||
addrs, err2 := r.LookupHost(ctx, q)
|
||||
return len(addrs), err2
|
||||
}
|
||||
|
||||
n, err := lookup(cfg.Name)
|
||||
if err != nil || n == 0 {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "dns",
|
||||
evidence: fmt.Sprintf("DNS resolution failed for %q: %v", cfg.Name, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
|
||||
// checkDisk performs a disk usage check.
|
||||
func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
@@ -502,6 +587,12 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
// Dial is an optional explicit dial address (the TLS terminator's IP)
|
||||
// for when the hostname doesn't resolve/reach from the scheduler — the
|
||||
// container has no mesh interface and the host resolver doesn't know
|
||||
// the split-horizon zone, so *.hubris.network dials Caddy's lab IP
|
||||
// directly while SNI/cert-read still uses Host.
|
||||
Dial string `json:"dial"`
|
||||
Port int `json:"port"`
|
||||
WarnDays int `json:"warn_days"`
|
||||
CritDays int `json:"crit_days"`
|
||||
@@ -522,9 +613,13 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
||||
timeout = 30 * time.Second
|
||||
}
|
||||
|
||||
addr := net.JoinHostPort(cfg.Host, fmt.Sprintf("%d", cfg.Port))
|
||||
dialHost := cfg.Host
|
||||
if cfg.Dial != "" {
|
||||
dialHost = cfg.Dial
|
||||
}
|
||||
addr := net.JoinHostPort(dialHost, fmt.Sprintf("%d", cfg.Port))
|
||||
|
||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true}}
|
||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true, ServerName: cfg.Host}}
|
||||
conn, err := d.DialContext(ctx, "tcp", addr)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
@@ -569,11 +664,61 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// checkVMStatus reports whether a VM is powered on, via `qm status <pve_id>`
|
||||
// run on its Proxmox host. This is the right reachability probe for a VM that
|
||||
// blocks ICMP (haos) and has no guest agent: it doesn't need the VM's network
|
||||
// at all — "status: running" means the VM is up. The command runs on the host
|
||||
// (not inside the VM), so it uses the host's address with identity wrap.
|
||||
func checkVMStatus(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
if cd.TargetID == nil {
|
||||
return checkResult{health: "unknown", evidence: "vm-status needs a target VM"}
|
||||
}
|
||||
var pveID, hostAttr string
|
||||
if err := pool.QueryRow(ctx,
|
||||
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||
*cd.TargetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||
return checkResult{health: "unknown", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("vm %s has no pve_id", cd.EntitySlug)}
|
||||
}
|
||||
hostSlug := remote.ResolveProxmoxHostSlug(ctx, pool, *cd.TargetID, hostAttr)
|
||||
addr, user, err := remote.ResolveHost(ctx, pool, hostSlug, sshUser)
|
||||
if err != nil {
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("resolve proxmox host for %s: %v", cd.EntitySlug, err), err: err}
|
||||
}
|
||||
|
||||
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||
if timeout <= 0 {
|
||||
timeout = 15 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||
defer cancel()
|
||||
|
||||
out, err := sshExec(ctx, addr, "22", user, "qm status "+pveID, timeout)
|
||||
if err != nil {
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("qm status %s on %s: %v", pveID, addr, err), err: err}
|
||||
}
|
||||
// `qm status <id>` prints "status: running" (or stopped/paused).
|
||||
if strings.Contains(string(out), "status: running") {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
trimmed := strings.TrimSpace(string(out))
|
||||
if trimmed == "" {
|
||||
trimmed = "(no output)"
|
||||
}
|
||||
return checkResult{health: "down", signalKind: "vm-status",
|
||||
evidence: fmt.Sprintf("%s not running: %s", cd.EntitySlug, trimmed)}
|
||||
}
|
||||
|
||||
// checkPing performs an ICMP ping check using the system ping command.
|
||||
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Count int `json:"count"`
|
||||
// Port for the TCP fallback below. Defaults to 22; set it for hosts
|
||||
// that answer on something else (a Home Assistant VM has no sshd).
|
||||
Port int `json:"port"`
|
||||
}{}
|
||||
if len(cd.Config) > 0 {
|
||||
_ = json.Unmarshal(cd.Config, &cfg)
|
||||
@@ -607,9 +752,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
|
||||
output, err := cmd.Output()
|
||||
if err != nil {
|
||||
// ICMP failing does not mean the host is down — it may mean ICMP is
|
||||
// simply unavailable from here. On this deployment the scheduler runs
|
||||
// in Docker on macOS, whose VM network stack does not route ICMP to
|
||||
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
|
||||
// ping checks reported "down" for hosts that were demonstrably up
|
||||
// (including the Docker host itself). Under health aggregation that one
|
||||
// broken probe was enough to drag each entity to down.
|
||||
//
|
||||
// The question this check exists to answer is "is it reachable", and
|
||||
// ICMP is only one way to ask. Fall back to a TCP connect before
|
||||
// concluding anything.
|
||||
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
|
||||
return checkResult{
|
||||
health: "healthy",
|
||||
metrics: map[string]float64{},
|
||||
}
|
||||
}
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ping",
|
||||
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
|
||||
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -623,6 +785,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
||||
return checkResult{health: "healthy", metrics: metrics}
|
||||
}
|
||||
|
||||
// tcpReachable reports whether a TCP handshake completes, used as the
|
||||
// reachability answer when ICMP is unavailable rather than unanswered.
|
||||
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
|
||||
if port == 0 {
|
||||
port = 22
|
||||
}
|
||||
if timeout <= 0 {
|
||||
timeout = 5 * time.Second
|
||||
}
|
||||
d := net.Dialer{Timeout: timeout}
|
||||
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
conn.Close()
|
||||
return true
|
||||
}
|
||||
|
||||
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
||||
|
||||
func parsePingLatency(output []byte) float64 {
|
||||
@@ -637,8 +817,16 @@ func parsePingLatency(output []byte) float64 {
|
||||
return val
|
||||
}
|
||||
|
||||
// checkSSHScript executes an allowlisted script on a remote host via SSH.
|
||||
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
// checkSSHScript executes an allowlisted script on a remote target via SSH.
|
||||
//
|
||||
// Routing follows the canonical access model (internal/remote): an LXC or VM
|
||||
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
|
||||
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
|
||||
// key of its own. Hosts and workstations are reached by direct SSH, resolved
|
||||
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
|
||||
// a re-seed. Services and other entities fall back to the host address baked
|
||||
// into check config at seed time (their hosting container's address).
|
||||
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||
cfg := struct {
|
||||
Host string `json:"host"`
|
||||
Port int `json:"port"`
|
||||
@@ -656,12 +844,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
if cfg.Host == "" || cfg.Script == "" {
|
||||
return checkResult{health: "healthy"}
|
||||
}
|
||||
if cfg.Port == 0 {
|
||||
cfg.Port = 22
|
||||
}
|
||||
if cfg.User == "" {
|
||||
cfg.User = sshUser
|
||||
}
|
||||
|
||||
if !allowlistedScript(cfg.Script) {
|
||||
return checkResult{
|
||||
@@ -684,13 +866,51 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
// the remote command. The script name itself is allowlisted above.
|
||||
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
|
||||
}
|
||||
port := strconv.Itoa(cfg.Port)
|
||||
|
||||
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
|
||||
// Resolve the execution endpoint. The resolver handles every target kind:
|
||||
// LXC/VM host-hop via pct/qm exec; hosts/workstations direct at their own
|
||||
// address; services route through their hosting compute entity (via the
|
||||
// provides edge) so a service check reaches the right machine with the
|
||||
// right user instead of baking an LXC lan_ip and SSHing it as root.
|
||||
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
|
||||
wrap := func(cmd string) string { return cmd }
|
||||
targetType := ""
|
||||
if cd.TargetType != nil {
|
||||
targetType = *cd.TargetType
|
||||
}
|
||||
// target_type was omitted by older writeCheck inserts, so resolve it from
|
||||
// the target entity when the column is blank — otherwise the guest routing
|
||||
// below (IsGuest) never triggers and a guest check falls back to its baked
|
||||
// (often mesh-only) address.
|
||||
if targetType == "" && cd.TargetID != nil {
|
||||
if err := pool.QueryRow(ctx, "SELECT type FROM entities WHERE id = $1", *cd.TargetID).Scan(&targetType); err != nil {
|
||||
targetType = ""
|
||||
}
|
||||
}
|
||||
if cd.TargetID != nil && targetType != "" {
|
||||
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
|
||||
if err != nil {
|
||||
if remote.IsGuest(targetType) {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ssh-script",
|
||||
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
|
||||
}
|
||||
}
|
||||
// Non-guest: log the resolution failure so an opaque ssh "down"
|
||||
// doesn't hide that the real cause was host/user resolution, then
|
||||
// fall back to the baked config below.
|
||||
slog.Warn("scheduler: target resolution failed, using baked config",
|
||||
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
|
||||
} else {
|
||||
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
|
||||
}
|
||||
}
|
||||
|
||||
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
|
||||
if err != nil {
|
||||
return checkResult{
|
||||
health: "down", signalKind: "ssh-script",
|
||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
|
||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
|
||||
err: err,
|
||||
}
|
||||
}
|
||||
@@ -728,6 +948,20 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
||||
}
|
||||
}
|
||||
|
||||
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
|
||||
func oru(v, def int) int {
|
||||
if v != 0 {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
func orStr(v, def string) string {
|
||||
if v != "" {
|
||||
return v
|
||||
}
|
||||
return def
|
||||
}
|
||||
|
||||
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
||||
|
||||
func allowlistedScript(name string) bool {
|
||||
|
||||
28
migrations/027_check_last_health.up.sql
Normal file
28
migrations/027_check_last_health.up.sql
Normal file
@@ -0,0 +1,28 @@
|
||||
-- 027_check_last_health.up.sql
|
||||
-- Aggregate an entity's health across its checks instead of last-writer-wins.
|
||||
--
|
||||
-- runCheck wrote entity_status.health on every check completion, so an
|
||||
-- entity's health was simply whichever of its checks finished most recently.
|
||||
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
|
||||
-- five passing ones produced a permanent flap: 226 health.changed events for
|
||||
-- host:strong in a single hour, oscillating down/healthy, while the host was
|
||||
-- fine the whole time.
|
||||
--
|
||||
-- On this fleet the trigger is a known false positive: the scheduler's network
|
||||
-- vantage point cannot ICMP host:strong, so its ping check fails while every
|
||||
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
|
||||
-- to declare the whole host down, twice a minute.
|
||||
--
|
||||
-- Storing each check's own verdict lets entity health be derived as the worst
|
||||
-- current result across that entity's enabled checks — so a single failing
|
||||
-- probe degrades the entity honestly without erasing what the other five say,
|
||||
-- and a passing probe cannot mask a genuine failure elsewhere.
|
||||
|
||||
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
|
||||
|
||||
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
|
||||
|
||||
-- The aggregation reads every enabled check for one target on each completion.
|
||||
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
|
||||
ON check_defs (target_id)
|
||||
WHERE enabled AND target_id IS NOT NULL;
|
||||
75
migrations/028_relationship_blast_direction.up.sql
Normal file
75
migrations/028_relationship_blast_direction.up.sql
Normal file
@@ -0,0 +1,75 @@
|
||||
-- 028_relationship_blast_direction.up.sql
|
||||
-- Make blast_radius answer the question it is named after.
|
||||
--
|
||||
-- blast_radius walked source_id -> target_id for every relationship type. But
|
||||
-- which end of an edge is the DEPENDENT differs per type:
|
||||
--
|
||||
-- machine --hosts--> container if the machine dies, the container dies
|
||||
-- -> dependent is the TARGET (forward)
|
||||
-- service --depends-on--> service if the target dies, the SOURCE breaks
|
||||
-- -> dependent is the SOURCE (backward)
|
||||
-- ingress --routes-to--> service if the service dies, the route 502s
|
||||
-- -> dependent is the SOURCE (backward)
|
||||
-- document --documents--> entity neither breaks the other
|
||||
-- -> no runtime dependency at all
|
||||
--
|
||||
-- Walking everything forwards meant the answer was right for `hosts` and
|
||||
-- `provides` and wrong for every backward edge, while `documents`, `involves`
|
||||
-- and `targets` (2,800+ edges of pure bookkeeping) polluted the result with
|
||||
-- tasks and executions that cannot "break".
|
||||
--
|
||||
-- Direction is therefore a property of the relationship type, declared in
|
||||
-- seeds/ontology.yaml — the same shape as the `monitoring:` declaration on
|
||||
-- entity types.
|
||||
--
|
||||
-- forward : if the SOURCE fails, the TARGET is affected
|
||||
-- backward : if the TARGET fails, the SOURCE is affected
|
||||
-- none : no runtime dependency (default — bookkeeping and documentation)
|
||||
--
|
||||
-- Defaulting to 'none' is deliberate: an undeclared edge contributes nothing
|
||||
-- rather than silently producing a wrong answer, which is how the old
|
||||
-- everything-is-forward behaviour went unnoticed.
|
||||
|
||||
ALTER TABLE relationship_types
|
||||
ADD COLUMN IF NOT EXISTS blast_direction TEXT NOT NULL DEFAULT 'none'
|
||||
CHECK (blast_direction IN ('forward', 'backward', 'none'));
|
||||
|
||||
COMMENT ON COLUMN relationship_types.blast_direction IS
|
||||
'Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().';
|
||||
|
||||
-- Walk the dependency graph in the direction each edge type declares.
|
||||
--
|
||||
-- Returns everything that is affected when start_id fails, with the number of
|
||||
-- hops. Cycles are guarded by the path array, as before.
|
||||
CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3,
|
||||
rel_types TEXT[] DEFAULT NULL)
|
||||
RETURNS TABLE(entity_id UUID, depth INT) AS $$
|
||||
WITH RECURSIVE walk AS (
|
||||
SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path
|
||||
UNION ALL
|
||||
SELECT next_id, w.depth + 1, w.path || next_id
|
||||
FROM walk w
|
||||
JOIN LATERAL (
|
||||
-- forward: this entity is the source, so the target depends on it
|
||||
SELECT r.target_id AS next_id
|
||||
FROM relationships r
|
||||
JOIN relationship_types rt ON rt.name = r.type
|
||||
WHERE r.source_id = w.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND rt.blast_direction = 'forward'
|
||||
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||
UNION ALL
|
||||
-- backward: this entity is the target, so the source depends on it
|
||||
SELECT r.source_id AS next_id
|
||||
FROM relationships r
|
||||
JOIN relationship_types rt ON rt.name = r.type
|
||||
WHERE r.target_id = w.entity_id
|
||||
AND r.valid_to IS NULL
|
||||
AND rt.blast_direction = 'backward'
|
||||
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||
) nxt ON TRUE
|
||||
WHERE w.depth < LEAST(max_depth, 5)
|
||||
AND NOT nxt.next_id = ANY(w.path)
|
||||
)
|
||||
SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id;
|
||||
$$ LANGUAGE sql STABLE;
|
||||
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
@@ -0,0 +1,33 @@
|
||||
-- 029_plan_generation_relative_seq.up.sql
|
||||
-- Make plan-step seq generation-relative: 1..N within each
|
||||
-- (session_id, generation). Before this, seq was globally increasing across
|
||||
-- generations (gen1: 1..6, gen2: 7..12), so the model's 1-based
|
||||
-- update_plan_step calls — which the prompt and schema explicitly tell it to
|
||||
-- use — landed on superseded gen-1 rows after a re-plan while the live gen-2
|
||||
-- work went unrecorded (or, worse, resurrected a `replaced` row as `done`).
|
||||
-- The addressing key is now (session_id, generation, seq); updatePlanStep
|
||||
-- resolves against MAX(generation), so a 1-based seq always maps to the
|
||||
-- CURRENT plan. See plans/2026-07-30-session-review-plan-drift-and-dead-
|
||||
-- activity-panel.md P0.1.
|
||||
|
||||
-- Renumber existing rows so seq resets to 1..N per (session, generation),
|
||||
-- preserving each generation's step order.
|
||||
WITH ranked AS (
|
||||
SELECT id,
|
||||
ROW_NUMBER() OVER (
|
||||
PARTITION BY session_id, generation
|
||||
ORDER BY seq, created_at
|
||||
) AS new_seq
|
||||
FROM session_plan_steps
|
||||
)
|
||||
UPDATE session_plan_steps s
|
||||
SET seq = ranked.new_seq
|
||||
FROM ranked
|
||||
WHERE s.id = ranked.id AND s.seq <> ranked.new_seq;
|
||||
|
||||
-- (session_id, seq) is no longer unique once seq resets per generation; the
|
||||
-- store resolves via (session_id, generation, seq). Drop the old composite
|
||||
-- index (it now collides on seq) and add the generation-scoped unique index.
|
||||
DROP INDEX IF EXISTS idx_plan_steps_session;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_plan_steps_session_gen_seq
|
||||
ON session_plan_steps (session_id, generation, seq);
|
||||
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
@@ -0,0 +1,8 @@
|
||||
-- 030_plan_step_replaced_reason.up.sql
|
||||
-- Add replaced_reason to session_plan_steps so the agent must explain why
|
||||
-- a step was replaced (wrong_diagnosis, scope_change, blocked, superseded,
|
||||
-- operator_override) rather than silently replacing entire plans. The column
|
||||
-- is also set by bulk-replace operations (proposePlan, setGoal, reopenSession)
|
||||
-- for auditability.
|
||||
|
||||
ALTER TABLE session_plan_steps ADD COLUMN IF NOT EXISTS replaced_reason TEXT;
|
||||
@@ -47,6 +47,24 @@ Read-only commands auto-run (no approval). Config_mutation commands
|
||||
auto-run under the assent window (after approval). Destructive commands
|
||||
always need explicit typed confirmation.
|
||||
|
||||
**Never mark a step `done` if its tool calls errored.** If `run` timed out,
|
||||
`update_entity_attributes` returned "not found", `create_relationship` returned
|
||||
"source entity not found", or any tool returned an error — the step is NOT done.
|
||||
Diagnose the error, try an alternative (e.g. use `create_entity` when
|
||||
`update_entity_attributes` reports the entity doesn't exist), and only advance
|
||||
to `done` when the step's intended work actually completed. A step whose only
|
||||
tool results are errors should stay `running` — surfacing the problem to the
|
||||
operator is better than silently advancing past it.
|
||||
|
||||
**Complete or skip steps — don't replace silently.** Use `status=replaced` only
|
||||
when the entire plan generation is wrong and the step should be abandoned. When
|
||||
you replace a step, provide `replaced_reason` with the cause
|
||||
(`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`).
|
||||
Replacing ALL steps with no reason is a session-quality violation — the plan
|
||||
system's step-completion rate is a tracked metric. Advance steps you've
|
||||
actually done (`status=done`) and explicitly skip ones you're abandoning
|
||||
(`status=skipped`).
|
||||
|
||||
### 6. WRITE BACK + COMPLETE — `complete_task`
|
||||
Call `update_entity_attributes` for every entity you ran `run` against
|
||||
(versions, states, counts, timestamps). Call `create_relationship` for any
|
||||
@@ -59,6 +77,15 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
|
||||
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
||||
no writeback needed.
|
||||
|
||||
**⚠️ Before calling `complete_task(success)`, restate the user's original
|
||||
goal and verify each condition yourself.** "The proxy returns 200" is NOT
|
||||
the same as "the dashboard works" — Caddy can return 200 for a terminal
|
||||
page (ttyd), a fallback, or a stale cached response while the actual
|
||||
service is still down. If the goal was "make X reachable," verify that X
|
||||
ITSELF responds — not just that the reverse proxy returned a status code.
|
||||
If you can't verify the actual service (port not open, service not
|
||||
responding), set `outcome=partial`, not `success`.
|
||||
|
||||
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
||||
on success, → skipped on partial/failure). You do NOT need to call
|
||||
`update_plan_step` for every step right before completing — once your work
|
||||
@@ -85,6 +112,13 @@ generation — the panel will show it as a new list), execute, write back,
|
||||
the panel in your reply. Never re-run `run` just to fix a display mismatch.
|
||||
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
|
||||
answer → present the existing knowledge, propose a targeted refresh only.
|
||||
- Pivot to a subsystem unrelated to the user's expressed goal without asking →
|
||||
when investigation leads to a different subsystem or root cause (e.g.
|
||||
debugging DHCP reservations when the goal was "make the dashboard reachable"),
|
||||
call `session_questions` with the discovery and options BEFORE taking action.
|
||||
Example: "The dashboard hasn't started since July 19 — this predates my work.
|
||||
Do you want me to debug the dashboard service [A], skip it and stabilize the
|
||||
current state [B], or stop here [C]?"
|
||||
|
||||
## Source of truth
|
||||
|
||||
@@ -205,6 +239,21 @@ disappear.
|
||||
`list_lxcs` answers the same question in one call. Use it.
|
||||
- When a bulk tool's summary isn't enough for a specific entity, call the
|
||||
per-entity tool for that one entity — not for every entity in the fleet.
|
||||
- **Cap pre-plan exploration:** prefer `list_entities(limit)` +
|
||||
`get_entity_knowledge` (context for one entity, one call) over N+1
|
||||
`get_entity`/`get_relations` chains. If you've already called
|
||||
`get_entity_knowledge(slug)` and need more, call `get_entity(slug)` +
|
||||
`get_relations(slug)` — not `list_entities` without a limit scanning the
|
||||
whole entity table.
|
||||
- **Group parallel reads:** `get_entity_knowledge`, `search_knowledge`,
|
||||
`get_entity`, and `get_relations` are all read-only DB calls that can
|
||||
be batched in a single tool-call block. Do not sequentialize them one
|
||||
per turn when they are independent.
|
||||
- **Source-reading on prod (`run cat/grep/find /opt/…`) is NOT the way to
|
||||
learn how the platform works.** The MCP tools ARE the interface. If you
|
||||
need to understand a check lifecycle or a scheduler behavior, search
|
||||
`search_knowledge("oikos check lifecycle")` or ask the operator — do
|
||||
not treat the prod host as a code repository you grep.
|
||||
|
||||
## Policy awareness
|
||||
|
||||
@@ -355,6 +404,26 @@ port is busy, find a free one. Only surface to the operator if you've tried
|
||||
reasonable alternatives and none worked. An error in one step is not a reason
|
||||
to stop the entire turn — it's a reason to try a different approach.
|
||||
|
||||
**When you hit a genuine missing capability — STOP and ask, don't bypass:**
|
||||
If a tool returns `entity … not found` when you're trying to create something
|
||||
(a check, an ingress, a cert, a new service), the entity doesn't exist yet —
|
||||
use `create_entity`. If you need to retire/delete an entity, use
|
||||
`set_entity_state`. If you need to remove a relationship, use
|
||||
`end_relationship`. If NONE of these fit and you truly lack a tool, **tell the
|
||||
operator directly: "I need to X, but no MCP tool does that — can you create it
|
||||
via the API?"** Do NOT pivot to `run find/grep/cat` on `/opt/homelab-context`
|
||||
to reverse-engineer how the platform works — MCP tools are the interface, not
|
||||
the prod source tree.
|
||||
|
||||
**Self-grounding — use the DB, don't invent:**
|
||||
- `run` targets must be `host:<slug>`, `lxc:<slug>`, or `vm:<slug>` — never
|
||||
`ws:`, raw container names, or Docker Compose service aliases.
|
||||
- Never invent an IP address or subnet. Query `get_entity("service:oikos")` for
|
||||
the real API address, `get_entity("host:<name>")` for a host's real LAN IP,
|
||||
`list_lxcs` for container addresses. The DB is authoritative; your guess is
|
||||
wrong (the homelab has multiple subnets — `192.168.8.0/24`, `192.168.178.0/24`,
|
||||
etc. — and guessing the wrong one wastes turns).
|
||||
|
||||
**A hung command is not a failed command — investigate before retrying.**
|
||||
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
|
||||
gateway timeout), DO NOT immediately retry the same command with different
|
||||
@@ -391,6 +460,21 @@ before producing the plan. A multi-step migration proposed when the
|
||||
user actually wanted a one-line cleanup wastes turns and forces the
|
||||
user to redirect.
|
||||
|
||||
**Scope gate — ask before chasing unrelated subsystems.** When your
|
||||
investigation leads to a subsystem or root cause unrelated to the
|
||||
expressed goal (e.g. the user asked "why is X unreachable?" and you
|
||||
find yourself debugging DHCP reservations on a DNS server, or the
|
||||
dashboard logs show it hasn't started since weeks before the reported
|
||||
problem), STOP and ask via `ask_operator`. Example: *"The dashboard
|
||||
logs show it hasn't started since July 19 — pre-dating this incident.
|
||||
Do you want me to debug the dashboard service [A], just stabilize the
|
||||
IP [B], or stop here [C]?"* Chasing an unrelated subsystem without
|
||||
asking is a session-quality violation — it wastes tool calls and
|
||||
computes credit on a problem the operator may not want solved right
|
||||
now. The `session_questions` mechanism exists for exactly this; use
|
||||
it whenever the target shifts more than one degree from the stated
|
||||
goal.
|
||||
|
||||
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
||||
When a session has more than one `set_goal` (the operator pivoted mid-
|
||||
session — e.g. "actually, just keep ludo-library"), the final
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
# 2026-08-03 — Session review: `service:haos` monitoring + agent capability gaps
|
||||
|
||||
**Status:** Plan (audit complete; ready to implement).
|
||||
**Reviewed session:** `23da10db-46a9-444c-bbde-ca9457bd9087` — *"Work out what
|
||||
monitoring checks service:haos should have and configure them."*
|
||||
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
|
||||
`agent_activity`/`session_plan_steps` on the prod mac-mini (oikos prod runs here
|
||||
in docker compose project `oikos`; gateway `:8092`), cross-referenced with the
|
||||
code paths in `internal/mcp`, `internal/httpapi`, `internal/policy`,
|
||||
`internal/checkdefaults`, `internal/db/seed.go`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Session audit (objective vs outcome)
|
||||
|
||||
| Dimension | Finding |
|
||||
|---|---|
|
||||
| Objective | Determine + configure monitoring checks for `service:haos` (HAOS VM 108, `home.hubris.network`, `192.168.8.101:8123`). |
|
||||
| Outcome | ❌ **Failed/stuck.** `status=executing`, `outcome=null` ~4 min after last activity (UTC); never reached a terminal state. Only the *existing* `check:vm-status:vm:haos:0` stub got populated; the three **new** checks (`http:service`, `http:ingress`, `cert-expiry`) and their `ingress:`/`cert:` entities were never created. |
|
||||
| Tool calls | **116** (vs the >30 N+1 failure signature). ~45 redundant `list_entities`/`get_entity`/`get_relations`, then a ~15-min storm of `run` doing `find`/`grep`/`cat` on prod source. |
|
||||
| Plan | 8 steps proposed; steps 1–4 genuinely done; **step 5 falsely marked "done"** after both its tool calls errored `entity not found`; steps 6–8 never started. |
|
||||
| Operator friction | 3 manual interventions: `status`, `proceed`, *"why dony you use the mcp?"*; plus a **44-minute approval stall** (19:52→20:36) on two trivial reachability curls. |
|
||||
| Severity | **blocker** (capability gap) + **friction** (classifier, plan-state, reaping). |
|
||||
|
||||
### Timeline (UTC)
|
||||
- **19:45–19:50** — read-only exploration; `run` correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
|
||||
- **19:50** — `propose_plan` (8 steps).
|
||||
- **19:52** — two `curl … -o /dev/null -w '%{http_code}'` reachability probes → both classified `config_mutation` → one queued for approval (`019fc92e…`), second blocked ("approval already pending").
|
||||
- **19:52 → 20:36 (44 min)** — idle, waiting on operator approval.
|
||||
- **20:36** — approval granted ("auto via assent window"); both curls → 200/200.
|
||||
- **20:37** — step 4 ✅: populated `check:vm-status:vm:haos:0` + `checks` edge.
|
||||
- **20:37:58** — step 5 ❌: `update_entity_attributes("check:http:service:haos:0")` → **`entity not found`**; `create_relationship` → **`source entity not found`**. *(There is no create tool.)*
|
||||
- **20:38–20:47** — spiral: `search_knowledge` (empty), then `run find/grep/cat` across `/opt/homelab-context/**/*.go` to reverse-engineer check creation. Reads `checkdefaults.go`, `monitoring.go`, `default_checks.go`, `checks.go`, `coverage.go`.
|
||||
- **20:42** — sets `service:haos` `monitoring: ["http"]` via `update_entity_attributes`, hoping `checkdefaults.Ensure()` auto-generates. **It does not** (see A2).
|
||||
- **20:42–20:50** — tries to reach the REST API directly: `psql` on hubris (cmd 127), `docker exec` on hubris (docker absent), `curl http://192.168.178.25:8090` (wrong subnet; real net is `192.168.8.x`; exit 7), `curl http://oikos-api:8090` (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), `ssh root@192.168.178.25` (no route). Final `update_entity_attributes` on `ingress:`/`cert:` → `not found`.
|
||||
- **20:50:38** — last activity: a failed 30s `run`. Session goes silent, never terminates.
|
||||
|
||||
---
|
||||
|
||||
## 2. Root-cause findings (with code evidence)
|
||||
|
||||
### A1 — No entity-creation capability in the MCP toolset *(the blocker)*
|
||||
`internal/mcp/tools.go` registers **37 tools**; the only entity-mutation surface is
|
||||
`update_entity_attributes` (merge into an **existing** entity) and
|
||||
`create_relationship` (needs **existing** source+target). Neither can create a new
|
||||
entity. The capability **does** exist at the HTTP layer — `CreateEntity`
|
||||
(`internal/httpapi/impl.go:865`, `POST /api/v1/entities`) — it is simply not exposed
|
||||
to the agent. Every "set up / onboard / configure entity X" task that needs a new
|
||||
check/ingress/cert/service hits this wall.
|
||||
|
||||
### A2 — MCP `update_entity_attributes` bypasses `ensureDefaultChecks`
|
||||
`ensureDefaultChecks` (`internal/httpapi/default_checks.go:20`) is invoked **only**
|
||||
from the HTTP handlers: `CreateEntity` (`impl.go:1012`) and `PatchEntity`
|
||||
(`impl.go:1280`). `grep ensureDefaultChecks internal/mcp/` → **no matches**: the MCP
|
||||
tool writes attributes straight to the store, so flipping `service:haos`
|
||||
`monitoring:["http"]` never regenerated its checks. The agent's fallback strategy
|
||||
was structurally doomed via MCP.
|
||||
|
||||
### A3 — `-o /dev/null` curl idiom misclassified as `config_mutation`
|
||||
`internal/policy/command.go:106` `curlMutateRe` matches `(?:^|\s)-(?:d|F|T|o)\b` — so
|
||||
`-o` (output-file) is treated as mutation. The canonical read-only reachability probe
|
||||
`curl -sS -o /dev/null -w '%{http_code}' …` therefore escalates to approval. This is
|
||||
the entire 44-minute stall. (`curlIsReadOnly` at `command.go:307` only passes for GET
|
||||
with no `-o`/`-d`/`-X`/`>`.) A pure GET that discards the body is the single most
|
||||
common health probe and shouldn't need approval.
|
||||
|
||||
### A4 — No platform self-knowledge doc for the check lifecycle
|
||||
`search_knowledge("create check entity how to add new check monitoring")` → empty.
|
||||
The agent re-derived the whole mechanism from source on prod (~15 min, dozens of
|
||||
`run`). There is no agent/operator runbook explaining: check slugs are
|
||||
`check:<kind>:<target>:<n>`; `check_defs` are derived from the type's `monitoring`
|
||||
spec by `checkdefaults.Ensure`; Ensure runs at **seed/deploy** and on **HTTP
|
||||
create/patch**, not via MCP.
|
||||
|
||||
### A5 — False plan progress (step marked done on failure)
|
||||
At 20:37:58 both tool calls for step 5 returned `error: entity not found`, yet the
|
||||
agent advanced step 5→`done`. Plan-state integrity hole: a step whose actions error
|
||||
should not transition to `done`. (`session_plan_steps` confirms seq 5 = `done`.)
|
||||
|
||||
### A6 — No "missing-capability" escalation; self-grounding failures
|
||||
On detecting the dead-end (no create tool) the agent never told the operator *"I lack
|
||||
a tool to create entities — please create them"*; instead it tried to bypass its own
|
||||
platform. Grounding errors: invented IP `192.168.178.25` (real LAN is `192.168.8.x`),
|
||||
ran `run` against `ws:mac-mini` ("unsupported target — must be host:/lxc:/vm:"),
|
||||
assumed `docker` exists on hubris, assumed the docker-alias `oikos-api` resolves from
|
||||
hubris. The agent didn't query `get_entity("service:oikos")` for the real address.
|
||||
|
||||
### A7 — Sessions never reap from `executing`
|
||||
Last activity 20:50; status still `executing` with no turn running. There is no
|
||||
idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide:
|
||||
176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly
|
||||
approval-stalls and entity-not-found — same families.)
|
||||
|
||||
### A8 — N+1 tool fan-out (116 calls)
|
||||
Dozens of redundant `list_entities`/`get_entity`/`get_relations` before proposing a
|
||||
plan, plus the source-reading `run` storm. Above the >30-per-turn signature; indicates
|
||||
weak bulk-tool use and under-constrained exploration before planning.
|
||||
|
||||
---
|
||||
|
||||
## 3. Improvement plan (ordered)
|
||||
|
||||
**Scope decision (confirmed with operator):** general `create_entity` MCP tool **+
|
||||
wire regen** — solves this case and the 67-entity blast radius (§4).
|
||||
|
||||
### Task 1 — `create_entity` MCP tool *(fixes A1; the centerpiece)*
|
||||
- Register a new tool `create_entity(slug, type, name, attributes?)` in
|
||||
`internal/mcp/tools.go` that **reuses** `httpapi.CreateEntity`
|
||||
(`impl.go:865`) / the same store path — do not hand-roll. It must run
|
||||
`ensureDefaultChecks` (free, since it goes through the create path).
|
||||
- **Approval policy:** no approval required for the entity itself — it mutates the
|
||||
knowledge graph, matching the existing no-approval stance of
|
||||
`update_entity_attributes`/`create_relationship`/`upsert_knowledge`. (Derived checks
|
||||
are safe/read-side; if a check kind is ever deemed mutating, gate *that* in the
|
||||
scheduler, not here.)
|
||||
- Validate `type` against `entity_types`; reject unknown slugs/types with a clear
|
||||
error. Idempotent on existing slug (return the existing entity, mirroring the HTTP
|
||||
`ETag`/conflict behavior).
|
||||
- Expose to the agent via the tool-list build path used by `cmd/nomos/agent.go`.
|
||||
|
||||
### Task 2 — MCP `update_entity_attributes` triggers `ensureDefaultChecks` *(fixes A2)*
|
||||
- After the attribute merge in the MCP handler, call `ensureDefaultChecks` with the
|
||||
post-merge entity (same args as `impl.go:1280`). This makes "set monitoring → checks
|
||||
regenerate" work via MCP, matching HTTP semantics.
|
||||
- Mind the `default_checks.go:14-19` caveat: a service whose address comes from its
|
||||
host edge may still produce no checks until the hosting edge exists — log/return
|
||||
that as an explicit result so the agent knows to create the edge next.
|
||||
|
||||
### Task 3 — Classifier: read-only `curl` with `-o /dev/null` *(fixes A3)*
|
||||
- In `internal/policy/command.go` `curlIsReadOnly`, treat `-o /dev/null` (and
|
||||
`--output /dev/null`) as read-only — it's a no-op sink. Keep `-o <realpath>` as
|
||||
mutation. Add `TestClassifyCommand_CurlDevNull_ReadOnly` next to the existing
|
||||
`TestClassifyCommand_CurlPipeSh_ConfigMutation`.
|
||||
- Coach complement: in `nomos/SOUL.md`, note that reachability probes should use
|
||||
`curl -I` or `-o /dev/null` GETs (now read-only) rather than POSTs.
|
||||
|
||||
### Task 4 — Plan-state integrity: don't mark `done` on errored actions *(fixes A5)*
|
||||
- In `cmd/nomos` (`agent.go`/`tasks.go` where `update_plan_step` is emitted), a step
|
||||
whose turn ended with only error/`not-found` tool results must **not** auto-advance
|
||||
to `done`; leave it `running`/`blocked` and surface the failure to the operator.
|
||||
Minimal: if every tool call in the step returned an `error:*` result, hold the step.
|
||||
|
||||
### Task 5 — Stuck-session reaping *(fixes A7)*
|
||||
- Add an idle sweep (extend the existing continuation/idle worker in `cmd/nomos`) that
|
||||
transitions a session from `executing`→`failed` (or a new `stuck`) when no turn has
|
||||
run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal
|
||||
handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.
|
||||
|
||||
### Task 6 — Missing-capability escalation + grounding *(fixes A6)*
|
||||
- `nomos/SOUL.md`: when a mutation tool returns `entity … not found` on a create
|
||||
intent, the agent must **stop and ask the operator** (or now use `create_entity`)
|
||||
rather than pivot to `run`/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead
|
||||
`get_entity("service:oikos")` for the real API address. `run` targets must be
|
||||
`host:/lxc:/vm:` slugs (state the contract explicitly).
|
||||
|
||||
### Task 7 — Runbook: "how checks work / how to add monitoring" *(fixes A4)*
|
||||
- Upsert a knowledge doc (via `upsert_knowledge`, linked to the `agent:nomos` and
|
||||
`document:infrastructure/monitoring` entities) covering: check slug grammar,
|
||||
`checkdefaults.Ensure` triggers (seed + HTTP create/patch, now also MCP), the
|
||||
`monitoring` per-entity override, the host-edge caveat, and the canonical way to add
|
||||
monitoring to an entity (create/patch entity → checks derive).
|
||||
|
||||
### Task 8 — (Lower priority) exploration budget / bulk-tool use *(A8)*
|
||||
- `nomos/SOUL.md`: prefer `list_entities(limit)` + `get_entity_knowledge` bulk calls
|
||||
over N+1 `get_entity`/`get_relations` fans; cap pre-plan exploration. Optional
|
||||
guardrail in `agent.go` (warn at >N same-tool calls per turn).
|
||||
|
||||
### Recommended sequence
|
||||
1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the
|
||||
approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona
|
||||
hardening.)
|
||||
|
||||
---
|
||||
|
||||
## 4. Uncovered cases — the capability-gap blast radius
|
||||
|
||||
The existing F1–F8 plans (`2026-08-03-nomos-chat-reliability-and-ux-audit.md`,
|
||||
shipped v0.15.0) and the turn-scheduler review cover **only** UI / streaming / turn
|
||||
serialization / connection UX. **None** addresses agent *capability* or
|
||||
MCP↔HTTP integration. This session exposes the uncovered class:
|
||||
|
||||
- **67 entities currently have no `check:` relationship** (DB query): 40 `lxc`, 26
|
||||
`service`, 1 `vm`. Any "add monitoring to X" task fails identically until Tasks 1+2.
|
||||
- **Whole task families blocked by the no-create gap:** onboarding a new host/LXC/VM,
|
||||
declaring a new service/ingress/cert/dns, adding any check that doesn't already
|
||||
exist, registering a relationship target that doesn't exist yet. All currently
|
||||
require an operator to hand-edit `seeds/inventory.yaml` and re-seed.
|
||||
- **MCP↔HTTP semantic drift (generalize A2):** audit other MCP mutation tools for
|
||||
side-effects that the HTTP handlers perform but the MCP path skips (check regen,
|
||||
drift-flagging, audit fields, idempotency). Each is a latent "agent did the right
|
||||
thing but nothing happened" bug.
|
||||
- **Classifier read-only false-positives (generalize A3):** beyond `-o /dev/null`,
|
||||
review other common read-only idioms that escalate (`curl` with benign flags,
|
||||
compound read-only commands) — friction compounds into approval stalls and stuck
|
||||
sessions.
|
||||
- **No terminal/`stuck` reaping (generalize A7):** any turn that ends unresolved
|
||||
leaves the session `executing` forever; the UI never shows "done/failed".
|
||||
|
||||
---
|
||||
|
||||
## 5. Validation
|
||||
|
||||
- **Task 1/2:** `go test ./internal/mcp/... ./internal/httpapi/...` — new test creates
|
||||
`check:http:service:haos:0` via `create_entity`, asserts the entity exists **and**
|
||||
that a `check_def` row was derived; then `update_entity_attributes(service:haos,
|
||||
monitoring:["http"])` via MCP and assert checks regenerate (currently absent).
|
||||
- **Task 3:** `go test ./internal/policy/` — `curl -sS -o /dev/null -w '%{http_code}'
|
||||
URL` ⇒ `read_only`; `curl -o /tmp/x URL` ⇒ `config_mutation`.
|
||||
- **Task 4:** `cmd/nomos` test — a step whose only tool result is `error:*` stays
|
||||
non-`done`.
|
||||
- **Task 5:** idle-sweep test — session with no turn for N min and no pending approval
|
||||
⇒ `failed` (+ event emitted).
|
||||
- **End-to-end re-run:** replay the haos goal against a local nomos; expect the three
|
||||
checks + `ingress:`/`cert:` entities created in <15 tool calls with **zero**
|
||||
approvals and a `done` outcome.
|
||||
|
||||
## 6. Out of scope / open questions
|
||||
- Whether `create_entity` for sensitive types (e.g. `secret`, `key`) should require
|
||||
approval even though it's graph-only — recommend: same no-approval stance now, add
|
||||
type-specific gating later if abused.
|
||||
- The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct
|
||||
`stuck` status vs reuse `failed`.
|
||||
- Whether to also expose a `delete_entity`/`retire_entity` MCP tool (not needed for
|
||||
this case; lifecycle retirement is a separate flow).
|
||||
@@ -1,5 +1,9 @@
|
||||
# 2026-07-21 Chat window full polish
|
||||
|
||||
**Status:** Implemented. Streaming affordance, inline tool rendering, message
|
||||
timestamps, code-copy buttons, and per-session store isolation all landed in
|
||||
`web/src/lib/components/ChatThread.svelte` + the chat stores (v0.8.x–0.10.x).
|
||||
|
||||
## Context
|
||||
|
||||
After fixing the streaming reactivity bug and merging the double thinking
|
||||
@@ -0,0 +1,413 @@
|
||||
# Plan: Make health reflect reality + complete the knowledge graph
|
||||
|
||||
Status: Implemented (v0.14.x–0.16.x). Shipped across `c9a00a9` (per-entity
|
||||
monitoring override), `a3914eb`/`8eb1ca2` (process check opt-in + probe_unit),
|
||||
`0929c17` (discover_infra_drift), and the vm-status/layered-probe/route-via-
|
||||
proxmox-host decisions now in project memory. Created 2026-07-29.
|
||||
|
||||
## Context
|
||||
|
||||
`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
|
||||
Investigation showed the problem is systemic, not local: **49 enabled checks report
|
||||
`down`**, almost all `ssh-script`, because the resource/updates probes assume
|
||||
**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target** —
|
||||
both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
|
||||
also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
|
||||
topology view).
|
||||
|
||||
The DB is the source of truth; live state was verified via the REST API
|
||||
(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
|
||||
and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.
|
||||
|
||||
## Decisions (confirmed with operator)
|
||||
|
||||
1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
|
||||
`qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
|
||||
plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
|
||||
use direct SSH with the correct per-target user.
|
||||
2. **Canonical host-hop access** — `pct exec`/`qm guest exec` through the proxmox host is
|
||||
the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
|
||||
guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
|
||||
3. **Auto-provision monitoring for new entities** — wire script-deploy + the
|
||||
`health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
|
||||
becomes monitorable with zero manual steps (Track E).
|
||||
4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
|
||||
permanent false alarms from retired things.
|
||||
5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
|
||||
re-export seeds, seed skills, raise graph cap.
|
||||
6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
|
||||
it against the DB graph, producing a ranked drift report; the operator acts on findings
|
||||
via existing lifecycle runbooks. No auto-fix. (Track F.)
|
||||
|
||||
## Findings (evidence)
|
||||
|
||||
### A. Health-check reality gaps (49 checks `down`)
|
||||
|
||||
**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
|
||||
`-i /etc/oikos/ssh_key`, not a default-key test):**
|
||||
|
||||
The MCP `run` tool already reaches every guest correctly via
|
||||
`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
|
||||
(`attributes.host` → `hosts` edge → hubris default), SSH there, run
|
||||
`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
|
||||
That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.
|
||||
|
||||
The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
|
||||
entity's own resolved address (`internal/scheduler/scheduler.go:758`,
|
||||
`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
|
||||
`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:
|
||||
|
||||
| Class | Targets (verified) | Root cause |
|
||||
|---|---|---|
|
||||
| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
|
||||
| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
|
||||
| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
|
||||
| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
|
||||
| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |
|
||||
|
||||
**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
|
||||
the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
|
||||
(it's a homelab client with root key + scripts). So the host hop is the reliable path.
|
||||
|
||||
**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
|
||||
strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
|
||||
script deployment, not topology.
|
||||
|
||||
Health aggregation itself is correct: `WorstHealthForTarget`
|
||||
(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
|
||||
ssh-script drags an otherwise-healthy entity to `down`.
|
||||
|
||||
### B. Dead/stale data
|
||||
|
||||
- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
|
||||
(e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
|
||||
Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
|
||||
- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
|
||||
routes to it and alarms permanently.
|
||||
|
||||
### C. Knowledge-graph gaps
|
||||
|
||||
- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
|
||||
checker all exist, but **0** certificate entities. Cert expiry is invisible.
|
||||
- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
|
||||
checker exists → every zone is an `unmonitored` signal.
|
||||
- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
|
||||
- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
|
||||
- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
|
||||
299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.
|
||||
|
||||
---
|
||||
|
||||
## Work breakdown
|
||||
|
||||
### Track A — Make ssh-script checks work everywhere (route through the proxmox host)
|
||||
|
||||
Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
|
||||
tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
|
||||
host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
|
||||
(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
|
||||
hop already has working root SSH. Hosts/workstations keep direct SSH.
|
||||
|
||||
**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**
|
||||
|
||||
Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
|
||||
execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
|
||||
owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
|
||||
credential per host (root key, already authorized on hubris/strong), no per-guest keys,
|
||||
sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
|
||||
the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).
|
||||
|
||||
- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
|
||||
execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
|
||||
they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
|
||||
- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
|
||||
package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
|
||||
(`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
|
||||
other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
|
||||
the scheduler SSHes guests directly (broken), MCP host-hops (works).
|
||||
- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
|
||||
proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
|
||||
base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
|
||||
keep direct SSH (they ARE the host).
|
||||
- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
|
||||
approval. Verify one LXC end-to-end (rclone) before fanning out.
|
||||
|
||||
**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
|
||||
- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
|
||||
romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
|
||||
scripts must live in the guest.
|
||||
- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
|
||||
host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
|
||||
`checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
|
||||
- Backfill once now (all guests + mac-mini). See Track E for the automated version.
|
||||
|
||||
**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
|
||||
- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
|
||||
`attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
|
||||
for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
|
||||
- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).
|
||||
|
||||
**A4. macOS-aware check scripts.**
|
||||
- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
|
||||
(`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
|
||||
(`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
|
||||
with `security_updates=0` or read `softwareupdate --list`).
|
||||
- Each must still emit `{"health":..,"metrics":{..}}` JSON
|
||||
(`internal/scheduler/scheduler.go:767`).
|
||||
|
||||
**A5. Reachability for external/mesh-only hosts.**
|
||||
- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
|
||||
for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
|
||||
Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
|
||||
key or proxy via hubris. Confirm before assuming direct SSH works.
|
||||
- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
|
||||
keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
|
||||
Open Questions.)
|
||||
- `lxc:rclone` no longer a special case — handled by A1's pct routing.
|
||||
|
||||
**A6. ICMP-blocked VMs.**
|
||||
- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
|
||||
(`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
|
||||
Lower priority — confirm haos blocks ICMP before building.
|
||||
|
||||
### Track B — Lifecycle monitoring gate
|
||||
|
||||
**B1. Skip monitoring for deprecated/destroyed targets.**
|
||||
- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
|
||||
`state` ∈ {`deprecated`,`destroyed`}.
|
||||
- Implement by joining target state in `ListEnabledCheckDefs`
|
||||
(`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
|
||||
whose target is retired — **or** in a `housekeeping` sweep
|
||||
(`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
|
||||
(no write needed at runtime).
|
||||
- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
|
||||
- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.
|
||||
|
||||
### Track C — Dead-data cleanup
|
||||
|
||||
**C1. Delete 24 orphan check_defs + check entities.**
|
||||
- Direct SQL (have psql access): delete `check_defs` then `entities` matching
|
||||
`slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
|
||||
`enabled=false` first (already verified).
|
||||
- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
|
||||
the rows first; this is `config_mutation` → operator approval.
|
||||
|
||||
**C2. Retire the secrets route.**
|
||||
- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
|
||||
→ `deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
|
||||
(or keep as archaeology). Decide with operator.
|
||||
|
||||
**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
|
||||
- DB shows these with live `hosts` edges on strong, never cleaned up:
|
||||
`lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
|
||||
`lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
|
||||
- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
|
||||
entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
|
||||
container still exists, destroy via `pct destroy` first (destructive → approval).
|
||||
- They currently generate checks and pollute the graph/health view.
|
||||
|
||||
### Track D — Knowledge graph
|
||||
|
||||
**D1. Model TLS certificates.**
|
||||
- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
|
||||
cert) + `uses-certificate` edges from each `ingress-route`.
|
||||
- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
|
||||
checker's discovery, or seed from Caddyfile and backfill `expires` live.
|
||||
- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
|
||||
against real entities instead of nothing.
|
||||
|
||||
**D2. dns-zone monitoring gap.**
|
||||
- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]` → `monitoring: none`
|
||||
with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
|
||||
`unmonitored` noise. Re-seed.
|
||||
|
||||
**D3. Re-export seeds to fix drift.**
|
||||
- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
|
||||
other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.
|
||||
|
||||
**D4. Seed skills from disk.**
|
||||
- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
|
||||
exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
|
||||
ingest. `get_skills()` then returns data.
|
||||
|
||||
**D5. Raise graph node cap.**
|
||||
- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
|
||||
paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
|
||||
confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
|
||||
the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.
|
||||
|
||||
### Track E — Auto-provision monitoring when a new entity is created
|
||||
|
||||
Goal: the operator's request — "make sure this is handled automatically in the future
|
||||
when the agent creates new entities." Today `ensureDefaultChecks`
|
||||
(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
|
||||
NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
|
||||
admits the gap. A new entity should become monitorable with zero manual steps.
|
||||
|
||||
**E1. Hook script-deploy into entity creation / provisioning.**
|
||||
- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
|
||||
after writing check_defs, it also ensures the target can answer:
|
||||
- **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
|
||||
(reuse the host resolution from A1). Idempotent (skip if present + unchanged).
|
||||
- **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
|
||||
over SSH; locally on mac-mini).
|
||||
- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
|
||||
sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
|
||||
automated. mac-mini still needs its `dtoro` key (A3) once.
|
||||
|
||||
**E2. Tie into the lifecycle `provisioning → active` gate.**
|
||||
- The ontology already requires `health-check-answering` for `provisioning → active`
|
||||
(`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
|
||||
- Make that gate actually run one check against the new entity and require a non-`down`
|
||||
verdict before the transition is allowed. This closes the loop: an entity isn't "active"
|
||||
(and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.
|
||||
|
||||
**E3. Re-run on re-seed / attribute change.**
|
||||
- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
|
||||
(`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
|
||||
for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
|
||||
and re-deploy scripts to the new guest.
|
||||
|
||||
**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
|
||||
entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
|
||||
script-pushed + check_defs + a passing `health-check-answering` gate before going active.
|
||||
|
||||
### Track F — Read-only knowledge-graph audit skill
|
||||
|
||||
Goal: the operator's request — a skill that auto-discovers live infra and validates the
|
||||
knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
|
||||
producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
|
||||
finding to the relevant lifecycle runbook.
|
||||
|
||||
**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
|
||||
knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
|
||||
- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
|
||||
- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
|
||||
- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
|
||||
`get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.
|
||||
|
||||
**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
|
||||
- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
|
||||
discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
|
||||
entities, evidence, suggested_runbook}`.
|
||||
- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
|
||||
config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
|
||||
API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
|
||||
what's deployed at `/opt/oikos/checks/` per target.
|
||||
- Report categories (the gaps this investigation found):
|
||||
1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
|
||||
2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
|
||||
3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
|
||||
rclone class — though rclone's parent is correct; this catches real migrations).
|
||||
4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
|
||||
orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
|
||||
5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
|
||||
the guest (the strong-guest/rclone class).
|
||||
6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
|
||||
7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
|
||||
via `oikos export` diff.
|
||||
8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
|
||||
9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).
|
||||
|
||||
**F2. Author the skill.**
|
||||
- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
|
||||
`risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
|
||||
Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
|
||||
the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
|
||||
`config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
|
||||
tracks for cert/seed/graph-cap work). No mutating steps.
|
||||
- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
|
||||
(bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
|
||||
empty-skills-table gap, Track D4).
|
||||
|
||||
**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
|
||||
sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
|
||||
unless the operator wants continuous drift signals; the on-demand skill is the deliverable.
|
||||
|
||||
**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
|
||||
DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
After each track, verify via API (read-only, no approval):
|
||||
|
||||
- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
|
||||
- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
|
||||
rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
|
||||
route via `pct exec 132` on hubris.
|
||||
- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
|
||||
after scripts pushed inside + routed via strong's `pct exec`.
|
||||
- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
|
||||
genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
|
||||
- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
|
||||
- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
|
||||
WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
|
||||
- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
|
||||
('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
|
||||
- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
|
||||
`health-check-answering` before reaching `active` (E1/E2).
|
||||
- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
|
||||
- `GET /api/v1/entities?type=skill&limit=50` → >0.
|
||||
- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
|
||||
- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
|
||||
- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
|
||||
- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
|
||||
`grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
|
||||
go through the shared `pct exec`/`qm guest exec` resolver.
|
||||
- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
|
||||
the 9 categories; running it against current state reproduces this plan's findings
|
||||
(orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
|
||||
is read-only — confirm it performs no DB writes (audit-log shows only reads).
|
||||
|
||||
Unit/integration tests to add/update:
|
||||
- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
|
||||
`pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
|
||||
`public_ipv4` preferred for standalone-server (`defaults_test.go`).
|
||||
- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
|
||||
(new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
|
||||
- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
|
||||
- E1: new-entity creation triggers script push (mock pct/SSH in test).
|
||||
- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
|
||||
category counts (ghost, missing, orphan, undeployed, drift).
|
||||
|
||||
## Risks
|
||||
|
||||
- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
|
||||
guests. This is already true (pct exec requires the host up) and is a net improvement
|
||||
(one credential vs many), but a host outage now fails all its guest checks together —
|
||||
which is the *correct* blast radius (guests are unreachable when their host is down).
|
||||
- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
|
||||
one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
|
||||
`resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
|
||||
- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
|
||||
Idempotent + content-checked; never clobber a same-named operator script without diffing.
|
||||
- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
|
||||
- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
|
||||
proxying via hubris; confirm before assuming direct SSH works (A5).
|
||||
- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
|
||||
only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
|
||||
reachable probe, or grant an operator override.
|
||||
- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
|
||||
read_only; ensure discovery commands are in the read-only allowlist (no state change).
|
||||
- **Seed re-export** can surface large diffs (cognition entities) — scope export to
|
||||
topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
|
||||
- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.
|
||||
|
||||
## Open questions (none blocking; confirm during implementation)
|
||||
|
||||
- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
|
||||
`monitoring: none`? (A5)
|
||||
- secrets ingress: keep as archaeology or destroy the route? (C2)
|
||||
- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
|
||||
- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
|
||||
- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
|
||||
stacks only? (F1)
|
||||
|
||||
## Suggested order
|
||||
|
||||
A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
|
||||
guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
|
||||
→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
|
||||
→ D1 → D4 → D3 → D5. Validate after each track.
|
||||
@@ -0,0 +1,293 @@
|
||||
# 2026-07-30 — Session review: plan drift & a dead activity panel
|
||||
|
||||
**Status:** Done — 2026-08-03. Shipped in `467589d` (v0.14.1), deployed to
|
||||
production. The two operator-reported complaints are resolved and verified on
|
||||
the bug-report session itself (`398f5eda`); see [Resolution](#resolution-2026-08-03)
|
||||
at the end. Items P0.1, P0.2 (fix 1+2), P1.1, P1.2 are complete; P0.2 fix 3,
|
||||
P2.1, P2.2 are deferred (the reported symptoms no longer reproduce).
|
||||
|
||||
**Scope:** The five most-recently-active `agent:nomos` sessions by
|
||||
`last_active_at`, pulled from the live Postgres on 2026-07-30, plus the
|
||||
code paths they exercise (`cmd/nomos/store.go`, `cmd/nomos/tasks.go`,
|
||||
`web/src/lib/stores/{activity,workspace,chat}.ts`,
|
||||
`web/src/lib/components/UnifiedTimeline.svelte`).
|
||||
**Trigger:** Operator report — "the plan was off, the activity sidepanel
|
||||
was not kept up to date and feels off, not live."
|
||||
|
||||
Both complaints are real, both reproduce deterministically, and both have
|
||||
a single-line root cause. They are *not* the same bug, but they compound:
|
||||
the plan bug produces the exact event stream that the activity panel
|
||||
silently discards.
|
||||
|
||||
---
|
||||
|
||||
## Sessions reviewed
|
||||
|
||||
| # | sid | goal (short) | outcome | activity rows | plan gens | re-planned? |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 1 | `398f5eda` | hubris recurring network outage → EEE mitigation | success | 48 | 2 | **yes** |
|
||||
| 2 | `0a49ba3d` | triage active signals on host:strong | success | 30 | 1 | no |
|
||||
| 3 | `9368633d` | sensor temperatures on host:strong | success | 12 | 1 | no |
|
||||
| 4 | `2065a29a` | temps → pivot to "fun fact about chickens" | success | 18 | 2 | **yes** |
|
||||
| 5 | `bad26076` | greeting / responsiveness test | success | 6 | 1 | no |
|
||||
|
||||
**Score: 5 success / 0 partial / 0 failed.** The agent's *reasoning* was
|
||||
fine in all five. Every defect below is in the bookkeeping and the
|
||||
rendering — the parts the operator actually looks at.
|
||||
|
||||
**The correlation that matters: both sessions that re-planned (`398f5eda`,
|
||||
`2065a29a`) recorded a corrupt plan. Neither of the three that didn't
|
||||
re-plan did.** Re-planning was a 100% failure path (pre-fix).
|
||||
|
||||
---
|
||||
|
||||
## P0.1 — `update_plan_step` addresses the wrong plan generation
|
||||
|
||||
This is "the plan was off," and it was fully deterministic.
|
||||
|
||||
`proposePlan` numbered a new generation's steps *continuing* from the old
|
||||
one (`store.go:937`):
|
||||
|
||||
```go
|
||||
seq := startSeq + i + 1 // startSeq = MAX(seq) of all prior steps
|
||||
```
|
||||
|
||||
So on generation 2 of `398f5eda`, the six new steps landed at **seq 7–12**.
|
||||
|
||||
But the tool result the model got back never mentioned those numbers
|
||||
(`tasks.go:282`):
|
||||
|
||||
```
|
||||
"Plan set (6 steps). If all steps are read-only, execute now — …"
|
||||
```
|
||||
|
||||
…while `update_plan_step`'s schema told it (`tasks.go:81`):
|
||||
|
||||
```go
|
||||
"seq": "1-based step number from propose_plan."
|
||||
```
|
||||
|
||||
The model had no way to learn the real seq numbers and was explicitly told
|
||||
to use 1-based ones. It did exactly that.
|
||||
|
||||
**What the DB recorded for `398f5eda`:**
|
||||
|
||||
```
|
||||
20:07:40 propose_plan → gen 2 created at seq 7..12
|
||||
gen 1 (seq 1..6) marked `replaced`
|
||||
20:08:10 update_plan_step seq=1 running ← hits gen-1 step 1
|
||||
20:08:18 update_plan_step seq=1 done ← resurrects a `replaced` row
|
||||
20:08:18 update_plan_step seq=2 running
|
||||
…
|
||||
20:09:57 complete_task
|
||||
```
|
||||
|
||||
Result — the persisted plan was a lie in three separate ways:
|
||||
|
||||
- **Steps 1–5 (the abandoned "force 1Gbps" plan) show `done`** with real
|
||||
start/finish timestamps. Work that was never performed was recorded as
|
||||
performed. `updatePlanStep` wrote status by seq with no guard, so it
|
||||
happily flipped `replaced` → `running` → `done`.
|
||||
- **Steps 7–12 (the actual EEE work that ran) had `started_at = NULL`**
|
||||
and were bulk-closed to `done` by `completeTask`'s auto-close sweep
|
||||
(`store.go:1096`) at 20:09:57 — all six sharing one timestamp.
|
||||
- **The panel shows 12 steps**, because `getPlanSteps` returned every
|
||||
generation unfiltered (`store.go:1356`) and the frontend never reads the
|
||||
`generation` field at all (`grep generation web/src` → zero hits outside
|
||||
the API type).
|
||||
|
||||
`2065a29a` had the identical signature: gen 2 at seq 4–5, gen-1 steps 1
|
||||
and 2 flipped to `done`/`skipped` four seconds later.
|
||||
|
||||
### Fix (implemented)
|
||||
|
||||
1. **Make seq generation-relative.** `proposePlan` resets seq to `1..N` per
|
||||
generation; `(session_id, generation, seq)` is the addressing key.
|
||||
`updatePlanStep` resolves against `MAX(generation)`. This matches what
|
||||
the model naturally does and what every prompt already says.
|
||||
2. **Return the seq numbers to the model.** The `propose_plan` result now
|
||||
enumerates them (`1=…; 2=…`).
|
||||
3. **Refuse writes to superseded rows.** `updatePlanStep` addresses only
|
||||
the current generation; a stale/out-of-range seq returns
|
||||
`errPlanStepNotFound` (never resurrects a `replaced` row).
|
||||
4. **Filter by generation on read.** `getPlanSteps` returns only
|
||||
`MAX(generation)` by default; `?all=true` for the audit/eval view.
|
||||
5. **Stamp `started_at` in the auto-close sweep.** `completeTask` closing
|
||||
a step sets `started_at = COALESCE(started_at, now())`.
|
||||
|
||||
A migration (`029`) renumbers existing rows to per-generation `1..N` and
|
||||
replaces the `(session_id, seq)` index with a unique
|
||||
`(session_id, generation, seq)`.
|
||||
|
||||
---
|
||||
|
||||
## P0.2 — The activity panel invents its own timestamps
|
||||
|
||||
This is "not live / feels off," and it was worse than a staleness bug: the
|
||||
times on screen were **fabricated at render time**.
|
||||
|
||||
`activity.ts:118` — every tool entry:
|
||||
|
||||
```ts
|
||||
timestamp: now - ($msgs.length - mi) * 1000
|
||||
```
|
||||
|
||||
`now` was `Date.now()` captured at the top of `computeActivityLog`. So a
|
||||
tool call's displayed time was *"the moment this function last ran, minus
|
||||
one second per message from the end."* Not when the call happened.
|
||||
|
||||
Three consequences, all of which read as "not live":
|
||||
|
||||
- **The clock was wrong.** `UnifiedTimeline` rendered these through
|
||||
`hhmm()` / `hhmmss()`, so opening yesterday's session showed every step
|
||||
timestamped *right now*, one second apart.
|
||||
- **It churned every 3 seconds.** The message poller re-set `messages`
|
||||
unconditionally on every tick, which re-derived `activityLog`, which
|
||||
re-captured `now`. Every entry's timestamp marched forward 3s at a time,
|
||||
forever. Motion with no information.
|
||||
- **Real and fake timestamps sorted together.** Plan steps used the
|
||||
genuine `started_at`; tool calls used the synthetic value; the final
|
||||
sort mixed them. Steps with no `started_at` fell back to `now` — **97 of
|
||||
339 non-pending steps in the DB (29%) had `started_at = NULL`** — so they
|
||||
landed at the bottom of the timeline regardless of when they ran.
|
||||
|
||||
The real data already existed and was already served: `agent_activity`
|
||||
holds true `ts`, `duration_ms`, `success`, and
|
||||
`correlation_id = session_id`, exposed at `GET /agent-activity`. The panel
|
||||
ignored it and reconstructed a worse version from the message blob.
|
||||
|
||||
### Fix (implemented — fix 1 + 2)
|
||||
|
||||
1. **Carry real timestamps on tool calls.** `computeActivityLog` uses each
|
||||
tool call's message `created_at` (a true persisted time). The
|
||||
`now - (len - mi) * 1000` expression is gone entirely.
|
||||
2. **Only fall back to wall-clock for genuinely-live entries, and freeze
|
||||
it once assigned** — a `Map<id, timestamp>` outside the derivation, so
|
||||
re-deriving never moves an existing entry. This is what kills the churn.
|
||||
|
||||
Deferred to a later pass: backing the panel with `agent_activity` for
|
||||
historical sessions (fix 3, unlocks `duration_ms`) — the two reported
|
||||
symptoms (wrong clock, churn) no longer reproduce without it.
|
||||
|
||||
---
|
||||
|
||||
## P1.1 — Plan-step events for a superseded generation were silently dropped
|
||||
|
||||
The frontend half of P0.1, and the reason the panel *froze* rather than
|
||||
merely showing wrong steps.
|
||||
|
||||
On `plan.proposed` with `appended: false`, the store replaced its step
|
||||
list wholesale — so after the re-plan it held seq 7–12. Every subsequent
|
||||
`plan.step.started` / `plan.step.finished` carried seq 1–5 and a gen-1
|
||||
`step_id`, and `applyPlanStepEventTo` bailed on no match:
|
||||
|
||||
```ts
|
||||
if (i === -1) return steps
|
||||
```
|
||||
|
||||
So for the entire second half of `398f5eda` — the half where all the real
|
||||
work happened — **the panel showed six pending steps and nothing ever
|
||||
moved.** Then `completeTask` closed them in the DB while emitting only
|
||||
`task.status`, no per-step events, so they stayed pending on screen even
|
||||
after the session finished.
|
||||
|
||||
### Fix (implemented)
|
||||
|
||||
- Fixing P0.1 removed the cause (the events now carry the correct
|
||||
generation-relative seq + the panel's current steps match). The `i === -1`
|
||||
branch now `console.warn`s and increments an exported
|
||||
`droppedPlanStepEvents` counter instead of returning silently, so the
|
||||
next divergence is visible instead of looking like a dead UI.
|
||||
- **`completeTask`'s auto-close sweep now emits `plan.step.finished` per
|
||||
closed step** (scoped to the current generation). General rule enforced:
|
||||
no plan-step status change without a corresponding event.
|
||||
|
||||
---
|
||||
|
||||
## P1.2 — Every plan carried a duplicate writeback step
|
||||
|
||||
In `398f5eda` gen 2, step 11 was the model's own writeback step and step 12
|
||||
was the auto-appended one. The detector substring-matched the literal tool
|
||||
names `update_entity_attributes` / `create_relationship` in the title or
|
||||
detail; the model wrote a natural-language equivalent, so the match failed
|
||||
and a redundant step was appended. Same pattern in `0a49ba3d` and
|
||||
`9368633d`.
|
||||
|
||||
### Fix (implemented)
|
||||
|
||||
Broadened the detector to a case-insensitive check for `write back` /
|
||||
`writeback` / `upsert_knowledge` in the title or detail, on top of the
|
||||
existing tool-name match.
|
||||
|
||||
---
|
||||
|
||||
## P2.1 — Long unexplained stalls, invisible in the UI *(deferred)*
|
||||
|
||||
- `bad26076`: a greeting took **16 minutes** wall-clock with 6 activity rows.
|
||||
- `2065a29a`: step 1 showed `started_at` → `finished_at` spanning **16 minutes**
|
||||
for a `sensors` call that returned in milliseconds.
|
||||
|
||||
The work took under a second; the step was *open* for 16 minutes. The panel
|
||||
has no way to distinguish "working" from "waiting for a nudge." Surfaces a
|
||||
step's idle time: mark a `running` step *stalled* when it has had no
|
||||
`agent_activity` row for >60s. Deferred — needs the `agent_activity`-backed
|
||||
panel (P0.2 fix 3).
|
||||
|
||||
## P2.2 — `agent_activity` is a single-type table *(deferred — decision)*
|
||||
|
||||
All rows are `activity_type = 'tool_call'`. Either start emitting the other
|
||||
types the schema anticipates (`reasoning`, `plan`, `error`) or drop the
|
||||
dimension. Worth a decision, not urgent.
|
||||
|
||||
---
|
||||
|
||||
## Recommended sequence (executed)
|
||||
|
||||
| Order | Item | Status |
|
||||
|---|---|---|
|
||||
| 1 | P0.1 fix 3 + 4 (refuse superseded writes, filter on read) | done |
|
||||
| 2 | P0.2 fix 1 + 2 (real timestamps, frozen fallback) | done |
|
||||
| 3 | P1.1 (emit events from the auto-close sweep) | done |
|
||||
| 4 | P0.1 fix 2 (generation-relative seq) + migration | done |
|
||||
| 5 | P1.2, P2.1 | P1.2 done; P2.1 deferred |
|
||||
| 6 | P0.2 fix 3 (back the panel with `agent_activity`) | deferred |
|
||||
| 7 | P2.2 | deferred |
|
||||
|
||||
## Regression coverage (added)
|
||||
|
||||
- `store_test.go`: `TestUpdatePlanStep_GenerationRelative` — re-plan →
|
||||
`update_plan_step(seq=1)` must address gen-2 and never resurrect a
|
||||
superseded gen-1 `replaced` row; out-of-range seq → `errPlanStepNotFound`.
|
||||
- `store_test.go`: `TestCompleteTask_AutoCloseEmitsEvents` — auto-close
|
||||
emits one `plan.step.finished` per closed step and stamps `started_at`.
|
||||
- `store_test.go`: `TestProposePlan_RefuseInFlight` — updated for
|
||||
generation-relative seq + `?all=true`.
|
||||
- `web/src/lib/stores/activity.test.ts`: `computeActivityLog` is pure w.r.t.
|
||||
wall-clock (two calls 50ms apart → identical output), persisted tool calls
|
||||
use real `created_at`, live entries freeze instead of churning.
|
||||
|
||||
---
|
||||
|
||||
## Resolution (2026-08-03)
|
||||
|
||||
Shipped in commit `467589d` (VERSION `0.14.0` → `0.14.1`), pushed to
|
||||
`origin/main`, deployed via the Gitea webhook (`scripts/deploy.sh`):
|
||||
pg_dump → pull → `docker compose build` → `up -d` → health check (healthy).
|
||||
|
||||
Verification on the bug-report session `398f5eda` post-migration:
|
||||
|
||||
```
|
||||
gen 1: seq 1..6 (the abandoned "force 1Gbps" plan — superseded)
|
||||
gen 2: seq 1..6 (the real EEE work — was seq 7..12, now normalized to 1..6)
|
||||
```
|
||||
|
||||
- `schema_migrations` v29 applied; old `idx_plan_steps_session` dropped,
|
||||
unique `idx_plan_steps_session_gen_seq` in place.
|
||||
- Containers recreated; `healthz` and `/agent/sessions/:id/plan` HTTP 200.
|
||||
- Full `cmd/nomos` suite (23 tests) + web suite (70 tests) green; `go vet`
|
||||
clean; ESLint/Prettier clean.
|
||||
|
||||
Note: historical `started_at = NULL` on already-completed steps (visible on
|
||||
`398f5eda` gen 2) is left as-is — backfilling would fabricate times. Going
|
||||
forward `completeTask` stamps `started_at`, and the frontend freezes
|
||||
NULL-started steps stably so they no longer churn.
|
||||
503
plans/done/2026-08-03-cyberspace-style-adoption.md
Normal file
503
plans/done/2026-08-03-cyberspace-style-adoption.md
Normal file
@@ -0,0 +1,503 @@
|
||||
# 2026-08-03 — Adopt cyberspace.online terminal aesthetic + dithered images
|
||||
|
||||
**Status:** Implemented in v0.16.0 (`757ef2f`). Shipped as a **full theme
|
||||
replacement** (Terracotta/Carbon → cyberspace BBS/terminal style), not the
|
||||
opt-in addition originally drafted below — the operator chose full replacement
|
||||
during execution (see decision `theme.replace_with_cyberspace`). The `<RasterImage>`
|
||||
Atkinson-dithering component and the warm-cream/JetBrains-Mono look landed as
|
||||
drafted; only the "opt-in vs replace" scope changed.
|
||||
|
||||
Adopt the look of https://cyberspace.online/ (a BBS / "social media
|
||||
de-imagined" terminal aesthetic) as a **new, opt-in theme family** in oikos,
|
||||
with **both light and dark variants**, plus a reusable **`<RasterImage>`**
|
||||
component that renders images to a `<canvas>` with Atkinson dithering (the
|
||||
"kinda dithered" image style). The existing Terracotta/Carbon themes stay the
|
||||
default; this adds, it does not replace.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. Add a third theme family — **"Cyberspace Dark"** and **"Cyberspace Light"** —
|
||||
wired through the same `--background` / `--foreground` / … token layer every
|
||||
component already uses, so nothing in the UI tree changes; only the tokens
|
||||
get new values. Square corners (`--radius: 0`), warm cream-on-black, mono
|
||||
everything.
|
||||
2. Extend `web/src/lib/stores/theme.svelte.ts` from a 2-state `'light'|'dark'`
|
||||
toggle to a named-theme model, keeping `.dark` class behavior for
|
||||
compatibility.
|
||||
3. Self-host JetBrains Mono (body) + a pixel/terminal face (VT323 or Departure
|
||||
Mono) for the logo/headings accents, replacing the Google Fonts `<link>`.
|
||||
4. Build `web/src/lib/components/RasterImage.svelte`: draws any image to a
|
||||
`<canvas>` reduced to a 2-color (theme `fg`/`bg`) palette via **Atkinson
|
||||
dithering**, with an `<img>` fallback and a skeleton placeholder — exactly
|
||||
the cyberspace pattern. Re-renders when the theme changes (palette flips).
|
||||
5. Optional cosmetic idioms (terminal-box focus ring, braille spinner, `<s>`
|
||||
strike lists) as small additive utilities, not a redesign.
|
||||
|
||||
The whole thing is **non-breaking and incremental**: each step ships behind the
|
||||
existing theme picker, so Terracotta/Carbon users see nothing until they opt in.
|
||||
|
||||
---
|
||||
|
||||
## 1. Extracted style spec (source of truth from cyberspace.online)
|
||||
|
||||
Captured from the live site's SSR HTML + inline boot script. This is the
|
||||
reference the tokens below are derived from.
|
||||
|
||||
### 1.1 Color model
|
||||
|
||||
Cyberspace defines **exactly three colors per theme** — `fg`, `bg`, `fgDim` —
|
||||
applied to CSS custom properties. Everything else (borders, primary, cards) is
|
||||
*derived* from those three. There are 11 named themes total; the two we care
|
||||
about:
|
||||
|
||||
| Theme | `fg` (text) | `bg` (canvas) | `fgDim` (muted) |
|
||||
|---------|--------------|---------------|-----------------|
|
||||
| Dark | `#efe5c0` | `#000000` | `#a89984` |
|
||||
| Light | `#000000` | `#efe5c0` | `#3a3a3a` |
|
||||
|
||||
Note the elegance: **light and dark are exact inverses** — they share the same
|
||||
warm cream (`#efe5c0`, a Gruvbox-ish paper tone) and just swap which side of it
|
||||
is ink vs. paper. The muted tone `#a89984` is straight out of the Gruvbox
|
||||
palette. This is why both themes read as "the same site" despite opposite
|
||||
polarity.
|
||||
|
||||
Boot-time fallback (the site's original/GRiD theme) is amber `#FF9810` on
|
||||
`#120900` — useful as a *third* optional accent if we ever want a true-phosphor
|
||||
variant.
|
||||
|
||||
### 1.2 Type
|
||||
|
||||
- **Body / mono:** JetBrains Mono (self-hosted `.woff2`, Regular).
|
||||
- **Boot + logo accents:** Departure Mono (self-hosted `.woff2`). A quirky
|
||||
monospace; VT323 (Google, free) is a close, easy substitute.
|
||||
- **Stylized wordmark** (`ᑕ¥βєяรקค¢є`, class `.font-vt`): a terminal/pixel face.
|
||||
Rule lives in their external `entry.*.css` (not in the SSR dump); VT323 is the
|
||||
safe assumption.
|
||||
|
||||
cyberspace sets `font-mono` on the root wrapper — the **entire UI is
|
||||
monospace**. There is no proportional body face. Headings use the same mono
|
||||
family at larger size / normal weight.
|
||||
|
||||
### 1.3 Layout & component idioms
|
||||
|
||||
- **Left rail nav:** fixed, icon-only when minimized (~80px), expands on click.
|
||||
Square buttons, Phosphor icons, uppercase `text-xs` labels.
|
||||
- **`.terminal-box`:** the universal card. Bordered (`border border-border`),
|
||||
**square corners** (`rounded-none` everywhere — `--radius` is effectively 0),
|
||||
and on focus/emphasis gets `ring-2 ring-fg` (a 2px ring in the foreground
|
||||
color).
|
||||
- **Emphasis by inversion:** active/primary state is `bg-fg text-bg` — fill with
|
||||
foreground ink, text becomes the canvas color. No separate "accent" hue; the
|
||||
accent *is* fg.
|
||||
- **Strikethrough as a feature list:** `<s>Ads</s> <s>Videos</s> …` — crossed-out
|
||||
`<s>` elements spell out what the product removes. Cheap, on-brand.
|
||||
- **Braille spinner:** `BrailleSpinner` component animates braille block chars
|
||||
(`⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏`) for loading states instead of a circle.
|
||||
- **Max content width** `max-w-4xl`, centered; generous vertical rhythm; thin
|
||||
2px scrollbars colored `--color-border`.
|
||||
- Borders are `1px solid` in a border color derived from `fg`/`fgDim` at low
|
||||
alpha (their `--color-border` is not literally in the dump, but every
|
||||
bordered surface uses it, and it tracks `fg`).
|
||||
|
||||
### 1.4 The dithered image (`RasterImage`) — what we actually know
|
||||
|
||||
From the SSR HTML the component is unambiguous about its *shape*, silent on its
|
||||
*algorithm* (the dither JS is in an external `/_nuxt/*.js` bundle not present in
|
||||
the page dump):
|
||||
|
||||
- Renders a **`<canvas>`** as primary output, with an **`<img>` fallback** as a
|
||||
sibling. Parent selectors `[&>canvas]:max-w-full [&>canvas]:h-auto` and
|
||||
`[&>img]:…` size both responsively.
|
||||
- Emits a **`.raster-image-skeleton`** placeholder (empty div, `background:
|
||||
var(--color-bg)`) during SSR/before hydration — no flash of the raw photo.
|
||||
- Scoped styles (`data-v-4d61df89`): `.raster-image { display:block }`,
|
||||
`.raster-image-skeleton { display:block; background:var(--color-bg) }`.
|
||||
|
||||
**Inferred technique** (standard for this look): Canvas 2D → `drawImage` →
|
||||
`getImageData` → per-pixel luminance reduction to a 2-color palette (`fg`/`bg`)
|
||||
with an **error-diffusion** pass (Atkinson or Floyd–Steinberg) → `putImageData`.
|
||||
This produces the characteristic speckled 1-bit halftone. Target is almost
|
||||
certainly the theme's own fg/bg, which is *why* the dithered art recolors
|
||||
correctly when you flip themes.
|
||||
|
||||
We will implement Atkinson (see §4) — it's the classic Mac/BBS dither, slightly
|
||||
softer than Floyd–Steinberg, and matches "kinda dithered" precisely.
|
||||
|
||||
---
|
||||
|
||||
## 2. Recommended approach: opt-in theme family (not a rebrand)
|
||||
|
||||
oikos today = Art-Nouveau / terracotta / rounded / serif-heading (Inknut
|
||||
Antiqua), floating-window desktop shell. cyberspace = BBS / mono / square /
|
||||
cream-on-black. These are **opposite poles**; a flat rebrand would discard the
|
||||
existing art direction and rework every component's rounding/spacing.
|
||||
|
||||
**Decision: add cyberspace as a new theme family, selectable in the existing
|
||||
theme picker.** This is low-risk, reversible, and lets the dithered images +
|
||||
terminal idioms land incrementally. The full-rebrand alternative is documented
|
||||
in §7 for if you later decide to make it the default.
|
||||
|
||||
Because every oikos component consumes colors through the Tailwind v4 token
|
||||
layer (`--background`, `--foreground`, `--card`, `--border`, `--primary`, …)
|
||||
defined in `web/src/app.css` `@theme inline`, a new theme is **just a new set
|
||||
of values for those same custom properties** — zero component edits required
|
||||
for the recolor. That indirection is the whole reason this is cheap.
|
||||
|
||||
---
|
||||
|
||||
## 3. Theme token additions (`web/src/app.css`)
|
||||
|
||||
Add two new blocks alongside the existing `:root` (Terracotta) and `.dark`
|
||||
(Carbon). They set the *same* token names to cyberspace's values, plus pin
|
||||
`--radius: 0` for square corners and remap fonts (see §5).
|
||||
|
||||
Driven by a `data-theme` attribute on `<html>` (set by the store, §6), so all
|
||||
four states — Terracotta, Carbon, Cyberspace Dark, Cyberspace Light — coexist:
|
||||
|
||||
```css
|
||||
/* ── Cyberspace Dark (cream on black) ── */
|
||||
:root[data-theme='cyber-dark'] {
|
||||
--radius: 0px;
|
||||
--background: #000000;
|
||||
--foreground: #efe5c0;
|
||||
--card: #000000; /* cyberspace has no card tint; cards are just bordered bg */
|
||||
--card-foreground: #efe5c0;
|
||||
--popover: #000000;
|
||||
--popover-foreground: #efe5c0;
|
||||
--primary: #efe5c0; /* emphasis = fg ink */
|
||||
--primary-foreground: #000000; /* inverted */
|
||||
--secondary: #1a1a1a;
|
||||
--secondary-foreground: #efe5c0;
|
||||
--muted: #141414;
|
||||
--muted-foreground: #a89984; /* fgDim */
|
||||
--accent: #efe5c0;
|
||||
--accent-foreground: #000000;
|
||||
--destructive: #cc241d; /* Gruvbox red, sits in the same palette */
|
||||
--destructive-foreground: #efe5c0;
|
||||
--border: color-mix(in oklab, #efe5c0 22%, transparent); /* fg-derived hairline */
|
||||
--input: color-mix(in oklab, #efe5c0 28%, transparent);
|
||||
--ring: #efe5c0; /* the ring-2 ring-fg look */
|
||||
--sidebar: #000000;
|
||||
--sidebar-foreground: #efe5c0;
|
||||
--sidebar-primary: #efe5c0;
|
||||
--sidebar-primary-foreground: #000000;
|
||||
--sidebar-accent: #1a1a1a;
|
||||
--sidebar-accent-foreground: #efe5c0;
|
||||
--sidebar-border: color-mix(in oklab, #efe5c0 22%, transparent);
|
||||
--sidebar-ring: #efe5c0;
|
||||
--chart-1: #efe5c0; --chart-2: #a89984; --chart-3: #fabd2f;
|
||||
--chart-4: #b8bb26; --chart-5: #83a598; /* Gruvbox for charts */
|
||||
--success: #b8bb26; --warning: #fabd2f;
|
||||
|
||||
/* oikos semantic aliases (app.css :root block) */
|
||||
--bg: var(--background); --bg-surface: var(--card); --bg-deeper: #050505;
|
||||
--bg-hover: var(--secondary); --bg-active: var(--accent);
|
||||
--text: var(--foreground); --text-muted: var(--muted-foreground);
|
||||
--accent-blue: #83a598; --accent-green: var(--success);
|
||||
--accent-red: var(--destructive); --accent-orange: var(--warning);
|
||||
|
||||
/* terminal face for this theme only (see §5) */
|
||||
--font-sans: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||
--font-mono: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||
--font-heading: 'VT323', 'JetBrains Mono', monospace; /* pixel wordmark feel */
|
||||
}
|
||||
|
||||
/* ── Cyberspace Light (black on cream paper) — exact inverse ── */
|
||||
:root[data-theme='cyber-light'] {
|
||||
--radius: 0px;
|
||||
--background: #efe5c0;
|
||||
--foreground: #000000;
|
||||
--card: #efe5c0;
|
||||
--card-foreground: #000000;
|
||||
--popover: #efe5c0;
|
||||
--popover-foreground: #000000;
|
||||
--primary: #000000;
|
||||
--primary-foreground: #efe5c0;
|
||||
--secondary: #e0d6b0;
|
||||
--secondary-foreground: #000000;
|
||||
--muted: #e6dcc0;
|
||||
--muted-foreground: #3a3a3a; /* fgDim */
|
||||
--accent: #000000;
|
||||
--accent-foreground: #efe5c0;
|
||||
--destructive: #9d0006;
|
||||
--destructive-foreground: #efe5c0;
|
||||
--border: color-mix(in oklab, #000000 22%, transparent);
|
||||
--input: color-mix(in oklab, #000000 28%, transparent);
|
||||
--ring: #000000;
|
||||
--sidebar: #efe5c0;
|
||||
--sidebar-foreground: #000000;
|
||||
--sidebar-primary: #000000;
|
||||
--sidebar-primary-foreground: #efe5c0;
|
||||
--sidebar-accent: #e0d6b0;
|
||||
--sidebar-accent-foreground: #000000;
|
||||
--sidebar-border: color-mix(in oklab, #000000 22%, transparent);
|
||||
--sidebar-ring: #000000;
|
||||
--chart-1: #000000; --chart-2: #3a3a3a; --chart-3: #b57614;
|
||||
--chart-4: #79740e; --chart-5: #076678;
|
||||
--success: #79740e; --warning: #b57614;
|
||||
|
||||
--bg: var(--background); --bg-surface: var(--card); --bg-deeper: #e6dcc0;
|
||||
--bg-hover: var(--secondary); --bg-active: var(--accent);
|
||||
--text: var(--foreground); --text-muted: var(--muted-foreground);
|
||||
--accent-blue: #076678; --accent-green: var(--success);
|
||||
--accent-red: var(--destructive); --accent-orange: var(--warning);
|
||||
|
||||
--font-sans: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||
--font-mono: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||
--font-heading: 'VT323', 'JetBrains Mono', monospace;
|
||||
}
|
||||
```
|
||||
|
||||
Two notes:
|
||||
|
||||
- **`.dark` vs `data-theme`.** The current store flips `.dark` on `<html>`. To
|
||||
keep Carbon working unchanged, leave `.dark` logic alone and layer
|
||||
`data-theme` on top: when a cyberspace theme is active the store sets
|
||||
`data-theme` and **removes** `.dark` (cyberspace themes are self-contained —
|
||||
they set both polarities explicitly). See §6.
|
||||
- **Borders from `fg`.** cyberspace's hairline tracks the foreground, not a
|
||||
fixed gray. `color-mix(in oklab, <fg> 22%, transparent)` reproduces that and
|
||||
auto-flips between the two themes. Tune the % after visual review.
|
||||
|
||||
---
|
||||
|
||||
## 4. The dithered image component (`RasterImage.svelte`)
|
||||
|
||||
**File:** `web/src/lib/components/RasterImage.svelte` (sibling of the existing
|
||||
`Spinner.svelte`).
|
||||
|
||||
### 4.1 API
|
||||
|
||||
```svelte
|
||||
<RasterImage src={entity.iconUrl} alt="host icon" width={320} />
|
||||
<!-- optional: scale (downsample factor), threshold bias, mono palette override -->
|
||||
```
|
||||
|
||||
- `src`, `alt` — as `<img>`.
|
||||
- `width` — render width in CSS px; canvas is sized to this × natural aspect.
|
||||
Downscaling before dithering is what sells the "lo-fi" look (defaults ~256–
|
||||
320). Expose `scale` (0–1) to control.
|
||||
- Reads the active theme's `--foreground` / `--background` via
|
||||
`getComputedStyle(document.documentElement)` so the dither palette **follows
|
||||
the theme** (cream/black in cyber-dark, black/cream in cyber-light, and
|
||||
perfectly sensible in Terracotta/Carbon too).
|
||||
|
||||
### 4.2 Behavior
|
||||
|
||||
1. Show `.raster-image-skeleton` (empty, `background: var(--background)`) until
|
||||
the source image loads — matches cyberspace's no-flash placeholder.
|
||||
2. On load: create an offscreen canvas at `width × (h/w*width)`, `drawImage`
|
||||
(with `imageSmoothingEnabled = true` for the downscale), pull
|
||||
`getImageData`.
|
||||
3. Run **Atkinson dithering** to 2 colors:
|
||||
- For each pixel: luminance `Y = 0.299R + 0.587G + 0.114B`.
|
||||
- Threshold at 128 (+ optional `bias`), snap to either `fg` or `bg`.
|
||||
- Push **1/8 of the quantization error** to each of 6 neighbors (Atkinson's
|
||||
kernel): right, below-left, below, below-right, and two pixels down on the
|
||||
next-next row. (Atkinson diffuses less than Floyd–Steinberg → softer, more
|
||||
"screen-printed" — exactly the cyberspace feel.)
|
||||
- Write `fg`/`bg` (read from CSS vars at render time) into the buffer.
|
||||
4. `putImageData`. Canvas is the visible output; the loaded `<img>` is kept as
|
||||
`aria-hidden` fallback for no-JS / copy-image / accessibility.
|
||||
5. **Re-dither on theme change**: subscribe to the theme store; when it flips,
|
||||
re-read `--foreground`/`--background` and re-run steps 3–4 (cheap — the
|
||||
decoded `ImageBitmap` is cached, only the palette pass reruns). This is the
|
||||
detail that makes the art flip polarity with the theme toggle.
|
||||
6. **Respect `prefers-reduced-data` / reduced motion?** Dithering is not motion,
|
||||
but offer a `plain` prop to skip the canvas and render the raw `<img>` for
|
||||
users who want crisp photos (e.g. entity detail screens where legibility
|
||||
beats aesthetic).
|
||||
|
||||
### 4.3 Reference dither kernel (Atkinson)
|
||||
|
||||
```
|
||||
* → 1/8 1/8
|
||||
1/8 1/8 1/8 (current pixel = *)
|
||||
1/8 1/8 (* is at top-left of this 4×? — see standard Atkinson spread)
|
||||
```
|
||||
|
||||
Spread pattern (error e from pixel at (x,y) distributed):
|
||||
|
||||
```
|
||||
px x+1 (1/8) x+2 (1/8)
|
||||
x-1 (1/8) x (1/8) x+1 (1/8)
|
||||
x+1 (1/8) x+2 (1/8) [next row offsets]
|
||||
```
|
||||
|
||||
Concretely, 6 neighbors each get `e/8`: `(x+1,y)`, `(x+2,y)`, `(x-1,y+1)`,
|
||||
`(x,y+1)`, `(x+1,y+1)`, `(x,y+2)`. (Clamp at edges — drop, don't wrap.)
|
||||
|
||||
### 4.4 Where to use it
|
||||
|
||||
- Entity icons / host thumbnails in the KB and entity desktop (the obvious win).
|
||||
- Mascot or login/Config background art (`ConfigBackground.svelte` already
|
||||
exists — a dithered backdrop there would be striking).
|
||||
- Any user-uploaded image in chat/knowledge where we want the "de-imagined"
|
||||
tone. Keep it **opt-in per call site** via the `plain` prop — don't dither
|
||||
diagrams/screenshots that need to stay readable.
|
||||
|
||||
### 4.5 Cross-origin caveat
|
||||
|
||||
`getImageData` throws on tainted canvases. If `src` is cross-origin and the
|
||||
server doesn't send CORS headers, fall back to the plain `<img>` (log once).
|
||||
For self-hosted assets (the common case here) it's a non-issue.
|
||||
|
||||
---
|
||||
|
||||
## 5. Fonts: self-host JetBrains Mono + VT323
|
||||
|
||||
cyberspace self-hosts both faces as `.woff2`. oikos currently pulls DM Sans /
|
||||
DM Mono / Inknut Antiqua from Google Fonts via a `<link>` in
|
||||
`web/index.html:10`.
|
||||
|
||||
- Drop `JetBrainsMono-Regular.woff2` and `VT323-Regular.woff2` under
|
||||
`web/static/fonts/` (or `web/public/fonts/` — match where static assets are
|
||||
served from; check `vite.config`).
|
||||
- Add `@font-face` blocks at the top of `app.css` with `font-display: swap`.
|
||||
- For the cyberspace themes only, the `--font-sans`/`--font-mono`/`--font-heading`
|
||||
overrides in §3 remap the families — Terracotta/Carbon keep DM Sans/Inknut
|
||||
untouched. This is the key trick: **font choice is part of the theme**, not a
|
||||
global swap, so the two art directions don't fight.
|
||||
- Leave the Google Fonts `<link>` in place for now (Terracotta/Carbon still need
|
||||
it); add a follow-up to self-host those too if we want to kill the external
|
||||
request entirely. Out of scope for this plan.
|
||||
|
||||
VT323 vs Departure Mono: VT323 is free on Google Fonts and trivial to self-host;
|
||||
Departure Mono is the authentic cyberspace face but needs a license check.
|
||||
**Recommend VT323** to start; swap to Departure Mono later if you want exact
|
||||
fidelity.
|
||||
|
||||
---
|
||||
|
||||
## 6. Theme store changes (`web/src/lib/stores/theme.svelte.ts`)
|
||||
|
||||
Current: `Theme = 'light' | 'dark'`, flips `.dark` class. Extend to a named set
|
||||
while preserving the existing API (callers of `toggleTheme`/`getTheme` keep
|
||||
working):
|
||||
|
||||
```ts
|
||||
export type ThemeName = 'terracotta' | 'carbon' | 'cyber-dark' | 'cyber-light'
|
||||
// Back-compat aliases used by existing callers:
|
||||
// 'light' -> 'terracotta', 'dark' -> 'carbon'
|
||||
```
|
||||
|
||||
- Store key stays `oikos-theme`; migrate old `'light'`/`'dark'` values on read.
|
||||
- `applyClass` becomes `applyTheme`: sets `data-theme` on `<html>` and toggles
|
||||
`.dark` **only** for `carbon` (so Terracotta and both cyberspace themes run
|
||||
with no `.dark`). This is important: the `.dark` block in `app.css` must not
|
||||
layer on top of the cyberspace token blocks — cyberspace sets its own
|
||||
polarities.
|
||||
- Update `THEME_LABELS` to the four names; update whatever UI surfaces the
|
||||
picker (search for `THEME_LABELS` / `toggleTheme` usages — likely
|
||||
`Settings.svelte` or the desktop shell's chrome) to a 4-option control instead
|
||||
of a binary toggle.
|
||||
|
||||
**Watch out:** any code that assumes `document.documentElement.classList.contains('dark')`
|
||||
≡ "dark colors" will be wrong for `cyber-dark`. Audit `grep -rn "classList.*dark\|\.dark" web/src` and prefer reading `getTheme()`/`data-theme` instead.
|
||||
|
||||
---
|
||||
|
||||
## 7. Optional cosmetic idioms (additive utilities)
|
||||
|
||||
Small, theme-aware utilities in `app.css` — usable in any theme but idiomatic
|
||||
for cyberspace:
|
||||
|
||||
- `.terminal-box` — `{ border:1px solid var(--border); border-radius:0 }` plus a
|
||||
`.terminal-box:focus-within { box-shadow: 0 0 0 2px var(--ring) }` to mirror
|
||||
the `ring-2 ring-fg` focus. Lets cards opt into the terminal look without a
|
||||
component rewrite.
|
||||
- `.braille-spinner` — keyframe cycling `⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏` as `::after` content, colored
|
||||
`var(--muted-foreground)`. Alternative to `Spinner.svelte` for loading states
|
||||
under cyberspace themes.
|
||||
- `.font-vt` — `{ font-family: var(--font-heading) }` so the wordmark class
|
||||
cyberspace uses maps to our heading var (VT323 under cyber themes, Inknut
|
||||
under Terracotta). Drop-in for any stylized title.
|
||||
- `.strike-list` — `li > s { color: var(--muted-foreground) }` convenience for
|
||||
the crossed-out feature-list pattern in marketing/empty states.
|
||||
|
||||
None of these are required for the theme to work; they're palette for the
|
||||
"de-imagined" voice where we want it.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation order (incremental, each step shippable)
|
||||
|
||||
1. **Fonts** (§5) — self-host JetBrains Mono + VT323, `@font-face` in app.css.
|
||||
No visual change yet (only cyberspace themes reference them).
|
||||
2. **Tokens** (§3) — add the two `:root[data-theme='cyber-*']` blocks.
|
||||
3. **Store** (§6) — extend `theme.svelte.ts` to named themes + `data-theme`;
|
||||
update the picker UI. **At this point both cyberspace themes are live and
|
||||
fully recolor the whole app** — the cheapest milestone, biggest visible win.
|
||||
4. **`RasterImage.svelte`** (§4) — build + wire into entity icons and
|
||||
`ConfigBackground`. This is the "dithered image" deliverable.
|
||||
5. **Idioms** (§7) — terminal-box, braille spinner, etc., applied opportunistically.
|
||||
|
||||
Each step is independently mergeable. Step 3 alone satisfies "light + dark
|
||||
cyberspace themes"; step 4 satisfies "dithered images."
|
||||
|
||||
---
|
||||
|
||||
## 9. Verification
|
||||
|
||||
- `cd web && npm run build` (or the repo's build command — confirm in
|
||||
`web/package.json`) — Tailwind v4 must accept the new `data-theme` selectors
|
||||
and `color-mix()` (both standard; no config change expected).
|
||||
- `npm run check` / `svelte-check` for the store + component TS.
|
||||
- Manual: cycle all four themes in the picker; confirm no `.dark` bleed on
|
||||
`cyber-light`; confirm `RasterImage` re-dithers on theme flip; confirm
|
||||
`prefers-reduced-data`/`plain` prop shows crisp image; confirm cross-origin
|
||||
`src` degrades to `<img>` without console errors.
|
||||
- Lighthouse / a11y: 1-bit dithered images still need a real `alt` (kept on the
|
||||
fallback `<img>`); contrast on `#a89984`-on-black passes WCAG AA for body text
|
||||
(ratio ≈ 7.4:1) — fine.
|
||||
|
||||
---
|
||||
|
||||
## 10. Alternatives considered
|
||||
|
||||
- **Full rebrand (replace Terracotta/Carbon).** Highest visual payoff, highest
|
||||
cost: every component's rounding/serif/spacing was authored for the Art
|
||||
ouveau
|
||||
direction; square + mono would need a component-level sweep, not just tokens.
|
||||
Defer unless you want cyberspace as *the* oikos look — then do it as a
|
||||
follow-up that deletes Terracotta/Carbon and makes `cyber-dark` the sole
|
||||
default.
|
||||
- **CSS-only image dither (filters / SVG turbulence).** Cheaper, but can't do
|
||||
true 1-bit error diffusion or recolor to theme fg/bg. Rejected — the canvas
|
||||
pass is the whole point and is ~60 lines.
|
||||
- **Ordered (Bayer) dither instead of Atkinson.** More regular/grid-like
|
||||
("newspaper halftone"). Atkinson is softer and more terminal-like; keep
|
||||
Bayer as a `algorithm='bayer'` prop option later if wanted.
|
||||
- **Server-side dithering.** Could pre-dither icons at ingest. Rejected for
|
||||
now — client canvas keeps one source of truth (the original image) and lets
|
||||
the palette follow the live theme, which a baked asset can't.
|
||||
|
||||
---
|
||||
|
||||
## 11. Non-goals / out of scope
|
||||
|
||||
- Replicating cyberspace's sidebar-rail *layout* (oikos uses a floating-window
|
||||
desktop shell; the rail is a different app model). We take the *visual*
|
||||
language, not the IA.
|
||||
- Porting the 9 other novelty themes (C64, Matrix, VT320, …). Two (light/dark)
|
||||
satisfy the request; the token model makes adding more trivial later.
|
||||
- Removing the Google Fonts dependency for Terracotta/Carbon (follow-up).
|
||||
- Licensing/redistributing Departure Mono (use VT323 unless cleared).
|
||||
|
||||
---
|
||||
|
||||
## 12. Risks
|
||||
|
||||
- **`.dark` coupling.** Existing code may equate `.dark` with "dark UI".
|
||||
Mitigation: audit in step 3; the grep is small.
|
||||
- **Dither perf on large images.** Atkinson is O(n) and runs on a downscaled
|
||||
canvas (≤~320px wide), so per-image cost is negligible; but batch-rendering
|
||||
many entity icons on first paint could jank. Mitigation: dither lazily (on
|
||||
intersection) and cache the result on the element.
|
||||
- **Tainted canvas** on cross-origin images → silent fallback to `<img>`
|
||||
(already handled in the design).
|
||||
- **Token drift.** If a component hardcodes a color instead of using a token,
|
||||
it won't recolor under cyberspace. This is the same risk Carbon already has;
|
||||
no new exposure, just more visible under a stronger theme.
|
||||
184
plans/done/2026-08-03-nomos-chat-changes-review.md
Normal file
184
plans/done/2026-08-03-nomos-chat-changes-review.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# 2026-08-03 — Review: nomos chat reliability/UX changes (F1–F7)
|
||||
|
||||
**Status:** Implemented (P0, P1, P2 all done). See
|
||||
[Resolution](#resolution) at the end.
|
||||
|
||||
A critical self-review of the uncommitted F1–F7 changeset
|
||||
(`plans/2026-08-03-nomos-chat-reliability-and-ux-audit.md` Resolution). The
|
||||
change set is mostly sound and builds/tests green, but **F1 introduced one
|
||||
real lost-work regression** by changing the contract of `resumeSession` (it can
|
||||
now skip) without updating two callers that mutate state *before* calling it.
|
||||
That must be fixed before this ships.
|
||||
|
||||
## What was changed (for orientation)
|
||||
- F1 `cmd/nomos/turngate.go` (+test): per-session single-flight; `resumeSession`
|
||||
acquires non-blocking and **skips** if a turn is active; `handleChat` live path
|
||||
acquires with a 5s wait.
|
||||
- F2/F3 `web/src/lib/stores/chat.ts`: humanized errors, `clearTurnState` on
|
||||
terminal `task.status`, turn-free reconnect.
|
||||
- F4 streaming in global `activityLog` + inline `ToolCallCard`.
|
||||
- F5 artifact/knowledge deep links; F6 step-first headline; F7 stable layout.
|
||||
|
||||
---
|
||||
|
||||
## P0 — F1 loses finished-execution continuations (must fix before shipping)
|
||||
|
||||
**Bug.** `processContinuations` (`cmd/nomos/continue.go:166-167`) calls
|
||||
`a.store.markContinued(ctx, p.ExecID)` **before** dispatching
|
||||
`continueSession → resumeSession`. `markContinued` sets `continued_at`, and
|
||||
`pendingContinuations` (`store.go:1763`) filters `WHERE continued_at IS NULL` —
|
||||
so a marked execution is **never re-queued**.
|
||||
|
||||
Before F1, `resumeSession` always ran, so marking-first was safe. F1 made
|
||||
`resumeSession` skip when a turn is already active for the session. Now:
|
||||
|
||||
- **Two executions for one session finish near-simultaneously** (the common
|
||||
multi-step case): the loop marks BOTH, spawns two goroutines; goroutine 1
|
||||
acquires and runs, goroutine 2's `resumeSession` **skips** → execution 2 is
|
||||
marked continued but its result is **never fed back to the agent. Lost.**
|
||||
- **A live turn is streaming when an async execution finishes**: continuation
|
||||
marks + dispatches; `resumeSession` skips (live turn holds the permit) →
|
||||
result lost.
|
||||
|
||||
This silently drops auto-continuation — worse than the interleaving F1 set out
|
||||
to fix.
|
||||
|
||||
**Fix.** Make `resumeSession` report whether it actually ran, and mark-continued
|
||||
only after a successful run; on a busy-skip, leave the execution pending for the
|
||||
next worker tick.
|
||||
|
||||
1. `cmd/nomos/continue.go` — change `resumeSession` to return `bool`:
|
||||
```go
|
||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||
if !a.gate.acquire(sessionID, 0) {
|
||||
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||
return false
|
||||
}
|
||||
defer a.gate.release(sessionID)
|
||||
…existing body…
|
||||
return true
|
||||
}
|
||||
```
|
||||
2. `continueSession` — mark only after a real run; on skip, leave pending:
|
||||
```go
|
||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||
return
|
||||
}
|
||||
a.store.markContinued(ctx, p.ExecID)
|
||||
}
|
||||
```
|
||||
3. `processContinuations` — **delete** the `a.store.markContinued(ctx, p.ExecID)`
|
||||
line at `continue.go:166` (the dispatch `safego.Go(... continueSession ...)`
|
||||
stays). The `markContinued` at `:162` (the no-assent-window branch, which
|
||||
saves a note and does **not** call resumeSession) stays as-is — that path
|
||||
intentionally consumes the item.
|
||||
4. Update every other `resumeSession` caller to ignore the new return value
|
||||
(`/resume`, `handleAnswerQuestion`, the empty-message reconnect in
|
||||
`handleChat`) — they don't need the bool; a bare call discards it. No behavior
|
||||
change for them (their skip semantics are already correct/desired).
|
||||
|
||||
**Why this preserves the original "no re-continue loop" guarantee:** a
|
||||
`resumeSession` that *runs* always returns `true` (even on its internal LLM
|
||||
failure path — it has already persisted a failure note), so it gets marked and
|
||||
won't loop. Only a *busy-skip* returns `false` and stays pending, which is
|
||||
correct (retry once the turn frees). Crash-safety also improves: a crash between
|
||||
acquire and mark leaves the item un-marked → re-queued on restart.
|
||||
|
||||
**Validation:**
|
||||
- New test: two `pendingContinuation`s for one session dispatched concurrently;
|
||||
assert both are eventually processed (both `continued_at` set) and at no point
|
||||
do two `resumeSession` bodies overlap (reuse the `turnGate` single-flight
|
||||
pattern, or assert via a shared counter in a stubbed `chatWith`).
|
||||
- Existing `cmd/nomos` suite stays green; `go vet` clean.
|
||||
|
||||
---
|
||||
|
||||
## P1 — F1 can false-auto-close a merely-busy session (low risk, fix for robustness)
|
||||
|
||||
**Bug.** `processIdleSweep` (`continue.go:78-89`) bumps `completion_nudges`
|
||||
**before** calling `resumeSession`. If `resumeSession` skips (busy), the nudge is
|
||||
counted as unanswered; the next sweep sees `CompletionNudges >= 1` and
|
||||
**auto-closes** a session that was just busy.
|
||||
|
||||
**Likelihood is low** because `staleGoalSessions` (`store.go:1336`) filters
|
||||
`last_active_at < now() - threshold` and an active turn keeps updating
|
||||
`last_active_at` — so a busy session shouldn't appear stale. But the coupling is
|
||||
the same shape as P0 and worth closing.
|
||||
|
||||
**Fix.** Gate the bump on the run, mirroring P0:
|
||||
```go
|
||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||
note := …
|
||||
if a.resumeSession(ctx, s.ID, note) {
|
||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil { … }
|
||||
}
|
||||
})
|
||||
```
|
||||
(If skipped, leave `completion_nudges` at 0 so a genuinely-stale sweep nudges
|
||||
again later.)
|
||||
|
||||
---
|
||||
|
||||
## P2 — Minor / hygiene (optional, can ship without)
|
||||
|
||||
- **Redundant catch-up turn on reconnect.** When the live turn *already ended*
|
||||
before a dropped-SSE reconnect fires, the empty-message path still runs a
|
||||
"report your state" `resumeSession` turn the operator didn't ask for. F1 makes
|
||||
it non-concurrent (good) but it's still a spare turn. Consider: in
|
||||
`handleChat`'s empty-message branch, skip the `resumeSession` if the session
|
||||
is already terminal (`done`/`failed`/`abandoned`) or had activity within the
|
||||
last few seconds — just return 202 and let the poller catch up.
|
||||
- **Top-level side-effect on import.** `chat.ts` now calls `subscribeEvents()` +
|
||||
`liveEvents.subscribe(...)` at module top level. It works (and `vitest` stays
|
||||
green because tests mock `./chat`), but a hidden SSE-connect-on-import is
|
||||
fragile for future tests. Prefer a lazy `ensureChatEventSync()` called from
|
||||
the window mount path, matching how `workspace.ts` subscribes inside
|
||||
`startWorkspace` rather than at import.
|
||||
- **F7 follow-up (already documented):** the `NewTaskChat → SessionChatWindow`
|
||||
window-swap on first send still flashes; an in-place handoff would remove it.
|
||||
- **Pre-existing, not introduced:** `a.chat` retries the LLM stream on
|
||||
`ctx`-cancellation (client disconnect) up to 3×, holding the turn permit a few
|
||||
extra seconds. Out of scope here.
|
||||
|
||||
---
|
||||
|
||||
## Out of scope
|
||||
- F8 (ordering toggle + live background tool-delta streaming) — deferred in the
|
||||
original plan; its main symptom is removed by F1.
|
||||
- `run` execution deep-links (need an execution-view opener).
|
||||
|
||||
## Recommended order
|
||||
1. **P0** (lost continuations) — blocks shipping F1.
|
||||
2. **P1** (idle-sweep nudge gate) — small, same pattern.
|
||||
3. P2 items as time allows.
|
||||
4. Re-run `go test ./cmd/nomos/`, `go vet`, web `vitest`, `vite build`; keep
|
||||
`VERSION` at `0.15.0` (these are correctness fixes to the same changeset, not
|
||||
a new bump) — or bump patch to `0.15.1` if shipped as a follow-up commit.
|
||||
|
||||
---
|
||||
|
||||
## Resolution
|
||||
|
||||
All review items implemented. The whole batch (F1–F7 + these review fixes)
|
||||
remains one uncommitted changeset at `VERSION 0.15.0`.
|
||||
|
||||
| Item | Fix | Where |
|
||||
|---|---|---|
|
||||
| **P0** | `resumeSession` returns `bool` (false on busy-skip). `continueSession` marks an execution `continued` **only after** the turn ran; on a skip it defers and the next worker tick retries (item stays pending). Removed the pre-dispatch `markContinued` in `processContinuations`. Other callers (`/resume`, answer-question, reconnect) ignore the return. | `cmd/nomos/continue.go` |
|
||||
| **P0 test** | `TestResumeSession_SkipsWhenBusy`, `TestContinueSession_DefersWhenBusy` — DB-free contract tests proving the skip path returns false without running the body (nil provider would panic otherwise). | `cmd/nomos/continue_test.go` |
|
||||
| **P1** | Idle sweep bumps `completion_nudges` only after `resumeSession` actually runs, so a busy-skip can't be counted as an unanswered nudge → no false auto-close. | `cmd/nomos/continue.go` (`processIdleSweep`) |
|
||||
| **P2.1** | Empty-message reconnect (now defensive — the frontend no longer POSTs empty messages post-F2) skips a terminal session instead of spawning a spare "report state" turn. | `cmd/nomos/main.go` (`handleChat`) |
|
||||
| **P2.2** | Event subscription armed lazily from `chatFor()` (`ensureChatEventSync`) instead of at module import — no SSE-connect-on-import side-effect. | `web/src/lib/stores/chat.ts` |
|
||||
|
||||
**Verification:** `go test -count=1 ./cmd/nomos/` green (incl. the two new
|
||||
contract tests); `go vet` clean. Web `vitest` 70/70; `vite build` succeeds; no
|
||||
new `tsc`/eslint errors in any touched file.
|
||||
|
||||
**Note on the P0 end-to-end test:** the full "two continuations both processed,
|
||||
no overlap" scenario needs a live LLM provider (chatWith isn't stubbable without
|
||||
a refactor) and was therefore covered at the contract level (the skip returns
|
||||
false without running the body) plus the existing `turnGate` single-flight test
|
||||
for serialization, rather than as a DB integration test.
|
||||
356
plans/done/2026-08-03-nomos-chat-reliability-and-ux-audit.md
Normal file
356
plans/done/2026-08-03-nomos-chat-reliability-and-ux-audit.md
Normal file
@@ -0,0 +1,356 @@
|
||||
# 2026-08-03 — Nomos chat: reliability & predictability audit
|
||||
|
||||
**Status:** Implemented (F1–F7) in v0.15.0; F8 deferred. See
|
||||
[Resolution](#resolution-2026-08-03) at the end.
|
||||
|
||||
**Scope:** The live chat/task UX across one production session, audited through
|
||||
the code paths behind each operator-reported symptom —
|
||||
`cmd/nomos/{main.go,agent.go,continue.go,store.go}`,
|
||||
`web/src/lib/stores/{chat,activity,execstream,events,workspace}.ts`,
|
||||
`web/src/lib/components/{ChatThread,AgentTrace,ToolCallCard,UnifiedTimeline,TaskContextPanel,SessionChatWindow}.svelte`.
|
||||
**Trigger:** Operator report — streaming invisible in the tool card; the
|
||||
activity/plan panel wrong about parallel/nested runs and timestamps with no clear
|
||||
sequence; no links to artifacts/knowledge referenced in chat; agent "thinking"
|
||||
flickers/overwrites itself; layout jumps when a chat goes from empty to content;
|
||||
"Agent connection lost / Error in input stream" messages that aren't actionable
|
||||
and don't self-resolve; overall flaky/disconnected feel where the task never
|
||||
cleanly ended.
|
||||
|
||||
The prior round (`2026-07-30-session-review-plan-drift-and-dead-activity-panel.md`,
|
||||
shipped in `467589d`) fixed the plan-seq and fabricated-timestamp rendering bugs.
|
||||
This round's symptoms are a different layer: **turn orchestration, streaming
|
||||
wiring, and connection-state UX**. One architectural gap (F1) is the common
|
||||
cause behind several of them.
|
||||
|
||||
---
|
||||
|
||||
## The one root cause that compounds everything: F1
|
||||
|
||||
### F1 — No per-session turn serialization (concurrent turns corrupt the view)
|
||||
|
||||
`handleChat` runs `a.chat(ctx, ...)` directly in the HTTP request goroutine, and
|
||||
every "resume" path (`resumeSession`, the reconnect empty-message path, the
|
||||
auto-continuation worker, the idle sweep, answer-question) launches **another
|
||||
goroutine** (`safego.Go`) running a full turn. There is **no mutex keyed on
|
||||
`sessionID`** anywhere. The codebase already knows this is a hazard —
|
||||
`agent.go:316-323` marks approved executions `continued` specifically because
|
||||
"two concurrent LLM calls for the same session cause empty responses and race
|
||||
conditions" — but the fix is per-path patching, not a general lock.
|
||||
|
||||
What this produces, deterministically:
|
||||
|
||||
- A network blip on the browser↔nomos stream fires `handleDisconnect`
|
||||
(`chat.ts:383`), which POSTs an **empty-message reconnect** →
|
||||
`main.go:194-206` spawns `resumeSession` as a **new goroutine**. If the
|
||||
original turn is still alive (or finishes its current tool call), **two turns
|
||||
now run for one session**: interleaved `tool_use`/`text_delta` events, a
|
||||
re-proposed plan, and "the agent is repeating itself."
|
||||
- The activity timeline (`activity.ts:119-185`) groups tools under a plan step
|
||||
by *inferring* `currentStepSeq` from `update_plan_step` calls in the message
|
||||
stream. Two interleaved turns make that inference wrong → tools land under the
|
||||
wrong step, steps appear to nest/parallelize that never did, the sequence
|
||||
reads as garbage. This is the "parallel runs / nesting / no clear sequence"
|
||||
report.
|
||||
- Two turns appending to the same session's messages is also the source of the
|
||||
duplicate-tool-call/empty-response class of bugs the prior plan docs keep
|
||||
patching individually.
|
||||
|
||||
**This is why the experience "felt flaky and disconnected" and "the task didn't
|
||||
end":** the panel is faithfully rendering a corrupted, interleaved event stream.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **One in-flight turn per session, server-side.** Add a per-`sessionID`
|
||||
turn mutex (a `sync.Map[string]*singleflight` or a keyed `sync.Mutex`) in
|
||||
`handleChat`/`resumeSession`/`continue.go`. A second attempt to start a turn
|
||||
for a session that already has one running must **queue** (preferred — the
|
||||
operator's message waits its turn) or **return 409 "turn in progress"** (the
|
||||
frontend then just re-polls; no new goroutine). This single change removes
|
||||
the interleaving that drives F2/F3/F8.
|
||||
2. **Make the empty-message reconnect a no-op when a turn is already running.**
|
||||
Today it *always* spawns `resumeSession`. Gate it on "is any turn active for
|
||||
this session?" — if yes, return 202 and let the existing turn + the poller do
|
||||
the work. A blip should never *create* work.
|
||||
|
||||
---
|
||||
|
||||
## F2 — Reconnect spawns a new turn and surfaces raw, non-actionable errors
|
||||
|
||||
`chat.ts:383-426` `handleDisconnect`: on a dropped SSE it sets
|
||||
`connectionState='disconnected'`, starts the 3s poller, shows
|
||||
`"Agent connection lost. The task is still running — retrying…"`, then calls
|
||||
`streamChat('', sid, …)` up to 3× — each of which is the empty-message POST that
|
||||
triggers F1's new `resumeSession` goroutine. Separately, the LLM stream errors
|
||||
surface verbatim: `agent.go:388` does `emitError("llm: %v", err)`, so an
|
||||
OpenRouter transport break reaches the operator as `llm: error in input stream:
|
||||
…` (the openai-go SDK's SSE-reader text), shown raw in `ChatThread`'s error bar.
|
||||
|
||||
Combined with F1, this is the exact "messages not actionable and not
|
||||
self-resolving" + "task didn't end" experience: a blip both invents a duplicate
|
||||
turn and paints a scary, unfixable error that lingers.
|
||||
|
||||
Secondary defects in the same path:
|
||||
|
||||
- `streaming` stays `true` for the entire reconnect window, so the composer is
|
||||
disabled and the poller's `if (streaming && connected) return` guard
|
||||
(`chat.ts:177`) suppresses updates except while disconnected — fragile.
|
||||
- The **per-window** error path (`sendSessionMessage`, `startTask`) does **not**
|
||||
auto-reconnect at all — it only polls. Its `onReconnect` in
|
||||
`SessionChatWindow.svelte:110` is `() => loadSessionChat(sessionId)`, which
|
||||
just *re-fetches the transcript* and never re-attaches to a live stream. And
|
||||
the global `reconnect()` (`chat.ts:428`) keys off the **global**
|
||||
`currentSession`, so a floating window's Reconnect button can target the wrong
|
||||
session. Two different, both-broken reconnect behaviors.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Stop the empty-message-reconnect from creating turns** (depends on F1.2).
|
||||
Reconnect should mean "catch up," not "run more."
|
||||
2. **Humanize + bucket error strings.** Map known transport errors to
|
||||
operator-readable, actionable copy with a single primary action:
|
||||
- `llm: …input stream…` / 502/503/timeout → "The model connection dropped.
|
||||
The task is still running in the background — it'll catch up
|
||||
automatically." (auto-dismiss when the next event/poll lands)
|
||||
- `HTTP 401/403` → "Session expired — reconnect." (action: re-auth)
|
||||
- unknown → show the raw text but behind a "Details" toggle, not as the
|
||||
headline.
|
||||
3. **Make errors self-resolving.** Clear the error + connection-lost banner the
|
||||
moment the poller sees a newer message or any live event for the session
|
||||
arrives (wire `eventsConnected` / a session-scoped event into the banner's
|
||||
visibility). Today the banner stays until manual dismiss even after recovery.
|
||||
4. **Unify reconnect.** One `reconnect(sessionId)` that (a) re-fetches the
|
||||
transcript, (b) if no turn is active, is a pure no-op refresh; used by both
|
||||
the main view and windows. Drop the global-`currentSession` coupling.
|
||||
|
||||
---
|
||||
|
||||
## F3 — The UI can't tell when a turn truly ended (so it never looks "done")
|
||||
|
||||
When the SSE stream ends without a `done` event, `streamChat`'s `onDone`
|
||||
(`chat.ts:355-368`) calls `handleDisconnect`. Even if the backend turn then
|
||||
finishes and persists its final message, the frontend only learns via the 3s
|
||||
poller re-setting `messages` — but nothing transitions `streaming`→`false` or
|
||||
`connectionState`→`connected` from that path, so the spinner/indicator and the
|
||||
"connection lost" banner can persist indefinitely. That is "the task didn't
|
||||
end / backend connection was lost."
|
||||
|
||||
The backend does emit a terminal signal — `task.status` events on
|
||||
`complete_task`/auto-complete (`workspace.ts:82-88` `STATUS_AFFECTING`) — but
|
||||
nothing in the chat store reacts to a terminal `task.status` to force
|
||||
`streaming=false` + clear the banner. The signal exists; the chat ignores it.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Treat a terminal `task.status` (done/failed) for the viewed session as
|
||||
authoritative end-of-turn** in `chat.ts`: set `streaming=false`,
|
||||
`connectionState='connected'`, dismiss any connection-lost error. The poller
|
||||
already refreshes messages; this just closes the loop on the *state* flags.
|
||||
2. **Add a `task.completed` / `turn.ended` SSE event** from the backend on every
|
||||
terminal path (today `done` is a chat-stream-only event; background turns
|
||||
have no equivalent). The always-on events stream already reaches the panel —
|
||||
route the same signal to the chat store so background-completed turns clear
|
||||
the UI without waiting on a poll.
|
||||
|
||||
---
|
||||
|
||||
## F4 — Command streaming isn't shown where the operator looks
|
||||
|
||||
Streaming **exists** (`execstream.ts` `liveExecutionOutputFor`, fed by
|
||||
`fetchExecutionLogs` via the always-on events stream) and the
|
||||
`UnifiedTimeline` **does** render `tool.liveOutput` with tail-pinned scroll
|
||||
(`UnifiedTimeline.svelte:451-457`). But:
|
||||
|
||||
- The **global** `activityLog` (`activity.ts:236`) — used by the main Chat page's
|
||||
panel — never calls `withLiveOutput`. Only the **per-window**
|
||||
`activityLogFor(sessionId)` (`activity.ts:271`) attaches live output. So the
|
||||
main chat view's timeline shows no streaming at all.
|
||||
- The **inline chat tool cards** — `ToolCallCard.svelte` (rendered inside
|
||||
`AgentTrace.svelte`) — show only args/result/error. They never read
|
||||
`liveOutput`. Expanding a running `run` call in the transcript (the natural
|
||||
place to "check the tool") shows nothing live; output appears all at once when
|
||||
the `tool_result` lands.
|
||||
|
||||
This is the report: "I expected checking on the tool to let me see the
|
||||
streaming."
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Wire live output into the global `activityLog`** so the main chat panel
|
||||
streams too (call `withLiveOutput` in the `activityLog` derivation, same as
|
||||
`activityLogFor`).
|
||||
2. **Show streaming in the inline tool card.** Pass the session's live-output
|
||||
store into `AgentTrace`/`ToolCallCard` (or attach `liveOutput` to the running
|
||||
`run` tool entry the way the timeline does) and render a tail-pinned `<pre>`
|
||||
while the call is `tool_use`/running. Reuse the UnifiedTimeline's scroll-pin
|
||||
pattern. Gated runs (queued-for-approval) should instead show a "queued —
|
||||
watch in entity detail" affordance (per `execstream.ts` header comment).
|
||||
|
||||
---
|
||||
|
||||
## F5 — Artifacts and knowledge referenced in chat aren't navigable
|
||||
|
||||
When the agent records knowledge, the activity panel shows `Recorded: <title>`
|
||||
(`activity.ts:188-203`) but it's plain text — no link. The backend already
|
||||
emits `knowledge.recorded` and links the note to the task
|
||||
(`store.go:1572 linkKnowledgeToTask`, `agent.go:594`), and the Wiki reader
|
||||
exists (`web/src/lib/components/knowledge/WikiReader.svelte`). Nothing connects
|
||||
them. Same for `get_entity`/`run` results: slugs and execution ids appear in
|
||||
tool output but aren't clickable to open the entity window or execution view.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Make activity/tool entries link-bearing.** Add an optional
|
||||
`link?: { kind: 'knowledge'|'entity'|'execution', id: string }` to
|
||||
`ActivityEntry`. Populate it from `upsert_knowledge` (title→knowledge id from
|
||||
the result), `get_entity` (slug), and `run` (execution id). Render a
|
||||
clickable chip that opens the right surface: knowledge → Wiki reader (new tab
|
||||
/ window), entity → entity detail window, execution → execution log pane
|
||||
(already fetched by `EntityDetailContent.svelte`).
|
||||
2. **Render entity/knowledge mentions in assistant markdown as links** when they
|
||||
resolve to known slugs (lightweight: a post-process pass on rendered text, or
|
||||
let the model emit explicit `[slug](entity:…)` markers it already has tools to
|
||||
discover).
|
||||
|
||||
---
|
||||
|
||||
## F6 — "Thinking" is an unstable single-line headline, not a predictable trace
|
||||
|
||||
`ChatThread`'s `indicatorLabel` (`ChatThread.svelte:83-89`) returns the **first**
|
||||
running activity entry's description; `AgentTrace`'s `headline` mirrors it. As
|
||||
tools fire sequentially the running entry changes, so the one line rewrites
|
||||
itself every call — "the thinking overwrites itself." There is no persistent,
|
||||
additive reasoning surface, and no predictable turn structure (plan → steps →
|
||||
answer) the operator can learn to read. Claude-Code-style predictability is
|
||||
absent.
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **A stable, additive per-turn reasoning block.** Keep the collapsed trace as
|
||||
a *summary* ("Step 2 of 4 · running `run`"), but when expanded show an
|
||||
**append-only** log of (a) the model's intermediate `text` (reasoning before
|
||||
each tool call — already emitted at `agent.go:458-460` and persisted) and
|
||||
(b) each tool call as a fixed row, instead of a single mutating headline.
|
||||
2. **Predictable turn shape.** Enforce/cue a consistent sequence in the UI —
|
||||
Goal → Plan → Steps (each with its tools nested) → Final answer — and render
|
||||
each phase as a stable section that fills in rather than a line that
|
||||
overwrites. The UnifiedTimeline already models most of this; surface the same
|
||||
model in the inline trace so chat and panel tell one story.
|
||||
|
||||
---
|
||||
|
||||
## F7 — Layout jumps when a chat goes from empty to content
|
||||
|
||||
`SessionChatWindow.svelte:58-63` gates the right rail on `hasContext`: empty
|
||||
task → `ChatThread` full-width; first activity/touched entity → switches to
|
||||
`Splitpanes` with the `TaskContextPanel` rail. The swap is instant and
|
||||
**reflows the chat column width** the moment the first event lands — "switching
|
||||
from empty to chat with something, the layout was off." Compounded by the
|
||||
`NewTaskChat` → real `SessionChatWindow` window-swap on first send
|
||||
(`NewTaskChat.svelte:17-22`).
|
||||
|
||||
### Fix (proposed)
|
||||
|
||||
1. **Reserve the rail's space from the start** (collapse to a thin sliver / icon
|
||||
rail when empty) instead of mounting it on demand, so adding content doesn't
|
||||
change the chat column width. Or animate the rail in.
|
||||
2. **Avoid the window swap on first send** — let the new-task window *become* the
|
||||
session window in place once the id is assigned (same component, swap the
|
||||
store source) rather than close+open.
|
||||
|
||||
---
|
||||
|
||||
## F8 — Activity/plan ordering & parallelism *(largely a symptom of F1)*
|
||||
|
||||
With F1 fixed (no interleaved turns) the heuristic step-grouping in
|
||||
`activity.ts` becomes reliable again. Remaining standalone items:
|
||||
|
||||
- The timeline is **newest-first** with ts-0 goal/pending parked at the bottom
|
||||
(`UnifiedTimeline.svelte:119-127`); for a long task this can read as
|
||||
"sequence is off." Consider an explicit **oldest-first / seq-ordered** mode
|
||||
toggle, and always show the step number prominently so order is unambiguous
|
||||
regardless of sort.
|
||||
- Background/auto-continued turns still rely on the 3s poller for their result
|
||||
to appear; until F3's terminal event lands, the panel can lag. The
|
||||
always-on events stream already carries `plan.*` and `entity.touched` live —
|
||||
extend it to carry per-tool `tool.*` deltas for background turns so the panel
|
||||
is live, not polled, during autonomous work.
|
||||
|
||||
---
|
||||
|
||||
## Recommended sequence
|
||||
|
||||
| Order | Item | Why first |
|
||||
|---|---|---|
|
||||
| 1 | **F1** per-session turn mutex + no-op reconnect-when-busy | Removes the interleaving that is the root cause of F2/F3/F8 symptoms; everything else is cosmetics on top of a corrupted stream. |
|
||||
| 2 | **F3** terminal-event → clear chat state | Once turns can't double, make "the task ended" unambiguous so the UI stops lingering. |
|
||||
| 3 | **F2** humanized/self-resolving errors + unified reconnect | Turns the scary, sticky "connection lost / input stream" into recoverable, auto-clearing UX. |
|
||||
| 4 | **F4** streaming in the global log + inline tool card | Highest-visibility "I can't see what it's doing" fix; small, isolated change. |
|
||||
| 5 | **F6** stable additive reasoning trace | Predictability of the interaction model (the Claude-Code feel). |
|
||||
| 6 | **F5** artifact/knowledge deep links | Navigation completeness. |
|
||||
| 7 | **F7** layout stability | Polish. |
|
||||
| 8 | **F8** ordering mode + live background deltas | Polish, partly free after F1. |
|
||||
|
||||
## Verification hooks (when implementing)
|
||||
|
||||
- `cmd/nomos`: a test that starts two turns for the same session and asserts the
|
||||
second queues/is-rejected (no interleaved `tool_use` order in persisted
|
||||
messages).
|
||||
- `web/src/lib/stores`: extend `activity.test.ts`/`execstream.test.ts` — global
|
||||
`activityLog` now carries `liveOutput`; tool-card live output renders while
|
||||
`tool_use` and clears on `tool_result`.
|
||||
- A reconnect/integration test: drop the SSE mid-turn, assert (a) no duplicate
|
||||
`resumeSession` goroutine, (b) banner auto-clears on next event, (c)
|
||||
`streaming` returns to false on terminal `task.status`.
|
||||
|
||||
---
|
||||
|
||||
## Note on method
|
||||
|
||||
This audit was done against the **code paths** behind the reported symptoms, not
|
||||
a single session transcript (no MCP/DB access from this session). To tie a
|
||||
specific finding to a specific past session, pull the session via
|
||||
`docker exec oikos-postgres-1 psql -U oikos oikos -c "select id,goal,outcome
|
||||
from agent_sessions order by last_active_at desc limit 5"` and cross-reference
|
||||
its `agent_activity` rows / persisted messages against the F1 interleaving
|
||||
signature (two assistant turns' tool ids interleaved in one message shell).
|
||||
|
||||
---
|
||||
|
||||
## Resolution (2026-08-03)
|
||||
|
||||
Implemented F1–F7 in v0.15.0 (`VERSION 0.14.2 → 0.15.0`). F8 deferred (its
|
||||
primary symptom — interleaved/out-of-order entries — is removed by F1; the
|
||||
ordering toggle and live background tool-delta streaming remain as nice-to-
|
||||
haves).
|
||||
|
||||
| Item | What shipped | Where |
|
||||
|---|---|---|
|
||||
| **F1** | Per-session single-flight turn gate (`turnGate`): at most one in-flight turn per session. Background resume paths (`resumeSession` — covers the continuation worker, idle sweep, answer-question, /resume, and the empty-message reconnect) skip non-blocking when busy; the live chat path waits briefly then bails with an actionable error instead of stacking a second turn. | `cmd/nomos/turngate.go` (+`turngate_test.go`), wired in `agent.go` (struct/init), `continue.go` (`resumeSession`), `main.go` (`handleChat`). |
|
||||
| **F3** | Terminal `task.status` events (done/failed/abandoned/awaiting_input) now clear a stuck chat view's `streaming`/`connectionState` and dismiss the connection-lost toasts — the authoritative "turn ended" signal the UI was ignoring. Poller safety net catches the edge where the event fired during the disconnect window. | `web/src/lib/stores/chat.ts` (`clearTurnState`, liveEvents subscription, `startSessionPolling`). |
|
||||
| **F2** | Raw errors humanized ("The model connection dropped. The task keeps running…") and bucketed; one connection surface per drop (not banner+toast+raw error); errors self-clear via F3. The turn-spawning reconnect attempt loop is gone (dead global path simplified to a turn-free refresh); window "Reconnect" re-fetches + resets state. | `web/src/lib/stores/chat.ts` (`humanizeChatError`, error handlers, `loadSessionChat`, `handleDisconnect`/`reconnect`), `web/src/lib/components/ChatThread.svelte` (banner copy). |
|
||||
| **F4** | Command streaming now shows (a) in the **global** activity timeline (live output wired into `activityLog`, was only per-window) and (b) in the **inline chat tool card** — expanding a running `run` shows live output auto-opened and tail-pinned. | `web/src/lib/types.ts` (`liveOutput`), `web/src/lib/stores/activity.ts` (`currentLiveOutput`), `web/src/lib/components/ChatThread.svelte` (`toolsWithLive`), `web/src/lib/components/ToolCallCard.svelte`. |
|
||||
| **F6** | The "thinking" headline is now step-first (stable across a step's many tool calls) instead of rewriting per command; falls back to the current tool / "thinking…" only when no step is active. | `web/src/lib/components/ChatThread.svelte` (`indicatorLabel`). |
|
||||
| **F5** | Activity entries now carry a deep link: recorded knowledge docs and `get_entity` lookups get an "open artifact" chip that opens the entity/knowledge window directly. | `web/src/lib/stores/activity.ts` (`link`, `knowledgeLinkFromResult`, `entityLinkFromArgs`), `web/src/lib/components/UnifiedTimeline.svelte`. |
|
||||
| **F7** | The empty→content layout reflow is gone: `SessionChatWindow` now has one stable `Splitpanes`+`ChatThread` from open (no more destroy/remount of the thread or column reflow when the rail appears). | `web/src/lib/components/SessionChatWindow.svelte`. |
|
||||
|
||||
**Verification:**
|
||||
- `go test ./cmd/nomos/` green (incl. new `turngate_test.go`: non-blocking skip,
|
||||
blocking-waits-for-release, timeout, and a 50-goroutine single-flight
|
||||
concurrency test asserting max in-flight = 1). `go vet` clean.
|
||||
- Web `vitest` 70/70 green (incl. `activity.test.ts`/`execstream.test.ts`); the
|
||||
`activity.test.ts` chat mock gained `currentSession` for the new
|
||||
`currentLiveOutput` derivation.
|
||||
- `vite build` succeeds (all Svelte components compile). Pre-existing `tsc`
|
||||
strictness errors in unrelated files (`ui/*`, `oidc.ts`, `windows.ts`,
|
||||
`workspace.ts`) are unchanged; no new errors in any touched file.
|
||||
|
||||
**Follow-ups (not in this pass):**
|
||||
- F8: oldest-first ordering toggle; emit per-tool `tool.*` events on the
|
||||
always-on stream during background `resumeSession` turns so the panel is live
|
||||
(not 3s-polled) during autonomous work.
|
||||
- F5: `run` execution deep-links (open the entity detail's execution pane) —
|
||||
needs an execution-view opener; knowledge/entity links shipped first as the
|
||||
explicit complaint.
|
||||
- F7: the `NewTaskChat → SessionChatWindow` window-swap on first send (a
|
||||
windows.ts open/close) still causes a brief flash; an in-place handoff
|
||||
(same window, swap store source) would remove it.
|
||||
184
plans/done/2026-08-03-nomos-chat-working-visibility.md
Normal file
184
plans/done/2026-08-03-nomos-chat-working-visibility.md
Normal file
@@ -0,0 +1,184 @@
|
||||
# 2026-08-03 — Nomos chat: working-visibility, message queue, generation-aware timeline
|
||||
|
||||
**Status:** Implemented (F1–F4) in v0.17.0. See
|
||||
[Resolution](#resolution-2026-08-03) at the end.
|
||||
|
||||
## Context (grounded in last-session logs + DB, not just code)
|
||||
|
||||
Operator report: *"On the chat window I can't tell the agent is working; it's
|
||||
making tool calls but no feedback. Typing returns 'Nomos is still finishing a
|
||||
previous step…'. Activity not up to date. Several plans at once, some don't
|
||||
execute."*
|
||||
|
||||
Verified against runtime state:
|
||||
|
||||
- **Last session `23da10db`** ran ONE live turn for **6m33s** (21 iterations,
|
||||
19:45:36→19:52:03, correlation `679566cb`). At 19:48:00 the operator typed
|
||||
`status`; at **19:48:05 the turn gate deferred it** (`turn already active,
|
||||
deferring operator message`). The operator could type at all only because the
|
||||
client had already lost the stream (`streaming=false`) while the server kept
|
||||
running — i.e. the client showed an *idle* window over a *working* turn. It
|
||||
ended `awaiting_input`.
|
||||
- **Turn runtimes are long**: sessions in the DB run 15-27 min
|
||||
(e.g. `44df8802` 24:24, `4319b9f8` 27:05). `handleChat` (main.go:170) has **no
|
||||
SSE keepalive**; inter-iteration gaps reach 20-40s, so a proxy/browser idle
|
||||
close mid-turn resets `streaming` while the turn continues on
|
||||
`context.Background()` (pctx).
|
||||
- **Re-proposing is real**: `44df8802` has **generation 1 (5 steps, all
|
||||
`replaced`) → generation 2 (25 steps, done)**, with **2 `propose_plan` + 52
|
||||
`update_plan_step`** calls persisted. The activity timeline renders every one
|
||||
of those across both generations.
|
||||
|
||||
## Root causes
|
||||
|
||||
- **G1 — "working" == `streaming`.** Every working-indication in the chat window
|
||||
(AgentTrace running status, indicator headline, stream cursor, panel spinner,
|
||||
`disabled={streaming}` input) is gated on the live SSE flag. A background turn
|
||||
(`resumeSession`/continuation worker) has no stream; a desynced long live turn
|
||||
has a dead stream. In both cases `streaming=false` while the server is actively
|
||||
working. The **session `status`** (`planning`/`executing`/`awaiting_input`) is
|
||||
the reliable "server is running a turn" signal and is already live-refreshed
|
||||
(`workspace.ts` `taskFor`, `STATUS_AFFECTING`), but the chat UI never uses it.
|
||||
- **G2 — busy-turn message is rejected, not queued.** main.go:292-302: the turn
|
||||
gate waits 5s then emits the "still finishing a previous step" error and
|
||||
returns. The user message *is* persisted (main.go:270) but is **inert** — the
|
||||
user must manually re-send.
|
||||
- **G3 — activity is poll/event laggy.** Tool-level activity derives from
|
||||
`messages`, refreshed only by the 3s poller; plan steps are **events-only**
|
||||
(`workspace.ts` `hydrateSession`) with no poll, so a missed `plan.proposed`
|
||||
event leaves the panel stuck on a stale generation.
|
||||
- **G4 — timeline is generation-unaware.** `activity.ts` `computeActivityLog`
|
||||
walks **all** messages' tool calls, so a re-proposed task renders N
|
||||
"Proposed plan" entries and attributes tools to steps via `currentStepSeq`
|
||||
inferred from `update_plan_step` calls across **every** generation — tools land
|
||||
under the wrong (current-gen) step or under steps that were `replaced`. This is
|
||||
the "several plans / some steps never run" view.
|
||||
|
||||
## Fixes (ordered)
|
||||
|
||||
### F1 — Status-driven `working` signal (fixes G1)
|
||||
Add a derived store `taskWorking(sessionId)` = `$streaming OR status ∈
|
||||
{planning, executing}` (explicitly **not** `awaiting_input` — that is paused for
|
||||
input), plus a global `currentWorking` for the main view backed by `currentTask`.
|
||||
Use it wherever `streaming` currently drives "is it working":
|
||||
- `ChatThread.svelte`: `traceStatus` last-message = `working ? 'running' : …`;
|
||||
`indicatorLabel` and the AgentTrace `status`/`label` props.
|
||||
- `TaskContextPanel.svelte:137` spinner and `UnifiedTimeline` `streaming` prop →
|
||||
`working`.
|
||||
- Keep a separate `streaming` for the literal "live text deltas are arriving"
|
||||
cursor; `working` is the superset for indicators/input.
|
||||
- Input stays **enabled** while `working` (the user must be able to interject);
|
||||
the send path queues when busy (F2). Show a muted "Nomos is working…" hint in
|
||||
the composer when `working && !streaming`.
|
||||
|
||||
### F2 — Queue operator messages; auto-run when free (fixes G2)
|
||||
- Server: in-memory per-session FIFO on the `agent` struct (mirrors `turnGate`),
|
||||
`{message, reply}` entries. `handleChat`: when the gate is busy, **enqueue**
|
||||
instead of rejecting, and emit a `queued` SSE event (replaces today's error at
|
||||
main.go:294-302). Persist the user message as today (already done pre-acquire).
|
||||
- Drain: arm a per-session drainer that, on gate release, acquires again and runs
|
||||
the next queued message as a normal turn (same persist/emit path as
|
||||
`handleChat`). Strictly one-at-a-time under the gate — this cannot stack turns
|
||||
(the hazard v0.15.0 F1 removed); background `resumeSession` keeps its
|
||||
non-blocking skip and never touches the queue.
|
||||
- If the session is terminal (`done`/`failed`) or `awaiting_input` when a queued
|
||||
message runs, `reopenSession`/answer handling applies as for any follow-up.
|
||||
- Frontend: on the `queued` event show an inline "Queued — will run when the
|
||||
current step finishes" chip on that user bubble; clear it when the turn's real
|
||||
events begin. Drop the humanized "still finishing" error for the busy case.
|
||||
|
||||
### F3 — SSE keepalive on `handleChat` (prevents the G1 desync at the source)
|
||||
Wrap `a.chat(...)` in a goroutine + `select` with a **10-15s ticker** that writes
|
||||
an SSE comment (`:keepalive\n\n`) and flushes, so 20-40s inter-iteration gaps no
|
||||
longer trip proxy/browser idle timeouts. Stop the ticker when `a.chat` returns.
|
||||
(EventSource ignores comment lines by spec — safe.)
|
||||
|
||||
### F4 — Generation-aware timeline + self-healing plan panel (fixes G3/G4)
|
||||
- `activity.ts` `computeActivityLog`: find the **last** `propose_plan` in the
|
||||
message stream; ignore `propose_plan`/`update_plan_step` calls **before** it
|
||||
for both rendering and `currentStepSeq` inference. Render at most one
|
||||
"Proposed plan" entry (the current generation). Steps continue to come from
|
||||
`$steps` (already current-gen via `fetchPlan` MAX(generation)). Optionally emit
|
||||
a single "Plan revised" entry when >1 generation exists.
|
||||
- Plan-panel resilience: on any `STATUS_AFFECTING` event (and on reconnect),
|
||||
re-fetch the plan (`fetchPlan`) in addition to the live `plan.proposed` handler,
|
||||
so a missed event self-heals instead of leaving a stale generation.
|
||||
|
||||
## Validation
|
||||
|
||||
- `go test ./cmd/nomos/`: extend `turngate_test.go`/new `messagequeue_test.go` —
|
||||
queued message runs strictly after release; FIFO order preserved across 3
|
||||
queued sends; a background `resumeSession` busy-skip does **not** consume or
|
||||
starve the queue; queued message runs even if session went `awaiting_input`.
|
||||
- Web `vitest`: `activity.test.ts` — add a 2-generation fixture (2× propose_plan,
|
||||
interleaved update_plan_step) asserting exactly one "Proposed plan" and correct
|
||||
step attribution to gen-2 steps; `chat`/store test — `working` is true from
|
||||
`status==='executing'` even with `streaming=false`; `queued` event renders the
|
||||
queued chip and clears on first tool_use.
|
||||
- Manual: (a) start a long task, **reload the window mid-turn** → the working
|
||||
indicator stays on (status-driven); (b) send a message mid-turn → "Queued" →
|
||||
runs after the turn; (c) open `44df8802`-style 2-gen session → timeline shows
|
||||
one plan, no ghost proposals.
|
||||
|
||||
## Risks
|
||||
|
||||
- **F2 must not reintroduce concurrent turns.** The queue drains one-at-a-time
|
||||
under the gate; background resume remains non-blocking and queue-agnostic.
|
||||
Existing `turngate_test.go` concurrency assertion (max in-flight = 1) must stay
|
||||
green.
|
||||
- **Status-driven `working` could stick on** if a terminal event is missed.
|
||||
Mitigated by the existing terminal `task.status` → `clearTurnState` recovery
|
||||
plus a `loadSessions` refresh on reconnect (F4).
|
||||
- **Keepalive comments** must stay SSE comments (`:` prefix) so they aren't
|
||||
parsed as events.
|
||||
|
||||
## Out of scope / follow-ups
|
||||
|
||||
- Model efficiency: the 8+ pure-exploration iterations (repeated
|
||||
`list_entities`/`get_relations`) that inflate turn length to 15-27 min —
|
||||
prompt/iteration-budget tuning, separate effort.
|
||||
- F8 from the prior plan (oldest-first timeline toggle; per-tool `tool.*` events
|
||||
for background turns). F1's status-driven `working` makes background work
|
||||
visible without live per-tool deltas, so this remains lower priority.
|
||||
|
||||
## Open implementation note
|
||||
|
||||
Host the per-session message queue on the `agent` struct (in-memory `map[string]
|
||||
[]queuedMsg` + per-session drainer goroutine), mirroring `turnGate`. No DB table
|
||||
needed — messages are already persisted by `handleChat` before enqueue; the queue
|
||||
only schedules *when* a turn runs, not *whether* the message is stored.
|
||||
|
||||
---
|
||||
|
||||
## Resolution (2026-08-03)
|
||||
|
||||
Implemented F1–F4 in v0.15.1 → v0.17.0 (the intermediate 0.16.0 was the
|
||||
cyberspace-aesthetic commit, landed via auto-pull during this work).
|
||||
|
||||
| Item | What shipped | Where |
|
||||
|---|---|---|
|
||||
| **F1** | Status-driven `working` signal (`taskWorking(sessionId)` / `currentWorking`) = live stream OR session status ∈ {planning, executing}. Drives the chat trace running state, the "thinking" headline, the activity spinner, and the timeline `streaming` prop — so a background/long/desynced turn still looks alive (the "can't tell it's working" symptom). The composer stays enabled during background work so the operator can interject. | `web/src/lib/stores/workspace.ts` (`isWorking`, `taskWorking`, `currentWorking`), `ChatThread.svelte` (`working` prop, `traceStatus`, indicator), `TaskContextPanel.svelte`, `SessionChatWindow.svelte`, `NewTaskChat.svelte`. |
|
||||
| **F2** | Operator messages sent during an in-flight turn are now QUEUED and auto-run when the gate frees, replacing the "still finishing a previous step… send it again" rejection. Per-session in-memory FIFO drained strictly one-at-a-time under the turn gate (no concurrent-turn reintroduction). A `queued` SSE event tells the client, which drops the optimistic bubble and shows a "Queued — will run when it finishes the current step" hint (derived from `working` + last-message shape, so it survives the poller). | `cmd/nomos/messagequeue.go` (+`messagequeue_test.go`), `agent.go` (queue field), `main.go` (`runChatTurn`, `drainQueued`, handleChat queue path), `continue.go` (resumeSession drains on release), `web/src/lib/types.ts` (`ChatQueuedEvent`), `chat.ts` (`queued` handling in sendSessionMessage/startTask). |
|
||||
| **F3** | SSE keepalive: a 12s `:keepalive` comment ticker during `handleChat` so 20-40s inter-iteration gaps no longer trip a proxy/browser idle timeout (the desync root cause). All SSE writes (events + keepalive) serialized through one mutex — `http.ResponseWriter` is not concurrency-safe. | `cmd/nomos/main.go` (`writeMu`/`writeEvent`, keepalive goroutine). |
|
||||
| **F4** | Generation-aware activity timeline: only the LAST `propose_plan` renders as "Proposed plan"; superseded ones collapse to a single "Earlier plan revised" marker, and step-attribution only follows the current generation's `update_plan_step` calls. Plus plan-panel self-heal: the plan is refetched (debounced) on any task-lifecycle event so a missed `plan.proposed` no longer freezes the panel on a stale generation. | `web/src/lib/stores/activity.ts` (`computeActivityLog`), `workspace.ts` (`schedulePlanRefetch`). |
|
||||
|
||||
**Verification:**
|
||||
- `go vet ./cmd/nomos/` clean; `go test ./cmd/nomos/` green, incl. new
|
||||
`messagequeue_test.go` (FIFO, requeueFront, per-session isolation, concurrency,
|
||||
drainQueued no-op-on-empty, drainQueued requeues-when-busy). Existing
|
||||
`turngate_test.go`/`continue_test.go` still green (single-flight guarantee
|
||||
intact).
|
||||
- Web `vitest` 72/72 green (added 2 F4 generation-awareness tests to
|
||||
`activity.test.ts`: one "Proposed plan" + revised marker + current-gen-only
|
||||
step attribution; plan-less Q&A attributes nothing).
|
||||
- `vite build` succeeds. `tsc --noEmit` shows only the pre-existing baseline
|
||||
errors (`ui/*`, `oidc.ts`, `windows.ts`, `workspace.ts:123/201/221`) noted in
|
||||
v0.15.0 — no new errors from this change. ESLint: no new errors (the one new
|
||||
`svelte/valid-compile` on `chatWorking` got the same disable its siblings have).
|
||||
|
||||
**Follow-ups (not in this pass):**
|
||||
- Model efficiency: the long (15-27 min) exploration-heavy turns that made the
|
||||
desync so painful — prompt / iteration-budget tuning, separate effort.
|
||||
- F8 from the prior plan (oldest-first timeline toggle; per-tool `tool.*` events
|
||||
for background turns). F1's status-driven `working` makes background work
|
||||
visible without live per-tool deltas, so this stays lower priority.
|
||||
191
plans/done/2026-08-04-chat-window-overhaul.md
Normal file
191
plans/done/2026-08-04-chat-window-overhaul.md
Normal file
@@ -0,0 +1,191 @@
|
||||
# 2026-08-04 — Chat interaction overhaul: inline progressive stream (Claude Code style)
|
||||
|
||||
**Status:** Planned — not started. (Refocused from the earlier feature-heavy
|
||||
draft; backend features deferred — see "Deferred".)
|
||||
|
||||
## Goal
|
||||
|
||||
Streamline agent interactions — thinking, plan, tool usage, responses — into
|
||||
**one linear progressive inline stream per turn** (the Claude Code / Cline /
|
||||
Roo pattern), instead of the current split where the transcript shows a
|
||||
collapsed trace and the real live activity lives in a separate rail timeline.
|
||||
The right rail becomes **graph-only** (and auto-zooms to fit all entities).
|
||||
|
||||
## Locked decisions (operator interview)
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Live activity layout | **Inline stream (Claude Code)** — one progressive column per turn; rail keeps ONLY the Scope graph; Activity timeline tab removed |
|
||||
| Tool-call detail | **Per-tool progressive lines** — each tool its own compact live line (spinner → one-line result summary), expandable to raw |
|
||||
| Feature phases | **Defer** — edit/resubmit, @mentions, attachments are later phases; this plan is interaction-focused + graph auto-zoom |
|
||||
|
||||
## Diagnosis (grounded in current code)
|
||||
|
||||
- The transcript (`ChatThread` → `AgentTrace`) collapses a whole turn's tool
|
||||
calls into one line ("Proposed plan" / "N tool calls"), raw-JSON detail on
|
||||
expand. Not progressive; you can't see what's happening without expanding.
|
||||
- The actual live plan + tool timeline lives in the **right rail**
|
||||
(`TaskContextPanel` → `UnifiedTimeline`): newest-first backbone + tool stubs.
|
||||
So "what is the agent doing" is in a **second place** — a cognitive split.
|
||||
- `UnifiedTimeline` is imported **only** by `TaskContextPanel` (grep confirms),
|
||||
so removing the Activity pane is self-contained.
|
||||
- The `activityLog` **store** stays required: it feeds inline labels
|
||||
(`toolActivityLabel`), live `run` output (`toolsWithLive`), and the mascot
|
||||
(`mascot/stimuli.ts`). Only the timeline *view* is removed.
|
||||
- Tool events already arrive separately (`tool_use` then `tool_result` in
|
||||
`chat.ts`), and the activity log already carries humanized labels + per-tool
|
||||
`stepSeq` attribution. So progressive per-tool lines + step grouping are a
|
||||
**presentation** change, not a data/model change.
|
||||
- `run` results are free-form text (e.g. `"run on lxc:caddy: ERROR exit status
|
||||
1"`) → one-line result summaries are best-effort text parsing, no backend.
|
||||
|
||||
## Design
|
||||
|
||||
### D1 — One progressive inline stream per turn
|
||||
Replace `AgentTrace` (one collapsed blob per turn) with a new
|
||||
**`TurnTrace.svelte`** rendered inline for each assistant turn, top-to-bottom:
|
||||
1. **Live plan checklist** (only on the most-recent/running turn — see D3).
|
||||
2. **Tool lines grouped by plan step** (D2), then orphan tools (no step).
|
||||
3. **Streamed text answer** (existing `markdown-body prose-chat`), with the
|
||||
blinking cursor while streaming (existing).
|
||||
4. A compact **"Thinking" line** while `working` and before any output: reuses
|
||||
the existing `indicatorLabel` (running step → tool → "Agent is thinking…").
|
||||
Fades once text/tools arrive; reappears between steps.
|
||||
|
||||
### D2 — Per-tool progressive lines (the Claude-Code signature)
|
||||
One `ToolLine.svelte` per tool call (replaces `ToolCallCard`'s row style):
|
||||
- Left: state icon — spinner while `tool_use`-only, ✓ on result, ✗ on error.
|
||||
- Label: existing `toolActivityLabel(tool)` (humanized action).
|
||||
- **One-line result summary** on completion — new `toolResultSummary(tool)`
|
||||
in `activity.ts` (see plumbing). E.g.:
|
||||
- `run` → `exit 0 · <first line>` (parse "exit status N" / "ERROR")
|
||||
- `get_entity` → `host:hubris (healthy)`; `get_health_summary` → `healthy X · degraded Y · down Z`
|
||||
- `list_entities`/`list_lxcs` → `N entities`; `get_relations` → `N relations`
|
||||
- `search_knowledge` → `N results`; `upsert_knowledge` → `recorded document:…`
|
||||
- `update_plan_step` → `step <seq> → <status>`; `propose_plan` → `N steps`
|
||||
- default → first non-empty line of stringified result (≤80ch); `done` if empty
|
||||
- Live `run` output: while streaming, the line auto-expands a pinned-tail mini
|
||||
pane (reuse the `liveOutput` path from `toolsWithLive`).
|
||||
- Click → expand raw args/result (border-driven `<pre>`, cyberspace-square).
|
||||
- Border-driven, no rounded/shadow (per `border_driven_language`).
|
||||
|
||||
### D3 — Live plan checklist (TodoWrite-style)
|
||||
On the **running/last** turn, render the current-generation `planSteps`
|
||||
(already generation-aware via `workspace.ts`) as a checklist: pending = hollow,
|
||||
running = spinner + highlight, done = ✓, failed = ✗, blocked = pause. Steps
|
||||
check off live as `plan.step.*` events land. This is the unified timeline's
|
||||
plan view, moved inline and scoped to the active turn. Past turns render only
|
||||
their tool lines + text (the plan is session-level; the running turn carries
|
||||
its current state, mirroring how TodoWrite re-displays state each turn). On a
|
||||
terminal task state (`done`/`failed`), the checklist collapses to one line:
|
||||
`Plan complete — N steps` / `Plan failed — step K`.
|
||||
|
||||
### D4 — Rail → graph only
|
||||
`TaskContextPanel`: remove the Activity pane and the `UnifiedTimeline` import;
|
||||
the panel becomes the Scope graph full-height (keep the collapsible "Scope"
|
||||
header + the `nowTouching` strip). The graph is now the rail's entire job, so
|
||||
auto-fit (D6) matters more. `activityLog*` stores remain imported only where
|
||||
the inline stream/mascot need them.
|
||||
|
||||
### D5 — Cyberspace cohesion of the stream
|
||||
Apply alongside the rewrite so the new inline view is on-system from day one:
|
||||
- Transcript → **terminal log rows** (square, full-width, `YOU`/`NOMOS`
|
||||
role-tags, hairline `divide-y` separators; no bubbles, no soft shadow).
|
||||
Delete `.user-msg { box-shadow }`.
|
||||
- Tool lines + expanded `<pre>`: border-driven, square, opaque.
|
||||
- Composer: opaque `bg-background`, square (remove `rounded-2xl`/`bg-card/50`).
|
||||
- Rewrite the stale "Art Nouveau" `<style>` comments → "cyberspace/terminal".
|
||||
- Per `central_css_override`: drive surface styling centrally in `app.css`
|
||||
where it's a primitive concern; no ad-hoc `rounded-*`/`shadow-*`/`backdrop-blur`.
|
||||
|
||||
### D6 — Graph auto-fit + drag-pan (`SessionGraph.svelte`) (carried over)
|
||||
- Wrap nodes+links in `<g transform="translate(tx,ty) scale(s)">`; fit the bbox
|
||||
of all nodes (radius + label + padding) into `cw`/`ch`; cap `s ∈ [0.2, 2.5]`.
|
||||
- Re-fit on: mount, node-set change, container resize, sim-settle
|
||||
(`alpha > 0.05`), background double-click. **Not** every tick (fights pan).
|
||||
A `userPanned` flag pauses auto-follow after a manual pan until next
|
||||
membership/resize/double-click.
|
||||
- Background drag = pan (`tx`/`ty`); node drag converts screen→graph via the
|
||||
inverse transform before setting `fx`/`fy`. Dot-grid stays in screen space.
|
||||
- Keep: open-on-click, `touched` pulse, health-diff label, selection ring.
|
||||
Respect `scrollIntoView` pitfall (transform, not scroll).
|
||||
|
||||
## Phased task list (each independently shippable; all frontend)
|
||||
|
||||
- **P1 — Inline progressive stream.** `TurnTrace.svelte` + `ToolLine.svelte`;
|
||||
wire into `ChatThread` per turn; "Thinking" line; tool→step grouping via
|
||||
activity-log `stepSeq` matched by tool id; keep `toolsWithLive` for `run`.
|
||||
- **P2 — Live plan checklist.** Inline current-gen `planSteps` on the running
|
||||
turn; collapse-to-summary at terminal state.
|
||||
- **P3 — Rail → graph only.** Strip Activity pane + `UnifiedTimeline` from
|
||||
`TaskContextPanel`; verify no other importers (grep: only TaskContextPanel).
|
||||
- **P4 — Cyberspace cohesion.** Terminal log rows; remove rounded/shadow/
|
||||
translucency; square composer; centralize in `app.css`; fix stale comments.
|
||||
- **P5 — Graph auto-fit + drag-pan.** D6.
|
||||
- **Polish (small, frontend-only):** per-message/tool **copy**; **scroll-to-
|
||||
bottom** button (uses `container.scrollTo`, never `scrollIntoView`).
|
||||
|
||||
## Plumbing specifics (grounded, no backend)
|
||||
|
||||
- New `toolResultSummary(t: ToolCallResult): string` in `activity.ts`, beside
|
||||
`toolActivityLabel`. Per-name switch (D2 list), graceful fallback.
|
||||
- Tool→step grouping: build `id → stepSeq` from the activity log once per turn;
|
||||
tools with no step render as orphans.
|
||||
- Reuse: `planSteps` (generation-aware), `indicatorLabel`, `toolsWithLive`,
|
||||
`toolActivityLabel`, `liveOutput` streaming path.
|
||||
|
||||
## Constraints honored (saved decisions)
|
||||
|
||||
- `design_system.central_css_override`, `border_driven_language`: square,
|
||||
hairline, opaque, focus-by-color, no soft shadows/glows.
|
||||
- `chat_thread.pane_layout`: dynamic status (Thinking line, live checklist)
|
||||
lives in the **message Pane**, never the input Pane.
|
||||
- `wmkit.scrollintoview_reflow_pitfall`: `container.scrollTo` for scroll-to-
|
||||
bottom; transform (not scroll) for graph pan.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Removing the rail timeline loses the "overview" view.** Mitigation: the
|
||||
inline checklist + per-turn tool lines carry the same info progressively; the
|
||||
graph still shows fleet scope. If operators miss the overview, a collapsed
|
||||
"full timeline" can return as a toggle (follow-up).
|
||||
- **Auto-fit vs manual pan** — handled by `userPanned` + settle-alpha gate.
|
||||
- **Inline stream length on long turns** (15–27 min, many tools) — progressive
|
||||
lines can get long; mitigate by auto-collapsing finished steps (keep the
|
||||
running step + its tools expanded, prior steps as one-line summaries).
|
||||
- **Best-effort result summaries** may misformat unusual payloads — fallback is
|
||||
always a truncated raw line + expandable raw detail, never a blank.
|
||||
|
||||
## Validation
|
||||
|
||||
- `npm run lint`, `tsc --noEmit` (no NEW errors beyond the known baseline in
|
||||
`ui/*`, `oidc.ts`, `windows.ts`, `workspace.ts`), `vite build`, `vitest`
|
||||
(add a `toolResultSummary` unit test per tool name + fallback).
|
||||
- Manual matrix: (a) start a long task → Thinking line → plan checklist
|
||||
appears and checks off live → each tool streams as its own line with a
|
||||
one-line summary → text streams; (b) reload mid-turn → working still shows;
|
||||
(c) `run` tool → live output pins to tail then collapses to summary; (d)
|
||||
graph auto-fits at settle + on new entity + drag-pan + double-click reset;
|
||||
(e) no rounded/soft-shadow remains on chat surfaces; (f) rail shows graph only.
|
||||
|
||||
## Deferred (later phases, after this lands + validates)
|
||||
|
||||
- **Edit-and-resubmit** — `truncateFrom` store method + `POST /sessions/{id}/edit`
|
||||
(extract `streamTurn` from `handleChat`); reuse `reopenSession` (already
|
||||
exists, store.go:838 — marks prior `session_plan_steps` `replaced`, clears
|
||||
outcome) for the reset. Reject edit while the gate is busy (HTTP 409); edit
|
||||
cannot queue (truncation must be atomic). Regenerate = no-op-edit case.
|
||||
- **@entity mentions** — small `GET /api/v1/entities/search?q=` + composer
|
||||
autocomplete inserting `type:name` slugs the agent/graph already parse.
|
||||
- **Attachments** — multipart upload + `agent_attachments` table + configured
|
||||
`OIKOS_ATTACHMENTS_DIR` (explicit volume, not relative) + capped text inlining.
|
||||
- **Continue button** — needs `/resume` to `reopenSession` first for terminal
|
||||
sessions (today `/resume` does not reopen `done`/`failed`; `handleChat`'s
|
||||
follow-up path does). Small backend tweak.
|
||||
- **Image vision** pending provider confirmation.
|
||||
|
||||
## Out of scope / follow-ups
|
||||
|
||||
- A collapsible "full timeline" overview toggle if the rail removal is missed.
|
||||
- `read_attachment` MCP tool (lazy full-content fetch, lower context than inlining).
|
||||
- Oldest-first timeline toggle / per-tool `tool.*` events for background turns.
|
||||
145
plans/done/2026-08-04-hermes-mcp-client-integration.md
Normal file
145
plans/done/2026-08-04-hermes-mcp-client-integration.md
Normal file
@@ -0,0 +1,145 @@
|
||||
# 2026-08-04 — Hermes MCP client integration: native tool surface for oikos
|
||||
|
||||
**Status:** Plan.
|
||||
**Context:** Hermes Agent (mac-mini workstation) now connects to oikos's MCP server
|
||||
as a native MCP client (`mcp_servers.oikos` in `~/.hermes/config.yaml`). All 37+ MCP
|
||||
tools are available as `mcp__oikos__*` first-class Hermes tool calls — no more raw
|
||||
curl with batch-initialize SSE parsing. The integration works; this plan tightens the
|
||||
remaining seams.
|
||||
|
||||
**Trigger:** First-use retrospective identified three areas that make the integration
|
||||
harder to use than it should be.
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation
|
||||
|
||||
The oikos MCP server (`internal/mcp/`) speaks Streamable HTTP at
|
||||
`https://mcp.hubris.network/mcp`. Hermes Agent's native MCP client connects to it on
|
||||
startup, discovers tools, and registers them as callable functions. This replaces the
|
||||
previous pattern where agents fired raw curl requests with batch `initialize` +
|
||||
`tools/call` envelopes.
|
||||
|
||||
Three friction points observed:
|
||||
|
||||
- **No lightweight connectivity check.** The `/healthz` HTTP endpoint exists but isn't
|
||||
exposed at the MCP protocol layer. An agent that wants to verify the MCP server is
|
||||
reachable must call a real tool (e.g. `list_entities` with a limit of 1) — every call
|
||||
carries the Streamable HTTP session-initialization overhead.
|
||||
- **Bearer token in plaintext.** `~/.hermes/config.yaml` stores the token directly in the
|
||||
`mcp_servers.oikos.headers.Authorization` value. Hermes does not support env-var
|
||||
interpolation in MCP server configs, so the token can't live only in `.env`.
|
||||
- **Zero-visibility streaming overhead.** Streamable HTTP batches `initialize` +
|
||||
`tools/call` per request. This adds ~2KB of transport per tool call that the agent
|
||||
never sees. For a single `get_health_summary` call this is negligible; for a 10-tool
|
||||
exploration pass it's 20KB of invisible overhead.
|
||||
|
||||
---
|
||||
|
||||
## 2. Changes
|
||||
|
||||
### I — MCP health/ping tool (`mcp__oikos__ping`)
|
||||
|
||||
**Why:** Agents need a zero-cost connectivity check before calling production tools.
|
||||
Currently every check incurs the full Streamable HTTP initialize + tools/call round-trip.
|
||||
|
||||
**What:**
|
||||
|
||||
Add a `ping` tool that returns `{"ok": true, "server": "oikos", "version": "dev"}`.
|
||||
No arguments. No DB hit. No auth check (already protected by the MCP transport's auth
|
||||
layer — the request won't arrive if the bearer token is missing).
|
||||
|
||||
```go
|
||||
// internal/mcp/tools.go
|
||||
{
|
||||
Name: "ping",
|
||||
Description: "Lightweight connectivity check. Returns immediately with server identity, no DB hit.",
|
||||
InputSchema: jsonschema.Must(nil), // no params
|
||||
Handler: func(ctx context.Context, args json.RawMessage, caller CallerInfo) (json.RawMessage, error) {
|
||||
return json.RawMessage(`{"ok":true,"server":"oikos","version":"` + version.Version + `"}`), nil
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**Risk class:** read-only. No auth, no DB, no state. Auto-approves.
|
||||
|
||||
**Test:** `hermes mcp test oikos` (from the Hermes CLI) verifies MCP server reachability
|
||||
independently; the `ping` tool gives agent code the same signal programmatically.
|
||||
|
||||
### II — Tool name documentation in server metadata
|
||||
|
||||
**Why:** Hermes prefixes MCP tools as `mcp_{server}_{tool}`, so `get_health_summary`
|
||||
becomes `mcp__oikos__get_health_summary`. Agents discover tool names at runtime via
|
||||
`tools/list`, but there's no short summary of what each tool group does that survives
|
||||
into the MCP tool description.
|
||||
|
||||
**What:**
|
||||
|
||||
Audit and tighten every tool's `Description` field in `internal/mcp/tools.go` so the
|
||||
first 8–12 words are a searchable one-liner an agent can pattern-match against.
|
||||
Current descriptions that are vague or redundant get a prefix rewrite:
|
||||
|
||||
| Tool | Current description | Revised |
|
||||
|------|-------------------|---------|
|
||||
| `get_entity` | "Get entity metadata" | "Look up one entity by slug or UUID — type, state, attributes, health" |
|
||||
| `list_entities` | "List entities" | "Browse entities by type, state, or name substring — paginated" |
|
||||
| `upsert_knowledge` | "Record what you learned" | "Write a document/investigation/runbook to the knowledge graph — idempotent" |
|
||||
| `run` | "Run ANY shell command" | "Execute a shell command on any host/LXC/VM — auto-classified by risk" |
|
||||
|
||||
Existing tools pass through unchanged if their description is already crisp. ~15 tools
|
||||
get description rewrites.
|
||||
|
||||
**Risk class:** read-only (config change). No runtime effect.
|
||||
|
||||
### III — Env-var interpolation docs for Hermes config (oikos-side documentation)
|
||||
|
||||
**Why:** The bearer token lives in `~/.hermes/config.yaml` in plaintext because Hermes
|
||||
does not support `${VAR}` interpolation in MCP server configs. This is a Hermes
|
||||
upstream feature request, not an oikos change — but oikos should document the
|
||||
workaround and track the upstream ask.
|
||||
|
||||
**What:**
|
||||
|
||||
Add a `### Hermes MCP client` subsection to `docs/infrastructure/mcp-server.md` (or
|
||||
create it if it doesn't exist) that covers:
|
||||
|
||||
1. The config block to add to `~/.hermes/config.yaml` (already done — record it
|
||||
for the next person).
|
||||
2. The token exposure caveat: Hermes doesn't support env-var interpolation in
|
||||
`mcp_servers` `headers` yet (upstream issue nousresearch/hermes-agent#TODO — file
|
||||
once).
|
||||
3. Workaround: `hermes config set security.redact_secrets true` (already default) so
|
||||
the token value is stripped from tool output and logs even if it appears in
|
||||
diagnostic text.
|
||||
4. How to verify the connection: `hermes mcp list` → `hermes mcp test oikos`.
|
||||
|
||||
**Risk class:** docs-only.
|
||||
|
||||
---
|
||||
|
||||
## 3. Open questions
|
||||
|
||||
| Question | Decision |
|
||||
|----------|----------|
|
||||
| Should `ping` bypass auth entirely or still require a valid bearer token? | **Still requires auth.** The MCP transport layer validates the token before routing to `ping` — no special treatment needed. If the token is missing, the request never reaches the handler. |
|
||||
| Who files the Hermes upstream feature request for `${VAR}` interpolation? | **Oikos operator** (dtoro). The need is specific to this deployment. File at https://github.com/NousResearch/hermes-agent/issues. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Not doing (yet)
|
||||
|
||||
- **Persistent MCP sessions** — Streamable HTTP stateless mode is fine for the
|
||||
current tool-call volume (~1–5 calls per agent turn). Persistent sessions would
|
||||
save ~2KB per call but add connection lifecycle complexity. Revisit if per-turn
|
||||
tool calls exceed 20.
|
||||
- **`tools/list` caching** — Hermes already caches tool discovery at session start.
|
||||
The 37-tool list is ~4KB; caching adds complexity for negligible savings.
|
||||
|
||||
---
|
||||
|
||||
## 5. Verification
|
||||
|
||||
1. `curl -X POST https://mcp.hubris.network/mcp ... -d '...ping...'` returns `{"ok":true,"server":"oikos","version":"dev"}`
|
||||
2. `hermes mcp list` shows `ping` among oikos tools
|
||||
3. `hermes doctor` passes
|
||||
4. Tool descriptions are crisp: `hermes mcp list` output for oikos shows prefixed summaries
|
||||
369
plans/done/2026-08-04-session-audit-agent-reliability.md
Normal file
369
plans/done/2026-08-04-session-audit-agent-reliability.md
Normal file
@@ -0,0 +1,369 @@
|
||||
# 2026-08-04 — Session audit: agent reliability, plan system, and learning loop gaps
|
||||
|
||||
**Status:** Done — all 12 tasks implemented, tested, deployed in v0.21.0. Verified in production
|
||||
with live session tests (classifications, feedback, token tracking, execution linkage all confirmed).
|
||||
**Reviewed sessions:** Past 5 completed plus ZimaOS continuation (268895a5)
|
||||
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
|
||||
`agent_activity`/`session_plan_steps`/`executions`/`classifications`/`feedback`/
|
||||
`patterns`/`skills`/`approvals`/`nomos_plan_executions`/`audit_log` on the prod
|
||||
mac-mini. Cross-referenced with `internal/audit/`, `internal/mcp/`,
|
||||
`internal/httpapi/`, `internal/policy/`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Sessions audited
|
||||
|
||||
| # | Session | Turns | Run calls | Failures | Plan steps (done/total) | Duration | Outcome |
|
||||
|---|---------|-------|-----------|----------|-------------------------|----------|---------|
|
||||
| S1 | Pocket-pascal deploy (a433b386) | 8 | 111 | 1 | 10/11 | 1.5h | Success (truncated by turn limit) |
|
||||
| S2 | SSH re-investigation (6f0ade08) | 4 | 42 | 0 | 0/4 | 11m | Success (all steps replaced) |
|
||||
| S3 | Webhook HMAC (0f509508) | 2 | 40 | 0 | 5/5 | 2m | Success (clean; best session) |
|
||||
| S4 | Check scripts (d458a5f8) | 23 | 96 | 4 | 0/14 | 2h | Success (3 plan gens; 0 steps done) |
|
||||
| S5 | ZimaOS outage (268895a5) | 18+10 | 209 | 8 | 0/5 | 2h30m | **Marked success; dashboard still broken** |
|
||||
|
||||
**Headline:** 85% session success rate, but plan adherence is ~38% (15/39 steps
|
||||
ever reached `done`). The ZimaOS session was the worst: marked `success` while
|
||||
the dashboard was still down, then burned 81 more calls and 30 minutes chasing
|
||||
irrelevant DHCP reservations.
|
||||
|
||||
---
|
||||
|
||||
## 2. Cross-cutting findings
|
||||
|
||||
### F1 — MCP 30s client timeout kills every long-running command (P0)
|
||||
|
||||
All **9** `run` failures across the audit are `Post "http://api:8090/mcp":
|
||||
context deadline exceeded` at exactly 30s. Commands with `sleep 30`, `qm
|
||||
shutdown`+wait, or async poll loops always hit this. The agent retries with
|
||||
longer sleeps and hits the same wall.
|
||||
|
||||
**Root cause:** `cmd/nomos` MCP client uses a 30s timeout; the `run` tool
|
||||
blocks synchronously waiting for command completion. No async path exists for
|
||||
long-running commands.
|
||||
|
||||
### F2 — Premature success: `complete_task` fires before the goal is actually met (P0)
|
||||
|
||||
Session 268895a5 called `complete_task(success)` at 18:29 with summary "zimaos
|
||||
returns 200." The endpoint was serving ttyd (terminal), not the ZimaOS
|
||||
dashboard. The agent conflated "HTTPS 200" with "dashboard works." The user had
|
||||
to resume the session.
|
||||
|
||||
**Root cause:** No pre-completion validation. The agent can mark success with a
|
||||
summary that doesn't match reality. `complete_task` is a write-and-forget
|
||||
operation with no state check.
|
||||
|
||||
### F3 — Wrong target: `run` doesn't validate the target can execute the command (P0)
|
||||
|
||||
In the ZimaOS continuation, `qm stop 100 --skiplock` was executed on
|
||||
`lxc:dns`. The command failed (`qm: command not found`) because the agent
|
||||
copied the command from a previous run but forgot to change the target.
|
||||
Similarly, `cmd/nomos` tried `docker exec` on `host:hubris` (no docker).
|
||||
|
||||
**Root cause:** `run(target, command)` in `internal/mcp/server.go` doesn't
|
||||
validate that the target type can execute the given command. A simple
|
||||
allowlist would catch `qm`/`pct`/`pvesh` on non-host targets.
|
||||
|
||||
### F4 — Plan system is decorative: 62% of steps never reach `done` (P1)
|
||||
|
||||
Across 5 sessions: 39 plan steps. 24 (62%) were `replaced`, 15 (38%) reached
|
||||
`done`. Session d458a5f8 had 3 complete plan regenerations with **zero**
|
||||
completed steps. The agent replaces plans instead of completing or explicitly
|
||||
skipping steps.
|
||||
|
||||
**Root cause:** Plan steps carry status (`pending`/`running`/`done`/`failed`/
|
||||
`skipped`/`blocked`/`replaced`) but `replaced` has no `replaced_reason`
|
||||
field. The model can silently replace every step and the system doesn't flag
|
||||
it. No constraint ties `propose_plan` to existing plan state.
|
||||
|
||||
### F5 — No plan on session resume: continuation sessions run ad-hoc (P1)
|
||||
|
||||
The ZimaOS continuation (18:29→19:01) had 81 calls with **zero**
|
||||
`propose_plan` calls. The agent ran ad-hoc tool calls with no structure.
|
||||
|
||||
**Root cause:** When a session resumes, the `must_have_plan` guard is already
|
||||
satisfied by the old (completed) plan. The agent doesn't re-plan on resume.
|
||||
|
||||
### F6 — Scope expansion / rabbit holes: agent chases irrelevant sub-goals (P2)
|
||||
|
||||
In the ZimaOS continuation, the agent spent ~40 calls trying to fix Technitium
|
||||
DHCP reservations — a completely different subsystem from the goal ("make the
|
||||
dashboard reachable"). The ZimaOS dashboard hadn't started since July 19
|
||||
(3-week-old issue), making the DHCP reservation effort moot. The agent never
|
||||
surfaced a question like "This is pre-existing — should I still fix DHCP?"
|
||||
|
||||
**Root cause:** No scope gate. When the agent pivots to a subsystem unrelated
|
||||
to the stated goal, nothing stops it. The `session_questions` mechanism exists
|
||||
(2 calls in 193 sessions) but the model never uses it.
|
||||
|
||||
### F7 — Command generation errors: malformed bash from the LLM (P2)
|
||||
|
||||
In the ZimaOS continuation, the agent generated:
|
||||
- `head - n` instead of `head -n` → bash syntax error
|
||||
- `echo "---" && \n curl ...` → literal `\n` in command → ambiguous redirect
|
||||
|
||||
**Root cause:** The LLM generates bash commands inline in content blocks. No
|
||||
syntax validation, no escape-character handling. The `run` tool should reject
|
||||
malformed commands before execution.
|
||||
|
||||
### F8 — Learning pipeline completely empty (P4)
|
||||
|
||||
| Table | Rows |
|
||||
|-------|------|
|
||||
| `classifications` | **0** |
|
||||
| `feedback` | **0** |
|
||||
| `patterns` | **0** |
|
||||
| `skills` | **0** |
|
||||
|
||||
Despite 1,884 executions and 263 approvals, the system learns nothing from
|
||||
outcomes. The ZimaOS session discovered that the dashboard hadn't started since
|
||||
July 19 — this was never persisted. The HMAC trailing-newline discovery was
|
||||
persisted manually; if the agent forgot `upsert_knowledge`, it would be lost.
|
||||
|
||||
### F9 — Observability gaps (P3)
|
||||
|
||||
| Gap | Detail |
|
||||
|-----|--------|
|
||||
| Token tracking | `agent_activity.token_count` is NULL for every row |
|
||||
| Execution linkage | `nomos_plan_executions` is empty; `audit_log.session_id` is null |
|
||||
| Plan quality | No metric for step completion rate (currently 38%) |
|
||||
| MCP timeout rate | No counter for `run` calls that hit the client timeout |
|
||||
|
||||
### F10 — Execution success rate is 74% (P2)
|
||||
|
||||
1,884 executions: 1,400 completed (74%), 295 failed (15.6%), 152 cancelled
|
||||
(8%), 37 denied (2%). One in four execution attempts doesn't complete.
|
||||
|
||||
---
|
||||
|
||||
## 3. Improvement plan (ordered by impact/effort)
|
||||
|
||||
### Task 1 — Prevent premature `complete_task(success)` *(fixes F2)*
|
||||
|
||||
**`internal/mcp/tools.go`** — In the `complete_task` handler, when
|
||||
`outcome=success`: require the summary field to contain a verifiable state
|
||||
assertion. Minimum: if the goal mentions a URL, check that the summary doesn't
|
||||
contradict known state. Lightweight: log a warning if the summary says "returns
|
||||
200" but the last `ping_service` or `run` result says otherwise.
|
||||
|
||||
**Coach:** `nomos/SOUL.md` — explicit rule: *"Before calling
|
||||
complete_task(success), restate the user's original goal in your own words and
|
||||
verify each condition. If any condition is 'probably works' rather than
|
||||
'verified,' ask the operator or set outcome=partial."*
|
||||
|
||||
### Task 2 — Target validation in `run` *(fixes F3)*
|
||||
|
||||
**`internal/mcp/server.go`** — In the `run` handler, before dispatching:
|
||||
validate that the command prefix matches the target type.
|
||||
|
||||
```
|
||||
pct/qm/pvesh/iptables → only host:* targets
|
||||
systemctl/docker → host:* or lxc:* targets
|
||||
curl/nmap/ss → any target
|
||||
```
|
||||
|
||||
If mismatched, return a clear error: *"Cannot run `qm` on lxc:dns — `qm` is a
|
||||
Proxmox host command. Use target host:hubris or host:strong."* Do not classify
|
||||
or execute.
|
||||
|
||||
**Test:** `TestClassifyCommand_WrongTarget` → commands with host-only prefixes
|
||||
on LXC targets return error without execution.
|
||||
|
||||
### Task 3 — Raise MCP client timeout; add async path for long-running commands *(fixes F1)*
|
||||
|
||||
**`cmd/nomos`** — Raise the MCP client timeout from 30s to 120s.
|
||||
|
||||
**`internal/mcp/server.go`** — For `run` commands that the classifier
|
||||
determines will exceed the client timeout (presence of `sleep`, `wait`,
|
||||
`timeout` in the command), return immediately with an `execution_id` and status
|
||||
`running`. The agent already has `get_execution_status` — use it:
|
||||
|
||||
1. Classify the command; if it contains `sleep`, `wait`, or shell constructs
|
||||
that imply polling, flag it as `async_potential`.
|
||||
2. Start the command, return the `execution_id` immediately.
|
||||
3. Agent polls with `get_execution_status(execution_id)`.
|
||||
4. If the client timeout is hit mid-poll, the execution continues on the server
|
||||
— it's not lost.
|
||||
|
||||
**Test:** `run` with `sleep 60; echo done` on host:hubris → returns
|
||||
immediately (not 30s timeout), `get_execution_status` eventually returns
|
||||
`completed`.
|
||||
|
||||
### Task 4 — Plan step integrity: require `replaced_reason` on replacement *(fixes F4)*
|
||||
|
||||
**`session_plan_steps` migration** — Add `replaced_reason TEXT` column.
|
||||
|
||||
**`cmd/nomos`** — When the agent emits `update_plan_step` with
|
||||
`status=replaced`, require a non-empty `replaced_reason`. Valid reasons:
|
||||
`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`.
|
||||
|
||||
**Coach:** `nomos/SOUL.md` — explicit rule: *"Complete (status=done) or
|
||||
explicitly skip (status=skipped) steps. Use status=replaced only when the
|
||||
entire plan generation is wrong; include the reason. Replacing all steps with
|
||||
no reason is a session-quality violation."*
|
||||
|
||||
### Task 5 — Force `propose_plan` on session resume *(fixes F5)*
|
||||
|
||||
**`cmd/nomos`** — When a session with status `done` or `failed` receives a new
|
||||
user message, reset the plan state: clear step status, require a new
|
||||
`propose_plan` call before any `run` calls. The "must have plan" guard should
|
||||
consider the resumed session as plan-less until a fresh `propose_plan` is
|
||||
called.
|
||||
|
||||
**Guard:** `set_goal` + `propose_plan` must be called before any `run` in a
|
||||
resumed session. Reuse the existing "No plan — call set_goal then propose_plan"
|
||||
error from `internal/mcp/server.go`.
|
||||
|
||||
### Task 6 — Scope gate: surface `session_questions` on context switch *(fixes F6)*
|
||||
|
||||
**Coach:** `nomos/SOUL.md` — explicit rule: *"Before pivoting to a subsystem
|
||||
not mentioned in the user's goal, ask via session_questions. Example: 'The
|
||||
dashboard logs show it hasn't started since July 19. Do you want me to debug
|
||||
the dashboard service itself [A], skip it and just stabilize the IP [B], or
|
||||
stop here [C]?'"*
|
||||
|
||||
**`cmd/nomos` prompt** — Add to the system prompt: *"When the investigation
|
||||
leads to a subsystem or root cause unrelated to the expressed goal, surface a
|
||||
session_question before taking action."*
|
||||
|
||||
### Task 7 — Auto-upsert knowledge on session close *(fixes F8)*
|
||||
|
||||
**`cmd/nomos`** — On `complete_task` (any outcome: success, partial, failure),
|
||||
auto-generate a knowledge entry:
|
||||
|
||||
```
|
||||
title: "<date>: <goal summary>"
|
||||
content: "## Outcome\n<outcome>\n## Root cause\n<extracted>\n## What was done\n<summary>\n## What was left\n<unresolved>"
|
||||
tags: [session:<id>]
|
||||
about: [entities involved]
|
||||
```
|
||||
|
||||
This ensures every session leaves a trace regardless of whether the agent
|
||||
remembered to call `upsert_knowledge`.
|
||||
|
||||
### Task 8 — Token tracking *(fixes F9)*
|
||||
|
||||
**`cmd/nomos`** — After each LLM call, extract `usage.prompt_tokens`,
|
||||
`usage.completion_tokens`, `usage.total_tokens` from the response and write to
|
||||
`agent_activity.token_count`. Currently the field exists but is never populated
|
||||
(NULL for all rows).
|
||||
|
||||
### Task 9 — Execution linkage *(fixes F9)*
|
||||
|
||||
**`internal/mcp/server.go`** — When `run` creates an execution, write a row
|
||||
into `nomos_plan_executions` linking `session_id`, `plan_step_seq`, and
|
||||
`execution_id`.
|
||||
|
||||
**`internal/mcp/server.go`** — Pass `session_id` (from MCP request headers)
|
||||
into `audit_log` writes. Currently `audit_log.session_id` is NULL — the
|
||||
`createAuditLog` function in `internal/httpapi/impl.go` receives the
|
||||
correlation_id but not the session_id from the MCP path.
|
||||
|
||||
### Task 10 — Plan quality metric *(fixes F9)*
|
||||
|
||||
**`cmd/nomos`** — At session close, compute: `completed_steps /
|
||||
total_steps_per_plan` (currently ~38%). Log as a metric or write as a session
|
||||
attribute. Track over time to measure plan-adherence improvements from Tasks
|
||||
4+5.
|
||||
|
||||
### Task 11 — Auto-classify every `run` call *(fixes F8)*
|
||||
|
||||
**`internal/mcp/server.go`** — The `run` handler already calls the classifier
|
||||
(`classifyCommand` in `internal/policy/command.go`) to determine risk_class and
|
||||
approval route. Write the result to the `classifications` table. Currently the
|
||||
table is empty (0 rows) despite 1,884 executions being classified.
|
||||
|
||||
### Task 12 — Auto-feedback on session close *(fixes F8)*
|
||||
|
||||
**`cmd/nomos`** — On `complete_task`, generate a `feedback` entry:
|
||||
|
||||
```
|
||||
session_id: <id>
|
||||
outcome: <outcome>
|
||||
observation: <summary>
|
||||
lesson: <extracted from complete_task.summary>
|
||||
side_effects: <entities created/modified during session>
|
||||
```
|
||||
|
||||
**`cmd/oikos`** — Add a daily cron or scheduler job that reads recent
|
||||
`feedback` entries and extracts `patterns` (recurring root causes, same-fix
|
||||
applied multiple times, known-broken services). Seed the pattern table.
|
||||
|
||||
### Task 13 — Command syntax validation in `run` *(fixes F7)*
|
||||
|
||||
**`internal/mcp/server.go`** — Before executing a `run` command, do
|
||||
lightweight bash syntax validation:
|
||||
|
||||
```
|
||||
- Reject literal \n in commands (should be ; or &&)
|
||||
- Reject commands where the last line ends with \ (backslash-continuation)
|
||||
but no next line
|
||||
- Warn on common typos: "head - n", "grep - i", spaces before flags
|
||||
- Reject `&& \n` patterns (the LLM sometimes inserts literal \n between && chains)
|
||||
```
|
||||
|
||||
### Task 14 — Stuck-session reaping (from prior plan; re-confirmed)
|
||||
|
||||
This session exhibited the same idle zombie pattern (23da10db — no closed_at,
|
||||
status `failed` but outcome `failure`). Task 5 from the 2026-08-03 plan is
|
||||
still open. Copying here for completeness.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommended sequence
|
||||
|
||||
```
|
||||
P0 (blocks operational waste):
|
||||
1 → 2 → 3
|
||||
|
||||
P1 (fixes plan architecture):
|
||||
4 → 5
|
||||
|
||||
P2 (cognitive guardrails):
|
||||
6 → 7 → 13
|
||||
|
||||
P3 (observability):
|
||||
8 → 9 → 10
|
||||
|
||||
P4 (learning loop):
|
||||
11 → 12
|
||||
```
|
||||
|
||||
Sequence rationale: Tasks 1-3 stop the worst outcomes (premature success,
|
||||
wrong-target execution, MCP timeouts). Tasks 4-5 make the plan system actually
|
||||
useful instead of decorative. Tasks 6-7 add guardrails that prevent the ZimaOS
|
||||
rabbit-hole class of failure. Tasks 8-10 give us visibility into whether any of
|
||||
the previous tasks are working. Tasks 11-12 close the learning loop.
|
||||
|
||||
---
|
||||
|
||||
## 5. Validation
|
||||
|
||||
| Task | Test |
|
||||
|------|------|
|
||||
| 1 | Session with goal "make X reachable" where last ping shows 502 → `complete_task(success)` is rejected or warns |
|
||||
| 2 | `run("lxc:dns", "qm stop 100")` → error: "qm is a Proxmox host command" |
|
||||
| 3 | `run` with `sleep 45; echo done` → returns execution_id immediately, `get_execution_status` shows final result |
|
||||
| 4 | `update_plan_step(status=replaced)` with no reason → rejected; with reason → accepted |
|
||||
| 5 | Resumed session calls `run` before `propose_plan` → blocked: "No plan — call propose_plan" |
|
||||
| 6 | Agent pivots to unrelated subsystem → `session_questions` is called before action |
|
||||
| 7 | `complete_task` → knowledge entry created automatically with session link |
|
||||
| 8 | `agent_activity.token_count` is non-NULL after any LLM call |
|
||||
| 9 | `nomos_plan_executions` has rows linking session + step + execution |
|
||||
| 10 | Session close writes `plan_adherence` attribute (step-completion %) |
|
||||
| 11 | `classifications` table has 1 row per `run` call with risk_class + route |
|
||||
| 12 | `complete_task` → auto `feedback` entry; daily pattern job finds recurring issues |
|
||||
| 13 | `run` with `head - n /etc/hosts` → rejected with clear error about malformed command |
|
||||
|
||||
---
|
||||
|
||||
## 6. Out of scope / open questions
|
||||
|
||||
- Whether to raise `auto_act` from `off` for `reversible_low` actions (separate
|
||||
policy decision; would reduce approval pileup without code changes).
|
||||
- Whether to add a `delete_entity` MCP tool for lifecycle management (separate
|
||||
from this reliability plan).
|
||||
- The exact TTL for stuck-session reaping (30 min recommended, confirmed in
|
||||
2026-08-03 plan).
|
||||
- Whether `run` async mode should be opt-in (command contains sleep/wait) or
|
||||
universal (every run returns immediately, agent always polls). Recommend
|
||||
opt-in for now — most commands complete in <5s.
|
||||
304
plans/done/2026-08-04-unified-mcp-agents.md
Normal file
304
plans/done/2026-08-04-unified-mcp-agents.md
Normal file
@@ -0,0 +1,304 @@
|
||||
# 2026-08-04 — Unified plan: external MCP agents + Nomos reliability
|
||||
|
||||
**Status:** Complete. All 5 stages implemented (2026-08-04).
|
||||
|
||||
---
|
||||
|
||||
## 1. Context
|
||||
|
||||
Two audiences, two gaps:
|
||||
|
||||
| Audience | Current state | Goal |
|
||||
|---|---|---|
|
||||
| **Nomos** (internal agent) | 85% session success, but 38% plan adherence, 9 timeout failures, premature success, empty learning table | Reliable, self-correcting, leaves a trace |
|
||||
| **External agents** (Claude, Goose, etc.) | 40 MCP tools — heavy on observe, light on act. Can't manage signals, checks, executions, or knowledge beyond `upsert`. | Full Oikos surface: observe + act + curate |
|
||||
|
||||
The two plans share infrastructure (`internal/mcp/server.go`) and have cross-task
|
||||
dependencies. This document merges them into one sequenced plan.
|
||||
|
||||
---
|
||||
|
||||
## 2. Cross-plan dependencies
|
||||
|
||||
Three audit tasks are **prerequisites** for external-agent mutation tools:
|
||||
|
||||
| Audit task | Enables | Why |
|
||||
|---|---|---|
|
||||
| **Task 3** — async `run` + timeout | `cancel_execution`, `list_executions` with live status | Without async `run`, every command >30s hits a client timeout. The agent loses track of the execution and can't cancel it. |
|
||||
| **Task 9** — execution linkage | `list_executions` filtered by session/entity | `nomos_plan_executions` table is empty; `audit_log.session_id` is NULL. Without linkage, execution queries are blind. |
|
||||
| **Task 2** — target validation in `run` | All Phase 2 mutation tools | Can't let external agents mutate state on targets that can't execute the command (audit found `qm` run on `lxc:dns`). |
|
||||
|
||||
Two items were **dropped** from the original MCP expansion:
|
||||
|
||||
- **`approve_execution` / `deny_execution`** — separation-of-duties violation. An
|
||||
MCP agent approving its own queued commands breaks the approval model. The
|
||||
correct fix is raising `auto_act` for `reversible_low` (policy change, zero
|
||||
code — listed in the audit plan's out-of-scope). The existing assent-window
|
||||
path in `run()` already auto-approves reversible-low commands.
|
||||
- **Execution log streaming** — MCP has no push model. `get_execution_status`
|
||||
returns the latest output; full streaming stays WebUI-only.
|
||||
|
||||
---
|
||||
|
||||
## 3. Unified sequence (5 stages, 31 tasks)
|
||||
|
||||
### Stage 1: Foundation — shared infrastructure (3 tasks)
|
||||
|
||||
These unlock everything downstream. Do first.
|
||||
|
||||
**T1 — Target validation in `run`** *(audit Task 2)*
|
||||
`internal/mcp/server.go`: before dispatching a `run` command, validate prefix
|
||||
against target type. `qm`/`pct`/`pvesh` → `host:*` only. `systemctl`/`docker` →
|
||||
`host:*` or `lxc:*`. Mismatch returns error without execution.
|
||||
|
||||
**T2 — Async `run` + 120s timeout** *(audit Task 3)*
|
||||
`cmd/nomos`: raise MCP client timeout 30s → 120s.
|
||||
`internal/mcp/server.go`: when classifier detects `sleep`/`wait`/poll loops in
|
||||
the command, start execution and return `execution_id` immediately. Agent polls
|
||||
with `get_execution_status`. Execution continues server-side even if client
|
||||
timeout.
|
||||
|
||||
**T3 — Execution linkage** *(audit Task 9)*
|
||||
`internal/mcp/server.go`: when `run` creates an execution, write into
|
||||
`nomos_plan_executions` (session_id + plan_step_seq + execution_id). Pass
|
||||
`session_id` from MCP request headers into `audit_log` writes.
|
||||
|
||||
---
|
||||
|
||||
### Stage 2: External agent observe (8 tasks)
|
||||
|
||||
All read-only MCP tools. Zero risk, ship fast. Unblocks external agents from
|
||||
understanding system state.
|
||||
|
||||
**T4 — `get_dashboard_summary`**
|
||||
Fleet health counts, signals by severity, pending approvals, event rate. One
|
||||
call instead of 4. Wraps existing `GetDashboardSummary` DB query.
|
||||
|
||||
**T5 — `get_ontology`**
|
||||
Entity types, relationship types, lifecycle states, monitoring specs. Agents
|
||||
need this to reason about the schema.
|
||||
|
||||
**T6 — `list_checks`**
|
||||
Per-entity health checks with verdict, last run, probe output. Filter by entity
|
||||
slug or check state.
|
||||
|
||||
**T7 — `list_executions`**
|
||||
Cursor-paginated execution history. Filter by entity slug, status, risk class.
|
||||
Depends on T3 (execution linkage) for session/entity filtering.
|
||||
|
||||
**T8 — `get_knowledge_revisions`**
|
||||
Version history for a knowledge entity. Agent can see what changed and when.
|
||||
|
||||
**T9 — `get_knowledge_duplicates`**
|
||||
Near-duplicate knowledge entries via trigram clustering. Wraps existing
|
||||
`knowledgeDuplicates` query.
|
||||
|
||||
**T10 — `get_knowledge_orphans`**
|
||||
Knowledge entries not linked to any entity. Agent can suggest cleanup.
|
||||
|
||||
**T11 — `list_knowledge_tags`**
|
||||
All tags with counts. Agent can see the taxonomy.
|
||||
|
||||
**T12 — `list_entity_sessions`** *(Phase 3 polish)*
|
||||
Active Nomos sessions linked to an entity. Depends on T3 (execution linkage).
|
||||
|
||||
**T13 — `find_entities_by`** *(Phase 3 polish)*
|
||||
Search entities by attribute (IP, port, version, tag). More flexible than
|
||||
`list_entities` (type/state only).
|
||||
|
||||
**T14 — MCP resources**
|
||||
Expose entities, knowledge entries, and executions as MCP resource templates:
|
||||
`oikos://entity/{slug}`, `oikos://knowledge/{id}`, `oikos://execution/{id}`.
|
||||
MCP clients that support resources (Claude Desktop, Goose) can browse and attach
|
||||
them to conversations.
|
||||
|
||||
---
|
||||
|
||||
### Stage 3: Nomos reliability (6 tasks)
|
||||
|
||||
Fixes the worst failure modes found in the 5-session audit.
|
||||
|
||||
**T15 — Prevent premature `complete_task(success)`** *(audit Task 1)*
|
||||
`internal/mcp/tools.go`: when `complete_task` with `outcome=success`, verify the
|
||||
summary doesn't contradict known state. If goal mentions a URL but the last
|
||||
probe shows non-200, log a warning.
|
||||
`nomos/SOUL.md`: explicit rule — restate goal, verify every condition before
|
||||
calling success. If any condition is "probably works," use `outcome=partial`.
|
||||
|
||||
**T16 — Plan step integrity: require `replaced_reason`** *(audit Task 4)*
|
||||
Schema: add `replaced_reason TEXT` to `session_plan_steps`.
|
||||
`cmd/nomos`: when agent emits `update_plan_step(status=replaced)`, require
|
||||
non-empty reason (enum: `wrong_diagnosis`, `scope_change`, `blocked`,
|
||||
`superseded`, `operator_override`).
|
||||
`nomos/SOUL.md`: explicit rule — complete or skip steps. Replacing all steps
|
||||
with no reason is a session-quality violation.
|
||||
|
||||
**T17 — Force `propose_plan` on session resume** *(audit Task 5)*
|
||||
`cmd/nomos`: when a session with status `done`/`failed` receives a new user
|
||||
message, reset plan state. Require fresh `propose_plan` before any `run`.
|
||||
The "must have plan" guard treats resumed sessions as plan-less.
|
||||
|
||||
**T18 — Scope gate: surface `session_questions` on context switch** *(audit Task 6)*
|
||||
`cmd/nomos` system prompt: "When investigation leads to a subsystem unrelated to
|
||||
the expressed goal, call `session_questions` before taking action."
|
||||
`nomos/SOUL.md`: explicit rule — ask before pivoting.
|
||||
|
||||
**T19 — Command syntax validation in `run`** *(audit Task 13)*
|
||||
`internal/mcp/server.go`: before executing, reject literal `\n` in commands,
|
||||
backslash-continuation on last line, `head - n`/`grep - i` space-before-flag
|
||||
typos, and `&& \n` patterns from LLM formatting errors.
|
||||
|
||||
**T20 — Stuck-session reaping** *(audit Task 14, from 2026-08-03 plan)*
|
||||
Reap sessions with `closed_at IS NULL` and no message in 30 minutes. Set
|
||||
status=failed, outcome=failure.
|
||||
|
||||
---
|
||||
|
||||
### Stage 4: External agent act (9 tasks)
|
||||
|
||||
Mutation MCP tools. Each writes audit log + emits event. Requires Stage 1
|
||||
infrastructure (T1 target validation, T2 async run, T3 execution linkage).
|
||||
Follows existing direct-DB patterns — no HTTP API calls.
|
||||
|
||||
**T21 — `ack_signal(signal_id)`**
|
||||
Acknowledge an open signal. Agent investigating an alert marks it acknowledged.
|
||||
|
||||
**T22 — `resolve_signal(signal_id, resolution?)`**
|
||||
Resolve a signal with optional resolution note.
|
||||
|
||||
**T23 — `mute_signal(signal_id, duration?)`**
|
||||
Temporarily mute a signal. Optional duration (default 1h).
|
||||
|
||||
**T24 — `cancel_execution(execution_id, reason)`**
|
||||
Cancel a queued/running execution. Depends on T2 (async `run` returns
|
||||
`execution_id`) and T3 (execution linkage for audit context).
|
||||
|
||||
**T25 — `update_check(check_id, enabled)`**
|
||||
Enable/disable a health check. Agent suppresses a noisy probe.
|
||||
|
||||
**T26 — `delete_knowledge(knowledge_id)`**
|
||||
Soft-delete a knowledge entry (move to trash, restorable).
|
||||
|
||||
**T27 — `restore_knowledge(knowledge_id)`**
|
||||
Restore a trashed knowledge entry.
|
||||
|
||||
**T28 — `merge_knowledge(source_id, target_id)`**
|
||||
Fold one knowledge entry into another. Source gets soft-deleted, content
|
||||
appended to target.
|
||||
|
||||
**T29 — `rename_knowledge_tag(old_name, new_name)`**
|
||||
Bulk-rename a tag across all knowledge entries.
|
||||
|
||||
---
|
||||
|
||||
### Stage 5: Close the learning loop (5 tasks)
|
||||
|
||||
Turn execution data into persistent knowledge. Currently all learning tables are
|
||||
empty (0 classifications, 0 feedback, 0 patterns, 0 skills).
|
||||
|
||||
**T30 — Auto-classify every `run` → `classifications` table** *(audit Task 11)*
|
||||
`internal/mcp/server.go`: `run` already calls `classifyCommand`. Write the
|
||||
result to the `classifications` table (risk_class + route + patterns matched).
|
||||
Currently 0 rows despite 1,884 executions.
|
||||
|
||||
**T31 — Auto-upsert knowledge on session close** *(audit Task 7)*
|
||||
`cmd/nomos`: on `complete_task` (any outcome), auto-generate a knowledge entry:
|
||||
title=`<date>: <goal>`, content with Outcome/Root cause/What was done/Unresolved
|
||||
sections, tags=`[session:<id>]`, linked to involved entities.
|
||||
|
||||
**T32 — Auto-feedback on session close** *(audit Task 12)*
|
||||
`cmd/nomos`: on `complete_task`, generate a `feedback` entry: session_id,
|
||||
outcome, observation, lesson, side_effects. Daily cron job reads recent feedback
|
||||
and extracts patterns (recurring root causes, same-fix-applied-multiple-times).
|
||||
|
||||
**T33 — Token tracking** *(audit Task 8)*
|
||||
`cmd/nomos`: after each LLM call, extract `usage.total_tokens` from the response
|
||||
and write to `agent_activity.token_count`. Currently NULL for all rows.
|
||||
|
||||
**T34 — Plan quality metric** *(audit Task 10)*
|
||||
`cmd/nomos`: at session close, compute `completed_steps / total_steps` (currently
|
||||
~38%). Write as session attribute. Track over time to measure impact of T16+T17.
|
||||
|
||||
---
|
||||
|
||||
## 4. Validation
|
||||
|
||||
| Task | Test |
|
||||
|---|---|
|
||||
| T1 | `run("lxc:dns", "qm stop 100")` → error: "qm is a Proxmox host command" |
|
||||
| T2 | `run` with `sleep 45; echo done` → returns `execution_id` immediately; `get_execution_status` eventually shows completed |
|
||||
| T3 | After `run`, `nomos_plan_executions` has row linking session + step + execution |
|
||||
| T4 | `get_dashboard_summary()` returns health counts, signal counts, approval count in one call |
|
||||
| T5 | `get_ontology()` returns entity_types, relationship_types, lifecycle_states |
|
||||
| T6 | `list_checks(entity_slug="lxc:jellyfin")` returns all checks with verdict + last run |
|
||||
| T7 | `list_executions(entity_slug="host:hubris", limit=10)` returns cursor-paginated list |
|
||||
| T8 | `get_knowledge_revisions(id)` returns ordered revision list with timestamps |
|
||||
| T9 | `get_knowledge_duplicates()` returns clusters with similarity scores |
|
||||
| T10 | `get_knowledge_orphans()` returns knowledge entries with zero entity links |
|
||||
| T11 | `list_knowledge_tags()` returns {name, count} for all tags |
|
||||
| T12 | `list_entity_sessions("lxc:jellyfin")` returns active sessions with goal + status |
|
||||
| T13 | `find_entities_by(ip="10.0.0.5")` returns matching entities |
|
||||
| T14 | MCP client can browse `oikos://entity/*` resources |
|
||||
| T15 | Session with goal "make X reachable" where last ping shows 502 → `complete_task(success)` warns or rejects |
|
||||
| T16 | `update_plan_step(status=replaced)` with no reason → rejected |
|
||||
| T17 | Resumed session calls `run` before `propose_plan` → blocked |
|
||||
| T18 | Agent pivots to unrelated subsystem → `session_questions` is called |
|
||||
| T19 | `run` with `head - n /etc/hosts` → rejected with syntax error |
|
||||
| T20 | Session idle for 30+ min with no `closed_at` → reaped (status=failed) |
|
||||
| T21 | `ack_signal(id)` → signal status transitions to acknowledged, audit logged |
|
||||
| T22 | `resolve_signal(id, "fixed DNS")` → resolved with note |
|
||||
| T23 | `mute_signal(id, 3600)` → muted for 1 hour, auto-unmutes |
|
||||
| T24 | `cancel_execution(id, "wrong target")` → execution cancelled, audit logged |
|
||||
| T25 | `update_check(id, false)` → check disabled, scheduler stops probing |
|
||||
| T26 | `delete_knowledge(id)` → soft-deleted (trashed), restorable |
|
||||
| T27 | `restore_knowledge(id)` → restored from trash, reappears in list |
|
||||
| T28 | `merge_knowledge(src, dst)` → src deleted, content appended to dst |
|
||||
| T29 | `rename_knowledge_tag("old", "new")` → all entries updated |
|
||||
| T30 | After any `run`, `classifications` has row with risk_class + route |
|
||||
| T31 | `complete_task` → knowledge entry created automatically with session link |
|
||||
| T32 | `complete_task` → feedback entry created; daily job extracts pattern if same root cause appears ≥3 times |
|
||||
| T33 | `agent_activity.token_count` is non-NULL after LLM call |
|
||||
| T34 | Session close writes `plan_adherence` attribute (% steps completed) |
|
||||
|
||||
---
|
||||
|
||||
## 5. Files touched
|
||||
|
||||
| File | Tasks |
|
||||
|---|---|
|
||||
| `internal/mcp/server.go` | T1, T2, T3, T19, T24, T30 |
|
||||
| `internal/mcp/tools.go` | T4–T14, T21–T29 |
|
||||
| `internal/mcp/discover.go` | (no changes — references for T10/T11 patterns) |
|
||||
| `cmd/nomos/main.go` (or config) | T2 (timeout), T16, T17, T18, T31, T32, T33, T34 |
|
||||
| `nomos/SOUL.md` | T15, T16, T18 |
|
||||
| `internal/httpapi/` | T3 (audit_log.session_id plumbing) |
|
||||
| DB migrations | T16 (replaced_reason column) |
|
||||
|
||||
---
|
||||
|
||||
## 6. What stays WebUI-only
|
||||
|
||||
| Feature | Reason |
|
||||
|---|---|
|
||||
| FleetMap visual graph | Canvas rendering — not an MCP concern |
|
||||
| uPlot metric charts | Raw data available via `query_metrics`/`get_trend` |
|
||||
| Desktop shell, Cluck, App Store | Pure UI layer |
|
||||
| SSE event streaming | MCP has no push model; polling covers it |
|
||||
| Execution log streaming | `get_execution_status` returns latest output |
|
||||
| Nomos session chat | MCP is a tool interface, not a chat agent |
|
||||
| Knowledge wiki editor (revision browse, cleanup UI) | MCP tools expose the data + mutations; UI provides the editing experience |
|
||||
| Approval queue with Approve/Deny buttons | Approvals stay operator-gated via WebUI/Matrix |
|
||||
| Client enrollment flow | Enrollment is IP-gated, not an MCP tool |
|
||||
|
||||
---
|
||||
|
||||
## 7. Out of scope
|
||||
|
||||
- **`approve_execution` / `deny_execution` MCP tools** — dropped. The fix is raising
|
||||
`auto_act` for `reversible_low` (policy change, no code).
|
||||
- **`delete_entity` MCP tool** — separate lifecycle management concern.
|
||||
- **Per-client bearer tokens** — open item tracked in CLIENTS.md. Until they
|
||||
exist, external agents share the same `OIKOS_MCP_BEARER_TOKEN`.
|
||||
- **Exactly-once pattern extraction from feedback** — T32 seeds the pipeline;
|
||||
the full pattern-mining algorithm (TF-IDF clustering, causal inference from
|
||||
event timelines) is future work.
|
||||
@@ -0,0 +1,158 @@
|
||||
# 2026-08-05 — Agent execution safety: QEMU guest agent guardrails + host-mutation gate
|
||||
|
||||
**Status:** Plan.
|
||||
**Context:** ZimaOS NFS recovery session (2026-08-04/05) surfaced three systemic
|
||||
failures in how agents drive oikos mutations. The `run` tool queued an execution
|
||||
against a VM whose QEMU guest agent was down — it sat `pending_approval` forever,
|
||||
never executed, and the agent silently fell back to raw SSH. That same raw-SSH
|
||||
fallback was then used to `apt-get install nfs-kernel-server` directly on the
|
||||
hubris **PVE host**, crashing it and taking the whole homelab subnet down for
|
||||
~15 minutes.
|
||||
|
||||
**Trigger:** Incident `investigation:nomos/incident-hubris-crash-from-nfs-kernel-server-on-pve-host-2026-08-05`
|
||||
(2026-08-05) + session audit. The crash was caused by an agent bypassing the
|
||||
`run` approval gate, which exists precisely to catch that kind of mistake.
|
||||
|
||||
---
|
||||
|
||||
## 1. Motivation
|
||||
|
||||
The OODA loop's Act phase is the security boundary (ADR 0012: "Hermes has no
|
||||
direct SSH access... all mutations go through the execution queue"). This session
|
||||
proved the boundary has three leaks:
|
||||
|
||||
1. **`run` on a VM with a dead QEMU guest agent queues silently.** The execution
|
||||
is classified `config_mutation`, queued for approval, and *never fails* — it
|
||||
just sits in `pending_approval` while the agent assumes progress. There is no
|
||||
feedback that the underlying transport (`qm guest exec`) cannot work.
|
||||
|
||||
2. **No guardrail against host-level package/kernel mutations.** `apt-get install`
|
||||
targeting a `proxmox-host` entity is classified `config_mutation` and gated —
|
||||
*if* the agent routes it through `run`. When the first `run` call stalls
|
||||
(leak #1), the agent falls back to raw SSH, which has no classification at all.
|
||||
The crash was the direct result of that fallback.
|
||||
|
||||
3. **Agents are trusted to self-report the `health` attribute.** `update_entity_attributes`
|
||||
let the agent set `health:"healthy"` on `lxc:nfs-export`, which derived 4
|
||||
spurious health checks. Health is scheduler-owned; agents shouldn't write it.
|
||||
|
||||
---
|
||||
|
||||
## 2. Changes
|
||||
|
||||
### I — `run` pre-flights the execution transport before queueing
|
||||
|
||||
**Why:** A queued execution that can never run is worse than a failed one — it
|
||||
looks like progress, stalls the agent, and (this session) pushed the agent into
|
||||
the unsafe raw-SSH fallback.
|
||||
|
||||
**What:**
|
||||
|
||||
In the `run` tool handler (`internal/mcp/`), before inserting the execution row:
|
||||
|
||||
- If target type is `vm`, read the target entity's attributes. If
|
||||
`qemu_guest_agent` is missing or `not_running`, return an immediate error:
|
||||
`"run on vm:zimaos blocked: QEMU guest agent is not running (qm guest exec
|
||||
unavailable). Start the agent first or use a different target."`
|
||||
- Same check for `lxc` targets whose `pct exec` path is known-broken (optional —
|
||||
start with VM only).
|
||||
|
||||
This converts "queued forever" into a fast, actionable failure the agent can
|
||||
recover from immediately.
|
||||
|
||||
**Risk class:** read-only (validation only, no execution row created).
|
||||
|
||||
**Test:** Unit test with a fake VM entity that has `qemu_guest_agent: not_running`
|
||||
→ assert the tool returns the blocking error and inserts no execution row.
|
||||
|
||||
### II — Host-mutation command guardrail in `run` classification
|
||||
|
||||
**Why:** `apt-get install` on a Proxmox host is the exact class of mutation that
|
||||
must always hit the approval gate. The classifier already escalates `apt`/kernel
|
||||
touches; this makes the escalation explicit and documented so agents stop
|
||||
second-guessing it.
|
||||
|
||||
**What:**
|
||||
|
||||
- In `seeds/policy.yaml`, add an explicit rule: `proxmox-host` targets + commands
|
||||
matching `(apt-get install|apt install|dpkg|modprobe|kernel)` ⇒ `config_mutation`
|
||||
(operator approval required), never `reversible_low`.
|
||||
- Extend the classifier to also flag `update-rc.d`, `systemctl enable` on host
|
||||
targets if not already covered.
|
||||
- Add a note in the `run` tool description: "Host-level package/kernel mutations
|
||||
always require operator approval."
|
||||
|
||||
**Risk class:** policy change — knowledge/DB, deploy via seed ingest.
|
||||
|
||||
**Test:** `classify_command("apt-get install -y nfs-kernel-server", declared_risk=read_only)`
|
||||
on `host:hubris` → must return `config_mutation`, not read_only. Add a fixture test
|
||||
in the classifier suite.
|
||||
|
||||
### III — `health` attribute is read-only for agents
|
||||
|
||||
**Why:** This session's `update_entity_attributes({"health":"healthy"})` on
|
||||
`lxc:nfs-export` derived 4 checks. Health is computed by the scheduler from probe
|
||||
results; an agent asserting it creates false monitoring.
|
||||
|
||||
**What:**
|
||||
|
||||
- In `update_entity_attributes` handler: reject (or strip with a warning) the
|
||||
`health` key. Return a message: `"health is scheduler-owned; attribute ignored.
|
||||
Use get_health_summary/list_checks to observe it."`
|
||||
- Document in the tool description: `"Do not write health — it is derived from
|
||||
probes."`
|
||||
|
||||
**Risk class:** knowledge-graph mutation (existing), no infra impact.
|
||||
|
||||
**Test:** `update_entity_attributes(slug=lxc:nfs-export, attributes={"health":"healthy"})`
|
||||
→ response shows health ignored, other keys merged.
|
||||
|
||||
### IV — Agent-side: record discovered dependency edges
|
||||
|
||||
**Why:** The session discovered `vm:zimaos` depends on `lxc:nfs-export` for
|
||||
`/media/library`, but no `depends-on` edge was recorded. A future agent
|
||||
investigating a ZimaOS mount failure would have no graph signal pointing at the
|
||||
NFS server.
|
||||
|
||||
**What (agent behaviour, not code):** After confirming a runtime dependency, call
|
||||
`create_relationship(source, target, type)` immediately. Concretely this session:
|
||||
`create_relationship("vm:zimaos", "lxc:nfs-export", "depends-on")`.
|
||||
|
||||
**Where to enforce:** Update the homelab-context `HERMES.md` / SOUL.md agent
|
||||
instructions with a one-line rule: "When you discover a dependency between two
|
||||
entities (a service consumes another's export/mount/API), record it with
|
||||
`create_relationship` in the same session." Plus a runbook in
|
||||
`.agents/skills/` if one doesn't exist.
|
||||
|
||||
**Risk class:** knowledge-graph mutation, auto-approves.
|
||||
|
||||
**Test:** Manual — after recording the edge, `get_relations("vm:zimaos")` shows
|
||||
`depends-on → lxc:nfs-export`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Rollout
|
||||
|
||||
| Step | Item | When |
|
||||
| ---- | ---- | ---- |
|
||||
| 1 | II — policy.yaml classifier rule + tests | next seed ingest |
|
||||
| 2 | I — `run` VM transport pre-flight + test | next mcp server deploy |
|
||||
| 3 | III — health read-only guard + test | same deploy as I |
|
||||
| 4 | IV — agent instruction update in homelab-context | commit + sync |
|
||||
| 5 | Verify: re-run `classify_command` + manual `run` on vm:zimaos (agent now up) | after deploy |
|
||||
|
||||
---
|
||||
|
||||
## 4. Out of scope
|
||||
|
||||
- Per-client MCP bearer tokens (separate track, ADR 0012 note).
|
||||
- `request_execution` re-introduction — the unified `run` primitive stays.
|
||||
- Automating the Technitium DHCP reservation UI (this session's leftover — the
|
||||
reservation for `BC:24:11:22:C2:F2 → 192.168.8.102` was added manually in the
|
||||
web UI; consider a `runbook:technitium-dhcp-reservation` doc in a follow-up).
|
||||
|
||||
---
|
||||
|
||||
## 5. Changelog
|
||||
|
||||
- 2026-08-05 — plan created from ZimaOS/NFS session audit + hubris crash incident.
|
||||
@@ -17,11 +17,10 @@ went sideways, open an investigation.
|
||||
| 2026-07-14 | [Activity gaps](2026-07-14-activity-gaps.md) | In Progress |
|
||||
| 2026-07-14 | [Activity timeline](2026-07-14-activity-timeline.md) | In Progress |
|
||||
| 2026-07-17 | [Codebase review, lint audit, and documentation maintenance](2026-07-17-codebase-review-and-cleanup.md) | Report delivered — doc/tooling fixes applied; code refactors pending |
|
||||
| 2026-07-18 | [Session review: three recent sessions](2026-07-18-session-review-three-sessions.md) | Implemented in v0.7.12 — P0.1/P0.2/P1.3/P1.4/P1.5/P1.6/P1.8/P2.10; P1.7 and P2.9 deferred (retry cap covers) |
|
||||
| 2026-07-20 | [Desktop mascot ("Cluck")](2026-07-20-desktop-mascot.md) | Implemented in v0.8.0 — see deviation note; physics/window-interaction follow-ups tracked separately |
|
||||
| 2026-07-20 | [Session review: past 10 sessions](2026-07-20-session-review-ten-sessions.md) | Implemented in v0.7.13 — all P0/P1/P2 items landed |
|
||||
| 2026-07-20 | [Mascot physics/window-interaction audit](2026-07-20-mascot-physics-audit.md) | P0–P2 implemented; P3 ("cool stuff") ideas open |
|
||||
| 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready |
|
||||
| 2026-08-04 | [Hermes MCP client integration](done/2026-08-04-hermes-mcp-client-integration.md) | Done — deployed |
|
||||
| 2026-08-05 | [Agent execution safety: QEMU guest agent gate + host-mutation guard](done/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md) | Done — implemented (1b9c761) |
|
||||
|
||||
## Done
|
||||
|
||||
@@ -59,10 +58,21 @@ See [`done/`](done/) for executed plans:
|
||||
| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](done/2026-07-14-post-fix-session-remainders.md) |
|
||||
| 2026-07-15 | [Plan-first and iteration](done/2026-07-15-plan-first-and-iteration.md) |
|
||||
| 2026-07-15 | [WhatsApp session audit](done/2026-07-15-whatsapp-session-audit.md) |
|
||||
| 2026-07-18 | [Session review: three recent sessions](done/2026-07-18-session-review-three-sessions.md) |
|
||||
| 2026-07-20 | [Desktop mascot ("Cluck")](done/2026-07-20-desktop-mascot.md) |
|
||||
| 2026-07-20 | [Session review: past 10 sessions](done/2026-07-20-session-review-ten-sessions.md) |
|
||||
| 2026-07-21 | [Chat window full polish](done/2026-07-21-chat-full-polish.md) |
|
||||
| 2026-07-29 | [Make health reflect reality + complete the knowledge graph](done/2026-07-29-health-check-reality-and-knowledge-graph.md) |
|
||||
| 2026-07-30 | [Session review: plan drift & dead activity panel](done/2026-07-30-session-review-plan-drift-and-dead-activity-panel.md) |
|
||||
| 2026-08-03 | [Nomos chat changes review (P0/P1/P2)](done/2026-08-03-nomos-chat-changes-review.md) |
|
||||
| 2026-08-03 | [Nomos chat: reliability & predictability audit](done/2026-08-03-nomos-chat-reliability-and-ux-audit.md) |
|
||||
| 2026-08-03 | [Adopt cyberspace.online terminal aesthetic + dithered images](done/2026-08-03-cyberspace-style-adoption.md) |
|
||||
| 2026-08-03 | [Nomos chat: working-visibility, message queue, generation-aware timeline](done/2026-08-03-nomos-chat-working-visibility.md) |
|
||||
| 2026-08-04 | [Chat interaction overhaul: inline progressive stream + thinking blocks](done/2026-08-04-chat-window-overhaul.md) |
|
||||
|
||||
## Conventions
|
||||
|
||||
- File name: `YYYY-MM-DD-<slug>.md`. Use the *target* date if known, otherwise the planning date.
|
||||
- Status: `Planned` → `In Progress` → `Done` (move to `done/` on completion).
|
||||
- When done: add a changelog entry on every affected node page, then move the file to `done/`.
|
||||
- Plans are append-only once execution starts — don't rewrite pre-flight intent after the fact.
|
||||
- Plans are append-only once execution starts — don't rewrite pre-flight intent after the fact.
|
||||
70
scripts/cleanup-orphan-checks.sh
Executable file
70
scripts/cleanup-orphan-checks.sh
Executable file
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env bash
|
||||
# cleanup-orphan-checks.sh — remove orphan check entities + check_defs.
|
||||
#
|
||||
# These are leftovers from the old shortSlug() collision bug: check entities
|
||||
# with truncated 8-hex slugs (e.g. check:ssh-script:0d31fdd1) that have no
|
||||
# live target and are disabled. They pollute the entity table and the checks
|
||||
# view. The audit_knowledge_graph tool reports them as `orphan_checks`.
|
||||
#
|
||||
# Risk class: config_mutation (deletes rows). DRY-RUN by default; pass --apply
|
||||
# to actually delete. Review the listed slugs first — they must all match the
|
||||
# legacy random-slug pattern and be disabled.
|
||||
#
|
||||
# Usage:
|
||||
# cleanup-orphan-checks.sh # dry-run: list what would be deleted
|
||||
# cleanup-orphan-checks.sh --apply # delete check_defs rows, then entities
|
||||
#
|
||||
# Connects via the OIKOS_TEST... no — via the running postgres container by
|
||||
# default, or OIKOS_PSQL if set.
|
||||
set -euo pipefail
|
||||
|
||||
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
|
||||
PATTERN='^check:(ping|ssh-script|disk):[0-9a-f]{8}$'
|
||||
|
||||
# Orphan = matches the legacy random-slug pattern AND has no enabled check_def
|
||||
# pointing at a real target. A random-slug check that IS enabled and has a live
|
||||
# target is a working check with a bad slug — keep it (deleting would drop
|
||||
# monitoring), and flag it for a slug fix instead.
|
||||
ORPHAN_PRED="e.type='check' AND e.slug ~ '$PATTERN'
|
||||
AND NOT EXISTS (SELECT 1 FROM check_defs cd
|
||||
WHERE cd.entity_id = e.id AND cd.enabled AND cd.target_id IS NOT NULL)"
|
||||
|
||||
echo "== orphan checks matching /$PATTERN/ (no enabled check_def w/ target) =="
|
||||
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
|
||||
|
||||
echo "== details (slug, state, enabled) =="
|
||||
$PSQL_CMD -F ' | ' -Ac "
|
||||
SELECT e.slug, COALESCE(e.state,'(null)'),
|
||||
COALESCE((SELECT cd.enabled::text FROM check_defs cd WHERE cd.entity_id=e.id LIMIT 1),'no-check_def')
|
||||
FROM entities e
|
||||
WHERE $ORPHAN_PRED
|
||||
ORDER BY e.slug;" | head -60
|
||||
|
||||
if [ "${1:-}" != "--apply" ]; then
|
||||
echo
|
||||
echo "DRY RUN — no rows deleted. Re-run with --apply to delete:"
|
||||
echo " check_defs whose check entity is an orphan, then those entities."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "== applying (config_mutation) =="
|
||||
# check_defs first, then the dependent rows an entity owns (status, signals,
|
||||
# metrics), then the orphan check entities. FKs prevent a plain entity delete.
|
||||
$PSQL_CMD -v ON_ERROR_STOP=1 <<SQL
|
||||
BEGIN;
|
||||
DELETE FROM check_defs WHERE entity_id IN (SELECT id FROM entities e WHERE $ORPHAN_PRED);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM metric_samples WHERE entity_id IN (SELECT id FROM ids);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM signals WHERE entity_id IN (SELECT id FROM ids) OR target_entity_id IN (SELECT id FROM ids);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM entity_status WHERE entity_id IN (SELECT id FROM ids);
|
||||
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||
DELETE FROM relationships WHERE source_id IN (SELECT id FROM ids) OR target_id IN (SELECT id FROM ids);
|
||||
DELETE FROM entities e WHERE $ORPHAN_PRED;
|
||||
COMMIT;
|
||||
SQL
|
||||
|
||||
echo "== remaining orphans (should be 0) =="
|
||||
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
|
||||
79
scripts/fix-zimaos-nfs.sh
Normal file
79
scripts/fix-zimaos-nfs.sh
Normal file
@@ -0,0 +1,79 @@
|
||||
#!/bin/bash
|
||||
# Fix NFS mounts on ZimaOS — both library shares
|
||||
# Run: bash /tmp/fix-zimaos-nfs.sh
|
||||
# You'll be prompted for sudo password once
|
||||
|
||||
set -e
|
||||
|
||||
echo "=== Step 1: Grant passwordless sudo for mount/umount ==="
|
||||
echo "dtoro ALL=(ALL) NOPASSWD: /usr/sbin/mount.nfs, /usr/sbin/umount.nfs, /bin/mount, /bin/umount, /usr/bin/mount" | sudo tee /etc/sudoers.d/zimaos-nfs > /dev/null
|
||||
sudo chmod 440 /etc/sudoers.d/zimaos-nfs
|
||||
echo " ✓ sudoers drop-in created"
|
||||
|
||||
echo "=== Step 2: Mount library from hubris (nfs-export LXC) ==="
|
||||
sudo mkdir -p /media/library
|
||||
sudo mount -t nfs -o nfsvers=4,rw,hard,intr 192.168.8.200:/mnt/library /media/library
|
||||
echo " ✓ /media/library ← 192.168.8.200:/mnt/library"
|
||||
|
||||
echo "=== Step 3: Mount ludo-library from strong ==="
|
||||
sudo mkdir -p /media/ludo-library
|
||||
sudo mount -t nfs -o nfsvers=4,rw,hard,intr 192.168.8.241:/mnt/media_local /media/ludo-library
|
||||
echo " ✓ /media/ludo-library ← 192.168.8.241:/mnt/media_local"
|
||||
|
||||
echo "=== Step 4: Verify ==="
|
||||
echo ""
|
||||
df -h | grep -E 'nfs|192.168'
|
||||
echo ""
|
||||
echo "--- /media/library contents ---"
|
||||
ls /media/library/ | head -10
|
||||
echo ""
|
||||
echo "--- /media/ludo-library contents ---"
|
||||
ls /media/ludo-library/ | head -10
|
||||
|
||||
echo ""
|
||||
echo "=== Step 5: Persistent systemd mount units ==="
|
||||
|
||||
# Library mount unit
|
||||
sudo tee /etc/systemd/system/media-library.mount > /dev/null << 'MOUNTUNIT'
|
||||
[Unit]
|
||||
Description=Mount nfs-export:/mnt/library as library
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Mount]
|
||||
What=192.168.8.200:/mnt/library
|
||||
Where=/media/library
|
||||
Type=nfs
|
||||
Options=nfsvers=4,rw,hard,intr
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
MOUNTUNIT
|
||||
|
||||
# Ludo-library mount unit
|
||||
sudo tee /etc/systemd/system/media-ludo\x2dlibrary.mount > /dev/null << 'MOUNTUNIT2'
|
||||
[Unit]
|
||||
Description=Mount strong:/mnt/media_local as ludo-library
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Mount]
|
||||
What=192.168.8.241:/mnt/media_local
|
||||
Where=/media/ludo-library
|
||||
Type=nfs
|
||||
Options=nfsvers=4,rw,hard,intr
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
MOUNTUNIT2
|
||||
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable media-library.mount
|
||||
sudo systemctl enable media-ludo\x2dlibrary.mount
|
||||
echo " ✓ Both systemd mount units created and enabled"
|
||||
|
||||
echo ""
|
||||
echo "=== DONE ==="
|
||||
echo "Both NFS mounts active and persistent across reboots:"
|
||||
echo " /media/library ← 192.168.8.200:/mnt/library (hubris nfs-export LXC)"
|
||||
echo " /media/ludo-library ← 192.168.8.241:/mnt/media_local (strong)"
|
||||
31
scripts/report-stray-test-lxcs.sh
Executable file
31
scripts/report-stray-test-lxcs.sh
Executable file
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env bash
|
||||
# report-stray-test-lxcs.sh — list leftover test/scratch LXC entities.
|
||||
#
|
||||
# Provisioning experiments leave active `lxc:test-*` / `lxc:preflight-*`
|
||||
# entities in the graph long after the containers are gone or repurposed.
|
||||
# They generate checks and pollute health/graph views. This reports them and
|
||||
# their DB state + which Proxmox host each is parented on, so the operator can
|
||||
# confirm the container is really gone and retire the entity via the
|
||||
# lifecycle-destroy-node runbook (a destructive, approval-gated action).
|
||||
#
|
||||
# Read-only. Pair with: lifecycle-destroy-node (mark destroyed) or
|
||||
# lifecycle-deprecate-node.
|
||||
set -euo pipefail
|
||||
|
||||
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
|
||||
|
||||
echo "== stray test/scratch LXC entities =="
|
||||
$PSQL_CMD -F ' | ' -Ac "
|
||||
SELECT e.slug, COALESCE(e.state,'active') AS state,
|
||||
e.attributes->>'pve_id' AS pve_id,
|
||||
COALESCE(h.slug,'(no host)') AS host
|
||||
FROM entities e
|
||||
LEFT JOIN relationships r ON r.target_id = e.id AND r.type='hosts' AND r.valid_to IS NULL
|
||||
LEFT JOIN entities h ON h.id = r.source_id
|
||||
WHERE e.type='lxc' AND e.slug ~ '^lxc:(test|preflight)'
|
||||
ORDER BY e.slug;"
|
||||
|
||||
echo
|
||||
echo "Next: for each, confirm the container is gone in Proxmox (pct list on its"
|
||||
echo "host), then retire via lifecycle-destroy-node (destructive) or mark"
|
||||
echo "deprecated. If a container still exists, pct destroy it first."
|
||||
@@ -87,6 +87,12 @@ entities:
|
||||
mesh: {netbird: {ip: 100.122.165.149, fqdn: netbird-ionos.netbird.selfhosted}}
|
||||
ssh: {user: root}
|
||||
note: netbird mgmt+signal+relay+dashboard + coturn; sshd locked to hubris pubkey
|
||||
monitoring: [http] # public HTTPS probe via https://mcp.hubris.network
|
||||
# (2026-08-05: was `none` — the VPS went silent for
|
||||
# 7 days because nothing probed it. The standalone-server
|
||||
# type inherits [ping,resource,updates] from machine, but
|
||||
# SSH/ICMP don't reach it from the lab; an HTTP probe on
|
||||
# the public endpoint is the reachable liveness signal).
|
||||
- slug: "ws:mac-mini"
|
||||
type: workstation
|
||||
name: mac-mini
|
||||
@@ -238,8 +244,8 @@ entities:
|
||||
doc_page: knowledge/wiki/containers/101-jellyfin.md,
|
||||
risk_notes: "native Authentik OIDC (no forward-auth gate); VAAPI depends on GPU passthrough on strong"}}
|
||||
- {slug: "service:nextcloud", type: service, name: nextcloud,
|
||||
attributes: {url: "https://cloud.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
|
||||
attributes: {url: "https://cloud.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
|
||||
- {slug: "service:paperless", type: service, name: paperless,
|
||||
attributes: {url: "https://paperless.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/103-paperless.md,
|
||||
@@ -249,8 +255,8 @@ entities:
|
||||
doc_page: knowledge/wiki/containers/118-elementsynapse.md,
|
||||
risk_notes: "alert/approval channel for Oikos — outage silences agent escalation"}}
|
||||
- {slug: "service:photos", type: service, name: photos,
|
||||
attributes: {url: "https://photos.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/120-mule-images.md}}
|
||||
attributes: {url: "https://photos.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/120-mule-images.md}}
|
||||
- {slug: "service:arr-stack", type: service, name: arr_stack,
|
||||
attributes: {doc_page: knowledge/wiki/containers/122-arriman.md,
|
||||
note: "jellyseerr / qbit / sab on docker compose"}}
|
||||
@@ -265,11 +271,21 @@ entities:
|
||||
attributes: {url: "https://zimaos.hubris.network",
|
||||
doc_page: knowledge/wiki/vms/100-zimaos.md}}
|
||||
- {slug: "service:haos", type: service, name: haos,
|
||||
attributes: {doc_page: knowledge/wiki/vms/108-haos.md}}
|
||||
attributes: {doc_page: knowledge/wiki/vms/108-haos.md,
|
||||
monitoring: none}} # redundant: vm:haos covers liveness via vm-status; haos blocks SSH so a process check can't reach it
|
||||
- {slug: "service:teddycloud", type: service, name: teddycloud,
|
||||
attributes: {url: "https://teddy.hubris.network",
|
||||
doc_page: knowledge/wiki/containers/131-teddycloud.md,
|
||||
risk_notes: "no forward-auth gate — reachable by anyone on LAN/mesh"}}
|
||||
# The Go control plane itself: api/scheduler/notifier/web on the mac-mini,
|
||||
# and what mcp.hubris.network fronts since the cutover. It existed in the
|
||||
# database (created outside the seed) but was never declared here, so a
|
||||
# fresh seed could not resolve the routes-to edge below.
|
||||
- {slug: "service:oikos", type: service, name: oikos,
|
||||
attributes: {url: "https://oikos.hubris.network",
|
||||
host: "ws:mac-mini",
|
||||
ports: {api: 8090, web: 8091, nomos_gateway: 8092},
|
||||
note: "homelab automation platform — api/scheduler/notifier/web on mac-mini docker compose (project name oikos)"}}
|
||||
- {slug: "service:homelab-mcp", type: service, name: homelab_mcp,
|
||||
attributes: {port: 9810, systemd_unit: homelab-mcp,
|
||||
endpoint: "https://mcp.hubris.network/mcp",
|
||||
@@ -330,6 +346,32 @@ entities:
|
||||
- {slug: "ingress:sab.hubris.network", type: ingress-route, name: sab.hubris.network,
|
||||
attributes: {forward_auth: true}}
|
||||
|
||||
# ─── TLS certificates (Caddy-managed, *.hubris.network) ───────────
|
||||
# Each cert's expiry is probed by dialing Caddy's lab IP (`dial`) with SNI
|
||||
# set to the hostname — the scheduler container has no mesh/split-horizon
|
||||
# DNS, so it can't resolve *.hubris.network, but it CAN reach Caddy on the
|
||||
# lab LAN.
|
||||
- {slug: "cert:proxmox.hubris.network", type: certificate, name: proxmox.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:git.hubris.network", type: certificate, name: git.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:auth.hubris.network", type: certificate, name: auth.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:media.hubris.network", type: certificate, name: media.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:cloud.hubris.network", type: certificate, name: cloud.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:paperless.hubris.network", type: certificate, name: paperless.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:matrix.hubris.network", type: certificate, name: matrix.hubris.network}
|
||||
- {slug: "cert:photos.hubris.network", type: certificate, name: photos.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:artifacto.hubris.network", type: certificate, name: artifacto.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:trmnl.hubris.network", type: certificate, name: trmnl.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:zimaos.hubris.network", type: certificate, name: zimaos.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:teddy.hubris.network", type: certificate, name: teddy.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:mcp.hubris.network", type: certificate, name: mcp.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:house.hubris.network", type: certificate, name: house.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:books.hubris.network", type: certificate, name: books.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:seanime.hubris.network", type: certificate, name: seanime.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:roms.hubris.network", type: certificate, name: roms.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:jellyseerr.hubris.network", type: certificate, name: jellyseerr.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:qbit.hubris.network", type: certificate, name: qbit.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
- {slug: "cert:sab.hubris.network", type: certificate, name: sab.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||
|
||||
# ─── Governance ────────────────────────────────────────────────────
|
||||
- {slug: "person:dtoro", type: person, name: dtoro,
|
||||
attributes: {matrix_id: "@dtoro:avispero"}}
|
||||
@@ -451,7 +493,54 @@ relationships:
|
||||
# fronts the Go api. Nomos recorded this correctly on 2026-07-12; the seed
|
||||
# was the stale one, and re-asserting the old edge alongside it is what made
|
||||
# ingress:mcp a cardinality violation.
|
||||
- {source: "ws:mac-mini", target: "service:oikos", type: provides}
|
||||
- {source: "ingress:mcp.hubris.network", target: "service:oikos", type: routes-to}
|
||||
# Every public hostname is terminated by caddy. Without these the
|
||||
# reverse proxy — the single widest point of failure in the lab —
|
||||
# had a blast radius of one.
|
||||
- {source: "ingress:proxmox.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:git.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:auth.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:media.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:cloud.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:paperless.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:matrix.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:photos.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:artifacto.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:trmnl.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:zimaos.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:teddy.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:mcp.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:secrets.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:house.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:books.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:seanime.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:roms.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:jellyseerr.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:qbit.hubris.network", target: "service:caddy", type: served-by}
|
||||
- {source: "ingress:sab.hubris.network", target: "service:caddy", type: served-by}
|
||||
# Each public route is served with its TLS certificate.
|
||||
- {source: "ingress:proxmox.hubris.network", target: "cert:proxmox.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:git.hubris.network", target: "cert:git.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:auth.hubris.network", target: "cert:auth.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:media.hubris.network", target: "cert:media.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:cloud.hubris.network", target: "cert:cloud.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:paperless.hubris.network", target: "cert:paperless.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:matrix.hubris.network", target: "cert:matrix.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:photos.hubris.network", target: "cert:photos.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:artifacto.hubris.network", target: "cert:artifacto.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:trmnl.hubris.network", target: "cert:trmnl.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:zimaos.hubris.network", target: "cert:zimaos.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:teddy.hubris.network", target: "cert:teddy.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:mcp.hubris.network", target: "cert:mcp.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:house.hubris.network", target: "cert:house.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:books.hubris.network", target: "cert:books.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:seanime.hubris.network", target: "cert:seanime.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:roms.hubris.network", target: "cert:roms.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:jellyseerr.hubris.network", target: "cert:jellyseerr.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:qbit.hubris.network", target: "cert:qbit.hubris.network", type: uses-certificate}
|
||||
- {source: "ingress:sab.hubris.network", target: "cert:sab.hubris.network", type: uses-certificate}
|
||||
|
||||
- {source: "ingress:secrets.hubris.network", target: "service:secrets-issuance", type: routes-to}
|
||||
- {source: "ingress:house.hubris.network", target: "service:house", type: routes-to}
|
||||
- {source: "ingress:books.hubris.network", target: "service:grimmory", type: routes-to}
|
||||
|
||||
@@ -6372,6 +6372,23 @@ investigations:
|
||||
tags:
|
||||
- investigation
|
||||
runbooks:
|
||||
- slug: knowledge-graph-audit
|
||||
name: Knowledge-graph audit
|
||||
risk_class: read_only
|
||||
entity_type: entity
|
||||
procedure: {}
|
||||
content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\
|
||||
\ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\
|
||||
\ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\
|
||||
\ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\
|
||||
\ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\
|
||||
\ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\
|
||||
\ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\
|
||||
\ each with `get_entity`/`get_relations` before acting. This skill makes no changes\
|
||||
\ \u2014 route confirmed findings to their remediation runbook and re-run the audit\
|
||||
\ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\
|
||||
\ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\
|
||||
\ follow-up until that machinery lands.\n"
|
||||
- slug: client-enrollment
|
||||
name: Client enrollment
|
||||
risk_class: read_only
|
||||
|
||||
@@ -241,6 +241,11 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Machine outside PVE management (e.g. external VPS).
|
||||
# Override inherited [ping, resource, updates] — standalone servers may
|
||||
# not be SSH/ICMP-reachable from the scheduler. HTTP is the least-
|
||||
# common-denominator liveness signal. Entities with full SSH access
|
||||
# can override to [ping, resource, http].
|
||||
monitoring: [http]
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -272,7 +277,9 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: Virtual machine.
|
||||
monitoring: [ping] # no guest agent assumed; reachability only
|
||||
monitoring: [vm-status] # `qm status` from the host: powered-on liveness
|
||||
# that works even when the VM blocks ICMP and has
|
||||
# no guest agent (haos). ping is unreliable for VMs.
|
||||
attributes:
|
||||
type: object
|
||||
properties:
|
||||
@@ -379,8 +386,13 @@ entity_types:
|
||||
layer: infrastructure
|
||||
lifecycle: infrastructure
|
||||
description: DNS zone (e.g. split-horizon hubris.network).
|
||||
monitoring: [dns] # NOTE: no `dns` checker exists yet — this is a
|
||||
# real gap and coverageSweep will report it
|
||||
monitoring: [dns] # resolves the zone's apex via the configured
|
||||
# resolver, verifying the zone is authoritatively
|
||||
# reachable. (2026-08-05: was `none` — the DNS layer
|
||||
# had zero checks, so a stale record like
|
||||
# matrix→82.165.190.79 went unnoticed. Requires
|
||||
# ontology re-ingest; coverageSweep clears stale
|
||||
# signals after.)
|
||||
attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
|
||||
dns-record:
|
||||
parent: entity
|
||||
@@ -699,36 +711,42 @@ relationship_types:
|
||||
target: compute-entity
|
||||
cardinality: one-to-many
|
||||
description: Machine hosts a VM/container (hubris hosts lxc:apps).
|
||||
blast_direction: forward
|
||||
runs-hypervisor:
|
||||
inverse: hypervisor-on
|
||||
source: machine
|
||||
target: hypervisor
|
||||
cardinality: one-to-one
|
||||
description: Machine runs hypervisor software.
|
||||
blast_direction: forward
|
||||
member-of:
|
||||
inverse: has-member
|
||||
source: proxmox-host
|
||||
target: cluster
|
||||
cardinality: many-to-one
|
||||
description: PVE host belongs to a cluster.
|
||||
blast_direction: backward
|
||||
part-of:
|
||||
inverse: comprises
|
||||
source: docker-container
|
||||
target: compose-stack
|
||||
cardinality: many-to-one
|
||||
description: Docker container belongs to a compose stack.
|
||||
blast_direction: backward
|
||||
provides:
|
||||
inverse: provided-by
|
||||
source: compute-entity
|
||||
target: service
|
||||
cardinality: one-to-many
|
||||
description: Compute entity provides a service (lxc:gitea provides service:gitea).
|
||||
blast_direction: forward
|
||||
runs:
|
||||
inverse: run-by
|
||||
source: service
|
||||
target: application
|
||||
cardinality: one-to-many
|
||||
description: Service runs an application.
|
||||
blast_direction: backward
|
||||
configured-by:
|
||||
inverse: configures
|
||||
source: entity
|
||||
@@ -741,36 +759,52 @@ relationship_types:
|
||||
target: entity
|
||||
cardinality: many-to-one
|
||||
description: Pipeline deploys to a service/host.
|
||||
blast_direction: forward
|
||||
routes-to:
|
||||
inverse: routed-via
|
||||
source: ingress-route
|
||||
target: service
|
||||
cardinality: many-to-one
|
||||
description: Public hostname routes to a service.
|
||||
blast_direction: backward
|
||||
served-by:
|
||||
inverse: serves
|
||||
source: ingress-route
|
||||
target: service
|
||||
cardinality: many-to-one
|
||||
description: Ingress route is terminated by this reverse proxy. Distinct
|
||||
from routes-to, which names the BACKEND the route forwards to — without
|
||||
this edge the proxy's blast radius is invisible, and lxc:caddy reported
|
||||
one affected entity despite terminating every *.hubris.network route.
|
||||
blast_direction: backward
|
||||
secured-by:
|
||||
inverse: secures
|
||||
source: ingress-route
|
||||
target: identity-provider
|
||||
cardinality: many-to-one
|
||||
description: Route gated by forward-auth.
|
||||
blast_direction: backward
|
||||
uses-certificate:
|
||||
inverse: certifies
|
||||
source: ingress-route
|
||||
target: certificate
|
||||
cardinality: many-to-one
|
||||
description: Route served with this certificate.
|
||||
blast_direction: backward
|
||||
authenticates-via:
|
||||
inverse: authenticates-service
|
||||
source: service
|
||||
target: identity-provider
|
||||
cardinality: many-to-one
|
||||
description: Service uses native OIDC (jellyfin authenticates-via authentik).
|
||||
blast_direction: backward
|
||||
in-zone:
|
||||
inverse: contains-record
|
||||
source: dns-record
|
||||
target: dns-zone
|
||||
cardinality: many-to-one
|
||||
description: Record belongs to a zone.
|
||||
blast_direction: backward
|
||||
resolves-to:
|
||||
inverse: resolved-from
|
||||
source: dns-record
|
||||
@@ -783,24 +817,28 @@ relationship_types:
|
||||
target: service
|
||||
cardinality: many-to-many
|
||||
description: Runtime dependency (blast-radius edge).
|
||||
blast_direction: backward
|
||||
connects-via:
|
||||
inverse: connects
|
||||
source: compute-entity
|
||||
target: network
|
||||
cardinality: many-to-many
|
||||
description: Coarse network membership (host on LAN / mesh).
|
||||
blast_direction: backward
|
||||
has-interface:
|
||||
inverse: interface-of
|
||||
source: compute-entity
|
||||
target: network-interface
|
||||
cardinality: one-to-many
|
||||
description: Optional per-interface refinement.
|
||||
blast_direction: backward
|
||||
interface-on:
|
||||
inverse: has-endpoint
|
||||
source: network-interface
|
||||
target: network
|
||||
cardinality: many-to-one
|
||||
description: Interface attaches to a network.
|
||||
blast_direction: backward
|
||||
|
||||
# Storage
|
||||
mounts:
|
||||
@@ -810,24 +848,28 @@ relationship_types:
|
||||
cardinality: many-to-many
|
||||
description: Compute entity mounts a volume. Edge attributes carry
|
||||
mount_point and options.
|
||||
blast_direction: backward
|
||||
stores-on:
|
||||
inverse: stores-for
|
||||
source: compute-entity
|
||||
target: storage-pool
|
||||
cardinality: many-to-many
|
||||
description: Rootfs/data lives on a pool.
|
||||
blast_direction: backward
|
||||
contains:
|
||||
inverse: contained-in
|
||||
source: storage-pool
|
||||
target: volume
|
||||
cardinality: one-to-many
|
||||
description: Pool contains a volume.
|
||||
blast_direction: forward
|
||||
holds-dataset:
|
||||
inverse: dataset-on
|
||||
source: volume
|
||||
target: dataset
|
||||
cardinality: one-to-many
|
||||
description: Volume holds a tracked dataset.
|
||||
blast_direction: backward
|
||||
backs-up-to:
|
||||
inverse: backup-of
|
||||
source: entity
|
||||
@@ -842,12 +884,14 @@ relationship_types:
|
||||
target: ups
|
||||
cardinality: many-to-one
|
||||
description: Machine on UPS power.
|
||||
blast_direction: backward
|
||||
located-at:
|
||||
inverse: location-of
|
||||
source: machine
|
||||
target: site
|
||||
cardinality: many-to-one
|
||||
description: Machine's physical site.
|
||||
blast_direction: backward
|
||||
registered-with:
|
||||
inverse: registrar-of
|
||||
source: domain-registration
|
||||
@@ -880,12 +924,14 @@ relationship_types:
|
||||
target: secret
|
||||
cardinality: many-to-one
|
||||
description: Grant covers a secret.
|
||||
blast_direction: forward
|
||||
can-decrypt:
|
||||
inverse: readable-by
|
||||
source: compute-entity
|
||||
target: secret
|
||||
cardinality: many-to-many
|
||||
description: Host can decrypt a secret (legacy SOPS; Infisical grants later).
|
||||
blast_direction: backward
|
||||
|
||||
# Cognition
|
||||
checks:
|
||||
|
||||
@@ -70,6 +70,11 @@ approval_rules:
|
||||
- {entity_type: docker-container, action: restart, risk_class: reversible_low, autonomy_level: auto}
|
||||
- {entity_type: machine, action: apt-upgrade, risk_class: config_mutation, autonomy_level: escalate}
|
||||
- {entity_type: machine, action: reboot, risk_class: config_mutation, autonomy_level: escalate}
|
||||
# host-level package/kernel install (apt-get install, dpkg, modprobe, systemctl enable)
|
||||
# always classifies as config_mutation — the classifier defaults to config_mutation
|
||||
# for any command not in the read-only allowlist, so apt-get install reaches this
|
||||
# tier naturally. Documented explicitly here so agents stop second-guessing:
|
||||
# host mutations always need operator approval.
|
||||
- {entity_type: machine, action: format-disk, risk_class: destructive, autonomy_level: never}
|
||||
- {entity_type: config-repo, action: edit, risk_class: config_mutation, autonomy_level: escalate}
|
||||
- {entity_type: deploy-pipeline, action: trigger, risk_class: config_mutation, autonomy_level: escalate}
|
||||
|
||||
94
tools/deploy-checks.sh
Executable file
94
tools/deploy-checks.sh
Executable file
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy-checks.sh — push the check scripts into every monitored target.
|
||||
#
|
||||
# Background: an ssh-script check runs the script INSIDE the target, so the
|
||||
# script must exist at /opt/oikos/checks/ on the target itself — not just on
|
||||
# the proxmox host. The scheduler routes LXC/VM checks through the host via
|
||||
# `pct exec`/`qm guest exec`, so it never SSHes a guest directly, but the
|
||||
# script still has to be present inside the guest. This script deploys them.
|
||||
#
|
||||
# Run from a Proxmox host (it uses `pct`/`qm`) to populate every local guest,
|
||||
# and/or pass --host to install on a host/workstation over SSH.
|
||||
#
|
||||
# Usage:
|
||||
# deploy-checks.sh # on a proxmox host: push to every LXC/VM here
|
||||
# deploy-checks.sh --host ws:mac-mini # ssh-install scripts on a host/workstation
|
||||
# deploy-checks.sh --checks /path # override the source checks dir
|
||||
#
|
||||
# Idempotent: skips a script whose deployed copy is byte-identical.
|
||||
set -euo pipefail
|
||||
|
||||
CHECKS_DIR="${OIKOS_CHECK_DIR:-${HOMELAB_CONTEXT_DIR:-/opt/homelab}/checks}"
|
||||
DEST=/opt/oikos/checks
|
||||
|
||||
die() { echo "deploy-checks: $*" >&2; exit 1; }
|
||||
|
||||
deploy_to_guest() {
|
||||
local id="$1" vm="$2" # vm=0 for LXC, 1 for VM
|
||||
local kind=pct; [ "$vm" = "1" ] && kind=qm
|
||||
echo "[deploy-checks] $kind $id"
|
||||
# Ensure the destination dir exists inside the guest.
|
||||
if [ "$kind" = "pct" ]; then
|
||||
pct exec "$id" -- mkdir -p "$DEST" 2>/dev/null || { echo " skip (pct exec failed)"; return; }
|
||||
else
|
||||
# qm guest exec returns JSON; best-effort for VMs (guest agent required).
|
||||
qm guest exec "$id" -- mkdir -p "$DEST" >/dev/null 2>&1 || { echo " skip (qm guest exec failed)"; return; }
|
||||
fi
|
||||
local pushed=0 skipped=0
|
||||
for script in "$CHECKS_DIR"/*.sh; do
|
||||
local name; name=$(basename "$script")
|
||||
[ "$name" = "deploy-checks.sh" ] && continue
|
||||
[ "$name" = "install.sh" ] && continue
|
||||
if [ "$kind" = "pct" ]; then
|
||||
pct push "$id" "$script" "$DEST/$name" --perms 755 2>/dev/null && pushed=$((pushed+1)) || skipped=$((skipped+1))
|
||||
else
|
||||
# qm has no push; copy via the guest agent file write if available.
|
||||
qm guest exec "$id" -- /bin/sh -c "cat > $DEST/$name" < "$script" >/dev/null 2>&1 && pushed=$((pushed+1)) || skipped=$((skipped+1))
|
||||
fi
|
||||
done
|
||||
echo " pushed=$pushed skipped=$skipped"
|
||||
}
|
||||
|
||||
deploy_to_host() {
|
||||
local target="$1" # user@ip or slug resolved by caller
|
||||
echo "[deploy-checks] host $target"
|
||||
ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$target" "bash -s" < "$CHECKS_DIR/install.sh" \
|
||||
|| echo " WARNING: install on $target failed"
|
||||
}
|
||||
|
||||
if [ ! -d "$CHECKS_DIR" ]; then die "checks dir not found: $CHECKS_DIR"; fi
|
||||
|
||||
# Host/workstation install mode.
|
||||
if [ "${1:-}" = "--host" ]; then
|
||||
[ $# -ge 2 ] || die "--host needs a target (user@ip)"
|
||||
deploy_to_host "$2"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Proxmox-host mode: push to every local LXC and VM.
|
||||
if command -v pct >/dev/null 2>&1; then
|
||||
# LXC containers: ID and status. Skip stopped ones.
|
||||
while IFS= read -r line; do
|
||||
[ -z "$line" ] && continue
|
||||
id=$(awk '{print $1}' <<<"$line")
|
||||
status=$(awk '{print $2}' <<<"$line")
|
||||
[ "$status" = "running" ] || { echo "[deploy-checks] skip LXC $id ($status)"; continue; }
|
||||
deploy_to_guest "$id" 0
|
||||
done < <(pct list 2>/dev/null | tail -n +2)
|
||||
else
|
||||
echo "deploy-checks: 'pct' not found — not a Proxmox host."
|
||||
echo " On a host/workstation, use: deploy-checks.sh --host user@ip"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if command -v qm >/dev/null 2>&1; then
|
||||
while IFS= read -r line; do
|
||||
[ -z "$line" ] && continue
|
||||
id=$(awk '{print $1}' <<<"$line")
|
||||
status=$(awk '{print $2}' <<<"$line")
|
||||
[ "$status" = "running" ] || continue
|
||||
deploy_to_guest "$id" 1
|
||||
done < <(qm list 2>/dev/null | tail -n +2)
|
||||
fi
|
||||
|
||||
echo "[deploy-checks] done"
|
||||
@@ -1,13 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
# setup-checks.sh — deploy check scripts to /opt/oikos/checks on each host.
|
||||
# setup-checks.sh — deploy check scripts to /opt/oikos/checks.
|
||||
# Auto-setup hook: tools/setup-*.sh runs after every git pull.
|
||||
#
|
||||
# On a plain host/workstation this installs the scripts locally (the pulling
|
||||
# host). On a Proxmox host it ALSO pushes the scripts into every running LXC/VM
|
||||
# guest, because an ssh-script check runs the script INSIDE the target — a
|
||||
# script present only on the host does nothing for a guest reached via
|
||||
# pct/qm exec. Guest deployment is delegated to deploy-checks.sh.
|
||||
set -euo pipefail
|
||||
|
||||
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
|
||||
CHECK_SETUP="$CLONE_DIR/checks/install.sh"
|
||||
DEPLOY="$CLONE_DIR/tools/deploy-checks.sh"
|
||||
|
||||
if [ -f "$CHECK_SETUP" ]; then
|
||||
bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
|
||||
else
|
||||
echo "[setup-checks] no checks/install.sh found, skipping"
|
||||
fi
|
||||
|
||||
# On a Proxmox host, keep every guest's scripts in sync too. Best-effort: a
|
||||
# failing push to one guest must not abort the whole hook. Warn (not skip
|
||||
# silently) if deploy-checks.sh itself is absent — without it guests never get
|
||||
# scripts and the pct-exec routing reports every guest check down.
|
||||
if command -v pct >/dev/null 2>&1; then
|
||||
if [ ! -f "$DEPLOY" ]; then
|
||||
echo "[setup-checks] WARNING: $DEPLOY missing — guest scripts will go stale. Commit tools/deploy-checks.sh alongside this hook."
|
||||
elif [ ! -x "$DEPLOY" ]; then
|
||||
echo "[setup-checks] WARNING: $DEPLOY not executable — running via bash"
|
||||
bash "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
|
||||
else
|
||||
"$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -4,12 +4,6 @@
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>Oikos</title>
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
||||
<link
|
||||
href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&family=Inknut+Antiqua:wght@300;400;500;600;700&display=swap"
|
||||
rel="stylesheet"
|
||||
/>
|
||||
<link rel="icon" type="image/svg+xml" href="favicon.svg" />
|
||||
<link rel="apple-touch-icon" href="apple-touch-icon.png" />
|
||||
<link rel="icon" type="image/png" sizes="192x192" href="android-chrome-192.png" />
|
||||
|
||||
653
web/package-lock.json
generated
653
web/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -42,6 +42,7 @@
|
||||
"vitest": "^2.0.0"
|
||||
},
|
||||
"dependencies": {
|
||||
"@joan/procedural-glyph-engine": "file:../vendor",
|
||||
"@surdeddd/wmkit": "^0.3.0",
|
||||
"clsx": "^2.1.1",
|
||||
"d3-force": "^3.0.0",
|
||||
|
||||
BIN
web/public/fonts/JetBrainsMono-Bold.woff2
Normal file
BIN
web/public/fonts/JetBrainsMono-Bold.woff2
Normal file
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user