Compare commits
55 Commits
42751623ea
...
chore/vend
| Author | SHA1 | Date | |
|---|---|---|---|
| 3cd4cf98c3 | |||
| 38c472a118 | |||
| 1d0197da69 | |||
| 0920c4cb6d | |||
| 2254a07baf | |||
| 1b9c761274 | |||
| a126cfa710 | |||
| 86fa57b5cd | |||
| 8e97d589af | |||
| 0dd8c28815 | |||
| 4e294b3630 | |||
| 85f0bb67fa | |||
| c3f478b8f8 | |||
| 1aaedf498a | |||
| 20adb89650 | |||
| 058f1afcdc | |||
| 2b73290994 | |||
| 428f4fe945 | |||
| 195d45a0e9 | |||
| 5b68bdc16c | |||
| 757ef2f34b | |||
| b27e1bf3ec | |||
| 39e9227fdb | |||
| bb05f215c6 | |||
| 467589d78a | |||
| e25e979757 | |||
| bc0ccb4cdc | |||
| c9a00a9532 | |||
| eb16796bf0 | |||
| a3914a1d41 | |||
| 8eb1ca2bac | |||
| e4104eb344 | |||
| 0929c17cbb | |||
| 6487032461 | |||
| fb6b6f9160 | |||
| 62c9fc5c86 | |||
| 6007e922b4 | |||
| 2d8eb91b25 | |||
| 3d88f52988 | |||
| 9016c3a43b | |||
| 04775192c1 | |||
| a104cb4bb4 | |||
| 72f0f46528 | |||
| b87735a111 | |||
| 1540f74342 | |||
| c7729b2ef6 | |||
| b8b4aa2aee | |||
| c10f6920cd | |||
| ad29295c93 | |||
| 6ca6d5b352 | |||
| af450dac2a | |||
| 6ed9dc39e8 | |||
| cc8eae4979 | |||
| 4f706fa65f | |||
| 50e899e5ee |
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
87
.agents/skills/knowledge-graph-audit/SKILL.md
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
---
|
||||||
|
name: knowledge-graph-audit
|
||||||
|
risk_class: read_only
|
||||||
|
inputs: []
|
||||||
|
verification: "audit_knowledge_graph returns a report with summary.total_findings"
|
||||||
|
docs_update_checklist: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Knowledge-graph audit
|
||||||
|
|
||||||
|
Goal: validate that the knowledge graph (entities, relationships, checks) and
|
||||||
|
the monitoring built on it reflect live reality — without mutating anything.
|
||||||
|
Read-only. Run this before trusting health, blast-radius, or coverage answers,
|
||||||
|
and whenever something feels off (a healthy host reports `down`, a retired
|
||||||
|
service still alarms, the graph looks thin).
|
||||||
|
|
||||||
|
## 1. Run the drift report
|
||||||
|
|
||||||
|
Call MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`). It returns a
|
||||||
|
ranked list of findings, each with `{category, severity, count, entities,
|
||||||
|
evidence, suggested_runbook}`, plus a `summary` with totals by category.
|
||||||
|
|
||||||
|
The DB-side categories:
|
||||||
|
|
||||||
|
- **orphan_checks** — check entities with truncated/random slugs left by the
|
||||||
|
old `shortSlug()` collision bug. Remediation: `scripts/cleanup-orphan-checks.sh`.
|
||||||
|
- **dead_checks** — enabled `check_defs` whose target entity is `deprecated`/
|
||||||
|
`destroyed`. Remediation: `lifecycle-deprecate-node` / `lifecycle-destroy-node`
|
||||||
|
(the scheduler already skips these, but the rows should be retired).
|
||||||
|
- **down_checks** — enabled probes reporting `down`. Remediation:
|
||||||
|
`service-health-check` (then check whether the failure is real or a
|
||||||
|
probe-config/routing problem — see step 3).
|
||||||
|
- **unknown_checks** — probes that ran but reported `unknown` (usually a
|
||||||
|
misconfigured or not-yet-deployed probe script).
|
||||||
|
- **unmonitored** — active entities whose type declares monitoring but have no
|
||||||
|
enabled `check_def`.
|
||||||
|
- **dangling_edges** — live `hosts`/`provides`/`mounts` edges still pointing at
|
||||||
|
destroyed/deprecated targets. Remediation: `lifecycle-destroy-node`.
|
||||||
|
|
||||||
|
## 2. Triage
|
||||||
|
|
||||||
|
`severity: critical` (down_checks) first. For each finding, read `evidence` and
|
||||||
|
open the entities with `get_entity` / `get_relations` to confirm the diagnosis
|
||||||
|
before acting — the report is a pointer, not a verdict.
|
||||||
|
|
||||||
|
## 3. Common probe-failure causes
|
||||||
|
|
||||||
|
A `down_checks` finding that is NOT a real outage is usually one of:
|
||||||
|
|
||||||
|
- **Guest reached wrong** — an LXC/VM check SSHed the guest directly instead of
|
||||||
|
routing through its Proxmox host. Confirm with `get_relations` that a `hosts`
|
||||||
|
edge exists and the guest has `pve_id`; checks route via `pct exec`/`qm guest
|
||||||
|
exec` automatically when both are present.
|
||||||
|
- **Script not deployed** — the probe script is absent at `/opt/oikos/checks/`
|
||||||
|
inside the target. Remediation: redeploy via `tools/deploy-checks.sh`.
|
||||||
|
- **macOS host** — a workstation check used the wrong SSH user or a Linux-only
|
||||||
|
script flag. The scheduler resolves `user: dtoro` from the entity attribute.
|
||||||
|
|
||||||
|
## 4. What this audit does NOT cover (follow-ups)
|
||||||
|
|
||||||
|
Live-infrastructure discovery has its own tool — run **`discover_infra_drift`**
|
||||||
|
alongside this one. It compares running Proxmox guests (`pct`/`qm list` on every
|
||||||
|
proxmox host) against the DB graph and returns:
|
||||||
|
|
||||||
|
- **missing entities** — a guest running in Proxmox with no DB entity.
|
||||||
|
- **ghost entities** — a DB lxc/vm whose `pve_id` is no longer live.
|
||||||
|
|
||||||
|
Still manual until that machinery lands:
|
||||||
|
|
||||||
|
- **Misplaced parent** — compare each guest's actual Proxmox host against its
|
||||||
|
`hosts` edge (migrations leave these stale).
|
||||||
|
- **Undeployed scripts** — per-guest `/opt/oikos/checks/` presence.
|
||||||
|
- **Unmodeled certs** — now modeled; verify with `audit_knowledge_graph` /
|
||||||
|
the cert-expiry checks.
|
||||||
|
- **Seed drift** — run `oikos export` and `git diff seeds/` to find
|
||||||
|
runtime-created entities not in version control.
|
||||||
|
|
||||||
|
## 5. Acting on findings
|
||||||
|
|
||||||
|
This skill is read-only — make no changes here. Route each confirmed finding to
|
||||||
|
its `suggested_runbook`, classify the action against `seeds/policy.yaml`, and
|
||||||
|
proceed through the normal lifecycle/approval flow. Re-run the audit afterward
|
||||||
|
to confirm the finding cleared.
|
||||||
|
|
||||||
|
Docs-update checklist: none — the audit reads state; it changes nothing. If a
|
||||||
|
finding reveals stale `risk_notes` or a wrong `doc_page`, fix `inventory.yaml`
|
||||||
|
in that remediation session.
|
||||||
12
.dockerignore
Normal file
12
.dockerignore
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
# Every docker build in this repo previously sent the whole directory as
|
||||||
|
# build context — including every OTHER git worktree under .claude/worktrees/
|
||||||
|
# (each with its own web/node_modules, ~200-300MB apiece). That's what
|
||||||
|
# starved the mac-mini's disk mid-build on 2026-07-27 (SHA 873b00a): the
|
||||||
|
# context alone crossed 390MB of pure worktree cruft before the host ran out
|
||||||
|
# of space. None of this ever belonged in an image.
|
||||||
|
.claude/worktrees/
|
||||||
|
.git/
|
||||||
|
**/node_modules/
|
||||||
|
**/dist/
|
||||||
|
**/build/
|
||||||
|
*.log
|
||||||
4
.gitignore
vendored
4
.gitignore
vendored
@@ -24,3 +24,7 @@ cmd/desktop/build/
|
|||||||
cmd/desktop/Oikos
|
cmd/desktop/Oikos
|
||||||
desktop
|
desktop
|
||||||
/eval
|
/eval
|
||||||
|
|
||||||
|
# Local tooling artifacts (Playwright MCP session logs, stray screenshots)
|
||||||
|
.playwright-mcp/
|
||||||
|
config-screen.png
|
||||||
|
|||||||
@@ -2284,6 +2284,23 @@ components:
|
|||||||
type: boolean
|
type: boolean
|
||||||
version:
|
version:
|
||||||
type: integer
|
type: integer
|
||||||
|
last_health:
|
||||||
|
type: string
|
||||||
|
description: >-
|
||||||
|
This check's own most recent verdict. An entity's health is the
|
||||||
|
worst of these across its enabled checks, so this is what explains
|
||||||
|
*why* an entity is degraded. Null until the check first runs.
|
||||||
|
nullable: true
|
||||||
|
enum:
|
||||||
|
- healthy
|
||||||
|
- degraded
|
||||||
|
- down
|
||||||
|
- unknown
|
||||||
|
last_run_at:
|
||||||
|
type: string
|
||||||
|
format: date-time
|
||||||
|
description: When this check last executed. Null = never run.
|
||||||
|
nullable: true
|
||||||
CheckCreate:
|
CheckCreate:
|
||||||
type: object
|
type: object
|
||||||
required:
|
required:
|
||||||
|
|||||||
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
104
archive/knowledge/infrastructure/oikos-check-lifecycle.md
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
# Oikos check lifecycle — how monitoring works
|
||||||
|
|
||||||
|
This runbook covers how Oikos health checks are derived, created, and wired so
|
||||||
|
an agent (Nomos) doesn't reverse-engineer source when asked to add monitoring to
|
||||||
|
an entity — the problem that stranded session `23da10db` (2026-08-03).
|
||||||
|
|
||||||
|
## Concepts
|
||||||
|
|
||||||
|
- **`check_defs`** (scheduler config, table `check_defs`): the row the scheduler
|
||||||
|
reads to know *what* to probe and *when*. One per check instance.
|
||||||
|
- **`check` entity** (type `check`, slug `check:<kind>:<target>:<n>`): the
|
||||||
|
knowledge-graph entity for that check. It carries attributes
|
||||||
|
(`check_type`, `target`, `port`, …) and `checks` edges to the probed target.
|
||||||
|
- **`monitoring` spec** on an entity type (`entity_types.monitoring_spec`): the
|
||||||
|
default list of check kinds (e.g. `[http, process]` for `service`).
|
||||||
|
- Per-entity override: set `monitoring` in the entity's attributes —
|
||||||
|
`"none"` for zero checks, `["http"]` to replace the type defaults.
|
||||||
|
- **`checkdefaults.Ensure`** (`internal/checkdefaults/defaults.go`): the
|
||||||
|
function that reads the monitoring spec, resolves host/port/URL from
|
||||||
|
attributes + relationships, and writes `check_defs` rows. Idempotent.
|
||||||
|
|
||||||
|
## When checks are derived
|
||||||
|
|
||||||
|
`checkdefaults.Ensure` runs in three situations (as of v0.17.1+):
|
||||||
|
|
||||||
|
1. **Seed/deploy ingest** — `internal/db/seed.go:231`. Every entity gets its
|
||||||
|
default checks once on initial ingest.
|
||||||
|
2. **HTTP `POST /api/v1/entities` (create)** — `ensureDefaultChecks` at
|
||||||
|
`internal/httpapi/impl.go:1012`. Creating an entity via the REST API derives
|
||||||
|
its checks in the same transaction.
|
||||||
|
3. **HTTP `PATCH /api/v1/entities` (patch)** — `ensureDefaultChecks` at
|
||||||
|
`internal/httpapi/impl.go:1280`. Changing an entity's attributes (especially
|
||||||
|
`monitoring`) via the REST API regenerates its checks.
|
||||||
|
4. **MCP `create_entity`** — SAME hook. Creating an entity via the MCP tool
|
||||||
|
derives checks. (Added 2026-08-03; previously MCP had no create.)
|
||||||
|
5. **MCP `update_entity_attributes`** — SAME hook. Changing an entity's
|
||||||
|
`monitoring` attribute via MCP now regenerates checks. (Added 2026-08-03;
|
||||||
|
previously MCP updates silently skipped check derivation — the exact bug
|
||||||
|
that stranded the haos session.)
|
||||||
|
|
||||||
|
## Check slug grammar
|
||||||
|
|
||||||
|
```
|
||||||
|
check:<kind>:<target-type>:<target-name>:<n>
|
||||||
|
```
|
||||||
|
|
||||||
|
Examples: `check:http:service:jellyfin:0`, `check:vm-status:vm:haos:0`,
|
||||||
|
`check:cert-expiry:cert:house.hubris.network:0`.
|
||||||
|
|
||||||
|
## Adding monitoring to an entity
|
||||||
|
|
||||||
|
**If the entity already exists:**
|
||||||
|
|
||||||
|
```
|
||||||
|
update_entity_attributes(slug="service:haos", attributes={"monitoring":["http"]})
|
||||||
|
```
|
||||||
|
|
||||||
|
This regenerates checks via `checkdefaults.Ensure`. The result message tells you
|
||||||
|
how many checks were derived and whether any kinds were skipped (and why).
|
||||||
|
|
||||||
|
**If the entity does not exist yet (a new check, ingress, cert, etc.):**
|
||||||
|
|
||||||
|
```
|
||||||
|
create_entity(type="check", name="HAOS http check",
|
||||||
|
slug="check:http:service:haos:0",
|
||||||
|
attributes={"check_type":"http:service","target":"service:haos","port":"8123"})
|
||||||
|
```
|
||||||
|
|
||||||
|
This creates the entity AND derives its `check_defs`. Same for a new `ingress`
|
||||||
|
(`type=ingress`, monitoring `[http]`) or `cert` (`type=cert`,
|
||||||
|
monitoring `[cert-expiry]`).
|
||||||
|
|
||||||
|
**To remove monitoring:** set `monitoring:["none"]` or transition the entity
|
||||||
|
to a terminal lifecycle state (`set_entity_state` → `deprecated`/`destroyed`).
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- **A service without a `url` attribute AND without a `probe_unit` gets no
|
||||||
|
process check** (the http check covers liveness; the process check would
|
||||||
|
be redundant without an opt-in `probe_unit`). The skip is logged.
|
||||||
|
- **A service whose address comes from a `hosts` edge** may produce no checks on
|
||||||
|
initial create because the edge doesn't exist yet — the next inventory ingest
|
||||||
|
(or a later `update_entity_attributes` after the edge is created) fills it in.
|
||||||
|
- **A `not found` error from `update_entity_attributes`** means the entity
|
||||||
|
doesn't exist — use `create_entity` instead.
|
||||||
|
- **`check_defs` has target columns** (`target_id`, `target_type`). A check
|
||||||
|
entity needs a `checks` relationship (`create_relationship(source=check:…,
|
||||||
|
target=service:…, type="checks")`) so the scheduler can resolve what to
|
||||||
|
probe. `create_entity` derives the check_def; `create_relationship` links
|
||||||
|
the check entity to its target in the graph.
|
||||||
|
|
||||||
|
## Related files
|
||||||
|
|
||||||
|
- `internal/checkdefaults/defaults.go` — `Ensure`, `Target`, `LogResult`
|
||||||
|
- `internal/httpapi/default_checks.go` — `ensureDefaultChecks` (HTTP hook)
|
||||||
|
- `internal/db/checks.go` — `db.EnsureEntityChecks` (shared hook)
|
||||||
|
- `internal/db/seed.go` — seed-time check derivation
|
||||||
|
- `internal/mcp/tools.go` — `create_entity`, `update_entity_attributes`
|
||||||
|
|
||||||
|
## Revision history
|
||||||
|
|
||||||
|
- **2026-08-03:** Created after session `23da10db` stranded for lack of entity-
|
||||||
|
creation tool and unawareness of check-derivation triggers. Covers the MCP
|
||||||
|
create_entity + update_entity_attributes regen paths added same day.
|
||||||
@@ -2,12 +2,27 @@
|
|||||||
# cpu_check.sh — CPU usage % and thermal temperature.
|
# cpu_check.sh — CPU usage % and thermal temperature.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
os=$(uname -s)
|
||||||
if [ -z "$USAGE" ]; then
|
|
||||||
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
if [ "$os" = "Darwin" ]; then
|
||||||
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
# `top -l 1 -n 0` prints "CPU usage: X% user, Y% sys, Z% idle".
|
||||||
|
# Usage is 100 minus the idle figure that precedes the literal `idle`.
|
||||||
|
USAGE=$(top -l 1 -n 0 -s 0 2>/dev/null | awk '
|
||||||
|
/^CPU usage/ {
|
||||||
|
for (i = 1; i <= NF; i++) {
|
||||||
|
if ($i == "idle") { gsub(/%/, "", $(i - 1)); printf "%.1f", 100 - $(i - 1) }
|
||||||
|
}
|
||||||
|
}' || true)
|
||||||
|
else
|
||||||
|
USAGE=$(top -bn1 2>/dev/null | awk '/^%Cpu/ {print 100 - $8}' || true)
|
||||||
|
if [ -z "$USAGE" ]; then
|
||||||
|
CORES=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
|
||||||
|
USAGE=$(awk -v cores="$CORES" '{print ($1+$2+$3)*100/cores}' /proc/loadavg 2>/dev/null || echo "0")
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
[ -z "$USAGE" ] && USAGE=0
|
||||||
|
|
||||||
TEMP=""
|
TEMP=""
|
||||||
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
if [ -f /sys/class/thermal/thermal_zone0/temp ]; then
|
||||||
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
TEMP=$(awk '{printf "%.1f", $1/1000}' /sys/class/thermal/thermal_zone0/temp 2>/dev/null || true)
|
||||||
|
|||||||
22
checks/disk_usage_check.sh
Normal file → Executable file
22
checks/disk_usage_check.sh
Normal file → Executable file
@@ -2,18 +2,34 @@
|
|||||||
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
# disk_usage_check.sh — disk usage and inode usage per mountpoint.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
MOUNTS=$(df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
# `timeout` caps each df so a single hung/stale mountpoint (a stale NFS
|
||||||
|
# export, a wedged ZFS pool) can't stall the whole check — that hung the
|
||||||
|
# scheduler's 30s budget on hubris. Available on Linux (coreutils); absent on
|
||||||
|
# Darwin, whose local mounts don't hang, so it degrades to an empty prefix.
|
||||||
|
TO=""
|
||||||
|
if command -v timeout >/dev/null 2>&1; then TO="timeout 8"; fi
|
||||||
|
|
||||||
|
# Build the mount list WITHOUT statting anything: reading /proc/mounts never
|
||||||
|
# blocks the way `df` does on a stuck filesystem, so the enumeration itself
|
||||||
|
# can't hang. Fall back to `df` on hosts without /proc/mounts (macOS).
|
||||||
|
if [ -r /proc/mounts ]; then
|
||||||
|
MOUNTS=$(awk '$1 ~ /^\// && $2 !~ /^\/(snap|dev|proc|sys|run|private)/ {print $2}' /proc/mounts || true)
|
||||||
|
else
|
||||||
|
MOUNTS=$($TO df -k 2>/dev/null | awk 'NR>1 && $1 ~ /^\// && $NF !~ /^\/(snap|dev|proc|sys|run|private)/ {print $NF}' || true)
|
||||||
|
fi
|
||||||
FIRST=1
|
FIRST=1
|
||||||
|
|
||||||
echo -n '{"health":"healthy","metrics":{'
|
echo -n '{"health":"healthy","metrics":{'
|
||||||
for m in $MOUNTS; do
|
for m in $MOUNTS; do
|
||||||
LINE=$(df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
# Each df is bounded: a stuck mount times out and is skipped (LINE empty)
|
||||||
|
# rather than hanging the probe.
|
||||||
|
LINE=$($TO df -k "$m" 2>/dev/null | awk 'NR==2 {print $3, $4, $5, $7}' | tr -d '%' || true)
|
||||||
if [ -z "$LINE" ]; then continue; fi
|
if [ -z "$LINE" ]; then continue; fi
|
||||||
USED=$(echo "$LINE" | awk '{print $1}')
|
USED=$(echo "$LINE" | awk '{print $1}')
|
||||||
FREE=$(echo "$LINE" | awk '{print $2}')
|
FREE=$(echo "$LINE" | awk '{print $2}')
|
||||||
PCT=$(echo "$LINE" | awk '{print $3}')
|
PCT=$(echo "$LINE" | awk '{print $3}')
|
||||||
|
|
||||||
INODE_LINE=$(df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
INODE_LINE=$($TO df -i "$m" 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%' || echo "0")
|
||||||
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
INODE_PCT=$(echo "${INODE_LINE:-0}" | sed 's/-/0/')
|
||||||
|
|
||||||
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
KEY=$(echo "$m" | sed 's|/|_|g' | sed 's|^_||')
|
||||||
|
|||||||
@@ -1,5 +1,17 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# process_check.sh — systemd service liveness.
|
# process_check.sh — service liveness.
|
||||||
|
#
|
||||||
|
# A service entity's name is a logical label, rarely the literal systemd unit
|
||||||
|
# or container name. matrix = matrix-synapse.service + element-web/mautrix-*
|
||||||
|
# containers; authentik = authentik-server/-worker containers. So checking
|
||||||
|
# `systemctl is-active matrix` reports "inactive" for a healthy service.
|
||||||
|
#
|
||||||
|
# Resolution order, any hit = healthy:
|
||||||
|
# 1. exact systemd unit `systemctl is-active <name>`
|
||||||
|
# 2. a systemd unit with the name as prefix `<name>*.service`
|
||||||
|
# 3. a running docker container whose name contains <name>
|
||||||
|
# An explicit probe target overrides the label — see checkdefaults, which
|
||||||
|
# passes a `probe_unit`/`container`/`systemd_unit` attribute as $1 when set.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SERVICE="${1:-}"
|
SERVICE="${1:-}"
|
||||||
@@ -8,15 +20,31 @@ if [ -z "$SERVICE" ]; then
|
|||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if ! command -v systemctl >/dev/null 2>&1; then
|
ok() { echo "{\"health\":\"healthy\"}"; exit 0; }
|
||||||
echo '{"health":"unknown","signalKind":"process-check","evidence":"systemctl not found"}'
|
|
||||||
exit 0
|
# 1. exact systemd unit
|
||||||
|
if command -v systemctl >/dev/null 2>&1; then
|
||||||
|
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null | head -1 || true)
|
||||||
|
[ "$STATE" = "active" ] && ok
|
||||||
|
|
||||||
|
# 2. prefix match: matrix -> matrix-synapse.service, house -> house.service, etc.
|
||||||
|
# --no-legend strips the header/footer so grep can see the unit rows; the
|
||||||
|
# pattern is a systemd unit glob.
|
||||||
|
if systemctl list-units --type=service --state=active --no-legend "$SERVICE*.service" 2>/dev/null \
|
||||||
|
| grep -q '\.service'; then
|
||||||
|
ok
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
STATE=$(systemctl is-active "$SERVICE" 2>/dev/null || echo "unknown")
|
# 3. a running docker container whose name contains the label.
|
||||||
|
if command -v docker >/dev/null 2>&1; then
|
||||||
if [ "$STATE" = "active" ]; then
|
if docker ps --filter "status=running" --filter "name=$SERVICE" --format '{{.Names}}' 2>/dev/null \
|
||||||
echo "{\"health\":\"healthy\"}"
|
| grep -q .; then
|
||||||
else
|
ok
|
||||||
echo "{\"health\":\"degraded\",\"signalKind\":\"$SERVICE\",\"evidence\":\"$SERVICE is $STATE\"}"
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
STATE=${STATE:-inactive}
|
||||||
|
STATE=${STATE//\"/}
|
||||||
|
SAFE_SERVICE=${SERVICE//\"/}
|
||||||
|
echo "{\"health\":\"degraded\",\"signalKind\":\"process\",\"evidence\":\"$SAFE_SERVICE is $STATE (no active unit/container matched)\"}"
|
||||||
|
|||||||
@@ -57,6 +57,13 @@ type agent struct {
|
|||||||
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
apiBase string // oikos HTTP API base, derived from NOMOS_MCP_URL, for chat-assent approvals
|
||||||
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
apiToken string // OIKOS_MCP_BEARER_TOKEN — api's combinedAuth requires it (no dev-open bypass)
|
||||||
httpClient *http.Client
|
httpClient *http.Client
|
||||||
|
// gate serializes turns per session (at most one in-flight turn per
|
||||||
|
// sessionID). See turngate.go and plan 2026-08-03 F1.
|
||||||
|
gate *turnGate
|
||||||
|
// queue holds operator messages that arrived while a turn was already
|
||||||
|
// running; they are auto-run when the gate frees (plan 2026-08-03 F2).
|
||||||
|
// See messagequeue.go.
|
||||||
|
queue *messageQueue
|
||||||
}
|
}
|
||||||
|
|
||||||
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug string) (*agent, error) {
|
||||||
@@ -117,6 +124,8 @@ func newAgent(ctx context.Context, clients *mcpClientPool, st *store, agentSlug
|
|||||||
apiBase: apiBase,
|
apiBase: apiBase,
|
||||||
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
apiToken: os.Getenv("OIKOS_MCP_BEARER_TOKEN"),
|
||||||
httpClient: &http.Client{Timeout: 15 * time.Second},
|
httpClient: &http.Client{Timeout: 15 * time.Second},
|
||||||
|
gate: newTurnGate(),
|
||||||
|
queue: newMessageQueue(),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -172,6 +181,11 @@ type agentEvent struct {
|
|||||||
Data any `json:"data,omitempty"`
|
Data any `json:"data,omitempty"`
|
||||||
SessionID string `json:"session_id,omitempty"`
|
SessionID string `json:"session_id,omitempty"`
|
||||||
Iteration int `json:"iteration,omitempty"`
|
Iteration int `json:"iteration,omitempty"`
|
||||||
|
// IsThinking marks text/text_delta events that carry the model's internal
|
||||||
|
// reasoning (text produced before tool calls in the same iteration), as
|
||||||
|
// distinct from the final response text. The frontend renders these as
|
||||||
|
// collapsible thinking blocks separated from the response.
|
||||||
|
IsThinking bool `json:"is_thinking,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
func (a *agent) chat(ctx context.Context, sessionID, message string, emit func(agentEvent)) {
|
||||||
@@ -368,7 +382,12 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
var msg openai.ChatCompletionMessage
|
var msg openai.ChatCompletionMessage
|
||||||
var acc openai.ChatCompletionAccumulator
|
var acc openai.ChatCompletionAccumulator
|
||||||
|
|
||||||
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
// Capture token usage from this LLM response for activity logging.
|
||||||
|
// Previously always NULL — every agent_activity row had no token
|
||||||
|
// count. Now each tool call in this iteration gets the same total.
|
||||||
|
totalTokens := 0
|
||||||
|
|
||||||
|
for attempt := 0; attempt <= maxLLMRetries; attempt++ {
|
||||||
acc = openai.ChatCompletionAccumulator{}
|
acc = openai.ChatCompletionAccumulator{}
|
||||||
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
|
stream := a.provider.Chat.Completions.NewStreaming(ctx, params, a.reqOpts...)
|
||||||
for stream.Next() {
|
for stream.Next() {
|
||||||
@@ -400,14 +419,19 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
msg = acc.Choices[0].Message
|
msg = acc.Choices[0].Message
|
||||||
finishReason := acc.Choices[0].FinishReason
|
finishReason := acc.Choices[0].FinishReason
|
||||||
|
|
||||||
|
// Capture token usage from this iteration.
|
||||||
|
if acc.Usage.TotalTokens > 0 {
|
||||||
|
totalTokens = int(acc.Usage.TotalTokens)
|
||||||
|
}
|
||||||
|
|
||||||
if len(msg.ToolCalls) == 0 {
|
if len(msg.ToolCalls) == 0 {
|
||||||
if isRefusalOrEmpty(msg.Content) {
|
if isRefusalOrEmpty(msg.Content) {
|
||||||
if attempt < maxLLMRetries {
|
if attempt < maxLLMRetries {
|
||||||
slog.Warn("nomos: empty or refusal response, retrying",
|
slog.Warn("nomos: empty or refusal response, retrying",
|
||||||
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
"session", sessionID, "iter", i+1, "attempt", attempt+1,
|
||||||
"content_len", len(msg.Content), "finish_reason", finishReason)
|
"content_len", len(msg.Content), "finish_reason", finishReason)
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
// B.4: surface the real error context (finish_reason +
|
// B.4: surface the real error context (finish_reason +
|
||||||
// refusal text) instead of a generic "empty response" —
|
// refusal text) instead of a generic "empty response" —
|
||||||
// the operator can tell "content_filter — rephrase" from
|
// the operator can tell "content_filter — rephrase" from
|
||||||
@@ -456,7 +480,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
// led to each step. Emitting it lets the persist layer accumulate
|
// led to each step. Emitting it lets the persist layer accumulate
|
||||||
// per-iteration reasoning into the row's text field.
|
// per-iteration reasoning into the row's text field.
|
||||||
if strings.TrimSpace(msg.Content) != "" {
|
if strings.TrimSpace(msg.Content) != "" {
|
||||||
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID})
|
emit(agentEvent{Type: "text", Data: msg.Content, SessionID: sessionID, IsThinking: true})
|
||||||
}
|
}
|
||||||
|
|
||||||
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
|
slog.Info("nomos: tool calls", "count", len(msg.ToolCalls), "iter", i+1, "correlation", correlationID)
|
||||||
@@ -491,7 +515,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
slog.Warn("nomos: run retry cap hit — refusing dispatch",
|
slog.Warn("nomos: run retry cap hit — refusing dispatch",
|
||||||
"target", t, "failures", n, "session", sessionID)
|
"target", t, "failures", n, "session", sessionID)
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args,
|
||||||
tc.Function.Arguments, directive, 0, false, correlationID)
|
tc.Function.Arguments, directive, 0, false, correlationID, totalTokens)
|
||||||
emit(agentEvent{
|
emit(agentEvent{
|
||||||
Type: "tool_result",
|
Type: "tool_result",
|
||||||
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
|
Data: map[string]any{"name": tc.Function.Name, "result": directive, "id": tc.ID, "retry_capped": true},
|
||||||
@@ -542,7 +566,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
inputStr := string(inputJSON)
|
inputStr := string(inputJSON)
|
||||||
|
|
||||||
if callErr != nil {
|
if callErr != nil {
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID)
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, callErr.Error(), elapsed, false, correlationID, totalTokens)
|
||||||
|
|
||||||
// Retry cap: dispatch errors (e.g. MCP client timeout)
|
// Retry cap: dispatch errors (e.g. MCP client timeout)
|
||||||
// count toward the cap too. A command that keeps timing
|
// count toward the cap too. A command that keeps timing
|
||||||
@@ -572,7 +596,7 @@ func (a *agent) chatWith(ctx context.Context, sessionID, message, systemInject s
|
|||||||
}
|
}
|
||||||
|
|
||||||
resultJSON, _ := json.Marshal(result)
|
resultJSON, _ := json.Marshal(result)
|
||||||
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID)
|
a.store.logActivity(ctx, a.agentID, sessionID, tc.Function.Name, args, inputStr, string(resultJSON), elapsed, true, correlationID, totalTokens)
|
||||||
|
|
||||||
// Link any execution this tool queued/started back to this
|
// Link any execution this tool queued/started back to this
|
||||||
// session, so the auto-continuation worker can feed its result
|
// session, so the auto-continuation worker can feed its result
|
||||||
|
|||||||
@@ -76,16 +76,20 @@ func (a *agent) processIdleSweep(ctx context.Context) {
|
|||||||
s := s
|
s := s
|
||||||
if s.CompletionNudges == 0 {
|
if s.CompletionNudges == 0 {
|
||||||
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||||
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
||||||
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
||||||
return
|
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
||||||
|
s.Goal, idleTaskThreshold)
|
||||||
|
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
||||||
|
// P1: only count the nudge if it actually delivered. resumeSession
|
||||||
|
// skips (returns false) when a turn is already active; bumping the
|
||||||
|
// counter anyway would make the next sweep auto-close a merely-busy
|
||||||
|
// session as "unanswered."
|
||||||
|
if a.resumeSession(ctx, s.ID, note) {
|
||||||
|
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil {
|
||||||
|
slog.Error("nomos: idle nudge bump failed", "session", s.ID, "error", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
note := fmt.Sprintf("[System: this task ('%s') has been idle for %s with no complete_task call. "+
|
|
||||||
"If the goal is done (or can't be completed), call complete_task now with the outcome and a "+
|
|
||||||
"one-line summary. If you're still genuinely working through the plan, ignore this and continue.]",
|
|
||||||
s.Goal, idleTaskThreshold)
|
|
||||||
note = a.store.enrichResumeNote(ctx, s.ID, note)
|
|
||||||
a.resumeSession(ctx, s.ID, note)
|
|
||||||
})
|
})
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -163,7 +167,9 @@ func (a *agent) processContinuations(ctx context.Context) {
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
a.store.markContinued(ctx, p.ExecID) // stamp first: a failure here must not cause a re-continue loop
|
// markContinued now happens inside continueSession, AFTER resumeSession
|
||||||
|
// actually runs (P0). Pre-marking here consumed the item even when
|
||||||
|
// resumeSession skipped on a busy session, losing the result.
|
||||||
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
safego.Go("nomos:continue-session:"+p.SessionID, func() { a.continueSession(ctx, p) })
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -179,7 +185,18 @@ func (a *agent) processContinuations(ctx context.Context) {
|
|||||||
// something new to poll for.
|
// something new to poll for.
|
||||||
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||||
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||||
a.resumeSession(ctx, p.SessionID, buildContinuationNote(p))
|
// P0 (plans/2026-08-03-nomos-chat-changes-review.md): mark the execution
|
||||||
|
// continued ONLY after the turn actually ran. resumeSession skips (returns
|
||||||
|
// false) when another turn is already active for this session; marking
|
||||||
|
// before that — as the old code did — consumed the item (continued_at set,
|
||||||
|
// never re-queued by pendingContinuations) and silently lost the result.
|
||||||
|
// On a skip, leave it pending so the next worker tick retries once the
|
||||||
|
// active turn frees the permit.
|
||||||
|
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||||
|
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.store.markContinued(ctx, p.ExecID)
|
||||||
}
|
}
|
||||||
|
|
||||||
// resumeSession re-invokes the agent for a session with a system-injected note —
|
// resumeSession re-invokes the agent for a session with a system-injected note —
|
||||||
@@ -187,7 +204,32 @@ func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
|||||||
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
// (handleAnswerQuestion) — persisting progress LIVE (a placeholder row updated
|
||||||
// in place as each tool call lands) so the frontend poller sees each step,
|
// in place as each tool call lands) so the frontend poller sees each step,
|
||||||
// instead of total silence until the whole resume concludes.
|
// instead of total silence until the whole resume concludes.
|
||||||
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
//
|
||||||
|
// F1 (plan 2026-08-03): this is the single entry point for EVERY background
|
||||||
|
// turn — the continuation worker, idle sweep, answer-question, /resume, and the
|
||||||
|
// empty-message reconnect all funnel through here. It acquires the session's
|
||||||
|
// turn permit non-blocking and SKIPS if a turn is already running. A duplicate
|
||||||
|
// resume while a turn (live or background) is active is exactly the
|
||||||
|
// interleaving that corrupted the activity panel and made tasks feel stuck.
|
||||||
|
//
|
||||||
|
// Returns whether the turn actually ran. Callers that mutate state before
|
||||||
|
// resuming (the continuation worker's markContinued, the idle sweep's nudge
|
||||||
|
// bump) MUST gate that mutation on a true return — otherwise a busy-skip leaves
|
||||||
|
// the state changed but the work undone (lost continuation / false auto-close).
|
||||||
|
// See plans/2026-08-03-nomos-chat-changes-review.md P0/P1.
|
||||||
|
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||||
|
if !a.gate.acquire(sessionID, 0) {
|
||||||
|
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// Release the gate, then drain any operator message that was queued while
|
||||||
|
// this background turn ran (plan 2026-08-03 F2). Queued messages are run as
|
||||||
|
// real user turns server-side; resumeSession itself never enqueues.
|
||||||
|
defer func() {
|
||||||
|
a.gate.release(sessionID)
|
||||||
|
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||||
|
}()
|
||||||
|
|
||||||
placeholder, _ := json.Marshal(map[string]any{
|
placeholder, _ := json.Marshal(map[string]any{
|
||||||
"role": "assistant",
|
"role": "assistant",
|
||||||
"text": "",
|
"text": "",
|
||||||
@@ -200,6 +242,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
|
|
||||||
var toolCalls []map[string]any
|
var toolCalls []map[string]any
|
||||||
var finalText, errText string
|
var finalText, errText string
|
||||||
|
var finalThinking string
|
||||||
|
|
||||||
persist := func() {
|
persist := func() {
|
||||||
if msgID == uuid.Nil {
|
if msgID == uuid.Nil {
|
||||||
@@ -212,6 +255,7 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
body, _ := json.Marshal(map[string]any{
|
body, _ := json.Marshal(map[string]any{
|
||||||
"role": "assistant",
|
"role": "assistant",
|
||||||
"text": text,
|
"text": text,
|
||||||
|
"thinking": finalThinking,
|
||||||
"tool_calls": toolCalls,
|
"tool_calls": toolCalls,
|
||||||
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
"auto": true, // marks this as an autonomous continuation, not an operator turn
|
||||||
})
|
})
|
||||||
@@ -242,15 +286,19 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
if attempt > 0 {
|
if attempt > 0 {
|
||||||
select {
|
select {
|
||||||
case <-cctx.Done():
|
case <-cctx.Done():
|
||||||
return
|
return true // a turn ran on an earlier attempt; consume, don't re-loop
|
||||||
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
case <-time.After(time.Duration(2<<attempt) * time.Second): // 4s, 8s
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
toolCalls, finalText, errText = nil, "", ""
|
toolCalls, finalText, errText = nil, "", ""
|
||||||
|
finalThinking = ""
|
||||||
// P3: accumulate per-iteration reasoning instead of overwriting
|
// P3: accumulate per-iteration reasoning instead of overwriting
|
||||||
// (same fix as main.go's chat handler). Without this, a resumed
|
// (same fix as main.go's chat handler). Without this, a resumed
|
||||||
// turn's intermediate thinking is lost on reload.
|
// turn's intermediate thinking is lost on reload.
|
||||||
|
// (same fix as main.go's chat handler). Without this, a resumed
|
||||||
|
// turn's intermediate thinking is lost on reload.
|
||||||
var textParts []string
|
var textParts []string
|
||||||
|
var thinkingParts []string
|
||||||
emit := func(ev agentEvent) {
|
emit := func(ev agentEvent) {
|
||||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||||
if m, ok := ev.Data.(map[string]any); ok {
|
if m, ok := ev.Data.(map[string]any); ok {
|
||||||
@@ -277,8 +325,13 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
}
|
}
|
||||||
if ev.Type == "text" {
|
if ev.Type == "text" {
|
||||||
if t, ok := ev.Data.(string); ok && t != "" {
|
if t, ok := ev.Data.(string); ok && t != "" {
|
||||||
textParts = append(textParts, t)
|
if ev.IsThinking {
|
||||||
finalText = strings.Join(textParts, "\n\n")
|
thinkingParts = append(thinkingParts, t)
|
||||||
|
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||||
|
} else {
|
||||||
|
textParts = append(textParts, t)
|
||||||
|
finalText = strings.Join(textParts, "\n\n")
|
||||||
|
}
|
||||||
persist()
|
persist()
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -314,9 +367,10 @@ func (a *agent) resumeSession(ctx context.Context, sessionID, note string) {
|
|||||||
// No placeholder was inserted (rare), save directly.
|
// No placeholder was inserted (rare), save directly.
|
||||||
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
a.store.saveMessage(context.Background(), sessionID, "assistant", body)
|
||||||
}
|
}
|
||||||
return // do not call persist() again — already persisted above
|
return true // do not call persist() again — already persisted above
|
||||||
}
|
}
|
||||||
persist() // final state — same row, updated one last time with the concluding text
|
persist() // final state — same row, updated one last time with the concluding text
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildContinuationNote frames the finished execution for the model: what
|
// buildContinuationNote frames the finished execution for the model: what
|
||||||
|
|||||||
@@ -1,6 +1,11 @@
|
|||||||
package main
|
package main
|
||||||
|
|
||||||
import "testing"
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
func TestExtractExecutionIDs(t *testing.T) {
|
func TestExtractExecutionIDs(t *testing.T) {
|
||||||
// Real tool-result phrasings that should yield an execution id.
|
// Real tool-result phrasings that should yield an execution id.
|
||||||
@@ -37,3 +42,37 @@ func TestExtractExecutionIDs(t *testing.T) {
|
|||||||
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
t.Errorf("expected de-dup to 1 id, got %v", ids)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestResumeSession_SkipsWhenBusy guards the P0 fix
|
||||||
|
// (plans/2026-08-03-nomos-chat-changes-review.md): resumeSession must skip —
|
||||||
|
// return false, body never executed — when a turn is already active for the
|
||||||
|
// session. continueSession relies on this so it only marks a continuation
|
||||||
|
// "continued" after a turn really ran (otherwise the result is lost: marked
|
||||||
|
// continued, never re-queued by pendingContinuations).
|
||||||
|
//
|
||||||
|
// A minimal agent with only a gate is enough: if the body ever ran, chatWith
|
||||||
|
// would dereference the nil provider and panic. Returning false cleanly proves
|
||||||
|
// the body was skipped.
|
||||||
|
func TestResumeSession_SkipsWhenBusy(t *testing.T) {
|
||||||
|
a := &agent{gate: newTurnGate()}
|
||||||
|
if !a.gate.acquire("sess", 0) {
|
||||||
|
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||||
|
}
|
||||||
|
ran := a.resumeSession(context.Background(), "sess", "note")
|
||||||
|
if ran {
|
||||||
|
t.Fatal("resumeSession must return false (skip) while a turn is active for the session")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestContinueSession_DefersWhenBusy guards the other half of P0: when the
|
||||||
|
// session is busy, continueSession defers (leaves the execution pending for the
|
||||||
|
// next worker tick) instead of running or marking it. It must return cleanly
|
||||||
|
// without reaching resumeSession's body (nil provider → panic) or markContinued.
|
||||||
|
func TestContinueSession_DefersWhenBusy(t *testing.T) {
|
||||||
|
a := &agent{gate: newTurnGate()}
|
||||||
|
if !a.gate.acquire("sess", 0) {
|
||||||
|
t.Fatal("precondition: initial acquire should succeed on a free session")
|
||||||
|
}
|
||||||
|
p := pendingContinuation{ExecID: uuid.New(), SessionID: "sess", Status: "completed"}
|
||||||
|
a.continueSession(context.Background(), p) // must not panic; must not run/mark
|
||||||
|
}
|
||||||
|
|||||||
@@ -319,8 +319,10 @@ func fetchTranscript(ctx context.Context, gateway, sid string) (transcript, sess
|
|||||||
}
|
}
|
||||||
// Fetch the plan (steps with generation numbers) for the
|
// Fetch the plan (steps with generation numbers) for the
|
||||||
// plan_generations assertion. A 404 or empty response is fine — a
|
// plan_generations assertion. A 404 or empty response is fine — a
|
||||||
// pure-DB Q&A with no propose_plan has no plan.
|
// pure-DB Q&A with no propose_plan has no plan. ?all=true returns every
|
||||||
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan"); perr == nil {
|
// generation so the assertion can count them (the default view returns
|
||||||
|
// only the current generation).
|
||||||
|
if planResp, perr := http.Get(gateway + "/sessions/" + sid + "/plan?all=true"); perr == nil {
|
||||||
if planResp.StatusCode == 200 {
|
if planResp.StatusCode == 200 {
|
||||||
pb, _ := io.ReadAll(planResp.Body)
|
pb, _ := io.ReadAll(planResp.Body)
|
||||||
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
_ = json.Unmarshal(pb, &t) // fills t.PlanSteps via "steps" field
|
||||||
|
|||||||
@@ -167,6 +167,147 @@ func sseEvent(w http.ResponseWriter, flusher http.Flusher, event agentEvent) {
|
|||||||
flusher.Flush()
|
flusher.Flush()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// runChatTurn is the shared core of an operator-initiated turn: insert an
|
||||||
|
// assistant placeholder, run a.chat with incremental persistence (so whatever
|
||||||
|
// happened before an abort is never lost), finalize the row, and derive a
|
||||||
|
// title. It is agnostic to the transport: `sink` receives every agent event
|
||||||
|
// for delivery (SSE for a live handleChat, a no-op for a queued turn that has
|
||||||
|
// no client attached — the frontend learns about those via the poller + the
|
||||||
|
// status-driven "working" signal). The caller MUST already hold the session's
|
||||||
|
// turn-gate permit.
|
||||||
|
func (a *agent) runChatTurn(pctx, ctx context.Context, sessionID, message string, sink func(agentEvent)) {
|
||||||
|
toolCalls := []map[string]any{}
|
||||||
|
// P3: accumulate per-iteration reasoning instead of overwriting with the
|
||||||
|
// final `text` event (see the original inline comment in handleChat).
|
||||||
|
var textParts []string
|
||||||
|
var thinkingParts []string
|
||||||
|
var finalText string
|
||||||
|
var finalThinking string
|
||||||
|
|
||||||
|
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
||||||
|
msgID, err := a.store.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
||||||
|
}
|
||||||
|
persist := func() {
|
||||||
|
if msgID == uuid.Nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
body, _ := json.Marshal(map[string]any{
|
||||||
|
"role": "assistant",
|
||||||
|
"text": finalText,
|
||||||
|
"thinking": finalThinking,
|
||||||
|
"tool_calls": toolCalls,
|
||||||
|
})
|
||||||
|
a.store.updateMessage(pctx, msgID, body)
|
||||||
|
}
|
||||||
|
|
||||||
|
a.chat(ctx, sessionID, message, func(ev agentEvent) {
|
||||||
|
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
||||||
|
if m, ok := ev.Data.(map[string]any); ok {
|
||||||
|
m["type"] = ev.Type
|
||||||
|
// One entry per tool call: tool_use creates it, tool_result
|
||||||
|
// merges the result into the same entry (matched by id).
|
||||||
|
id, _ := m["id"].(string)
|
||||||
|
if id != "" && ev.Type == "tool_result" {
|
||||||
|
for _, existing := range toolCalls {
|
||||||
|
if eID, _ := existing["id"].(string); eID == id {
|
||||||
|
for k, v := range m {
|
||||||
|
existing[k] = v
|
||||||
|
}
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
toolCalls = append(toolCalls, m)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
persist() // live: survives even if the client disconnects right after
|
||||||
|
}
|
||||||
|
if ev.Type == "text" {
|
||||||
|
if t, ok := ev.Data.(string); ok && t != "" {
|
||||||
|
if ev.IsThinking {
|
||||||
|
thinkingParts = append(thinkingParts, t)
|
||||||
|
finalThinking = strings.Join(thinkingParts, "\n\n")
|
||||||
|
} else {
|
||||||
|
textParts = append(textParts, t)
|
||||||
|
finalText = strings.Join(textParts, "\n\n")
|
||||||
|
}
|
||||||
|
persist()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sink(ev)
|
||||||
|
})
|
||||||
|
|
||||||
|
// B.6: if the turn ended with no text and no tool calls (the model
|
||||||
|
// empty-response'd and all retries failed), delete the placeholder row
|
||||||
|
// instead of persisting an empty bubble.
|
||||||
|
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
||||||
|
a.store.deleteMessage(pctx, msgID)
|
||||||
|
} else {
|
||||||
|
persist() // final state — same row, updated one last time
|
||||||
|
}
|
||||||
|
|
||||||
|
// Title: prefer the goal once set; else the first assistant answer.
|
||||||
|
if finalText != "" && sessionID != "ephemeral" {
|
||||||
|
var goalTitle string
|
||||||
|
if sess, gerr := a.store.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
||||||
|
goalTitle = truncate(sess.Goal, 120)
|
||||||
|
}
|
||||||
|
title := goalTitle
|
||||||
|
if title == "" {
|
||||||
|
title = truncate(finalText, 80)
|
||||||
|
}
|
||||||
|
if title != "" {
|
||||||
|
a.store.updateSessionTitle(pctx, sessionID, title)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// drainAcquireWait is how long drainQueued blocks for a busy gate before
|
||||||
|
// re-queuing and deferring to the holder's own release-drain. A package var so
|
||||||
|
// tests can shorten it; in production it just needs to outlast the brief
|
||||||
|
// release→drain handoff window.
|
||||||
|
var drainAcquireWait = 5 * time.Second
|
||||||
|
|
||||||
|
// drainQueued runs every queued operator message for a session as its own turn,
|
||||||
|
// one at a time, under the turn gate. Called (in a goroutine) whenever a turn
|
||||||
|
// releases the gate — from handleChat (live) and resumeSession (background) —
|
||||||
|
// so a message queued while the agent was busy is acted on as soon as it's
|
||||||
|
// free, without the operator re-sending. See messagequeue.go (plan 2026-08-03
|
||||||
|
// F2).
|
||||||
|
//
|
||||||
|
// Each queued turn is persisted incrementally and has no SSE client (the
|
||||||
|
// browser detached after receiving the `queued` event); the frontend sees the
|
||||||
|
// result via the 3s poller and the status-driven "working" indicator.
|
||||||
|
func (a *agent) drainQueued(ctx context.Context, sessionID string) {
|
||||||
|
for {
|
||||||
|
msg, ok := a.queue.dequeue(sessionID)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
// Block briefly for the gate. If a live turn grabbed it first, put the
|
||||||
|
// message back — that turn's release will drain it again. Never stack.
|
||||||
|
if !a.gate.acquire(sessionID, drainAcquireWait) {
|
||||||
|
a.queue.requeueFront(sessionID, msg)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
slog.Info("nomos: running queued operator message", "session", sessionID)
|
||||||
|
pctx := context.Background()
|
||||||
|
// Run the turn inside a per-iteration closure so the gate release is
|
||||||
|
// deferred to the end of THIS turn (and runs even if runChatTurn
|
||||||
|
// panics — safego recovers the panic at the goroutine boundary, so a
|
||||||
|
// non-deferred release would be skipped and the session's permit held
|
||||||
|
// forever, deadlocking all future turns). A bare `defer release` in
|
||||||
|
// the loop would be wrong too: Go defers run at function exit, not
|
||||||
|
// iteration exit, so the gate would stay held across iterations.
|
||||||
|
func() {
|
||||||
|
defer a.gate.release(sessionID)
|
||||||
|
a.runChatTurn(pctx, ctx, sessionID, msg, func(agentEvent) {})
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
||||||
if r.Method != http.MethodPost {
|
if r.Method != http.MethodPost {
|
||||||
http.Error(w, "method not allowed", 405)
|
http.Error(w, "method not allowed", 405)
|
||||||
@@ -186,12 +327,22 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Empty message with an existing session = reconnect/resume. The
|
// Empty message with an existing session = reconnect/resume. This path is
|
||||||
// frontend sends this after a dropped SSE stream to re-establish the
|
// defensive now — the frontend (post F2) recovers a dropped SSE via the
|
||||||
// connection and catch up on any auto-continuation work that happened
|
// poller + terminal task.status clearing, and no longer POSTs empty
|
||||||
// while disconnected. Route into resumeSession so the agent sees a
|
// messages. If a client ever does, route into resumeSession so the agent
|
||||||
// system note and reports current state.
|
// reports current state — but SKIP a terminal session (done/failed/
|
||||||
|
// abandoned): there's nothing to resume, and running a "report state"
|
||||||
|
// turn there is just a spare turn the operator never asked for (P2.1).
|
||||||
if req.Message == "" && req.SessionID != "" {
|
if req.Message == "" && req.SessionID != "" {
|
||||||
|
if sess, err := st.getSession(context.Background(), req.SessionID); err == nil {
|
||||||
|
switch sess.Status {
|
||||||
|
case "done", "failed", "abandoned":
|
||||||
|
slog.Info("nomos: reconnect skipped — session already terminal", "session", req.SessionID, "status", sess.Status)
|
||||||
|
w.WriteHeader(202)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
slog.Info("nomos: reconnect", "session", req.SessionID)
|
slog.Info("nomos: reconnect", "session", req.SessionID)
|
||||||
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
safego.Go("nomos:reconnect:"+req.SessionID, func() {
|
||||||
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
base := "[System: the operator's connection was re-established. The task may have progressed in the background.]"
|
||||||
@@ -200,7 +351,7 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
})
|
})
|
||||||
// Return 202 so the frontend doesn't try to consume an SSE stream
|
// Return 202 so the frontend doesn't try to consume an SSE stream
|
||||||
// from this POST — resumeSession writes to the DB directly and
|
// from this POST — resumeSession writes to the DB directly and
|
||||||
// the poller (already running from handleDisconnect) picks it up.
|
// the poller picks it up.
|
||||||
w.WriteHeader(202)
|
w.WriteHeader(202)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -217,6 +368,17 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
|
w.Header().Set("X-Accel-Buffering", "no") // disable proxy buffering
|
||||||
w.WriteHeader(200)
|
w.WriteHeader(200)
|
||||||
|
|
||||||
|
// All writes to w (events + the keepalive comment below) go through one
|
||||||
|
// mutex: http.ResponseWriter is NOT safe for concurrent use, and the
|
||||||
|
// keepalive ticker runs alongside the turn's event sink (plan 2026-08-03
|
||||||
|
// F3). Without this, interleaved writes corrupt the SSE stream.
|
||||||
|
var writeMu sync.Mutex
|
||||||
|
writeEvent := func(ev agentEvent) {
|
||||||
|
writeMu.Lock()
|
||||||
|
defer writeMu.Unlock()
|
||||||
|
sseEvent(w, flusher, ev)
|
||||||
|
}
|
||||||
|
|
||||||
ctx := r.Context()
|
ctx := r.Context()
|
||||||
sessionID := req.SessionID
|
sessionID := req.SessionID
|
||||||
|
|
||||||
@@ -268,111 +430,65 @@ func handleChat(w http.ResponseWriter, r *http.Request, a *agent, st *store) {
|
|||||||
st.answerQuestion(pctx, sessionID, qid, req.Message)
|
st.answerQuestion(pctx, sessionID, qid, req.Message)
|
||||||
}
|
}
|
||||||
|
|
||||||
sseEvent(w, flusher, agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
writeEvent(agentEvent{Type: "session", Data: sessionID, SessionID: sessionID})
|
||||||
|
|
||||||
toolCalls := []map[string]any{}
|
// F1/F2 (plan 2026-08-03): serialize turns per session. The user message is
|
||||||
// P3: accumulate per-iteration reasoning instead of overwriting with
|
// already persisted above, so it is never lost. Wait briefly for a finishing
|
||||||
// the final `text` event. The agent loop emits a `text` event for each
|
// background turn; if one is still running after that, QUEUE this message
|
||||||
// LLM iteration that produced text (intermediate reasoning before tool
|
// (don't reject it) and tell the client so it shows a "queued" state. The
|
||||||
// calls + the final answer). Without accumulation, only the last `text`
|
// in-flight turn's release drains the queue (drainQueued) and runs it as a
|
||||||
// survives in the persisted row — a reload shows the final summary but
|
// real turn server-side. This never stacks concurrent turns — the gate still
|
||||||
// not the thinking that led to each tool call.
|
// guarantees one in-flight turn per session.
|
||||||
var textParts []string
|
const turnWait = 5 * time.Second
|
||||||
var finalText string
|
if !a.gate.acquire(sessionID, turnWait) {
|
||||||
|
a.queue.enqueue(sessionID, req.Message)
|
||||||
// Incremental persistence, mirroring resumeSession's existing
|
slog.Info("nomos: turn already active, queued operator message", "session", sessionID)
|
||||||
// placeholder+update pattern (continue.go): insert a placeholder now,
|
writeEvent(agentEvent{Type: "queued", Data: sessionID, SessionID: sessionID})
|
||||||
// update the SAME row after every tool call, so whatever happened before
|
writeEvent(agentEvent{Type: "done", Data: map[string]any{
|
||||||
// an abort is never lost — only what hadn't happened yet is.
|
"session_id": sessionID,
|
||||||
placeholder, _ := json.Marshal(map[string]any{"role": "assistant", "text": ""})
|
"queued": true,
|
||||||
msgID, err := st.insertMessageReturningID(pctx, sessionID, "assistant", placeholder)
|
}, SessionID: sessionID})
|
||||||
if err != nil {
|
return
|
||||||
slog.Error("nomos: chat placeholder insert failed", "session", sessionID, "error", err)
|
|
||||||
}
|
|
||||||
persist := func() {
|
|
||||||
if msgID == uuid.Nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
body, _ := json.Marshal(map[string]any{
|
|
||||||
"role": "assistant",
|
|
||||||
"text": finalText,
|
|
||||||
"tool_calls": toolCalls,
|
|
||||||
})
|
|
||||||
st.updateMessage(pctx, msgID, body)
|
|
||||||
}
|
}
|
||||||
|
defer func() {
|
||||||
|
a.gate.release(sessionID)
|
||||||
|
// Run any message that was queued while this turn held the gate. In a
|
||||||
|
// goroutine so the HTTP response finishes without waiting on the next
|
||||||
|
// turn; the queued turn has no SSE client of its own.
|
||||||
|
safego.Go("nomos:drain:"+sessionID, func() { a.drainQueued(context.Background(), sessionID) })
|
||||||
|
}()
|
||||||
|
|
||||||
a.chat(ctx, sessionID, req.Message, func(ev agentEvent) {
|
// F3 (plan 2026-08-03): keep the SSE alive during long turns. A turn can
|
||||||
if ev.Type == "tool_use" || ev.Type == "tool_result" {
|
// run for many minutes (provisioning chains, deep research); the model
|
||||||
if m, ok := ev.Data.(map[string]any); ok {
|
// often takes 20-40s between tool iterations, and with nothing flushed in
|
||||||
m["type"] = ev.Type
|
// that gap a proxy/browser idle timeout silently closes the stream. The
|
||||||
// One entry per tool call: tool_use creates it, tool_result
|
// client then sees streaming=false while the server keeps working — the
|
||||||
// merges the result into the same entry (matched by id).
|
// "I can't tell it's working" desync. An SSE comment line (":keepalive") is
|
||||||
// Before this fix, both events appended separate entries,
|
// ignored by EventSource but resets idle timers.
|
||||||
// doubling every tool call in the persisted transcript
|
keepDone := make(chan struct{})
|
||||||
// (confirmed pre-existing in d9cdcee1, v0.3.x era).
|
go func() {
|
||||||
id, _ := m["id"].(string)
|
t := time.NewTicker(12 * time.Second)
|
||||||
if id != "" && ev.Type == "tool_result" {
|
defer t.Stop()
|
||||||
for _, existing := range toolCalls {
|
for {
|
||||||
if eID, _ := existing["id"].(string); eID == id {
|
select {
|
||||||
for k, v := range m {
|
case <-keepDone:
|
||||||
existing[k] = v
|
return
|
||||||
}
|
case <-t.C:
|
||||||
break
|
writeMu.Lock()
|
||||||
}
|
fmt.Fprintf(w, ":keepalive\n\n")
|
||||||
}
|
flusher.Flush()
|
||||||
} else {
|
writeMu.Unlock()
|
||||||
toolCalls = append(toolCalls, m)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
persist() // live: survives even if the client disconnects right after
|
|
||||||
}
|
|
||||||
if ev.Type == "text" {
|
|
||||||
// P3: accumulate. Each `text` event is one iteration's reasoning
|
|
||||||
// (or the final answer). Join with newlines so the persisted row
|
|
||||||
// reads as the full transcript of what the agent said, not just
|
|
||||||
// the last thing.
|
|
||||||
if t, ok := ev.Data.(string); ok && t != "" {
|
|
||||||
textParts = append(textParts, t)
|
|
||||||
finalText = strings.Join(textParts, "\n\n")
|
|
||||||
persist()
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
sseEvent(w, flusher, ev)
|
}()
|
||||||
|
// Defer the close (not a statement after runChatTurn) so the goroutine
|
||||||
|
// exits even if runChatTurn panics — net/http recovers handler panics, so
|
||||||
|
// a non-deferred close would be skipped and the ticker would keep writing
|
||||||
|
// to a dead ResponseWriter forever.
|
||||||
|
defer close(keepDone)
|
||||||
|
a.runChatTurn(pctx, ctx, sessionID, req.Message, func(ev agentEvent) {
|
||||||
|
writeEvent(ev)
|
||||||
})
|
})
|
||||||
|
|
||||||
// B.6: if the turn ended with no text and no tool calls (the model
|
|
||||||
// empty-response'd and all retries failed), delete the placeholder row
|
|
||||||
// instead of persisting an empty bubble. The error event was already
|
|
||||||
// streamed to the frontend via the 'done with error=true' event, so the
|
|
||||||
// operator sees the error inline — an empty assistant bubble in the
|
|
||||||
// transcript adds nothing and looks like the agent is broken.
|
|
||||||
if finalText == "" && len(toolCalls) == 0 && msgID != uuid.Nil {
|
|
||||||
st.deleteMessage(pctx, msgID)
|
|
||||||
} else {
|
|
||||||
persist() // final state — same row, updated one last time with the concluding text
|
|
||||||
}
|
|
||||||
|
|
||||||
// Generate a meaningful title from the assistant's first answer
|
|
||||||
// instead of reusing the raw user message for every session.
|
|
||||||
// P2.9 (2026-07-20): prefer the goal as the title when one is set —
|
|
||||||
// the first assistant text is often a greeting or narrative that
|
|
||||||
// doesn't describe the task ("Hey! 👋 Nomos here, running on
|
|
||||||
// mac-mini:8092..."). The goal is the operator's actual intent.
|
|
||||||
// Sessions that never call set_goal (pure Q&A) fall back to the
|
|
||||||
// assistant text, which is still better than the raw user message.
|
|
||||||
if finalText != "" && sessionID != "ephemeral" {
|
|
||||||
var goalTitle string
|
|
||||||
if sess, gerr := st.getSession(pctx, sessionID); gerr == nil && sess.Goal != "" {
|
|
||||||
goalTitle = truncate(sess.Goal, 120)
|
|
||||||
}
|
|
||||||
title := goalTitle
|
|
||||||
if title == "" {
|
|
||||||
title = truncate(finalText, 80)
|
|
||||||
}
|
|
||||||
if title != "" {
|
|
||||||
st.updateSessionTitle(pctx, sessionID, title)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
func handleSessionsList(w http.ResponseWriter, r *http.Request, st *store) {
|
||||||
@@ -483,7 +599,8 @@ func handleSessionDetail(w http.ResponseWriter, r *http.Request, st *store, a *a
|
|||||||
if len(parts) == 2 && r.Method == http.MethodGet {
|
if len(parts) == 2 && r.Method == http.MethodGet {
|
||||||
switch parts[1] {
|
switch parts[1] {
|
||||||
case "plan":
|
case "plan":
|
||||||
steps, err := st.getPlanSteps(r.Context(), id)
|
all := r.URL.Query().Has("all") && r.URL.Query().Get("all") != "0" && r.URL.Query().Get("all") != "false"
|
||||||
|
steps, err := st.getPlanSteps(r.Context(), id, all)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
http.Error(w, err.Error(), 500)
|
http.Error(w, err.Error(), 500)
|
||||||
return
|
return
|
||||||
@@ -701,7 +818,7 @@ func newMCPClient(baseURL, token string) (*mcpClient, error) {
|
|||||||
c := &mcpClient{
|
c := &mcpClient{
|
||||||
baseURL: baseURL,
|
baseURL: baseURL,
|
||||||
token: token,
|
token: token,
|
||||||
http: &http.Client{Timeout: 30 * time.Second},
|
http: &http.Client{Timeout: 120 * time.Second},
|
||||||
}
|
}
|
||||||
|
|
||||||
resp, err := c.doRequest("initialize", map[string]any{
|
resp, err := c.doRequest("initialize", map[string]any{
|
||||||
|
|||||||
82
cmd/nomos/messagequeue.go
Normal file
82
cmd/nomos/messagequeue.go
Normal file
@@ -0,0 +1,82 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log/slog"
|
||||||
|
"sync"
|
||||||
|
)
|
||||||
|
|
||||||
|
// maxQueuedPerSession caps a session's queue. A held turn plus unbounded
|
||||||
|
// enqueues would grow memory without limit; an operator nudging a long
|
||||||
|
// autonomous turn realistically queues only a handful, so a generous cap is
|
||||||
|
// pure insurance. Overflow drops the newest enqueue and logs (the message is
|
||||||
|
// already persisted in the DB by handleChat before enqueue, so it isn't lost
|
||||||
|
// from the transcript — it just won't auto-run).
|
||||||
|
const maxQueuedPerSession = 20
|
||||||
|
|
||||||
|
// messageQueue holds operator messages that arrived while a turn was already
|
||||||
|
// running for a session. Plan 2026-08-03 (F2): instead of rejecting the
|
||||||
|
// operator's message with "Nomos is still finishing a previous step… send it
|
||||||
|
// again", the message is queued and auto-run when the in-flight turn releases
|
||||||
|
// the session's turn-gate permit.
|
||||||
|
//
|
||||||
|
// The queue only schedules WHEN a turn runs, not WHETHER the message is stored
|
||||||
|
// — handleChat persists the user message before acquiring the gate, so a queued
|
||||||
|
// message is already in the transcript; this just makes sure a turn eventually
|
||||||
|
// acts on it.
|
||||||
|
//
|
||||||
|
// Draining is strictly one-at-a-time under the turn gate (see drainQueued in
|
||||||
|
// main.go), so this cannot stack concurrent turns — the exact hazard the gate
|
||||||
|
// itself exists to prevent. Background resumeSession callers never touch this
|
||||||
|
// queue; they keep their non-blocking skip.
|
||||||
|
type messageQueue struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
queue map[string][]string
|
||||||
|
}
|
||||||
|
|
||||||
|
func newMessageQueue() *messageQueue {
|
||||||
|
return &messageQueue{queue: map[string][]string{}}
|
||||||
|
}
|
||||||
|
|
||||||
|
// enqueue appends a message to the back of the session's FIFO. Returns false
|
||||||
|
// (and logs) if the session is already at maxQueuedPerSession — the caller's
|
||||||
|
// message is already persisted in the DB, so this only skips auto-running it.
|
||||||
|
func (q *messageQueue) enqueue(sessionID, msg string) bool {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
if len(q.queue[sessionID]) >= maxQueuedPerSession {
|
||||||
|
slog.Warn("nomos: message queue full; dropping auto-run for operator message", "session", sessionID, "cap", maxQueuedPerSession)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
q.queue[sessionID] = append(q.queue[sessionID], msg)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// dequeue pops the next message from the front of the session's FIFO. Returns
|
||||||
|
// ok=false when empty.
|
||||||
|
func (q *messageQueue) dequeue(sessionID string) (string, bool) {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
xs := q.queue[sessionID]
|
||||||
|
if len(xs) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
m := xs[0]
|
||||||
|
q.queue[sessionID] = xs[1:]
|
||||||
|
return m, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// requeueFront pushes a message back to the front — used when a drainer popped
|
||||||
|
// a message but lost the race for the gate to a live turn; that turn's own
|
||||||
|
// release will drain it again.
|
||||||
|
func (q *messageQueue) requeueFront(sessionID, msg string) {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
q.queue[sessionID] = append([]string{msg}, q.queue[sessionID]...)
|
||||||
|
}
|
||||||
|
|
||||||
|
// peek reports the queued depth for a session (test/diagnostic helper).
|
||||||
|
func (q *messageQueue) peek(sessionID string) int {
|
||||||
|
q.mu.Lock()
|
||||||
|
defer q.mu.Unlock()
|
||||||
|
return len(q.queue[sessionID])
|
||||||
|
}
|
||||||
142
cmd/nomos/messagequeue_test.go
Normal file
142
cmd/nomos/messagequeue_test.go
Normal file
@@ -0,0 +1,142 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestMessageQueue_FIFO(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
q.enqueue("s", "first")
|
||||||
|
q.enqueue("s", "second")
|
||||||
|
q.enqueue("s", "third")
|
||||||
|
|
||||||
|
want := []string{"first", "second", "third"}
|
||||||
|
for _, w := range want {
|
||||||
|
got, ok := q.dequeue("s")
|
||||||
|
if !ok || got != w {
|
||||||
|
t.Fatalf("dequeue = %q,%v want %q,true", got, ok, w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, ok := q.dequeue("s"); ok {
|
||||||
|
t.Fatal("dequeue on drained queue should return ok=false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_RequeueFront(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
q.enqueue("s", "a")
|
||||||
|
q.enqueue("s", "b")
|
||||||
|
// Pop "a", then push it back to the front; "a" must come out before "b".
|
||||||
|
a, _ := q.dequeue("s")
|
||||||
|
q.requeueFront("s", a)
|
||||||
|
got, _ := q.dequeue("s")
|
||||||
|
if got != "a" {
|
||||||
|
t.Fatalf("after requeueFront, dequeue = %q want %q", got, "a")
|
||||||
|
}
|
||||||
|
got2, _ := q.dequeue("s")
|
||||||
|
if got2 != "b" {
|
||||||
|
t.Fatalf("next dequeue = %q want %q", got2, "b")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_IsolatedPerSession(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
q.enqueue("s1", "one")
|
||||||
|
q.enqueue("s2", "two")
|
||||||
|
if got, _ := q.dequeue("s1"); got != "one" {
|
||||||
|
t.Fatalf("s1 = %q want one", got)
|
||||||
|
}
|
||||||
|
if got, _ := q.dequeue("s2"); got != "two" {
|
||||||
|
t.Fatalf("s2 = %q want two", got)
|
||||||
|
}
|
||||||
|
if q.peek("s1") != 0 || q.peek("s2") != 0 {
|
||||||
|
t.Fatal("both sessions should be drained")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_Concurrent(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
const n = maxQueuedPerSession // stay under the cap so every enqueue lands
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func(i int) {
|
||||||
|
defer wg.Done()
|
||||||
|
q.enqueue("s", "m")
|
||||||
|
}(i)
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
if q.peek("s") != n {
|
||||||
|
t.Fatalf("peek = %d want %d (all enqueues must be counted)", q.peek("s"), n)
|
||||||
|
}
|
||||||
|
seen := 0
|
||||||
|
for {
|
||||||
|
if _, ok := q.dequeue("s"); !ok {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
seen++
|
||||||
|
}
|
||||||
|
if seen != n {
|
||||||
|
t.Fatalf("drained %d want %d", seen, n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMessageQueue_CapsOverflow(t *testing.T) {
|
||||||
|
q := newMessageQueue()
|
||||||
|
for i := 0; i < maxQueuedPerSession; i++ {
|
||||||
|
if !q.enqueue("s", "m") {
|
||||||
|
t.Fatalf("enqueue #%d within cap should succeed", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if q.enqueue("s", "overflow") {
|
||||||
|
t.Fatal("enqueue past the cap should return false (dropped)")
|
||||||
|
}
|
||||||
|
if got := q.peek("s"); got != maxQueuedPerSession {
|
||||||
|
t.Fatalf("peek = %d want %d (overflow must not append)", got, maxQueuedPerSession)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// drainQueued on an empty queue must be a no-op: it returns immediately and
|
||||||
|
// never touches the gate (so the session stays free for the next turn).
|
||||||
|
func TestDrainQueued_NoOpOnEmpty(t *testing.T) {
|
||||||
|
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||||
|
a.drainQueued(context.Background(), "s")
|
||||||
|
if !a.gate.acquire("s", 0) {
|
||||||
|
t.Fatal("gate should be free after a no-op drain (drain must not hold it)")
|
||||||
|
}
|
||||||
|
a.gate.release("s")
|
||||||
|
}
|
||||||
|
|
||||||
|
// With a queued message but the gate held by another turn, drainQueued must
|
||||||
|
// re-queue the message and return WITHOUT running a turn (no store/provider → a
|
||||||
|
// real run would panic). This is the "never stack" property: a busy gate
|
||||||
|
// defers to the holder's own release-drain.
|
||||||
|
func TestDrainQueued_RequeuesWhenBusy(t *testing.T) {
|
||||||
|
prev := drainAcquireWait
|
||||||
|
drainAcquireWait = 10 * time.Millisecond
|
||||||
|
t.Cleanup(func() { drainAcquireWait = prev })
|
||||||
|
|
||||||
|
a := &agent{gate: newTurnGate(), queue: newMessageQueue()}
|
||||||
|
if !a.gate.acquire("s", 0) {
|
||||||
|
t.Fatal("precondition: hold the gate")
|
||||||
|
}
|
||||||
|
a.queue.enqueue("s", "queued-msg")
|
||||||
|
|
||||||
|
done := make(chan struct{})
|
||||||
|
go func() {
|
||||||
|
a.drainQueued(context.Background(), "s") // must not panic; must requeue
|
||||||
|
close(done)
|
||||||
|
}()
|
||||||
|
select {
|
||||||
|
case <-done:
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("drainQueued did not return promptly while the gate was busy")
|
||||||
|
}
|
||||||
|
if got := a.queue.peek("s"); got != 1 {
|
||||||
|
t.Fatalf("message should be re-queued while busy; peek = %d want 1", got)
|
||||||
|
}
|
||||||
|
a.gate.release("s")
|
||||||
|
}
|
||||||
@@ -13,6 +13,7 @@ import (
|
|||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
"github.com/jackc/pgx/v5/pgxpool"
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -25,6 +26,13 @@ const maxToolResultSize = 4096
|
|||||||
// The caller translates this into a directive tool result.
|
// The caller translates this into a directive tool result.
|
||||||
var errPlanInFlight = errors.New("plan already in flight")
|
var errPlanInFlight = errors.New("plan already in flight")
|
||||||
|
|
||||||
|
// errPlanStepNotFound is returned by updatePlanStep when no step matches the
|
||||||
|
// given seq in the CURRENT (MAX) generation — either the seq is out of range,
|
||||||
|
// or (after a re-plan) the model addressed a stale 1-based number. seq is
|
||||||
|
// generation-relative, so this never resurrects a superseded generation's row.
|
||||||
|
// The caller translates it into a directive tool result (P0.1).
|
||||||
|
var errPlanStepNotFound = errors.New("plan step not found in current generation")
|
||||||
|
|
||||||
type store struct {
|
type store struct {
|
||||||
pool *pgxpool.Pool
|
pool *pgxpool.Pool
|
||||||
}
|
}
|
||||||
@@ -799,10 +807,11 @@ func (s *store) setGoal(ctx context.Context, sessionID, goal string) error {
|
|||||||
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
|
// Replace any prior plan steps (done/running/pending/...) as `replaced`.
|
||||||
// The rows are kept for the generation counter + audit trail; proposePlan
|
// The rows are kept for the generation counter + audit trail; proposePlan
|
||||||
// excludes `replaced` from its in-flight check, so the next propose_plan
|
// excludes `replaced` from its in-flight check, so the next propose_plan
|
||||||
// takes the fresh-generation path.
|
// takes the fresh-generation path. replaced_reason records the cause
|
||||||
|
// (2026-08-04 plan-step integrity audit).
|
||||||
s.pool.Exec(ctx,
|
s.pool.Exec(ctx,
|
||||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||||
sessionID)
|
sessionID, "goal superseded")
|
||||||
if _, err := s.pool.Exec(ctx,
|
if _, err := s.pool.Exec(ctx,
|
||||||
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
|
`UPDATE agent_sessions SET goal = $2, status = 'executing', title = $2, last_active_at = now() WHERE id = $1`,
|
||||||
sessionID, goal); err != nil {
|
sessionID, goal); err != nil {
|
||||||
@@ -842,6 +851,14 @@ func (s *store) reopenSession(ctx context.Context, sessionID string) bool {
|
|||||||
s.pool.Exec(ctx,
|
s.pool.Exec(ctx,
|
||||||
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
|
`UPDATE agent_sessions SET status = 'executing', outcome = NULL, summary = NULL, last_active_at = now() WHERE id = $1`,
|
||||||
sessionID)
|
sessionID)
|
||||||
|
// Mark the prior plan's steps as replaced so the P1 plan-first gate in
|
||||||
|
// classifyAndGate forces a fresh propose_plan before any run. Without
|
||||||
|
// this, the agent could resume a session and call run against the old
|
||||||
|
// (completed) plan — exactly what caused the ZimaOS continuation to
|
||||||
|
// have 81 ad-hoc tool calls with zero plan structure (2026-08-04).
|
||||||
|
s.pool.Exec(ctx,
|
||||||
|
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status <> 'replaced'`,
|
||||||
|
sessionID, "session reopened — awaiting new plan")
|
||||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
|
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.reopened", s.taskEntityPtr(ctx, sessionID),
|
||||||
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
|
"info", "nomos", sessionID, map[string]any{"prior_status": currentStatus})
|
||||||
return true
|
return true
|
||||||
@@ -879,16 +896,15 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
}
|
}
|
||||||
defer tx.Rollback(ctx)
|
defer tx.Rollback(ctx)
|
||||||
|
|
||||||
var startSeq int
|
|
||||||
var anyStarted bool
|
var anyStarted bool
|
||||||
// `replaced` steps (from a prior plan generation superseded by a
|
// `replaced` steps (from a prior plan generation superseded by a
|
||||||
// follow-up sub-task — see reopenSession) are excluded: they prove a
|
// follow-up sub-task — see setGoal/reopenSession) are excluded: they
|
||||||
// prior plan was completed and superseded, not that a plan is in flight.
|
// prove a prior plan was completed and superseded, not that a plan is in
|
||||||
// Without this exclusion, reopenSession's `replaced` marking would be
|
// flight. Without this exclusion, setGoal's `replaced` marking would be
|
||||||
// useless — propose_plan would still refuse on the follow-up.
|
// useless — propose_plan would still refuse on the follow-up.
|
||||||
if err := tx.QueryRow(ctx, `
|
if err := tx.QueryRow(ctx, `
|
||||||
SELECT COALESCE(max(seq), 0), COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
SELECT COALESCE(bool_or(status NOT IN ('pending', 'replaced')), false)
|
||||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&startSeq, &anyStarted); err != nil {
|
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&anyStarted); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
if anyStarted {
|
if anyStarted {
|
||||||
@@ -898,35 +914,29 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
return nil, errPlanInFlight
|
return nil, errPlanInFlight
|
||||||
}
|
}
|
||||||
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
// Fresh/revise: mark any prior PENDING steps as `replaced` (not DELETE).
|
||||||
// This preserves the rows for the generation counter (MAX(generation)+1
|
// The rows are kept for the generation counter (MAX(generation)+1 below)
|
||||||
// below) and the plan_generations eval assertion. Without this, a first
|
// and the plan_generations eval assertion. `replaced` steps are excluded
|
||||||
// plan that was proposed but never executed (all pending) would be
|
// from the anyStarted check above, so they don't block this proposal.
|
||||||
// wiped, resetting the counter to 1 — making a follow-up's plan look
|
// replaced_reason records the cause — required by the plan-step integrity
|
||||||
// like generation 1 instead of 2. `replaced` steps are excluded from
|
// gate (2026-08-04 session audit).
|
||||||
// the anyStarted check above, so they don't block the fresh proposal.
|
|
||||||
if _, err := tx.Exec(ctx,
|
if _, err := tx.Exec(ctx,
|
||||||
`UPDATE session_plan_steps SET status = 'replaced', finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
`UPDATE session_plan_steps SET status = 'replaced', replaced_reason = $2, finished_at = COALESCE(finished_at, now()) WHERE session_id = $1 AND status = 'pending'`,
|
||||||
sessionID); err != nil {
|
sessionID, "superseded by new plan generation"); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
// startSeq keeps the max(seq) from the query above: if prior steps
|
|
||||||
// exist (replaced or done), the new generation's steps start after them
|
|
||||||
// (no seq collisions across generations). If no rows exist (first plan),
|
|
||||||
// startSeq is 0 and the first step is seq 1.
|
|
||||||
|
|
||||||
// Resolve the generation number for this plan. Generation 1 is the
|
// nextGen: generation 1 for the first plan, MAX(generation)+1 for every
|
||||||
// initial plan; a genuine revise (which currently goes through the same
|
// revise/follow-up (prior rows were marked `replaced` above, not deleted,
|
||||||
// fresh-start path above because all steps were pending) resets to 1
|
// so the counter survives). seq is generation-relative — it resets to
|
||||||
// since the DELETE wiped the prior rows. The column is wired here so a
|
// 1..N for this generation, so (session_id, generation, seq) is the
|
||||||
// future explicit mid-flight revise path can increment it.
|
// addressing key and the model's 1-based update_plan_step always maps to
|
||||||
|
// the CURRENT plan after a re-plan (P0.1).
|
||||||
var nextGen int
|
var nextGen int
|
||||||
if err := tx.QueryRow(ctx, `
|
if err := tx.QueryRow(ctx, `
|
||||||
SELECT COALESCE(MAX(generation), 0) + 1
|
SELECT COALESCE(MAX(generation), 0) + 1
|
||||||
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
FROM session_plan_steps WHERE session_id = $1`, sessionID).Scan(&nextGen); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
// After the DELETE above, no rows remain, so MAX(generation) is NULL →
|
|
||||||
// nextGen = 1. (Keep the query for the future revise path; it's cheap.)
|
|
||||||
|
|
||||||
out := make([]map[string]any, 0, len(steps))
|
out := make([]map[string]any, 0, len(steps))
|
||||||
for i, st := range steps {
|
for i, st := range steps {
|
||||||
@@ -934,7 +944,7 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
if st.TargetSlug != "" {
|
if st.TargetSlug != "" {
|
||||||
targetSlug = &st.TargetSlug
|
targetSlug = &st.TargetSlug
|
||||||
}
|
}
|
||||||
seq := startSeq + i + 1
|
seq := i + 1
|
||||||
var id uuid.UUID
|
var id uuid.UUID
|
||||||
if err := tx.QueryRow(ctx, `
|
if err := tx.QueryRow(ctx, `
|
||||||
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
INSERT INTO session_plan_steps (session_id, seq, title, detail, target_slug, generation)
|
||||||
@@ -973,10 +983,27 @@ func (s *store) proposePlan(ctx context.Context, sessionID string, steps []planS
|
|||||||
// be marked complete while an earlier step is still pending, preventing the
|
// be marked complete while an earlier step is still pending, preventing the
|
||||||
// agent from marking step 5 done before step 4 (observed in production: the
|
// agent from marking step 5 done before step 4 (observed in production: the
|
||||||
// agent rushed to close all steps in a final turn, in reverse order).
|
// agent rushed to close all steps in a final turn, in reverse order).
|
||||||
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID string) error {
|
func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, status, execID, replacedReason string) error {
|
||||||
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
if s == nil || sessionID == "" || sessionID == "ephemeral" {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
// Resolve the CURRENT generation: seq is generation-relative (1-based
|
||||||
|
// within the plan the model is working), so (session_id, MAX(generation),
|
||||||
|
// seq) is the addressing key. A re-plan's superseded generations have
|
||||||
|
// their own seq space and must never be touched by a follow-up's
|
||||||
|
// update_plan_step — that was the root cause of "the plan was off"
|
||||||
|
// (gen-1 `replaced` rows resurrected as `done` while gen-2 work went
|
||||||
|
// unrecorded). The MAX(generation) step is by construction the active
|
||||||
|
// plan, never `replaced`, so this can't resurrect a superseded row (P0.1).
|
||||||
|
var curGen int
|
||||||
|
if err := s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(MAX(generation), 0) FROM session_plan_steps WHERE session_id = $1`,
|
||||||
|
sessionID).Scan(&curGen); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if curGen == 0 {
|
||||||
|
return errPlanStepNotFound
|
||||||
|
}
|
||||||
stamp := ""
|
stamp := ""
|
||||||
switch status {
|
switch status {
|
||||||
case "running":
|
case "running":
|
||||||
@@ -984,16 +1011,18 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
|||||||
case "done", "failed", "skipped", "blocked", "replaced":
|
case "done", "failed", "skipped", "blocked", "replaced":
|
||||||
stamp = ", finished_at = now()"
|
stamp = ", finished_at = now()"
|
||||||
}
|
}
|
||||||
// Completion ordering: for terminal states, check that no earlier step
|
// Completion ordering, scoped to the CURRENT generation: for terminal
|
||||||
// is still pending. Running steps can start out of order (the agent
|
// states, no earlier step in THIS plan may still be pending. Running
|
||||||
// may dispatch parallel work), but completion must be sequential.
|
// steps can start out of order (the agent may dispatch parallel work),
|
||||||
|
// but completion must be sequential. Earlier generations are superseded
|
||||||
|
// and irrelevant.
|
||||||
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
if status == "done" || status == "failed" || status == "skipped" || status == "blocked" {
|
||||||
var blockedBy int
|
var blockedBy int
|
||||||
if err := s.pool.QueryRow(ctx, `
|
if err := s.pool.QueryRow(ctx, `
|
||||||
SELECT COALESCE(MIN(seq), 0)
|
SELECT COALESCE(MIN(seq), 0)
|
||||||
FROM session_plan_steps
|
FROM session_plan_steps
|
||||||
WHERE session_id = $1 AND seq < $2 AND status = 'pending'`,
|
WHERE session_id = $1 AND generation = $2 AND seq < $3 AND status = 'pending'`,
|
||||||
sessionID, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
sessionID, curGen, seq).Scan(&blockedBy); err == nil && blockedBy > 0 {
|
||||||
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
return fmt.Errorf("cannot complete step %d — step %d is still pending", seq, blockedBy)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1004,12 +1033,33 @@ func (s *store) updatePlanStep(ctx context.Context, sessionID string, seq int, s
|
|||||||
var stepID uuid.UUID
|
var stepID uuid.UUID
|
||||||
var targetSlug *string
|
var targetSlug *string
|
||||||
// stamp is a fixed literal from the switch above — never user input.
|
// stamp is a fixed literal from the switch above — never user input.
|
||||||
if err := s.pool.QueryRow(ctx, `
|
// status <> 'replaced' is defense-in-depth: MAX(generation) can't hold a
|
||||||
UPDATE session_plan_steps
|
// replaced row, but if it ever could, this refuses the write instead of
|
||||||
SET status = $3, execution_id = COALESCE($4, execution_id)`+stamp+`
|
// resurrecting it. No matching row → errPlanStepNotFound (stale/out-of-range seq).
|
||||||
WHERE session_id = $1 AND seq = $2
|
if status == "replaced" && replacedReason != "" {
|
||||||
RETURNING id, target_slug`, sessionID, seq, status, execPtr).Scan(&stepID, &targetSlug); err != nil {
|
err := s.pool.QueryRow(ctx, `
|
||||||
return err
|
UPDATE session_plan_steps
|
||||||
|
SET status = $4, execution_id = COALESCE($5, execution_id), replaced_reason = $6`+stamp+`
|
||||||
|
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||||
|
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr, replacedReason).Scan(&stepID, &targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return errPlanStepNotFound
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
err := s.pool.QueryRow(ctx, `
|
||||||
|
UPDATE session_plan_steps
|
||||||
|
SET status = $4, execution_id = COALESCE($5, execution_id)`+stamp+`
|
||||||
|
WHERE session_id = $1 AND generation = $2 AND seq = $3 AND status <> 'replaced'
|
||||||
|
RETURNING id, target_slug`, sessionID, curGen, seq, status, execPtr).Scan(&stepID, &targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, pgx.ErrNoRows) {
|
||||||
|
return errPlanStepNotFound
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
// Anchor the event to the step's target entity when it has one, else the task.
|
// Anchor the event to the step's target entity when it has one, else the task.
|
||||||
entPtr := s.taskEntityPtr(ctx, sessionID)
|
entPtr := s.taskEntityPtr(ctx, sessionID)
|
||||||
@@ -1098,13 +1148,58 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
|||||||
if outcome != "success" {
|
if outcome != "success" {
|
||||||
closeStatus = "skipped"
|
closeStatus = "skipped"
|
||||||
}
|
}
|
||||||
|
// Auto-close only the CURRENT generation's in-flight steps — superseded
|
||||||
|
// generations were already resolved when their plan was replaced. Stamp
|
||||||
|
// started_at so no `done` step is left with a NULL start time (P0.1 fix
|
||||||
|
// 5), and emit a plan.step.finished event per closed step so the panel
|
||||||
|
// converges instead of freezing on "running" after the task completes
|
||||||
|
// (P1.1: no bulk plan-step status write without a corresponding event).
|
||||||
|
type closingStep struct {
|
||||||
|
id uuid.UUID
|
||||||
|
seq int
|
||||||
|
targetSlug *string
|
||||||
|
}
|
||||||
|
var toClose []closingStep
|
||||||
|
if rows, qerr := s.pool.Query(ctx, `
|
||||||
|
SELECT id, seq, target_slug FROM session_plan_steps
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||||
|
AND status IN ('pending', 'running')`, sessionID); qerr == nil {
|
||||||
|
for rows.Next() {
|
||||||
|
var cs closingStep
|
||||||
|
if err := rows.Scan(&cs.id, &cs.seq, &cs.targetSlug); err == nil {
|
||||||
|
toClose = append(toClose, cs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
}
|
||||||
if _, err := s.pool.Exec(ctx, `
|
if _, err := s.pool.Exec(ctx, `
|
||||||
UPDATE session_plan_steps
|
UPDATE session_plan_steps
|
||||||
SET status = $2, finished_at = COALESCE(finished_at, now())
|
SET status = $2,
|
||||||
WHERE session_id = $1 AND status IN ('pending', 'running')`,
|
started_at = COALESCE(started_at, now()),
|
||||||
|
finished_at = COALESCE(finished_at, now())
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||||
|
AND status IN ('pending', 'running')`,
|
||||||
sessionID, closeStatus); err != nil {
|
sessionID, closeStatus); err != nil {
|
||||||
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
|
slog.Warn("nomos: completeTask failed to auto-close in-flight steps", "session", sessionID, "error", err)
|
||||||
}
|
}
|
||||||
|
// Emit one plan.step.finished per closed step so the live panel advances
|
||||||
|
// (mirrors updatePlanStep's event). A bulk UPDATE that skips the event
|
||||||
|
// bus guarantees a stale panel — the rule is: no plan-step status change
|
||||||
|
// without a corresponding event.
|
||||||
|
taskEnt := s.taskEntityPtr(ctx, sessionID)
|
||||||
|
for _, cs := range toClose {
|
||||||
|
evEnt := taskEnt
|
||||||
|
if cs.targetSlug != nil && *cs.targetSlug != "" {
|
||||||
|
var tid uuid.UUID
|
||||||
|
if s.pool.QueryRow(ctx, `SELECT id FROM entities WHERE slug = $1`, *cs.targetSlug).Scan(&tid) == nil {
|
||||||
|
evEnt = &tid
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ = observability.Event(ctx, sqlcgen.New(s.pool), "plan.step.finished", evEnt, "info", "nomos", sessionID,
|
||||||
|
map[string]any{"step_id": cs.id.String(), "seq": cs.seq, "status": closeStatus})
|
||||||
|
}
|
||||||
|
|
||||||
// Clean up assent and destructive window keys from autonomy_settings.
|
// Clean up assent and destructive window keys from autonomy_settings.
|
||||||
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
s.pool.Exec(ctx, `DELETE FROM autonomy_settings
|
||||||
@@ -1148,9 +1243,141 @@ func (s *store) completeTask(ctx context.Context, sessionID, outcome, summary st
|
|||||||
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
_ = observability.Event(ctx, sqlcgen.New(s.pool), "task.status", entPtr, severity, "nomos", sessionID,
|
||||||
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
map[string]any{"status": status, "outcome": outcome, "summary": summary,
|
||||||
"cancelled_executions": cancelledCount, "blocker": blocker})
|
"cancelled_executions": cancelledCount, "blocker": blocker})
|
||||||
|
// Auto-persist knowledge so the graph learns from this session regardless
|
||||||
|
// of whether the agent remembered to call upsert_knowledge (2026-08-04
|
||||||
|
// session audit: only 2.4% of sessions called upsert_knowledge manually).
|
||||||
|
if outcome == "success" || outcome == "partial" {
|
||||||
|
autoUpsertKnowledge(ctx, s, sessionID, outcome, summary)
|
||||||
|
}
|
||||||
|
// Plan quality metric: compute step completion rate for the session's
|
||||||
|
// current plan generation. Tracked as a task attribute so the trend
|
||||||
|
// can be monitored over time (2026-08-04 session audit: 38% baseline).
|
||||||
|
writePlanCompletionRate(ctx, s, sessionID)
|
||||||
|
// Auto-feedback: create a feedback entry linking the session's outcome
|
||||||
|
// to its last execution, feeding the pattern-extraction pipeline that
|
||||||
|
// has been empty since launch (2026-08-04 session audit: 0 feedback rows).
|
||||||
|
if outcome == "success" || outcome == "partial" {
|
||||||
|
autoFeedback(ctx, s, sessionID, outcome, summary)
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// autoUpsertKnowledge creates a knowledge entry for a completed session,
|
||||||
|
// capturing what was done and linking it to the entities involved. Called
|
||||||
|
// automatically from completeTask so every session leaves a trace, even if
|
||||||
|
// the agent forgot to call upsert_knowledge. Only fired for success/partial
|
||||||
|
// outcomes (failures don't have actionable discoveries).
|
||||||
|
func autoUpsertKnowledge(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||||
|
var goal string
|
||||||
|
if err := s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||||
|
sessionID).Scan(&goal); err != nil || goal == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
title := "Session " + sessionID[:8] + ": " + goal
|
||||||
|
if len(title) > 200 {
|
||||||
|
title = title[:200]
|
||||||
|
}
|
||||||
|
content := "## Outcome\n" + outcome + "\n\n## Summary\n" + summary
|
||||||
|
kind := "investigation"
|
||||||
|
slug := "investigation:nomos/" + sessionID
|
||||||
|
tags := []string{"nomos-session", "auto-generated"}
|
||||||
|
|
||||||
|
// Upsert the knowledge entity.
|
||||||
|
docID, _ := uuid.NewV7()
|
||||||
|
if err := s.pool.QueryRow(ctx, `
|
||||||
|
INSERT INTO entities (id, slug, type, name, attributes)
|
||||||
|
VALUES ($1, $2, $3, $4, '{}')
|
||||||
|
ON CONFLICT (slug) DO UPDATE SET name = EXCLUDED.name, updated_at = now()
|
||||||
|
RETURNING id`, docID, slug, kind, title).Scan(&docID); err != nil {
|
||||||
|
slog.Warn("nomos: autoUpsertKnowledge entity insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Upsert the knowledge content.
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO knowledge_entities (entity_id, title, content, source, tags, updated_at)
|
||||||
|
VALUES ($1, $2, $3, 'nomos-agent', $4, now())
|
||||||
|
ON CONFLICT (entity_id) DO UPDATE
|
||||||
|
SET title = EXCLUDED.title, content = EXCLUDED.content,
|
||||||
|
tags = EXCLUDED.tags, updated_at = now()`,
|
||||||
|
docID, title, content, tags); err != nil {
|
||||||
|
slog.Warn("nomos: autoUpsertKnowledge content insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// Link to the task entity.
|
||||||
|
var taskEntID uuid.UUID
|
||||||
|
if s.pool.QueryRow(ctx, `SELECT entity_id FROM agent_sessions WHERE id = $1`,
|
||||||
|
sessionID).Scan(&taskEntID) == nil && taskEntID != uuid.Nil {
|
||||||
|
s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO relationships (source_id, target_id, type, attributes, valid_from)
|
||||||
|
SELECT $1, $2, 'involves', '{"by":"nomos","auto":true}'::jsonb, now()
|
||||||
|
WHERE NOT EXISTS (
|
||||||
|
SELECT 1 FROM relationships
|
||||||
|
WHERE source_id = $1 AND target_id = $2 AND type = 'involves' AND valid_to IS NULL)`,
|
||||||
|
taskEntID, docID)
|
||||||
|
}
|
||||||
|
|
||||||
|
slog.Info("nomos: auto-upserted knowledge for session",
|
||||||
|
"session", sessionID, "outcome", outcome, "slug", slug)
|
||||||
|
}
|
||||||
|
|
||||||
|
// writePlanCompletionRate computes the step completion rate for the current
|
||||||
|
// plan generation and writes it as a task entity attribute so the trend can
|
||||||
|
// be tracked. Baseline from 2026-08-04 audit: 38% (15/39 steps reached done).
|
||||||
|
func writePlanCompletionRate(ctx context.Context, s *store, sessionID string) {
|
||||||
|
var total, completed int
|
||||||
|
s.pool.QueryRow(ctx, `
|
||||||
|
SELECT COUNT(*), COALESCE(SUM(CASE WHEN status = 'done' THEN 1 ELSE 0 END), 0)
|
||||||
|
FROM session_plan_steps
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)
|
||||||
|
AND status <> 'replaced'`, sessionID).Scan(&total, &completed)
|
||||||
|
if total > 0 {
|
||||||
|
rate := float64(completed) / float64(total)
|
||||||
|
attrs, _ := json.Marshal(map[string]any{"plan_completion_rate": rate, "plan_steps_total": total, "plan_steps_completed": completed})
|
||||||
|
s.pool.Exec(ctx, `
|
||||||
|
UPDATE entities SET attributes = attributes || $2::jsonb, updated_at = now()
|
||||||
|
WHERE id = (SELECT entity_id FROM agent_sessions WHERE id = $1)`,
|
||||||
|
sessionID, string(attrs))
|
||||||
|
slog.Info("nomos: plan completion rate", "session", sessionID, "rate", fmt.Sprintf("%.0f%%", rate*100),
|
||||||
|
"completed", completed, "total", total)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// autoFeedback creates a feedback entry linking the session's outcome to its
|
||||||
|
// last execution, feeding the pattern-extraction pipeline that has been empty
|
||||||
|
// since launch. Only created for success/partial outcomes (failures don't
|
||||||
|
// have a specific execution to tie to).
|
||||||
|
func autoFeedback(ctx context.Context, s *store, sessionID, outcome, summary string) {
|
||||||
|
// Find the last execution linked to this session.
|
||||||
|
var execID uuid.UUID
|
||||||
|
if err := s.pool.QueryRow(ctx, `
|
||||||
|
SELECT pe.execution_id FROM nomos_plan_executions pe
|
||||||
|
WHERE pe.session_id = $1::uuid
|
||||||
|
ORDER BY pe.created_at DESC LIMIT 1`, sessionID).Scan(&execID); err != nil || execID == uuid.Nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
fbID, _ := uuid.NewV7()
|
||||||
|
slug := "feedback:" + fbID.String()
|
||||||
|
if _, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'feedback', $3, '{}')`,
|
||||||
|
fbID, slug, "feedback for "+sessionID[:8]); err != nil {
|
||||||
|
slog.Warn("nomos: autoFeedback entity insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_, err := s.pool.Exec(ctx, `
|
||||||
|
INSERT INTO feedback (entity_id, execution_id, outcome, observation, lesson, tags, created_at)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, now())`,
|
||||||
|
fbID, execID, outcome, summary, summary, []string{"nomos-session", "auto-generated", "session:" + sessionID[:8]})
|
||||||
|
if err != nil {
|
||||||
|
slog.Warn("nomos: autoFeedback insert", "session", sessionID, "error", err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
slog.Info("nomos: auto-feedback created for session", "session", sessionID, "outcome", outcome)
|
||||||
|
}
|
||||||
|
|
||||||
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
|
// blockerPatterns maps a substring (case-insensitive) to a structured blocker
|
||||||
// reason. Order matters — earlier patterns take precedence. These are the
|
// reason. Order matters — earlier patterns take precedence. These are the
|
||||||
// recurring failure signatures from the 2026-07-20 session audit. A
|
// recurring failure signatures from the 2026-07-20 session audit. A
|
||||||
@@ -1240,6 +1467,53 @@ func (s *store) hadDiscovery(ctx context.Context, sessionID string) bool {
|
|||||||
return count > 0
|
return count > 0
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// sessionGoal returns the session's goal text, empty string if not found.
|
||||||
|
// Used by complete_task to check whether the goal involved a reachability
|
||||||
|
// verification before marking success.
|
||||||
|
func (s *store) sessionGoal(ctx context.Context, sessionID string) string {
|
||||||
|
if s == nil || sessionID == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var goal string
|
||||||
|
s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COALESCE(goal, '') FROM agent_sessions WHERE id = $1`,
|
||||||
|
sessionID).Scan(&goal)
|
||||||
|
return goal
|
||||||
|
}
|
||||||
|
|
||||||
|
// hadRecentVerification checks whether the session successfully verified
|
||||||
|
// reachability in recent turns — ping_service, or a run with curl/wget that
|
||||||
|
// returned successfully. Used by complete_task as a soft warning when the
|
||||||
|
// goal involved a reachability check but no recent verification occurred.
|
||||||
|
func (s *store) hadRecentVerification(ctx context.Context, sessionID string) bool {
|
||||||
|
if s == nil || sessionID == "" {
|
||||||
|
return true // fail safe: don't warn when we can't check
|
||||||
|
}
|
||||||
|
// Check for ping_service calls in the last 5 activity entries for this session.
|
||||||
|
var pingCount int
|
||||||
|
s.pool.QueryRow(ctx, `
|
||||||
|
SELECT COUNT(*) FROM (
|
||||||
|
SELECT 1 FROM agent_activity
|
||||||
|
WHERE session_id = $1 AND tool_name = 'ping_service' AND success = true
|
||||||
|
ORDER BY ts DESC LIMIT 5
|
||||||
|
) sub`, sessionID).Scan(&pingCount)
|
||||||
|
if pingCount > 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
// Check for run calls with curl/wget that returned successfully.
|
||||||
|
var curlCount int
|
||||||
|
s.pool.QueryRow(ctx, `
|
||||||
|
SELECT COUNT(*) FROM (
|
||||||
|
SELECT 1 FROM agent_activity
|
||||||
|
WHERE session_id = $1
|
||||||
|
AND tool_name = 'run'
|
||||||
|
AND success = true
|
||||||
|
AND (input_summary LIKE '%curl%' OR input_summary LIKE '%wget%')
|
||||||
|
ORDER BY ts DESC LIMIT 10
|
||||||
|
) sub`, sessionID).Scan(&curlCount)
|
||||||
|
return curlCount > 0
|
||||||
|
}
|
||||||
|
|
||||||
// staleGoalSession is a goal-bearing task that's gone idle without reaching
|
// staleGoalSession is a goal-bearing task that's gone idle without reaching
|
||||||
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
|
// a terminal state — the idle-sweep worker's work list (fix 2+3 of
|
||||||
// plans/2026-07-11-task-completion-safety-net.md).
|
// plans/2026-07-11-task-completion-safety-net.md).
|
||||||
@@ -1347,15 +1621,23 @@ type planStep struct {
|
|||||||
|
|
||||||
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
// getPlanSteps returns a task's plan in order — REST hydration for the context
|
||||||
// panel when it first opens a task (live events only carry deltas from then on).
|
// panel when it first opens a task (live events only carry deltas from then on).
|
||||||
func (s *store) getPlanSteps(ctx context.Context, sessionID string) ([]planStep, error) {
|
// By default only the CURRENT (MAX) generation is returned — the panel shows the
|
||||||
|
// live plan, not an archaeological record of every superseded generation. Pass
|
||||||
|
// all=true for the audit/eval view that needs every generation (the
|
||||||
|
// plan_generations assertion counts distinct generations across the full set).
|
||||||
|
func (s *store) getPlanSteps(ctx context.Context, sessionID string, all bool) ([]planStep, error) {
|
||||||
if s == nil {
|
if s == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
|
genFilter := ""
|
||||||
|
if !all {
|
||||||
|
genFilter = "AND generation = (SELECT MAX(generation) FROM session_plan_steps WHERE session_id = $1)"
|
||||||
|
}
|
||||||
rows, err := s.pool.Query(ctx, `
|
rows, err := s.pool.Query(ctx, `
|
||||||
SELECT id::text, seq, title, detail, status,
|
SELECT id::text, seq, title, detail, status,
|
||||||
execution_id::text, target_slug,
|
execution_id::text, target_slug,
|
||||||
started_at::text, finished_at::text, generation
|
started_at::text, finished_at::text, generation
|
||||||
FROM session_plan_steps WHERE session_id = $1 ORDER BY seq`, sessionID)
|
FROM session_plan_steps WHERE session_id = $1 `+genFilter+` ORDER BY generation, seq`, sessionID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -1835,7 +2117,7 @@ func (s *store) resolveArgEntityID(ctx context.Context, args map[string]any) uui
|
|||||||
// The (nullable) session_id column carries the conversation id. args is the
|
// The (nullable) session_id column carries the conversation id. args is the
|
||||||
// tool call's own arguments, used to best-effort tag the row with the
|
// tool call's own arguments, used to best-effort tag the row with the
|
||||||
// entity it acted on (see resolveArgEntityID).
|
// entity it acted on (see resolveArgEntityID).
|
||||||
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string) {
|
func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, toolName string, args map[string]any, inputSummary, outputSummary string, durationMs int, success bool, correlationID string, tokenCount int) {
|
||||||
if s == nil || agentID == uuid.Nil {
|
if s == nil || agentID == uuid.Nil {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -1847,8 +2129,8 @@ func (s *store) logActivity(ctx context.Context, agentID uuid.UUID, sessionID, t
|
|||||||
s.pool.Exec(ctx, `
|
s.pool.Exec(ctx, `
|
||||||
INSERT INTO agent_activity
|
INSERT INTO agent_activity
|
||||||
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
|
(agent_id, session_id, activity_type, tool_name, entity_id, input_summary, output_summary,
|
||||||
duration_ms, success, correlation_id)
|
duration_ms, success, correlation_id, token_count)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)`,
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)`,
|
||||||
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
|
agentID, sessionID, "tool_call", toolName, entityIDArg, inputSummary, outputSummary,
|
||||||
durationMs, success, correlationID)
|
durationMs, success, correlationID, tokenCount)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -192,7 +192,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Mark step 1 as started.
|
// Mark step 1 as started.
|
||||||
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", ""); err != nil {
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||||
t.Fatalf("updatePlanStep: %v", err)
|
t.Fatalf("updatePlanStep: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -205,7 +205,7 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The original step 1 must be untouched — not erased, not appended to.
|
// The original step 1 must be untouched — not erased, not appended to.
|
||||||
steps, err := s.getPlanSteps(ctx, sess.ID)
|
steps, err := s.getPlanSteps(ctx, sess.ID, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("getPlanSteps: %v", err)
|
t.Fatalf("getPlanSteps: %v", err)
|
||||||
}
|
}
|
||||||
@@ -217,7 +217,8 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Third call BEFORE anything runs on a fresh session: every step is
|
// Third call BEFORE anything runs on a fresh session: every step is
|
||||||
// still pending, so this must REPLACE, not refuse.
|
// still pending, so this must REPLACE (mark the prior plan `replaced`),
|
||||||
|
// not refuse. The new plan becomes generation 2.
|
||||||
sess2, err := s.createSession(ctx, "plan replace test")
|
sess2, err := s.createSession(ctx, "plan replace test")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
@@ -228,15 +229,146 @@ func TestProposePlan_RefuseInFlight(t *testing.T) {
|
|||||||
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
if _, err := s.proposePlan(ctx, sess2.ID, []planStepInput{{Title: "Revised"}}); err != nil {
|
||||||
t.Fatalf("proposePlan (revise before execution): %v", err)
|
t.Fatalf("proposePlan (revise before execution): %v", err)
|
||||||
}
|
}
|
||||||
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID)
|
// Default (current generation) view: only the revised step.
|
||||||
|
revisedSteps, err := s.getPlanSteps(ctx, sess2.ID, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("getPlanSteps: %v", err)
|
t.Fatalf("getPlanSteps: %v", err)
|
||||||
}
|
}
|
||||||
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
if len(revisedSteps) != 1 || revisedSteps[0].Title != "Revised" {
|
||||||
t.Fatalf("got %+v, want a single 'Revised' step (pre-execution revise must replace, not refuse)", revisedSteps)
|
t.Fatalf("got %+v, want a single 'Revised' step (current-generation view)", revisedSteps)
|
||||||
}
|
}
|
||||||
if revisedSteps[0].Generation != 1 {
|
if revisedSteps[0].Seq != 1 {
|
||||||
t.Fatalf("revised step generation = %d, want 1 (fresh-start after DELETE resets generation)", revisedSteps[0].Generation)
|
t.Fatalf("revised step seq = %d, want 1 (seq is generation-relative, resets to 1..N)", revisedSteps[0].Seq)
|
||||||
|
}
|
||||||
|
if revisedSteps[0].Generation != 2 {
|
||||||
|
t.Fatalf("revised step generation = %d, want 2 (prior pending plan is replaced, not deleted, so the counter increments)", revisedSteps[0].Generation)
|
||||||
|
}
|
||||||
|
// all=true audit view: both generations, the original marked `replaced`.
|
||||||
|
allSteps, err := s.getPlanSteps(ctx, sess2.ID, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("getPlanSteps(all): %v", err)
|
||||||
|
}
|
||||||
|
if len(allSteps) != 2 {
|
||||||
|
t.Fatalf("all=true got %d steps, want 2 (Original replaced gen1 + Revised gen2)", len(allSteps))
|
||||||
|
}
|
||||||
|
if allSteps[0].Title != "Original" || allSteps[0].Status != "replaced" || allSteps[0].Generation != 1 {
|
||||||
|
t.Errorf("gen1 step = %+v, want Original/replaced/gen1", allSteps[0])
|
||||||
|
}
|
||||||
|
if allSteps[1].Title != "Revised" || allSteps[1].Generation != 2 || allSteps[1].Seq != 1 {
|
||||||
|
t.Errorf("gen2 step = %+v, want Revised/gen2/seq1", allSteps[1])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdatePlanStep_GenerationRelative is the P0.1 regression proof: after a
|
||||||
|
// re-plan, update_plan_step(seq=N) — using the 1-based number the model
|
||||||
|
// naturally carries — must address the CURRENT generation and never resurrect
|
||||||
|
// a superseded generation's `replaced` row. Before the fix, seq was globally
|
||||||
|
// increasing across generations, so seq=1 after a re-plan flipped the gen-1
|
||||||
|
// `replaced` step back to `running`/`done` while the real gen-2 work went
|
||||||
|
// unrecorded.
|
||||||
|
func TestUpdatePlanStep_GenerationRelative(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
sess, err := s.createSession(ctx, "gen-relative seq test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("createSession: %v", err)
|
||||||
|
}
|
||||||
|
// Generation 1: two steps.
|
||||||
|
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||||
|
t.Fatalf("proposePlan #1: %v", err)
|
||||||
|
}
|
||||||
|
// Re-plan: setGoal marks the gen-1 plan `replaced`, proposePlan starts gen 2.
|
||||||
|
if err := s.setGoal(ctx, sess.ID, "follow-up sub-task"); err != nil {
|
||||||
|
t.Fatalf("setGoal: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "C"}, {Title: "D"}}); err != nil {
|
||||||
|
t.Fatalf("proposePlan #2: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The model addresses the new plan with 1-based seq. seq=1 must hit
|
||||||
|
// gen-2 "C", leaving gen-1 "A" (replaced) untouched.
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||||
|
t.Fatalf("updatePlanStep(seq=1, running): %v", err)
|
||||||
|
}
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "done", "", ""); err != nil {
|
||||||
|
t.Fatalf("updatePlanStep(seq=1, done): %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
all, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("getPlanSteps(all): %v", err)
|
||||||
|
}
|
||||||
|
byTitle := map[string]planStep{}
|
||||||
|
for _, st := range all {
|
||||||
|
byTitle[st.Title] = st
|
||||||
|
}
|
||||||
|
// gen-1 steps stay `replaced` — NOT resurrected to running/done.
|
||||||
|
if byTitle["A"].Status != "replaced" || byTitle["A"].Generation != 1 {
|
||||||
|
t.Errorf("A = %+v, want replaced/gen1 (a superseded row must never be touched)", byTitle["A"])
|
||||||
|
}
|
||||||
|
if byTitle["B"].Status != "replaced" || byTitle["B"].Generation != 1 {
|
||||||
|
t.Errorf("B = %+v, want replaced/gen1", byTitle["B"])
|
||||||
|
}
|
||||||
|
// gen-2 seq=1 advanced; seq=2 untouched.
|
||||||
|
if byTitle["C"].Status != "done" || byTitle["C"].Generation != 2 || byTitle["C"].Seq != 1 {
|
||||||
|
t.Errorf("C = %+v, want done/gen2/seq1 (the 1-based update must address the current generation)", byTitle["C"])
|
||||||
|
}
|
||||||
|
if byTitle["D"].Status != "pending" || byTitle["D"].Seq != 2 {
|
||||||
|
t.Errorf("D = %+v, want pending/seq2", byTitle["D"])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Out-of-range seq must be refused (no current-gen step there).
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 99, "running", "", ""); !errors.Is(err, errPlanStepNotFound) {
|
||||||
|
t.Fatalf("updatePlanStep(seq=99) err = %v, want errPlanStepNotFound", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCompleteTask_AutoCloseEmitsEvents is the P1.1 regression proof:
|
||||||
|
// completeTask's bulk auto-close of in-flight steps must emit one
|
||||||
|
// plan.step.finished event per closed step (so the live panel converges
|
||||||
|
// instead of freezing on "running" after the task completes) and must stamp
|
||||||
|
// started_at so no closed step is left un-timestamped (P0.1 fix 5).
|
||||||
|
func TestCompleteTask_AutoCloseEmitsEvents(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
sess, err := s.createSession(ctx, "auto-close events test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("createSession: %v", err)
|
||||||
|
}
|
||||||
|
if _, err := s.proposePlan(ctx, sess.ID, []planStepInput{{Title: "A"}, {Title: "B"}}); err != nil {
|
||||||
|
t.Fatalf("proposePlan: %v", err)
|
||||||
|
}
|
||||||
|
// A is running, B still pending at completion time.
|
||||||
|
if err := s.updatePlanStep(ctx, sess.ID, 1, "running", "", ""); err != nil {
|
||||||
|
t.Fatalf("updatePlanStep(1, running): %v", err)
|
||||||
|
}
|
||||||
|
if err := s.completeTask(ctx, sess.ID, "success", "done"); err != nil {
|
||||||
|
t.Fatalf("completeTask: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every auto-closed step should now carry both a started_at and a
|
||||||
|
// finished_at (no NULL-started `done` step).
|
||||||
|
steps, err := s.getPlanSteps(ctx, sess.ID, true)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("getPlanSteps: %v", err)
|
||||||
|
}
|
||||||
|
for _, st := range steps {
|
||||||
|
if st.Status == "done" && st.StartedAt == nil {
|
||||||
|
t.Errorf("step %q done but started_at is NULL (P0.1 fix 5: stamp it)", st.Title)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Exactly two plan.step.finished events — one per closed step (A and B).
|
||||||
|
var finished int
|
||||||
|
if err := s.pool.QueryRow(ctx,
|
||||||
|
`SELECT COUNT(*) FROM events WHERE type = 'plan.step.finished' AND correlation_id = $1`,
|
||||||
|
sess.ID).Scan(&finished); err != nil {
|
||||||
|
t.Fatalf("count events: %v", err)
|
||||||
|
}
|
||||||
|
if finished != 2 {
|
||||||
|
t.Fatalf("plan.step.finished events = %d, want 2 (one per auto-closed step)", finished)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -265,7 +397,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
|
|
||||||
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
|
// A `run` call (discovery) — should set hadDiscovery, not hadEntityWriteback.
|
||||||
agentID := uuid.New()
|
agentID := uuid.New()
|
||||||
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1")
|
s.logActivity(ctx, agentID, sess.ID, "run", nil, "", "uptime output", 100, true, "corr-1", 0)
|
||||||
if !s.hadDiscovery(ctx, sess.ID) {
|
if !s.hadDiscovery(ctx, sess.ID) {
|
||||||
t.Fatal("hadDiscovery = false after a successful run call, want true")
|
t.Fatal("hadDiscovery = false after a successful run call, want true")
|
||||||
}
|
}
|
||||||
@@ -278,7 +410,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
}
|
}
|
||||||
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2")
|
s.logActivity(ctx, agentID, sess2.ID, "run", nil, "", "ssh timeout", 100, false, "corr-2", 0)
|
||||||
if s.hadDiscovery(ctx, sess2.ID) {
|
if s.hadDiscovery(ctx, sess2.ID) {
|
||||||
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
|
t.Fatal("hadDiscovery = true after a failed run call, want false (no facts learned)")
|
||||||
}
|
}
|
||||||
@@ -288,7 +420,7 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
}
|
}
|
||||||
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3")
|
s.logActivity(ctx, agentID, sess3.ID, "get_entity", nil, "", "entity row", 10, true, "corr-3", 0)
|
||||||
if s.hadDiscovery(ctx, sess3.ID) {
|
if s.hadDiscovery(ctx, sess3.ID) {
|
||||||
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
|
t.Fatal("hadDiscovery = true after get_entity, want false (DB lookups are not discovery)")
|
||||||
}
|
}
|
||||||
@@ -298,12 +430,12 @@ func TestHadDiscoveryAndWriteback(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("createSession: %v", err)
|
t.Fatalf("createSession: %v", err)
|
||||||
}
|
}
|
||||||
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4")
|
s.logActivity(ctx, agentID, sess4.ID, "update_entity_attributes", nil, "", "ok", 10, true, "corr-4", 0)
|
||||||
if !s.hadEntityWriteback(ctx, sess4.ID) {
|
if !s.hadEntityWriteback(ctx, sess4.ID) {
|
||||||
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
|
t.Fatal("hadEntityWriteback = false after update_entity_attributes, want true")
|
||||||
}
|
}
|
||||||
// And the discovery+writeback combination (the conv3 scenario).
|
// And the discovery+writeback combination (the conv3 scenario).
|
||||||
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5")
|
s.logActivity(ctx, agentID, sess4.ID, "run", nil, "", "apt-get update output", 100, true, "corr-5", 0)
|
||||||
if !s.hadDiscovery(ctx, sess4.ID) {
|
if !s.hadDiscovery(ctx, sess4.ID) {
|
||||||
t.Fatal("hadDiscovery = false after run+writeback, want true")
|
t.Fatal("hadDiscovery = false after run+writeback, want true")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"log/slog"
|
"log/slog"
|
||||||
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
@@ -245,13 +246,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
// the seq-order enforcement (5.6) require it to be completed last,
|
// the seq-order enforcement (5.6) require it to be completed last,
|
||||||
// and D.1's complete_task gate enforces the actual calls. Together
|
// and D.1's complete_task gate enforces the actual calls. Together
|
||||||
// they close the loop structurally — neither relies on the agent
|
// they close the loop structurally — neither relies on the agent
|
||||||
// reading SOUL.md.
|
// reading SOUL.md. The match is broadened past the literal tool
|
||||||
|
// names so a natural-language step ("Write back: update entity
|
||||||
|
// attributes…") isn't doubled by an auto-appended duplicate (P1.2).
|
||||||
hasWritebackStep := false
|
hasWritebackStep := false
|
||||||
for _, st := range steps {
|
for _, st := range steps {
|
||||||
if strings.Contains(st.Title, "update_entity_attributes") ||
|
t := strings.ToLower(st.Title + " " + st.Detail)
|
||||||
strings.Contains(st.Title, "create_relationship") ||
|
if strings.Contains(t, "update_entity_attributes") ||
|
||||||
strings.Contains(st.Detail, "update_entity_attributes") ||
|
strings.Contains(t, "create_relationship") ||
|
||||||
strings.Contains(st.Detail, "create_relationship") {
|
strings.Contains(t, "upsert_knowledge") ||
|
||||||
|
strings.Contains(t, "write back") ||
|
||||||
|
strings.Contains(t, "writeback") {
|
||||||
hasWritebackStep = true
|
hasWritebackStep = true
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
@@ -280,8 +285,19 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
// The writeback step is now always present (D.2 auto-appends it if
|
// The writeback step is now always present (D.2 auto-appends it if
|
||||||
// the agent forgot), so the old advisory nudge is replaced by the
|
// the agent forgot), so the old advisory nudge is replaced by the
|
||||||
// structural gate: D.1 refuses complete_task without the actual
|
// structural gate: D.1 refuses complete_task without the actual
|
||||||
// update_entity_attributes/create_relationship calls.
|
// update_entity_attributes/create_relationship calls. Enumerate the
|
||||||
result := fmt.Sprintf("Plan set (%d steps)%s. If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), appendedNote)
|
// step seqs so the model knows exactly which numbers to address with
|
||||||
|
// update_plan_step (seq is 1-based within this plan — the addressing
|
||||||
|
// key, not a global counter).
|
||||||
|
var seqs strings.Builder
|
||||||
|
for i, p := range persisted {
|
||||||
|
if i > 0 {
|
||||||
|
seqs.WriteString("; ")
|
||||||
|
}
|
||||||
|
title := fmt.Sprint(p["title"])
|
||||||
|
fmt.Fprintf(&seqs, "%v=%s", p["seq"], title)
|
||||||
|
}
|
||||||
|
result := fmt.Sprintf("Plan set (%d steps): %s.%s Address them with update_plan_step(seq=N). If all steps are read-only, execute now — call update_plan_step(running) + run for each step, no approval needed. If any step is config_mutation/destructive, STOP and wait for operator approval (\"approved\", \"yes\", \"go\", \"proceed\", \"continue\", \"ok\", \"go ahead\"). Do not call propose_plan again.", len(persisted), seqs.String(), appendedNote)
|
||||||
return result, true
|
return result, true
|
||||||
|
|
||||||
case "update_plan_step":
|
case "update_plan_step":
|
||||||
@@ -291,7 +307,17 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
if seq <= 0 || status == "" {
|
if seq <= 0 || status == "" {
|
||||||
return "error: update_plan_step needs seq (>=1) and status", true
|
return "error: update_plan_step needs seq (>=1) and status", true
|
||||||
}
|
}
|
||||||
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID); err != nil {
|
reason, _ := args["replaced_reason"].(string)
|
||||||
|
if err := a.store.updatePlanStep(ctx, sessionID, seq, status, execID, reason); err != nil {
|
||||||
|
if errors.Is(err, errPlanStepNotFound) {
|
||||||
|
// The seq doesn't address a step in the CURRENT plan — most
|
||||||
|
// often a stale 1-based number the model carried across a
|
||||||
|
// re-plan, or an out-of-range seq. seq is generation-relative
|
||||||
|
// (1..N within the latest propose_plan), so a superseded
|
||||||
|
// generation's row is never touched (P0.1 fix 3). Direct the
|
||||||
|
// model instead of silently no-op'ing.
|
||||||
|
return fmt.Sprintf("Step %d is not in the current plan. seq is 1-based within your latest propose_plan (a re-plan resets it to 1..N, so an old step number no longer applies). The plan was not changed. Re-address with the correct 1-based seq, or if you've lost track, re-read the plan.", seq), true
|
||||||
|
}
|
||||||
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
return fmt.Sprintf("error updating step %d: %v", seq, err), true
|
||||||
}
|
}
|
||||||
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
return fmt.Sprintf("Step %d → %s. (Advance with update_plan_step + run; do not re-propose.)", seq, status), true
|
||||||
@@ -349,6 +375,18 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
|
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) && !a.store.hadEntityWriteback(ctx, sessionID) {
|
||||||
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
|
return "Refused: this session ran `run` against live targets (discovery) but did not call update_entity_attributes or create_relationship to persist what you learned. The knowledge graph will drift if you complete without writeback. Call update_entity_attributes for each entity you ran against (versions, states, counts, timestamps), and create_relationship for any edge you discovered, then call complete_task again. Outcome is held at 'executing' until you do.", true
|
||||||
}
|
}
|
||||||
|
// D.2: refuse success when the goal mentions a reachability/uptime
|
||||||
|
// check but no verification was done. The agent can't claim "X is
|
||||||
|
// reachable" based on a shell command alone — the proxy (Caddy) can
|
||||||
|
// return 200 for a terminal page (ttyd) or fallback while the actual
|
||||||
|
// dashboard is still down. Must call ping_service or run a successful
|
||||||
|
// curl before claiming success.
|
||||||
|
if outcome == "success" && a.store.hadDiscovery(ctx, sessionID) {
|
||||||
|
goal := a.store.sessionGoal(ctx, sessionID)
|
||||||
|
if mentionsReachability(goal) && !a.store.hadRecentVerification(ctx, sessionID) {
|
||||||
|
return "Refused: the goal involves a reachability or uptime check (\"make X reachable\", \"get X up\", etc.), but no ping_service call or successful curl/HTTP request against the target was detected. Caddy can return 200 for a terminal or fallback page while the actual service is still down — you must verify the service itself, not just the proxy. Call ping_service(target) or run a curl against the actual service URL, then call complete_task again. Outcome held until verified.", true
|
||||||
|
}
|
||||||
|
}
|
||||||
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
|
if err := a.store.completeTask(ctx, sessionID, outcome, summary); err != nil {
|
||||||
if errors.Is(err, errTaskAlreadyComplete) {
|
if errors.Is(err, errTaskAlreadyComplete) {
|
||||||
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
|
return "Task is already complete. Do not call complete_task again. If the operator pointed out a UI/sidebar inconsistency, fix it with update_plan_step (reconcile step states) or summarize the panel in your reply — do not re-execute the work.", true
|
||||||
@@ -365,6 +403,28 @@ func (a *agent) handleTaskTool(ctx context.Context, sessionID, name string, args
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// reachabilityPatterns matches goal text that involves making something
|
||||||
|
// reachable/accessible/working. Used by complete_task to surface a soft
|
||||||
|
// warning when the session goal was about reachability but no verification
|
||||||
|
// occurred before marking success.
|
||||||
|
var reachabilityPatterns = []*regexp.Regexp{
|
||||||
|
regexp.MustCompile(`(?i)https?://[^\s]+`),
|
||||||
|
regexp.MustCompile(`(?i)\.hubris\.net\w+`),
|
||||||
|
regexp.MustCompile(`(?i)(un)?reachable`),
|
||||||
|
regexp.MustCompile(`(?i)(not?\s+)?(accessible|reachable|responding|resolving)`),
|
||||||
|
regexp.MustCompile(`(?i)diagnose\s+why`),
|
||||||
|
regexp.MustCompile(`(?i)(fix|restore|bring\s+back).*(accessible|reachable|online)`),
|
||||||
|
}
|
||||||
|
|
||||||
|
func mentionsReachability(goal string) bool {
|
||||||
|
for _, p := range reachabilityPatterns {
|
||||||
|
if p.MatchString(goal) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// autoCompleteTrivialTask is the case-1 fix from
|
// autoCompleteTrivialTask is the case-1 fix from
|
||||||
// plans/2026-07-11-task-completion-safety-net.md: a session that never
|
// plans/2026-07-11-task-completion-safety-net.md: a session that never
|
||||||
// called set_goal never framed itself as a structured task, so a turn that
|
// called set_goal never framed itself as a structured task, so a turn that
|
||||||
|
|||||||
90
cmd/nomos/turngate.go
Normal file
90
cmd/nomos/turngate.go
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// turnGate enforces at most one in-flight agent turn per session.
|
||||||
|
//
|
||||||
|
// Why this exists (plan 2026-08-03, F1): handleChat runs a turn in the HTTP
|
||||||
|
// request goroutine, and every "resume" path (the empty-message reconnect,
|
||||||
|
// the auto-continuation worker, the idle sweep, answer-question, the /resume
|
||||||
|
// endpoint) launches ANOTHER goroutine running a full turn. Nothing prevented
|
||||||
|
// two turns for the SAME session at once, so a network blip that triggered a
|
||||||
|
// reconnect would spawn a duplicate resumeSession while the original turn was
|
||||||
|
// still alive — their tool calls interleaved on the wire and in the persisted
|
||||||
|
// transcript, which is the root cause behind the "parallel/nesting/sequence
|
||||||
|
// is off" and "task didn't end / flaky" reports.
|
||||||
|
//
|
||||||
|
// Model: one permit (buffered-1 channel seeded with a single token) per
|
||||||
|
// session id. Acquiring consumes the token; releasing puts it back.
|
||||||
|
// - Background/best-effort callers (resumeSession and everything it backs)
|
||||||
|
// use a non-blocking acquire and SKIP when busy — a duplicate nudge while a
|
||||||
|
// turn is already running adds nothing, and the continuation/idle tickers
|
||||||
|
// will retry on their own.
|
||||||
|
// - The live chat path (an operator message) waits briefly for a finishing
|
||||||
|
// background turn, then bails with an actionable error if still busy — see
|
||||||
|
// handleChat.
|
||||||
|
//
|
||||||
|
// The permits map grows one entry per session id seen. For this single-agent
|
||||||
|
// homelab process that set is small and bounded by real sessions; cleanup is
|
||||||
|
// intentionally omitted (a sweep would race with acquire/release and the
|
||||||
|
// memory is negligible).
|
||||||
|
type turnGate struct {
|
||||||
|
mu sync.Mutex
|
||||||
|
permits map[string]chan struct{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newTurnGate() *turnGate {
|
||||||
|
return &turnGate{permits: make(map[string]chan struct{})}
|
||||||
|
}
|
||||||
|
|
||||||
|
// permit returns the single token-channel for sessionID, creating and seeding
|
||||||
|
// it on first use. Creation is guarded so two concurrent first-callers for the
|
||||||
|
// same id share one channel.
|
||||||
|
func (g *turnGate) permit(sessionID string) chan struct{} {
|
||||||
|
g.mu.Lock()
|
||||||
|
defer g.mu.Unlock()
|
||||||
|
ch, ok := g.permits[sessionID]
|
||||||
|
if !ok {
|
||||||
|
ch = make(chan struct{}, 1)
|
||||||
|
ch <- struct{}{}
|
||||||
|
g.permits[sessionID] = ch
|
||||||
|
}
|
||||||
|
return ch
|
||||||
|
}
|
||||||
|
|
||||||
|
// acquire takes the session's permit. With wait <= 0 it is non-blocking
|
||||||
|
// (returns false immediately if a turn is active). With wait > 0 it blocks up
|
||||||
|
// to wait for the permit, returning false on timeout. Every true return MUST
|
||||||
|
// be paired with exactly one release.
|
||||||
|
func (g *turnGate) acquire(sessionID string, wait time.Duration) bool {
|
||||||
|
ch := g.permit(sessionID)
|
||||||
|
if wait <= 0 {
|
||||||
|
select {
|
||||||
|
case <-ch:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t := time.NewTimer(wait)
|
||||||
|
defer t.Stop()
|
||||||
|
select {
|
||||||
|
case <-ch:
|
||||||
|
return true
|
||||||
|
case <-t.C:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// release returns the session's permit. Idempotent: a release with no matching
|
||||||
|
// acquire (or a double release) is a no-op rather than a blocking send.
|
||||||
|
func (g *turnGate) release(sessionID string) {
|
||||||
|
ch := g.permit(sessionID)
|
||||||
|
select {
|
||||||
|
case ch <- struct{}{}:
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
}
|
||||||
114
cmd/nomos/turngate_test.go
Normal file
114
cmd/nomos/turngate_test.go
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sync"
|
||||||
|
"sync/atomic"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestTurnGate_NonBlockingSkipsWhenBusy(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
if !g.acquire("s1", 0) {
|
||||||
|
t.Fatal("first non-blocking acquire should succeed on a free session")
|
||||||
|
}
|
||||||
|
// A second non-blocking acquire (a background resume) must skip, not queue.
|
||||||
|
if g.acquire("s1", 0) {
|
||||||
|
t.Fatal("second non-blocking acquire should fail while a turn is active")
|
||||||
|
}
|
||||||
|
// A different session is independent.
|
||||||
|
if !g.acquire("s2", 0) {
|
||||||
|
t.Fatal("acquire on a different session should succeed")
|
||||||
|
}
|
||||||
|
g.release("s2")
|
||||||
|
g.release("s1")
|
||||||
|
// After release, the session is free again.
|
||||||
|
if !g.acquire("s1", 0) {
|
||||||
|
t.Fatal("acquire should succeed again after release")
|
||||||
|
}
|
||||||
|
g.release("s1")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTurnGate_BlockingAcquireWaitsForRelease(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
if !g.acquire("s1", 0) {
|
||||||
|
t.Fatal("first acquire should succeed")
|
||||||
|
}
|
||||||
|
|
||||||
|
got := make(chan bool, 1)
|
||||||
|
go func() { got <- g.acquire("s1", 2*time.Second) }()
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-got:
|
||||||
|
t.Fatal("blocking acquire should wait, not return before release")
|
||||||
|
case <-time.After(50 * time.Millisecond):
|
||||||
|
// expected: still waiting
|
||||||
|
}
|
||||||
|
|
||||||
|
g.release("s1")
|
||||||
|
select {
|
||||||
|
case ok := <-got:
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("blocking acquire should succeed after release")
|
||||||
|
}
|
||||||
|
case <-time.After(time.Second):
|
||||||
|
t.Fatal("blocking acquire did not return after release")
|
||||||
|
}
|
||||||
|
g.release("s1")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestTurnGate_BlockingAcquireTimesOut(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
g.acquire("s1", 0) // hold the permit
|
||||||
|
|
||||||
|
start := time.Now()
|
||||||
|
if g.acquire("s1", 60*time.Millisecond) {
|
||||||
|
t.Fatal("acquire should time out while permit is held")
|
||||||
|
}
|
||||||
|
if elapsed := time.Since(start); elapsed < 50*time.Millisecond {
|
||||||
|
t.Fatalf("acquire returned too fast (%v); expected to wait ~60ms", elapsed)
|
||||||
|
}
|
||||||
|
g.release("s1")
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTurnGate_SingleFlightConcurrent is the core F1 guarantee: many concurrent
|
||||||
|
// background acquirers on the SAME session, exactly one runs at a time. This is
|
||||||
|
// the property that prevents two turns interleaving tool calls.
|
||||||
|
func TestTurnGate_SingleFlightConcurrent(t *testing.T) {
|
||||||
|
g := newTurnGate()
|
||||||
|
const n = 50
|
||||||
|
var inFlight, maxInFlight int64
|
||||||
|
var runs int64
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
wg.Add(n)
|
||||||
|
start := make(chan struct{})
|
||||||
|
for i := 0; i < n; i++ {
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
<-start
|
||||||
|
if !g.acquire("shared", 0) { // background-style: skip if busy
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer g.release("shared")
|
||||||
|
cur := atomic.AddInt64(&inFlight, 1)
|
||||||
|
for {
|
||||||
|
m := atomic.LoadInt64(&maxInFlight)
|
||||||
|
if cur <= m || atomic.CompareAndSwapInt64(&maxInFlight, m, cur) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
atomic.AddInt64(&runs, 1)
|
||||||
|
time.Sleep(2 * time.Millisecond)
|
||||||
|
atomic.AddInt64(&inFlight, -1)
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
close(start)
|
||||||
|
wg.Wait()
|
||||||
|
|
||||||
|
if maxInFlight != 1 {
|
||||||
|
t.Fatalf("max in-flight turns = %d, want 1 (turns must not overlap)", maxInFlight)
|
||||||
|
}
|
||||||
|
if runs == 0 {
|
||||||
|
t.Fatal("expected at least one turn to run")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,5 +1,27 @@
|
|||||||
:80 {
|
:80 {
|
||||||
root * /srv
|
root * /srv
|
||||||
file_server
|
|
||||||
try_files {path} /index.html
|
# /wails/runtime.js is injected by the Wails desktop wrapper, which serves
|
||||||
|
# the same dist/ from its own asset handler. In a browser it does not
|
||||||
|
# exist, and the SPA fallback below answered it with index.html — so the
|
||||||
|
# browser parsed "<!doctype html>" as JavaScript and threw
|
||||||
|
# "SyntaxError: expected expression, got '<'" on every page load.
|
||||||
|
# Return a real 404 instead: the tag fails quietly, and the desktop app is
|
||||||
|
# unaffected because it never reaches this server.
|
||||||
|
handle /wails/* {
|
||||||
|
error 404
|
||||||
|
}
|
||||||
|
|
||||||
|
# Same reasoning for any other asset: a missing .js/.css/.map answered with
|
||||||
|
# HTML is always a confusing parse error rather than an honest 404. Only
|
||||||
|
# real routes should fall through to the SPA.
|
||||||
|
@asset path_regexp \.(js|mjs|css|map|json|png|jpg|svg|ico|woff2?)$
|
||||||
|
handle @asset {
|
||||||
|
file_server
|
||||||
|
}
|
||||||
|
|
||||||
|
handle {
|
||||||
|
file_server
|
||||||
|
try_files {path} /index.html
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,7 +8,8 @@ FROM node:22-alpine AS builder
|
|||||||
|
|
||||||
WORKDIR /build/web
|
WORKDIR /build/web
|
||||||
COPY web/package.json web/package-lock.json ./
|
COPY web/package.json web/package-lock.json ./
|
||||||
RUN npm ci
|
COPY web/vendor /build/vendor
|
||||||
|
RUN npm install --no-audit --no-fund
|
||||||
COPY VERSION ./
|
COPY VERSION ./
|
||||||
COPY web/ ./
|
COPY web/ ./
|
||||||
RUN npm run build
|
RUN npm run build
|
||||||
|
|||||||
@@ -83,6 +83,18 @@ services:
|
|||||||
command: ["api"]
|
command: ["api"]
|
||||||
stop_signal: SIGTERM
|
stop_signal: SIGTERM
|
||||||
stop_grace_period: 30s
|
stop_grace_period: 30s
|
||||||
|
# Exists so nomos can wait for the API to actually answer rather than just
|
||||||
|
# for its container to exist — see nomos's depends_on below. wget is
|
||||||
|
# BusyBox's, already in the alpine runtime image, so this adds no
|
||||||
|
# dependency. /healthz pings the DB, so "healthy" means genuinely ready.
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8090/healthz"]
|
||||||
|
interval: 5s
|
||||||
|
timeout: 3s
|
||||||
|
retries: 10
|
||||||
|
# Migrations and seed run before this container, but the first bind can
|
||||||
|
# still take a moment; failures inside the start period don't count.
|
||||||
|
start_period: 10s
|
||||||
|
|
||||||
# Scheduler (Phase 3) — observe loop
|
# Scheduler (Phase 3) — observe loop
|
||||||
scheduler:
|
scheduler:
|
||||||
@@ -139,7 +151,12 @@ services:
|
|||||||
profiles: ["full"]
|
profiles: ["full"]
|
||||||
depends_on:
|
depends_on:
|
||||||
api:
|
api:
|
||||||
condition: service_started
|
# service_started only waits for the container to exist, so nomos came
|
||||||
|
# up while the API was still binding :8090, failed its MCP initialize,
|
||||||
|
# exited 1, and crash-looped for ~25s on every single deploy. It always
|
||||||
|
# recovered, which is exactly why it went unnoticed. service_healthy
|
||||||
|
# waits for the API to actually answer.
|
||||||
|
condition: service_healthy
|
||||||
environment:
|
environment:
|
||||||
NOMOS_MCP_URL: http://api:8090/mcp
|
NOMOS_MCP_URL: http://api:8090/mcp
|
||||||
NOMOS_AGENT_SLUG: agent:nomos
|
NOMOS_AGENT_SLUG: agent:nomos
|
||||||
|
|||||||
@@ -234,9 +234,33 @@ sequenceDiagram
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
**2026-07-08 — renamed to Nomos.** The Hermes agent gateway was renamed to
|
**2026-07-08 — renamed to Nomos.**
|
||||||
Nomos (from *oikonomos*, the steward of the oikos) under the
|
Nomos (from *oikonomos*, the steward of the oikos) under the
|
||||||
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
|
[Nomos resident agent plan](../../plans/2026-07-08-nomos-resident-agent.md),
|
||||||
|
|
||||||
|
### Hermes MCP client setup
|
||||||
|
|
||||||
|
To connect a Hermes Agent instance to oikos as a native MCP client, add to
|
||||||
|
`~/.hermes/config.yaml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
mcp_servers:
|
||||||
|
oikos:
|
||||||
|
url: "https://mcp.hubris.network/mcp"
|
||||||
|
headers:
|
||||||
|
Authorization: "Bearer <OIKOS_MCP_BEARER_TOKEN>"
|
||||||
|
timeout: 180
|
||||||
|
```
|
||||||
|
|
||||||
|
Run `/reload-mcp` in-session or restart Hermes. Tools appear as
|
||||||
|
`mcp__oikos__*`.
|
||||||
|
|
||||||
|
**Caveat:** Hermes stores the bearer token in plaintext in `config.yaml` —
|
||||||
|
it does not support `${VAR}` interpolation in MCP server headers. Ensure
|
||||||
|
`security.redact_secrets: true` (default) so the token value is stripped
|
||||||
|
from tool output and logs. File an upstream feature request at
|
||||||
|
https://github.com/NousResearch/hermes-agent/issues for env-var
|
||||||
|
interpolation support.
|
||||||
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
|
N0 milestone. The gateway binary (`cmd/nomos`), Docker service, DB slug
|
||||||
(`agent:nomos`), and all referencing docs were updated. All architectural
|
(`agent:nomos`), and all referencing docs were updated. All architectural
|
||||||
principles in this ADR remain unchanged.
|
principles in this ADR remain unchanged.
|
||||||
133
internal/audit/audit.go
Normal file
133
internal/audit/audit.go
Normal file
@@ -0,0 +1,133 @@
|
|||||||
|
// Package audit produces read-only drift reports over the knowledge graph and
|
||||||
|
// monitoring state. It is the shared engine behind the
|
||||||
|
// /api/v1/audit/drift endpoint and the audit_knowledge_graph MCP tool.
|
||||||
|
//
|
||||||
|
// It surfaces the structural gaps an operator otherwise discovers only by
|
||||||
|
// accident: orphan check entities, checks targeting retired entities, probes
|
||||||
|
// stuck down/unknown, unmonitored declared types, and live edges pointing at
|
||||||
|
// destroyed/deprecated targets. Live-infra discovery (pct/docker/certs) is a
|
||||||
|
// follow-up that needs host-hop execution; these categories are pure DB
|
||||||
|
// queries, so the report is cheap, safe to run unattended, and testable.
|
||||||
|
package audit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Finding is one drift item the operator should look at.
|
||||||
|
type Finding struct {
|
||||||
|
Category string `json:"category"`
|
||||||
|
Severity string `json:"severity"` // info | warning | critical
|
||||||
|
Count int `json:"count"`
|
||||||
|
Entities []string `json:"entities"`
|
||||||
|
Evidence string `json:"evidence"`
|
||||||
|
SuggestedRunbook string `json:"suggested_runbook"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Summary tallies findings by category.
|
||||||
|
type Summary struct {
|
||||||
|
TotalFindings int `json:"total_findings"`
|
||||||
|
ByCategory map[string]int `json:"by_category"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Report runs every drift check and returns the findings plus a summary.
|
||||||
|
func Report(ctx context.Context, pool *db.Pool) ([]Finding, Summary) {
|
||||||
|
specs := []struct {
|
||||||
|
finding Finding
|
||||||
|
query string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
Finding{Category: "orphan_checks", Severity: "warning",
|
||||||
|
Evidence: "check entities with truncated/random slugs (legacy shortSlug bug), no live target",
|
||||||
|
SuggestedRunbook: "scripts/cleanup-orphan-checks.sh"},
|
||||||
|
`SELECT e.slug FROM entities e
|
||||||
|
WHERE e.type = 'check'
|
||||||
|
AND e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "dead_checks", Severity: "warning",
|
||||||
|
Evidence: "enabled check_defs whose target entity is deprecated/destroyed",
|
||||||
|
SuggestedRunbook: "lifecycle-deprecate-node / lifecycle-destroy-node"},
|
||||||
|
`SELECT e.slug FROM check_defs cd
|
||||||
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
|
JOIN entities tgt ON tgt.id = cd.target_id
|
||||||
|
WHERE cd.enabled AND tgt.state IN ('deprecated','destroyed')`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "down_checks", Severity: "critical",
|
||||||
|
Evidence: "enabled checks reporting health=down",
|
||||||
|
SuggestedRunbook: "service-health-check"},
|
||||||
|
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||||
|
WHERE cd.enabled AND cd.last_health = 'down'`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "unknown_checks", Severity: "warning",
|
||||||
|
Evidence: "enabled checks that ran but reported health=unknown (likely misconfigured probe)",
|
||||||
|
SuggestedRunbook: "knowledge-graph-audit"},
|
||||||
|
`SELECT e.slug FROM check_defs cd JOIN entities e ON e.id = cd.entity_id
|
||||||
|
WHERE cd.enabled AND cd.last_health = 'unknown'`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "unmonitored", Severity: "warning",
|
||||||
|
Evidence: "active entities whose type declares monitoring but have no enabled check_def",
|
||||||
|
SuggestedRunbook: "knowledge-graph-audit"},
|
||||||
|
`SELECT DISTINCT e.slug FROM signals sg
|
||||||
|
JOIN entities e ON e.id = sg.target_entity_id
|
||||||
|
WHERE sg.kind = 'unmonitored' AND sg.state IN ('raised','acknowledged','acting')`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "dangling_edges", Severity: "warning",
|
||||||
|
Evidence: "live relationships (hosts/provides/mounts) pointing at destroyed/deprecated targets",
|
||||||
|
SuggestedRunbook: "lifecycle-destroy-node"},
|
||||||
|
`SELECT src.slug || ' -' || r.type || '-> ' || tgt.slug FROM relationships r
|
||||||
|
JOIN entities src ON src.id = r.source_id
|
||||||
|
JOIN entities tgt ON tgt.id = r.target_id
|
||||||
|
WHERE r.valid_to IS NULL
|
||||||
|
AND src.state NOT IN ('destroyed','deprecated')
|
||||||
|
AND tgt.state IN ('destroyed','deprecated')`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Finding{Category: "polluted_attrs", Severity: "warning",
|
||||||
|
Evidence: "routing-critical attributes carrying prose (breaks resolution) — e.g. host='hubris (confirmed via pct…')",
|
||||||
|
SuggestedRunbook: "knowledge-graph-audit"},
|
||||||
|
`SELECT slug || ': host=' || (attributes->>'host') FROM entities
|
||||||
|
WHERE attributes->>'host' IS NOT NULL
|
||||||
|
AND (attributes->>'host') ~ '[ (]'`,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
findings := make([]Finding, 0, len(specs))
|
||||||
|
summary := Summary{ByCategory: map[string]int{}}
|
||||||
|
for _, sp := range specs {
|
||||||
|
f := runFinding(ctx, pool, sp.finding, sp.query)
|
||||||
|
findings = append(findings, f)
|
||||||
|
summary.TotalFindings += f.Count
|
||||||
|
summary.ByCategory[f.Category] = f.Count
|
||||||
|
}
|
||||||
|
return findings, summary
|
||||||
|
}
|
||||||
|
|
||||||
|
const entityCap = 50
|
||||||
|
|
||||||
|
// runFinding runs a single-column slug query and folds the rows into a Finding.
|
||||||
|
func runFinding(ctx context.Context, pool *db.Pool, f Finding, query string) Finding {
|
||||||
|
rows, err := pool.Query(ctx, query)
|
||||||
|
if err != nil {
|
||||||
|
f.Evidence = f.Evidence + " (query error: " + err.Error() + ")"
|
||||||
|
return f
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
for rows.Next() {
|
||||||
|
var slug string
|
||||||
|
if err := rows.Scan(&slug); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
f.Count++
|
||||||
|
if len(f.Entities) < entityCap {
|
||||||
|
f.Entities = append(f.Entities, slug)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return f
|
||||||
|
}
|
||||||
69
internal/audit/audit_test.go
Normal file
69
internal/audit/audit_test.go
Normal file
@@ -0,0 +1,69 @@
|
|||||||
|
package audit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Integration tests against a real Postgres, guarded by
|
||||||
|
// OIKOS_TEST_DATABASE_URL (same convention as internal/scheduler).
|
||||||
|
|
||||||
|
func newAuditPool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
base := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||||
|
if base == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
return createTestDB(t, base)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestReportFlagsOrphanAndDeadAndDown(t *testing.T) {
|
||||||
|
pool := newAuditPool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
// An orphan check entity (truncated random slug, the legacy bug shape).
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:ssh-script:0d31fdd1','check','check:ssh-script:0d31fdd1','active','{}'::jsonb,1,now(),now())`, uuid.New())
|
||||||
|
|
||||||
|
// An active entity + a check_def on it stuck down.
|
||||||
|
target := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'service:demo','service','demo','active','{}'::jsonb,1,now(),now())`, target)
|
||||||
|
checkE := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:http:service:demo:0','check','c','active','{}'::jsonb,1,now(),now())`, checkE)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at, last_health)
|
||||||
|
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now(),'down')`, checkE, target)
|
||||||
|
|
||||||
|
// A deprecated entity still carrying an enabled check (dead_checks).
|
||||||
|
dep := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'service:old','service','old','deprecated','{}'::jsonb,1,now(),now())`, dep)
|
||||||
|
depCheck := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'check:http:service:old:0','check','c','active','{}'::jsonb,1,now(),now())`, depCheck)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||||
|
VALUES ($1,$2,'service','http','{}'::jsonb,60,30,true,now())`, depCheck, dep)
|
||||||
|
|
||||||
|
findings, summary := Report(ctx, pool)
|
||||||
|
|
||||||
|
byCat := map[string]int{}
|
||||||
|
for _, f := range findings {
|
||||||
|
byCat[f.Category] = f.Count
|
||||||
|
}
|
||||||
|
if byCat["orphan_checks"] < 1 {
|
||||||
|
t.Errorf("orphan_checks = %d, want >=1", byCat["orphan_checks"])
|
||||||
|
}
|
||||||
|
if byCat["down_checks"] < 1 {
|
||||||
|
t.Errorf("down_checks = %d, want >=1", byCat["down_checks"])
|
||||||
|
}
|
||||||
|
if byCat["dead_checks"] < 1 {
|
||||||
|
t.Errorf("dead_checks = %d, want >=1", byCat["dead_checks"])
|
||||||
|
}
|
||||||
|
if summary.TotalFindings < 3 {
|
||||||
|
t.Errorf("TotalFindings = %d, want >=3", summary.TotalFindings)
|
||||||
|
}
|
||||||
|
}
|
||||||
67
internal/audit/testutil_test.go
Normal file
67
internal/audit/testutil_test.go
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
package audit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// createTestDB provisions a throwaway migrated database, same convention as
|
||||||
|
// internal/scheduler/coverage_test.go.
|
||||||
|
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_aud_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
at := strings.LastIndex(baseURL, "/")
|
||||||
|
testURL := baseURL[:at+1] + dbName
|
||||||
|
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||||
|
testURL += baseURL[at+q:]
|
||||||
|
}
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func getenvOrDefault(key, def string) string {
|
||||||
|
if v := os.Getenv(key); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||||
|
t.Fatalf("exec %s: %v", q, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -24,13 +24,16 @@ import (
|
|||||||
// check_defs.kind values — one semantic kind can expand to several concrete
|
// check_defs.kind values — one semantic kind can expand to several concrete
|
||||||
// checks (`resource` becomes four ssh-script rows).
|
// checks (`resource` becomes four ssh-script rows).
|
||||||
const (
|
const (
|
||||||
KindPing = "ping"
|
KindPing = "ping"
|
||||||
KindResource = "resource"
|
KindResource = "resource"
|
||||||
KindUpdates = "updates"
|
KindUpdates = "updates"
|
||||||
KindProcess = "process"
|
KindProcess = "process"
|
||||||
KindHTTP = "http"
|
KindHTTP = "http"
|
||||||
KindCapacity = "capacity"
|
KindCapacity = "capacity"
|
||||||
KindBackup = "backup-freshness"
|
KindBackup = "backup-freshness"
|
||||||
|
KindCertExpiry = "cert-expiry"
|
||||||
|
KindVMStatus = "vm-status"
|
||||||
|
KindDNS = "dns"
|
||||||
)
|
)
|
||||||
|
|
||||||
// defaultBackupMaxAge is how long a backup target may go without a new
|
// defaultBackupMaxAge is how long a backup target may go without a new
|
||||||
@@ -108,6 +111,18 @@ func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (
|
|||||||
attrs = map[string]any{}
|
attrs = map[string]any{}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Per-entity override: an explicit `monitoring` attribute wins over the
|
||||||
|
// type declaration. A single entity can opt out (monitoring: none) or pick
|
||||||
|
// different kinds without introducing a new type — e.g. service:haos opts
|
||||||
|
// out because its VM is already covered by a vm-status check and the
|
||||||
|
// service can't be SSH-probed (haos blocks SSH).
|
||||||
|
if mo, ok := attrs["monitoring"]; ok {
|
||||||
|
mon = resolveMonitoringAttr(mo, mon)
|
||||||
|
if mon.None() {
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// A service has no address of its own — it lives on the container that
|
// A service has no address of its own — it lives on the container that
|
||||||
// provides it. Fall back to the graph before giving up.
|
// provides it. Fall back to the graph before giving up.
|
||||||
host := resolveHost(attrs)
|
host := resolveHost(attrs)
|
||||||
@@ -146,6 +161,27 @@ func Ensure(ctx context.Context, tx pgx.Tx, tree *ontology.TypeTree, t Target) (
|
|||||||
return res, nil
|
return res, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// resolveMonitoringAttr turns an entity's `monitoring` attribute into a
|
||||||
|
// MonitoringResolution that overrides the type's declaration. Accepts the
|
||||||
|
// scalar "none" (or empty) to opt out, or a list of kind strings to override.
|
||||||
|
func resolveMonitoringAttr(v any, fallback ontology.MonitoringResolution) ontology.MonitoringResolution {
|
||||||
|
switch vv := v.(type) {
|
||||||
|
case string:
|
||||||
|
if vv == "none" || vv == "" {
|
||||||
|
return ontology.MonitoringResolution{Declared: true, Source: "attribute"}
|
||||||
|
}
|
||||||
|
case []any:
|
||||||
|
kinds := make([]string, 0, len(vv))
|
||||||
|
for _, k := range vv {
|
||||||
|
if s, ok := k.(string); ok && s != "" {
|
||||||
|
kinds = append(kinds, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return ontology.MonitoringResolution{Declared: true, Kinds: kinds, Source: "attribute"}
|
||||||
|
}
|
||||||
|
return fallback
|
||||||
|
}
|
||||||
|
|
||||||
// buildKind turns one declared semantic kind into concrete check_defs, or
|
// buildKind turns one declared semantic kind into concrete check_defs, or
|
||||||
// returns the reason it could not.
|
// returns the reason it could not.
|
||||||
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
|
func buildKind(kind string, t Target, attrs map[string]any, host, user string, port int) ([]checkDef, string) {
|
||||||
@@ -201,12 +237,36 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
|
|||||||
if host == "" {
|
if host == "" {
|
||||||
return nil, "no address on the entity or its host"
|
return nil, "no address on the entity or its host"
|
||||||
}
|
}
|
||||||
if t.Name == "" {
|
// A service's name is a logical label, not usually its systemd unit
|
||||||
|
// or container name (matrix = matrix-synapse.service + containers).
|
||||||
|
// Prefer an explicit probe target when declared; process_check.sh also
|
||||||
|
// matches a unit prefix or a docker container as a fallback.
|
||||||
|
unit := ""
|
||||||
|
for _, key := range []string{"probe_unit", "systemd_unit", "container"} {
|
||||||
|
if v, _ := attrs[key].(string); v != "" {
|
||||||
|
unit = v
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Ontology intent: "http when it has a url, else a process check." A
|
||||||
|
// url-fronted service is already liveness-probed via http (the real
|
||||||
|
// endpoint, through the TLS terminator); the process check is redundant
|
||||||
|
// and fragile (needs host access + the exact unit/container name), and
|
||||||
|
// under worst-of aggregation it lets a broken supplementary probe veto
|
||||||
|
// a working service. Emit it only for services WITHOUT a url, or when
|
||||||
|
// an explicit probe_unit opts into binary-level depth.
|
||||||
|
if unit == "" {
|
||||||
|
if httpURL(t, attrs) != "" {
|
||||||
|
return nil, "url present and no probe_unit; http check covers liveness"
|
||||||
|
}
|
||||||
|
unit = t.Name
|
||||||
|
}
|
||||||
|
if unit == "" {
|
||||||
return nil, "no name to check a process for"
|
return nil, "no name to check a process for"
|
||||||
}
|
}
|
||||||
// process_check.sh takes the unit name as $1 and reports "unknown"
|
// process_check.sh takes the unit/container name as $1 and reports
|
||||||
// without it.
|
// "unknown" without it.
|
||||||
return []checkDef{ssh("process_check.sh", t.Name)}, ""
|
return []checkDef{ssh("process_check.sh", unit)}, ""
|
||||||
|
|
||||||
case KindBackup:
|
case KindBackup:
|
||||||
// A backup target is checked from the machine that writes to it, so it
|
// A backup target is checked from the machine that writes to it, so it
|
||||||
@@ -246,11 +306,79 @@ func buildKind(kind string, t Target, attrs map[string]any, host, user string, p
|
|||||||
config: map[string]any{"url": url, "max_status": 500},
|
config: map[string]any{"url": url, "max_status": 500},
|
||||||
interval: 60,
|
interval: 60,
|
||||||
}}, ""
|
}}, ""
|
||||||
|
|
||||||
|
case KindDNS:
|
||||||
|
// Resolve the entity's name via DNS to verify the zone is reachable.
|
||||||
|
// Uses the entity name (zone apex) or falls back to the slug.
|
||||||
|
name := t.Name
|
||||||
|
if name == "" {
|
||||||
|
name = strings.TrimPrefix(t.Slug, "zone:")
|
||||||
|
}
|
||||||
|
if name == "" {
|
||||||
|
return nil, "no name to resolve"
|
||||||
|
}
|
||||||
|
return []checkDef{{
|
||||||
|
kind: "dns",
|
||||||
|
config: map[string]any{"name": name},
|
||||||
|
interval: 300, // 5 min — DNS changes are rare; the cost of a miss
|
||||||
|
// is a stale IP, not a service outage.
|
||||||
|
}}, ""
|
||||||
|
|
||||||
|
case KindCertExpiry:
|
||||||
|
// The host whose cert to read (SNI / cert CN). Prefer an explicit
|
||||||
|
// `hostname` attribute, then `cn`, then a dotted name. Hourly: expiry
|
||||||
|
// changes once a day, but a renewal or mis-issued cert is worth
|
||||||
|
// noticing within the hour.
|
||||||
|
host := certHost(t, attrs)
|
||||||
|
if host == "" {
|
||||||
|
return nil, "no hostname / cn / dotted name to dial for the cert"
|
||||||
|
}
|
||||||
|
// `dial` is the TLS terminator's address to connect to (Caddy's lab
|
||||||
|
// IP), used when the hostname doesn't resolve/reach from the scheduler.
|
||||||
|
// Without it the probe can't reach *.hubris.network from a container
|
||||||
|
// with no mesh / split-horizon DNS.
|
||||||
|
dial, _ := attrs["dial"].(string)
|
||||||
|
config := map[string]any{"host": host, "warn_days": 30, "crit_days": 7}
|
||||||
|
if dial != "" {
|
||||||
|
config["dial"] = dial
|
||||||
|
}
|
||||||
|
return []checkDef{{
|
||||||
|
kind: "cert-expiry",
|
||||||
|
config: config,
|
||||||
|
interval: 3600,
|
||||||
|
}}, ""
|
||||||
|
|
||||||
|
case KindVMStatus:
|
||||||
|
// "Is the VM powered on" via `qm status` on its Proxmox host — the
|
||||||
|
// right reachability probe for a VM, since many block ICMP and lack a
|
||||||
|
// guest agent. checkVMStatus re-reads pve_id + host at runtime.
|
||||||
|
if _, ok := attrs["pve_id"]; !ok {
|
||||||
|
return nil, "no pve_id to run qm status"
|
||||||
|
}
|
||||||
|
return []checkDef{{
|
||||||
|
kind: "vm-status",
|
||||||
|
config: map[string]any{},
|
||||||
|
interval: 60,
|
||||||
|
}}, ""
|
||||||
}
|
}
|
||||||
|
|
||||||
return nil, "no builder for this kind yet"
|
return nil, "no builder for this kind yet"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// certHost works out the hostname to TLS-dial for a certificate's expiry.
|
||||||
|
func certHost(t Target, attrs map[string]any) string {
|
||||||
|
for _, key := range []string{"hostname", "cn", "san"} {
|
||||||
|
if v, ok := attrs[key].(string); ok && v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// A dotted name is a hostname (hubris.network, media.hubris.network).
|
||||||
|
if strings.Contains(t.Name, ".") && !strings.Contains(t.Name, " ") {
|
||||||
|
return t.Name
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
// writeCheck upserts one check_def and its backing check entity.
|
// writeCheck upserts one check_def and its backing check entity.
|
||||||
//
|
//
|
||||||
// The entity upsert MUST return the row's id. The previous version generated
|
// The entity upsert MUST return the row's id. The previous version generated
|
||||||
@@ -299,14 +427,15 @@ func writeCheck(ctx context.Context, tx pgx.Tx, t Target, idx int, def checkDef)
|
|||||||
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
// absent from the DO UPDATE below — a re-seed must not reset the schedule
|
||||||
// and re-herd everything.
|
// and re-herd everything.
|
||||||
tag, err := tx.Exec(ctx,
|
tag, err := tx.Exec(ctx,
|
||||||
`INSERT INTO check_defs (entity_id, target_id, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
`INSERT INTO check_defs (entity_id, target_id, target_type, kind, config, interval_s, timeout_s, enabled, last_run_at)
|
||||||
VALUES ($1, $2, $3, $4, $5, 30, true,
|
VALUES ($1, $2, $6, $3, $4, $5, 30, true,
|
||||||
now() - make_interval(secs => random() * $5::int))
|
now() - make_interval(secs => random() * $5::int))
|
||||||
ON CONFLICT (entity_id) DO UPDATE
|
ON CONFLICT (entity_id) DO UPDATE
|
||||||
SET target_id = EXCLUDED.target_id, kind = EXCLUDED.kind,
|
SET target_id = EXCLUDED.target_id, target_type = EXCLUDED.target_type,
|
||||||
|
kind = EXCLUDED.kind,
|
||||||
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
config = EXCLUDED.config, interval_s = EXCLUDED.interval_s,
|
||||||
updated_at = now()`,
|
updated_at = now()`,
|
||||||
checkID, t.ID, def.kind, configJSON, def.interval)
|
checkID, t.ID, def.kind, configJSON, def.interval, t.Type)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
|
return false, fmt.Errorf("upsert check_def %s: %w", checkSlug, err)
|
||||||
}
|
}
|
||||||
@@ -380,6 +509,12 @@ func resolveHost(attrs map[string]any) string {
|
|||||||
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
if ip, ok := attrs["lan_ip"].(string); ok && ip != "" {
|
||||||
return ip
|
return ip
|
||||||
}
|
}
|
||||||
|
// public_ipv4 before mesh: the scheduler container has no mesh interface,
|
||||||
|
// so a standalone-server reachable only by mesh IP (netbird-vps) is
|
||||||
|
// unprobeable even though a public IPv4 is available.
|
||||||
|
if ip, ok := attrs["public_ipv4"].(string); ok && ip != "" {
|
||||||
|
return ip
|
||||||
|
}
|
||||||
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
if mesh, ok := attrs["mesh"].(map[string]any); ok {
|
||||||
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||||
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
if ip, ok := nb["ip"].(string); ok && ip != "" {
|
||||||
@@ -409,6 +544,13 @@ func resolveSSHUser(attrs map[string]any) string {
|
|||||||
return u
|
return u
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
// Workstations carry their login as a top-level `user` attribute
|
||||||
|
// (mac-mini: user: dtoro) rather than under ssh.user. Take it only when
|
||||||
|
// no explicit ssh.user was set, so a host that genuinely wants root still
|
||||||
|
// gets root.
|
||||||
|
if u, ok := attrs["user"].(string); ok && u != "" {
|
||||||
|
return u
|
||||||
|
}
|
||||||
return "root"
|
return "root"
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
34
internal/db/checks.go
Normal file
34
internal/db/checks.go
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// EnsureEntityChecks derives an entity's default check_defs from the
|
||||||
|
// monitoring spec of its type (resolving per-entity `monitoring` overrides).
|
||||||
|
//
|
||||||
|
// This is the single shared hook that keeps the check graph in sync with
|
||||||
|
// entity mutations. Both the HTTP create/patch handlers and the MCP
|
||||||
|
// entity-mutation tools (create_entity, update_entity_attributes) call it so
|
||||||
|
// that flipping an entity's `monitoring` attribute regenerates checks
|
||||||
|
// regardless of which surface made the change — previously only the HTTP
|
||||||
|
// path ran check derivation, so entities mutated via MCP silently produced no
|
||||||
|
// checks (see plans/2026-08-03-session-review-haos-monitoring-capability-gaps.md, A2).
|
||||||
|
func EnsureEntityChecks(ctx context.Context, tx pgx.Tx, id uuid.UUID, slug, entityType, name string, attrs []byte) (checkdefaults.Result, error) {
|
||||||
|
tree, err := LoadTypeTree(ctx, tx)
|
||||||
|
if err != nil {
|
||||||
|
return checkdefaults.Result{}, err
|
||||||
|
}
|
||||||
|
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
||||||
|
ID: id, Slug: slug, Type: entityType, Name: name, Attrs: attrs,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return res, err
|
||||||
|
}
|
||||||
|
checkdefaults.LogResult(slug, entityType, res)
|
||||||
|
return res, nil
|
||||||
|
}
|
||||||
@@ -306,7 +306,17 @@ func TestBlastRadiusTerminatesOnCycles(t *testing.T) {
|
|||||||
pool := newTestPool(t)
|
pool := newTestPool(t)
|
||||||
seedAll(t, pool, seedsDir())
|
seedAll(t, pool, seedsDir())
|
||||||
|
|
||||||
// Build a dependency cycle: gitea → caddy → authentik → gitea
|
// Build a dependency cycle: gitea → caddy → authentik → gitea.
|
||||||
|
//
|
||||||
|
// `depends-on` is declared blast_direction: backward — "A depends-on B"
|
||||||
|
// means B failing breaks A — so the blast radius of gitea walks the edges
|
||||||
|
// BACKWARDS: whoever depends on gitea is affected first. That is authentik
|
||||||
|
// (1 hop), then caddy which depends on authentik (2 hops).
|
||||||
|
//
|
||||||
|
// This test previously asserted caddy=1, authentik=2, which is the same
|
||||||
|
// cycle walked the wrong way round: blast_radius used to follow every edge
|
||||||
|
// source→target regardless of what the edge means, so it answered "what
|
||||||
|
// does gitea depend on" while being named for the opposite question.
|
||||||
cycle := []byte(`
|
cycle := []byte(`
|
||||||
version: 1
|
version: 1
|
||||||
relationships:
|
relationships:
|
||||||
@@ -342,14 +352,24 @@ relationships:
|
|||||||
}
|
}
|
||||||
got[slug] = depth
|
got[slug] = depth
|
||||||
}
|
}
|
||||||
want := map[string]int{"service:gitea": 0, "service:caddy": 1, "service:authentik": 2}
|
want := map[string]int{"service:gitea": 0, "service:authentik": 1, "service:caddy": 2}
|
||||||
for slug, depth := range want {
|
for slug, depth := range want {
|
||||||
if got[slug] != depth {
|
if got[slug] != depth {
|
||||||
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
t.Errorf("blast_radius[%s] = %d, want %d (full: %v)", slug, got[slug], depth, got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if len(got) != len(want) {
|
// Deliberately not an exact node count. Walking the right way round also
|
||||||
t.Errorf("blast_radius returned %d nodes, want %d: %v", len(got), len(want), got)
|
// surfaces the real seed's own dependents of gitea (homelab-mcp and what
|
||||||
|
// depends on it), which are correct answers — the old exact-count
|
||||||
|
// assertion only held because the forward walk found nothing real.
|
||||||
|
// What matters here is that the cycle terminates rather than recursing.
|
||||||
|
if len(got) > 20 {
|
||||||
|
t.Errorf("blast_radius did not terminate sensibly: %d nodes: %v", len(got), got)
|
||||||
|
}
|
||||||
|
for slug, depth := range got {
|
||||||
|
if depth > 5 {
|
||||||
|
t.Errorf("blast_radius[%s] = %d, beyond the max_depth bound", slug, depth)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
146
internal/db/lifecycle.go
Normal file
146
internal/db/lifecycle.go
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
package db
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ErrTransitionInvalid is a sentinel returned by ValidateTransition when the
|
||||||
|
// from→to pair is not a declared lifecycle transition or a precondition fails.
|
||||||
|
// Callers test with errors.Is to distinguish semantic validation failures
|
||||||
|
// (→ HTTP 409) from infrastructure errors (→ HTTP 500).
|
||||||
|
var ErrTransitionInvalid = errors.New("invalid lifecycle transition")
|
||||||
|
|
||||||
|
// ValidateTransition enforces an entity type's lifecycle: fromState → toState
|
||||||
|
// must be a declared transition, and every precondition it lists must hold. A
|
||||||
|
// type with no lifecycle defined allows any state. A no-op (fromState ==
|
||||||
|
// toState) passes immediately.
|
||||||
|
//
|
||||||
|
// Shared by the HTTP PATCH path and the MCP set_entity_state tool so both
|
||||||
|
// surfaces apply identical lifecycle rules — previously only the HTTP path
|
||||||
|
// validated transitions, so an agent changing state via MCP could skip the
|
||||||
|
// graph's retire/deprecate guardrails entirely.
|
||||||
|
func ValidateTransition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, fromState, toState string) error {
|
||||||
|
if toState == fromState {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, entityType)
|
||||||
|
if err != nil {
|
||||||
|
if err == pgx.ErrNoRows {
|
||||||
|
return nil // no lifecycle defined → any state allowed
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
var transitions map[string]map[string]json.RawMessage
|
||||||
|
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
||||||
|
return fmt.Errorf("parse lifecycle transitions: %w", err)
|
||||||
|
}
|
||||||
|
tos, ok := transitions[fromState]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("%w: no transitions defined from %q", ErrTransitionInvalid, fromState)
|
||||||
|
}
|
||||||
|
trans, ok := tos[toState]
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("%w: %s → %s is not a declared lifecycle transition", ErrTransitionInvalid, fromState, toState)
|
||||||
|
}
|
||||||
|
var gate struct {
|
||||||
|
Requires []string `json:"requires"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(trans, &gate); err == nil {
|
||||||
|
for _, check := range gate.Requires {
|
||||||
|
if err := checkPrecondition(ctx, tx, entityID, entityType, check); err != nil {
|
||||||
|
return fmt.Errorf("%w: precondition %q not met: %w", ErrTransitionInvalid, check, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkPrecondition evaluates one mechanical precondition named by a lifecycle
|
||||||
|
// transition's `requires` list. Soft/operator-confirmed checks pass; unknown
|
||||||
|
// checks are skipped (operator intent overrides). Moved here from httpapi so
|
||||||
|
// both surfaces share one implementation.
|
||||||
|
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
||||||
|
switch check {
|
||||||
|
case "no-inbound-edges":
|
||||||
|
var count int
|
||||||
|
if err := tx.QueryRow(ctx,
|
||||||
|
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if count > 0 {
|
||||||
|
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
||||||
|
}
|
||||||
|
case "backups-verified", "secrets-revoked", "ingress-dns-removed":
|
||||||
|
var attrs string
|
||||||
|
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
want := map[string]string{
|
||||||
|
"backups-verified": "backups_verified",
|
||||||
|
"secrets-revoked": "secrets_revoked",
|
||||||
|
"ingress-dns-removed": "ingress_dns_removed",
|
||||||
|
}[check]
|
||||||
|
if !strings.Contains(attrs, want) {
|
||||||
|
return fmt.Errorf("%s not recorded in entity attributes", want)
|
||||||
|
}
|
||||||
|
case "age-key-enrolled-if-needed":
|
||||||
|
if entityType == "workstation" {
|
||||||
|
var attrs string
|
||||||
|
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !strings.Contains(attrs, "age_pubkey") {
|
||||||
|
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "mesh-joined-if-needed":
|
||||||
|
if entityType == "workstation" {
|
||||||
|
var attrs string
|
||||||
|
if err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !strings.Contains(attrs, "mesh_ip") {
|
||||||
|
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
case "health-check-answering":
|
||||||
|
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
||||||
|
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
||||||
|
h := "unknown"
|
||||||
|
if err == nil {
|
||||||
|
h = st.Health
|
||||||
|
}
|
||||||
|
return fmt.Errorf("health check not answering (status: %s)", h)
|
||||||
|
}
|
||||||
|
case "doc-page-complete":
|
||||||
|
var count int
|
||||||
|
if err := tx.QueryRow(ctx, `
|
||||||
|
SELECT count(*) FROM relationships r
|
||||||
|
JOIN entities ke ON ke.id = r.source_id
|
||||||
|
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||||
|
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
||||||
|
entityID).Scan(&count); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if count == 0 {
|
||||||
|
return fmt.Errorf("no documentation linked to entity")
|
||||||
|
}
|
||||||
|
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
||||||
|
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
||||||
|
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
||||||
|
"ingress-live-if-public", "doc-page-stub", "un-deprecate-note", "write-off-note":
|
||||||
|
// Soft checks — always pass. Operator-confirmed via the transition
|
||||||
|
// request itself, or not mechanically enforceable.
|
||||||
|
default:
|
||||||
|
// Unknown preconditions are skipped (operator intent overrides).
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -25,8 +25,8 @@ ON CONFLICT (actor, key) DO NOTHING;
|
|||||||
|
|
||||||
-- name: InsertAuditEntry :exec
|
-- name: InsertAuditEntry :exec
|
||||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||||
status_code, detail, source_ip, correlation_id)
|
status_code, detail, source_ip, correlation_id, session_id)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10);
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11);
|
||||||
|
|
||||||
-- name: InsertEvent :one
|
-- name: InsertEvent :one
|
||||||
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
|
INSERT INTO events (type, entity_id, severity, source, data, correlation_id)
|
||||||
@@ -63,12 +63,28 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
|||||||
e.slug AS entity_slug
|
e.slug AS entity_slug
|
||||||
FROM check_defs cd
|
FROM check_defs cd
|
||||||
JOIN entities e ON e.id = cd.entity_id
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
|
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||||
WHERE cd.enabled = true
|
WHERE cd.enabled = true
|
||||||
|
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||||
AND (cd.last_run_at IS NULL
|
AND (cd.last_run_at IS NULL
|
||||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
|
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s));
|
||||||
|
|
||||||
-- name: MarkCheckRun :exec
|
-- name: MarkCheckRun :exec
|
||||||
UPDATE check_defs SET last_run_at = now() WHERE entity_id = $1;
|
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1;
|
||||||
|
|
||||||
|
-- name: WorstHealthForTarget :one
|
||||||
|
-- An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||||
|
-- yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||||
|
-- newly added check does not drag a known-good entity down before it has
|
||||||
|
-- produced a verdict.
|
||||||
|
SELECT COALESCE(
|
||||||
|
(SELECT last_health FROM check_defs
|
||||||
|
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||||
|
ORDER BY CASE last_health
|
||||||
|
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||||
|
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||||
|
LIMIT 1),
|
||||||
|
'unknown')::text AS health;
|
||||||
|
|
||||||
-- name: GetCheckDef :one
|
-- name: GetCheckDef :one
|
||||||
SELECT * FROM check_defs WHERE entity_id = $1;
|
SELECT * FROM check_defs WHERE entity_id = $1;
|
||||||
|
|||||||
@@ -67,12 +67,20 @@ func IngestOntologySeed(ctx context.Context, tx pgx.Tx, data map[string]any) (*S
|
|||||||
targetType, _ := rtMap["target"].(string)
|
targetType, _ := rtMap["target"].(string)
|
||||||
cardinality, _ := rtMap["cardinality"].(string)
|
cardinality, _ := rtMap["cardinality"].(string)
|
||||||
desc, _ := rtMap["description"].(string)
|
desc, _ := rtMap["description"].(string)
|
||||||
|
// Which end of the edge depends on the other; drives blast_radius().
|
||||||
|
// Absent means 'none' — an undeclared edge contributes nothing rather
|
||||||
|
// than silently producing a wrong dependency answer.
|
||||||
|
blastDirection, _ := rtMap["blast_direction"].(string)
|
||||||
|
if blastDirection == "" {
|
||||||
|
blastDirection = "none"
|
||||||
|
}
|
||||||
_, err := tx.Exec(ctx,
|
_, err := tx.Exec(ctx,
|
||||||
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description)
|
`INSERT INTO relationship_types (name, inverse, source_type, target_type, cardinality, description, blast_direction)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6)
|
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||||
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
ON CONFLICT (name) DO UPDATE SET inverse = $2, source_type = $3,
|
||||||
target_type = $4, cardinality = $5, description = $6`,
|
target_type = $4, cardinality = $5, description = $6,
|
||||||
name, nullableStr(inverse), sourceType, targetType, cardinality, desc)
|
blast_direction = $7`,
|
||||||
|
name, nullableStr(inverse), sourceType, targetType, cardinality, desc, blastDirection)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
return nil, fmt.Errorf("relationship_type %s: %w", name, err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -114,6 +114,8 @@ type CheckDef struct {
|
|||||||
UpdatedAt time.Time
|
UpdatedAt time.Time
|
||||||
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
// When this check last executed. NULL = never, due immediately. Compared against interval_s to decide due-ness.
|
||||||
LastRunAt *time.Time
|
LastRunAt *time.Time
|
||||||
|
// This check's own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target's enabled checks.
|
||||||
|
LastHealth *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type Classification struct {
|
type Classification struct {
|
||||||
@@ -255,6 +257,20 @@ type KnowledgeEntity struct {
|
|||||||
UpdatedAt time.Time
|
UpdatedAt time.Time
|
||||||
ContentHash *string
|
ContentHash *string
|
||||||
Search interface{}
|
Search interface{}
|
||||||
|
EditedBy string
|
||||||
|
DeletedAt *time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
type KnowledgeRevision struct {
|
||||||
|
ID int64
|
||||||
|
EntityID uuid.UUID
|
||||||
|
Title string
|
||||||
|
Content string
|
||||||
|
Source *string
|
||||||
|
Tags []string
|
||||||
|
EditedBy string
|
||||||
|
VersionAt time.Time
|
||||||
|
RevisedAt time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
type Ledger struct {
|
type Ledger struct {
|
||||||
@@ -364,6 +380,8 @@ type RelationshipType struct {
|
|||||||
Cardinality string
|
Cardinality string
|
||||||
Description *string
|
Description *string
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
|
// Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().
|
||||||
|
BlastDirection string
|
||||||
}
|
}
|
||||||
|
|
||||||
type RiskClass struct {
|
type RiskClass struct {
|
||||||
@@ -380,18 +398,19 @@ type SeedVersion struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type SessionPlanStep struct {
|
type SessionPlanStep struct {
|
||||||
ID uuid.UUID
|
ID uuid.UUID
|
||||||
SessionID uuid.UUID
|
SessionID uuid.UUID
|
||||||
Seq int32
|
Seq int32
|
||||||
Title string
|
Title string
|
||||||
Detail string
|
Detail string
|
||||||
Status string
|
Status string
|
||||||
ExecutionID *uuid.UUID
|
ExecutionID *uuid.UUID
|
||||||
TargetSlug *string
|
TargetSlug *string
|
||||||
StartedAt *time.Time
|
StartedAt *time.Time
|
||||||
FinishedAt *time.Time
|
FinishedAt *time.Time
|
||||||
CreatedAt time.Time
|
CreatedAt time.Time
|
||||||
Generation int32
|
Generation int32
|
||||||
|
ReplacedReason *string
|
||||||
}
|
}
|
||||||
|
|
||||||
type SessionQuestion struct {
|
type SessionQuestion struct {
|
||||||
|
|||||||
@@ -99,7 +99,7 @@ func (q *Queries) ListLifecycleDefs(ctx context.Context) ([]LifecycleDef, error)
|
|||||||
}
|
}
|
||||||
|
|
||||||
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
|
const listRelationshipTypes = `-- name: ListRelationshipTypes :many
|
||||||
SELECT name, inverse, source_type, target_type, cardinality, description, created_at FROM relationship_types ORDER BY name
|
SELECT name, inverse, source_type, target_type, cardinality, description, created_at, blast_direction FROM relationship_types ORDER BY name
|
||||||
`
|
`
|
||||||
|
|
||||||
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
|
func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType, error) {
|
||||||
@@ -119,6 +119,7 @@ func (q *Queries) ListRelationshipTypes(ctx context.Context) ([]RelationshipType
|
|||||||
&i.Cardinality,
|
&i.Cardinality,
|
||||||
&i.Description,
|
&i.Description,
|
||||||
&i.CreatedAt,
|
&i.CreatedAt,
|
||||||
|
&i.BlastDirection,
|
||||||
); err != nil {
|
); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ func (q *Queries) GetAutonomySetting(ctx context.Context, key string) (string, e
|
|||||||
}
|
}
|
||||||
|
|
||||||
const getCheckDef = `-- name: GetCheckDef :one
|
const getCheckDef = `-- name: GetCheckDef :one
|
||||||
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at FROM check_defs WHERE entity_id = $1
|
SELECT entity_id, target_id, target_type, kind, config, interval_s, timeout_s, zone, enabled, updated_at, last_run_at, last_health FROM check_defs WHERE entity_id = $1
|
||||||
`
|
`
|
||||||
|
|
||||||
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef, error) {
|
||||||
@@ -69,6 +69,7 @@ func (q *Queries) GetCheckDef(ctx context.Context, entityID uuid.UUID) (CheckDef
|
|||||||
&i.Enabled,
|
&i.Enabled,
|
||||||
&i.UpdatedAt,
|
&i.UpdatedAt,
|
||||||
&i.LastRunAt,
|
&i.LastRunAt,
|
||||||
|
&i.LastHealth,
|
||||||
)
|
)
|
||||||
return i, err
|
return i, err
|
||||||
}
|
}
|
||||||
@@ -352,8 +353,8 @@ func (q *Queries) InsertApproval(ctx context.Context, arg InsertApprovalParams)
|
|||||||
|
|
||||||
const insertAuditEntry = `-- name: InsertAuditEntry :exec
|
const insertAuditEntry = `-- name: InsertAuditEntry :exec
|
||||||
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
INSERT INTO audit_log (actor_type, actor_id, action, entity_id, method, path,
|
||||||
status_code, detail, source_ip, correlation_id)
|
status_code, detail, source_ip, correlation_id, session_id)
|
||||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11)
|
||||||
`
|
`
|
||||||
|
|
||||||
type InsertAuditEntryParams struct {
|
type InsertAuditEntryParams struct {
|
||||||
@@ -367,6 +368,7 @@ type InsertAuditEntryParams struct {
|
|||||||
Detail []byte
|
Detail []byte
|
||||||
SourceIp *string
|
SourceIp *string
|
||||||
CorrelationID *string
|
CorrelationID *string
|
||||||
|
SessionID *uuid.UUID
|
||||||
}
|
}
|
||||||
|
|
||||||
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
|
func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryParams) error {
|
||||||
@@ -381,6 +383,7 @@ func (q *Queries) InsertAuditEntry(ctx context.Context, arg InsertAuditEntryPara
|
|||||||
arg.Detail,
|
arg.Detail,
|
||||||
arg.SourceIp,
|
arg.SourceIp,
|
||||||
arg.CorrelationID,
|
arg.CorrelationID,
|
||||||
|
arg.SessionID,
|
||||||
)
|
)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -695,7 +698,9 @@ SELECT cd.entity_id, cd.target_id, cd.target_type, cd.kind, cd.config,
|
|||||||
e.slug AS entity_slug
|
e.slug AS entity_slug
|
||||||
FROM check_defs cd
|
FROM check_defs cd
|
||||||
JOIN entities e ON e.id = cd.entity_id
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
|
LEFT JOIN entities tgt ON tgt.id = cd.target_id
|
||||||
WHERE cd.enabled = true
|
WHERE cd.enabled = true
|
||||||
|
AND (tgt.id IS NULL OR tgt.state IS NULL OR tgt.state NOT IN ('deprecated', 'destroyed'))
|
||||||
AND (cd.last_run_at IS NULL
|
AND (cd.last_run_at IS NULL
|
||||||
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
|
OR cd.last_run_at <= now() - make_interval(secs => cd.interval_s))
|
||||||
`
|
`
|
||||||
@@ -1133,11 +1138,16 @@ func (q *Queries) ListSkills(ctx context.Context, status *string) ([]Skill, erro
|
|||||||
}
|
}
|
||||||
|
|
||||||
const markCheckRun = `-- name: MarkCheckRun :exec
|
const markCheckRun = `-- name: MarkCheckRun :exec
|
||||||
UPDATE check_defs SET last_run_at = now() WHERE entity_id = $1
|
UPDATE check_defs SET last_run_at = now(), last_health = $2 WHERE entity_id = $1
|
||||||
`
|
`
|
||||||
|
|
||||||
func (q *Queries) MarkCheckRun(ctx context.Context, entityID uuid.UUID) error {
|
type MarkCheckRunParams struct {
|
||||||
_, err := q.db.Exec(ctx, markCheckRun, entityID)
|
EntityID uuid.UUID
|
||||||
|
LastHealth *string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (q *Queries) MarkCheckRun(ctx context.Context, arg MarkCheckRunParams) error {
|
||||||
|
_, err := q.db.Exec(ctx, markCheckRun, arg.EntityID, arg.LastHealth)
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1462,3 +1472,25 @@ func (q *Queries) UpsertSignal(ctx context.Context, arg UpsertSignalParams) (Sig
|
|||||||
)
|
)
|
||||||
return i, err
|
return i, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const worstHealthForTarget = `-- name: WorstHealthForTarget :one
|
||||||
|
SELECT COALESCE(
|
||||||
|
(SELECT last_health FROM check_defs
|
||||||
|
WHERE enabled AND target_id = $1 AND last_health IS NOT NULL
|
||||||
|
ORDER BY CASE last_health
|
||||||
|
WHEN 'down' THEN 0 WHEN 'degraded' THEN 1 WHEN 'stale' THEN 2
|
||||||
|
WHEN 'unknown' THEN 3 ELSE 4 END
|
||||||
|
LIMIT 1),
|
||||||
|
'unknown')::text AS health
|
||||||
|
`
|
||||||
|
|
||||||
|
// An entity is as healthy as its unhealthiest check. Checks that have not run
|
||||||
|
// yet (last_health IS NULL) are ignored rather than counted as unknown, so a
|
||||||
|
// newly added check does not drag a known-good entity down before it has
|
||||||
|
// produced a verdict.
|
||||||
|
func (q *Queries) WorstHealthForTarget(ctx context.Context, targetID *uuid.UUID) (string, error) {
|
||||||
|
row := q.db.QueryRow(ctx, worstHealthForTarget, targetID)
|
||||||
|
var health string
|
||||||
|
err := row.Scan(&health)
|
||||||
|
return health, err
|
||||||
|
}
|
||||||
|
|||||||
@@ -89,6 +89,7 @@ func (s *Server) CreateApprovalRule(ctx context.Context, req gen.CreateApprovalR
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||||
&id, "POST", "/api/v1/policy/approval-rules", "",
|
&id, "POST", "/api/v1/policy/approval-rules", "",
|
||||||
|
nil,
|
||||||
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
|
map[string]any{"action": req.Body.Action, "risk_class": req.Body.RiskClass}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -148,6 +149,7 @@ func (s *Server) PatchApprovalRule(ctx context.Context, req gen.PatchApprovalRul
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
|
&id, "PATCH", "/api/v1/policy/approval-rules/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"action": req.Body.Action}); auditErr != nil {
|
map[string]any{"action": req.Body.Action}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -154,6 +154,7 @@ func (s *Server) DecideApproval(ctx context.Context, req gen.DecideApprovalReque
|
|||||||
|
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "decide",
|
||||||
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
|
&id, "POST", "/api/v1/approvals/"+req.Id+"/decision", "",
|
||||||
|
nil,
|
||||||
map[string]any{"decision": status}); auditErr != nil {
|
map[string]any{"decision": status}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
20
internal/httpapi/audit.go
Normal file
20
internal/httpapi/audit.go
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
package httpapi
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/audit"
|
||||||
|
)
|
||||||
|
|
||||||
|
// serveAuditDrift returns a read-only DB-side drift report: orphan check
|
||||||
|
// entities, checks on retired targets, probes stuck down/unknown, unmonitored
|
||||||
|
// declared types, and dangling edges. Companion to the knowledge-graph-audit
|
||||||
|
// skill. Live-infra discovery (pct/docker/certs) is a follow-up.
|
||||||
|
func (s *Server) serveAuditDrift(w http.ResponseWriter, req *http.Request) {
|
||||||
|
findings, summary := audit.Report(req.Context(), s.pool)
|
||||||
|
writeJSON(w, map[string]any{
|
||||||
|
"findings": findings,
|
||||||
|
"summary": summary,
|
||||||
|
"note": "read-only DB drift report; live-infra discovery (pct/docker/certs) is a follow-up",
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -81,6 +81,7 @@ func (s *Server) PatchAutonomySettings(ctx context.Context, req gen.PatchAutonom
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
nil, "PATCH", "/api/v1/policy/autonomy", "",
|
nil, "PATCH", "/api/v1/policy/autonomy", "",
|
||||||
|
nil,
|
||||||
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
|
map[string]any{"keys": keysOfMap(*req.Body)}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
|||||||
SELECT cd.entity_id, e.slug, cd.kind,
|
SELECT cd.entity_id, e.slug, cd.kind,
|
||||||
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
COALESCE(te.slug, '') AS target_slug, cd.target_type,
|
||||||
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
cd.config, cd.interval_s, cd.timeout_s, cd.zone, cd.enabled,
|
||||||
e.version
|
e.version, cd.last_health, cd.last_run_at
|
||||||
FROM check_defs cd
|
FROM check_defs cd
|
||||||
JOIN entities e ON e.id = cd.entity_id
|
JOIN entities e ON e.id = cd.entity_id
|
||||||
LEFT JOIN entities te ON te.id = cd.target_id
|
LEFT JOIN entities te ON te.id = cd.target_id
|
||||||
@@ -43,10 +43,19 @@ func (s *Server) ListChecks(ctx context.Context, req gen.ListChecksRequestObject
|
|||||||
var c gen.Check
|
var c gen.Check
|
||||||
var targetSlug string
|
var targetSlug string
|
||||||
var configBytes []byte
|
var configBytes []byte
|
||||||
|
// last_health is what turns a check list from configuration into an
|
||||||
|
// explanation: an entity's health is the worst of these, so this is
|
||||||
|
// the field that says which probe is responsible.
|
||||||
|
var lastHealth *string
|
||||||
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
if err := rows.Scan(&c.Id, &c.Slug, &c.Kind, &targetSlug, &c.TargetType,
|
||||||
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version); err != nil {
|
&configBytes, &c.IntervalS, &c.TimeoutS, &c.Zone, &c.Enabled, &c.Version,
|
||||||
|
&lastHealth, &c.LastRunAt); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
if lastHealth != nil {
|
||||||
|
h := gen.CheckLastHealth(*lastHealth)
|
||||||
|
c.LastHealth = &h
|
||||||
|
}
|
||||||
if targetSlug != "" {
|
if targetSlug != "" {
|
||||||
c.Target = &targetSlug
|
c.Target = &targetSlug
|
||||||
}
|
}
|
||||||
@@ -173,6 +182,7 @@ func (s *Server) CreateCheck(ctx context.Context, req gen.CreateCheckRequestObje
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||||
&id, "POST", "/api/v1/checks", "",
|
&id, "POST", "/api/v1/checks", "",
|
||||||
|
nil,
|
||||||
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
|
map[string]any{"kind": req.Body.Kind, "slug": slug}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -260,6 +270,7 @@ func (s *Server) PatchCheck(ctx context.Context, req gen.PatchCheckRequestObject
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
|
&id, "PATCH", "/api/v1/checks/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
|
map[string]any{"enabled": updated.Enabled}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -285,6 +296,15 @@ func checkDefToGen(cd sqlcgen.CheckDef) gen.Check {
|
|||||||
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
if len(cd.Config) > 0 && json.Unmarshal(cd.Config, &config) == nil && len(config) > 0 {
|
||||||
c.Config = &config
|
c.Config = &config
|
||||||
}
|
}
|
||||||
|
// Carried through so toggling a check does not blank its verdict in the
|
||||||
|
// UI — the entity window renders last_health to explain which probe is
|
||||||
|
// responsible for an entity's health, and a patch response missing it
|
||||||
|
// would erase that until the next poll.
|
||||||
|
c.LastRunAt = cd.LastRunAt
|
||||||
|
if cd.LastHealth != nil {
|
||||||
|
h := gen.CheckLastHealth(*cd.LastHealth)
|
||||||
|
c.LastHealth = &h
|
||||||
|
}
|
||||||
return c
|
return c
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,31 +3,22 @@ package httpapi
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
|
||||||
"github.com/dtoro/oikos/internal/checkdefaults"
|
|
||||||
"github.com/dtoro/oikos/internal/db"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/jackc/pgx/v5"
|
"github.com/jackc/pgx/v5"
|
||||||
)
|
)
|
||||||
|
|
||||||
// ensureDefaultChecks derives an entity's default checks from the monitoring
|
// ensureDefaultChecks derives an entity's default checks from the monitoring
|
||||||
// kinds its type declares.
|
// kinds its type declares. Thin wrapper over the shared db.EnsureEntityChecks
|
||||||
|
// hook so the HTTP create/patch paths and the MCP entity-mutation tools stay
|
||||||
|
// in lockstep.
|
||||||
//
|
//
|
||||||
// Note the ordering caveat: an entity created through the API usually has no
|
// Note the ordering caveat (carried from db.LoadTypeTree / checkdefaults.Ensure):
|
||||||
// edges yet, so a type whose address comes from its host (a service) will
|
// an entity created through the API usually has no edges yet, so a type whose
|
||||||
// produce no checks on this pass. That gap is real and deliberately visible —
|
// address comes from its host (a service) will produce no checks on this pass.
|
||||||
// coverageSweep reports it, and the next inventory ingest fills it in once
|
// That gap is real and deliberately visible — coverageSweep reports it, and
|
||||||
// the hosting edge exists.
|
// the next inventory ingest fills it in once the hosting edge exists.
|
||||||
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
|
func ensureDefaultChecks(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, slug, entityType, name string, attrsJSON []byte) error {
|
||||||
tree, err := db.LoadTypeTree(ctx, tx)
|
_, err := db.EnsureEntityChecks(ctx, tx, entityID, slug, entityType, name, attrsJSON)
|
||||||
if err != nil {
|
return err
|
||||||
return err
|
|
||||||
}
|
|
||||||
res, err := checkdefaults.Ensure(ctx, tx, tree, checkdefaults.Target{
|
|
||||||
ID: entityID, Slug: slug, Type: entityType, Name: name, Attrs: attrsJSON,
|
|
||||||
})
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
checkdefaults.LogResult(slug, entityType, res)
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -67,6 +67,7 @@ func (s *Server) CreateEntityType(ctx context.Context, req gen.CreateEntityTypeR
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||||
nil, "POST", "/api/v1/ontology/entity-types", "",
|
nil, "POST", "/api/v1/ontology/entity-types", "",
|
||||||
|
nil,
|
||||||
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
|
map[string]any{"name": req.Body.Name, "domain": req.Body.Domain}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -148,6 +149,7 @@ func (s *Server) PatchEntityType(ctx context.Context, req gen.PatchEntityTypeReq
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "patch",
|
||||||
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
|
nil, "PATCH", "/api/v1/ontology/entity-types/"+req.Name, "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": req.Body.Status}); auditErr != nil {
|
map[string]any{"status": req.Body.Status}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -240,6 +240,7 @@ func (s *Server) RequestExecution(ctx context.Context, req gen.RequestExecutionR
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||||
&id, "POST", "/api/v1/executions", "",
|
&id, "POST", "/api/v1/executions", "",
|
||||||
|
nil,
|
||||||
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
|
map[string]any{"action": req.Body.Action, "target": req.Body.Target}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -307,6 +308,7 @@ func (s *Server) CancelExecution(ctx context.Context, req gen.CancelExecutionReq
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "cancel",
|
||||||
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
|
&id, "POST", "/api/v1/executions/"+req.Id+"/cancel", "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": "cancelled"}); auditErr != nil {
|
map[string]any{"status": "cancelled"}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -86,6 +86,14 @@ const (
|
|||||||
CheckKindTcp CheckKind = "tcp"
|
CheckKindTcp CheckKind = "tcp"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// Defines values for CheckLastHealth.
|
||||||
|
const (
|
||||||
|
CheckLastHealthDegraded CheckLastHealth = "degraded"
|
||||||
|
CheckLastHealthDown CheckLastHealth = "down"
|
||||||
|
CheckLastHealthHealthy CheckLastHealth = "healthy"
|
||||||
|
CheckLastHealthUnknown CheckLastHealth = "unknown"
|
||||||
|
)
|
||||||
|
|
||||||
// Defines values for CheckCreateKind.
|
// Defines values for CheckCreateKind.
|
||||||
const (
|
const (
|
||||||
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
CheckCreateKindCertExpiry CheckCreateKind = "cert-expiry"
|
||||||
@@ -273,10 +281,10 @@ const (
|
|||||||
|
|
||||||
// Defines values for TrendDirection.
|
// Defines values for TrendDirection.
|
||||||
const (
|
const (
|
||||||
Degrading TrendDirection = "degrading"
|
TrendDirectionDegrading TrendDirection = "degrading"
|
||||||
Improving TrendDirection = "improving"
|
TrendDirectionImproving TrendDirection = "improving"
|
||||||
Stable TrendDirection = "stable"
|
TrendDirectionStable TrendDirection = "stable"
|
||||||
Unknown TrendDirection = "unknown"
|
TrendDirectionUnknown TrendDirection = "unknown"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Defines values for ListApprovalsParamsStatus.
|
// Defines values for ListApprovalsParamsStatus.
|
||||||
@@ -462,7 +470,13 @@ type Check struct {
|
|||||||
Id openapi_types.UUID `json:"id"`
|
Id openapi_types.UUID `json:"id"`
|
||||||
IntervalS int `json:"interval_s"`
|
IntervalS int `json:"interval_s"`
|
||||||
Kind CheckKind `json:"kind"`
|
Kind CheckKind `json:"kind"`
|
||||||
Slug string `json:"slug"`
|
|
||||||
|
// LastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||||
|
LastHealth *CheckLastHealth `json:"last_health"`
|
||||||
|
|
||||||
|
// LastRunAt When this check last executed. Null = never run.
|
||||||
|
LastRunAt *time.Time `json:"last_run_at"`
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
|
||||||
// Target Entity slug (instance-scoped)
|
// Target Entity slug (instance-scoped)
|
||||||
Target *string `json:"target"`
|
Target *string `json:"target"`
|
||||||
@@ -477,6 +491,9 @@ type Check struct {
|
|||||||
// CheckKind defines model for Check.Kind.
|
// CheckKind defines model for Check.Kind.
|
||||||
type CheckKind string
|
type CheckKind string
|
||||||
|
|
||||||
|
// CheckLastHealth This check's own most recent verdict. An entity's health is the worst of these across its enabled checks, so this is what explains *why* an entity is degraded. Null until the check first runs.
|
||||||
|
type CheckLastHealth string
|
||||||
|
|
||||||
// CheckCreate defines model for CheckCreate.
|
// CheckCreate defines model for CheckCreate.
|
||||||
type CheckCreate struct {
|
type CheckCreate struct {
|
||||||
Config *map[string]interface{} `json:"config,omitempty"`
|
Config *map[string]interface{} `json:"config,omitempty"`
|
||||||
@@ -8208,175 +8225,177 @@ func (sh *strictHandler) GetTrends(w http.ResponseWriter, r *http.Request, entit
|
|||||||
// Base64 encoded, gzipped, json marshaled Swagger object
|
// Base64 encoded, gzipped, json marshaled Swagger object
|
||||||
var swaggerSpec = []string{
|
var swaggerSpec = []string{
|
||||||
|
|
||||||
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpCn6MpPsyLU/FFljO7FjraXJt6mRiwK7D0mM0EAPgKbEuFyV",
|
"H4sIAAAAAAAC/+x963IbudXgq6C4WzXUpClq7JlkI9f3Q5E1thM71lqafJuKXBTYfUhihAZ6ADQlxuWq",
|
||||||
"X/sAW3nCPMlXuHY3iSabF1nOVP7YkhqNBs4FOPfzqZfyvOAMmJK9k0+9GeAMhPnx/ApP9f8ZyFSQQhHO",
|
"/NoH2MoT5km2cO1uEk02L7KcqfyxJTUaDZwLcO7nUy/lecEZMCV7p596M8AZCPPjxTWe6v8zkKkghSKc",
|
||||||
"eie9DyB5KVJAcxCScIYmXKA3k8E7rNJZL+nJdAY51u+pRQG9k55UgrBp7/Pnz0mvwALnoNwHzkohuVj9",
|
"9U57H0DyUqSA5iAk4QxNuEBvJoN3WKWzXtKT6QxyrN9TiwJ6pz2pBGHT3ufPn5NegQXOQbkPnJdCcrH6",
|
||||||
"xPsC/1oCSs1jNBE8RxgVAuaElxIJkAVnEr6RiMG9GtlhvaRH9Lu/liAWvaTHcK4/Hh62LyvpnTNF1OJN",
|
"ifcF/qUElJrHaCJ4jjAqBMwJLyUSIAvOJHwjEYMHNbLDekmP6Hd/KUEsekmP4Vx/PDxsX1bSu2CKqMWb",
|
||||||
"trqSn3568xJxgSQtp6gPx9NjdDPjUp3MyrEg8ubIf7bAalZ9lWS9pCfg15IIyHonSpTQZQWXtIwA3D5r",
|
"bHUlP/305iXiAklaTlEfjqfH6HbGpTqdlWNB5O2R/2yB1az6Ksl6SU/ALyURkPVOlSihywquaBkBuH3W",
|
||||||
"LOFOnuQ4HeSEkZsE3dD79CTFWbZoW49+d8sV/Sh4fkX025+igNVYaYB1wkWOVe+kl2EFA6VfTSLzvskg",
|
"WMK9PM1xOsgJI7cJuqUP6WmKs2zRth797pYr+lHw/Jrotz9FAaux0gDrhIscq95pL8MKBkq/mkTmfZNB",
|
||||||
"L7gCli7+DIvV3Z5RAkwNpsBAYAUZuoXFCySgoHgh0R1RM8LQs+9mSIAqBUNqBogLMiUM00AaHgqWmKtF",
|
"XnAFLF38CRaruz2nBJgaTIGBwAoydAeLF0hAQfFConuiZoShZ9/PkABVCobUDBAXZEoYpoE0PBQsMVeL",
|
||||||
"1z4+0F+vrz/H92+BTdWsd/L02f+KLn1iaXwVQ1d4WpEp4QK9Or96gb57+gxxFvgkJzJ3PBJfXMVD2yDq",
|
"rn18oL9eX3+OH94Cm6pZ7/S7Z/8ruvSJpfFVDF3jaUWmhAv06uL6Bfr+u2eIs8AnOZG545H44ioe2gZR",
|
||||||
"LcmJasMSNQ/rE2QwwSVVvZPvnyR6zyQv897Jsyf6N8Lsb0/D7glTMAVhPnTF19GD4ttTw2e9U4sxcx5c",
|
"b0lOVBuWqHlYnyCDCS6p6p3+cJLoPZO8zHunz070b4TZ374LuydMwRSE+dA1X0cPim9PDZ/1Ti3GzHlw",
|
||||||
"AMsIm54WheBzTPWfUs4UMLM/XBSUpFjDfPiL1ID/VPvg/xQw6Z30/sewOs6G9qkchgnNJ5fobYbZFJBU",
|
"CSwjbHpWFILPMdV/SjlTwMz+cFFQkmIN8+HPUgP+U+2D/1PApHfa+x/D6jgb2qdyGCY0n1yitxlmU0BS",
|
||||||
"eArZC4RRDgoPsHsD3WGJUgGGEvtZielAr0hwetT7nPQuBB9TyNcstLAjfrfdgv28kfWeC8EF6n/48Qz9",
|
"4SlkLxBGOSg8wO4NdI8lSgUYSuxnJaYDvSLB6VHvc9K7FHxMIV+z0MKO+M12C/bzRtZ7IQQXqP/hx3P0",
|
||||||
"8N33fzDLuCRThulPhYZ1djCo2Vlja3BfQtKP8Jg3WDydAlOnqSJzogyDF4IXIBSxSMbuychSw6ceME1z",
|
"++9/+J1ZxhWZMkx/KjSss4NBzc4aW4P7EpJ+hMe8weLZFJg6SxWZE2UYvBC8AKGIRTJ2T0aWGj71gGma",
|
||||||
"P/cU53SUYkoNA2DJmaYS/e2UaAbqJb08LUae7kCmmJp99T6ukFbSw3oVI5JFmCbppVwIsC+7IaykFI8p",
|
"+1tPcU5HKabUMACWnGkq0d9OiWagXtLL02Lk6Q5kiqnZV+/jCmklPaxXMSJZhGmSXsqFAPuyG8JKSvGY",
|
||||||
"eI5beSUrhR2fyzXjA78kPTDHdtfpGwutzUJYUaqRLPMci0WnmXiptn1FgpRbgEKWaQpyHRjGnFPATA9W",
|
"gue4lVeyUtjxuVwzPvBL0gNzbHedvrHQ2iyEFaUayTLPsVh0momXattXJEi5BShkmaYg14FhzDkFzPRg",
|
||||||
"/BbYKOWlJcfNcDNkYA+VDmuxQkvHu6c6Vn+2V7SSvRqlJEu0WZEVH/8CqdLfq59Nq3Rt2WuV3OwBMsKq",
|
"xe+AjVJeWnLcDDdDBvZQ6bAWK7R0vHuqY/Vv9opWslejlGSJNiuy4uOfIVX6e/WzaZWuLXutkps9QEZY",
|
||||||
"62It1Wfr39lMs26OcTcygPuCCJBbLdOSTBhblhauy8NuCcvqvA73kJbKMnXBKUkXg9QcxPp3rBQINjDI",
|
"dV2spfps/TubadbNMe5GBvBQEAFyq2Vakgljy9LCdXnYHWFZndfhAdJSWaYuOCXpYpCag1j/jpUCwQYG",
|
||||||
"aGfwAi8oxxGZzYhIGjdcQobs7Cgjk0k7yCr8CiJvRynFlrxXSd9JaKsPFFalrG+xsJeZpipDM5CZs4wR",
|
"Ge0MXuAF5TgisxkRSeOGS8iQnR1lZDJpB1mFX0Hk3Sil2JL3Kuk7CW31gcKqlPUtFvYy01RlaAYyc5Yx",
|
||||||
"84OFtRUT5/wWsugeZWkXtlYm1BJQuK9SzlIQTG4mjxg/ODnRkXIDGg6HYacNcmmQ+Dq2+VBS2Ip1cKk4",
|
"Yn6wsLZi4pzfQRbdoyztwtbKhFoCCvdVylkKgsnN5BHjBycnOlJuQMPhMOy0QS4NEl/HNh9KCluxDi4V",
|
||||||
"4/liRGEOtA5f/aS6Bgw/wBxEFI7uLPY3ThOW7/ACjQHhsVQCpwr1CZuBIEqijN+xoy6M1pELNhFXygsY",
|
"ZzxfjCjMgdbhq59U14DhB5iDiMLRncX+xmnC8h1eoDEgPJZK4FShPmEzEERJlPF7dtSF0TpywSbiSnkB",
|
||||||
"2bWuR7lWuQoQAzsW8TkIQTKQXdbqxNHYdRMjiTgtLKGlmnUT8s8MnTw+CeyLm/XM1AloUVCVGVHnTInF",
|
"I7vW9SjXKlcBYmDHIj4HIUgGsstanTgau25iJBGnhSW0VLNuQv65oZOnJ4F9cbOemToBLQqqMiPqgimx",
|
||||||
"diBKFRddr287eFn6MrdgL+npL2JlVeaFVOCVvKykLZDdRZgChQmtbaWCwGHEphzUjHebwmjKncQeY/YY",
|
"2A5EqeKi6/VtBy9LX+YW7CU9/UWsrMq8kAq8kpeVtAWyuwhToDChta1UEDiM2JSDmvFuUxhNuZPYY8we",
|
||||||
"kaLbaHNMjlKeQUe55wCSTIXZwLhxKrOEeAlK6RlXSO3WauZwj/NCr7k3pXyM6bGm4BFOVexwK61SsJX0",
|
"I1J0G22OyVHKM+go9xxAkqkwGxg3TmWWEK9AKT3jCqndWc0cHnBe6DX3ppSPMT3WFDzCqYodbqVVCraS",
|
||||||
"MMe0jLNj91Pq1ujxdqb159DZDNLb1c2mnE1IxO7yV0yJ1XP0Uetuvwi9arTWybAm/Ha8F/TWxBzTkYxT",
|
"HuaYlnF27H5K3Rk93s60/hw6n0F6t7rZlLMJidhd/oIpsXqOPmrd7RehV43WOhnWhN+O94LemphjOpJx",
|
||||||
"87L0NFOq0POk+t+MyFt9AYNQA3Mla3Bkgkw0lgorgUg5G9itxcWMNqlGYTGFDeJHnzCpMEthYM7IrNOF",
|
"al6WnmZKFXqeVP+bEXmnL2AQamCuZA2OTJCJxlJhJRApZwO7tSgHUyzVaAaYqohx43pGJEo16L6RiN8z",
|
||||||
"aSduuZDd7Poh6ut/t5qZ5MC1/hMH5Rq6Snp/56yL1rFGcnJUUkNofUUVtXQg1LabsiLXdbQYzDxtelmT",
|
"lHOpkIAUmEJzEBlJ1TE6Y8hy7jcS2ZkQkUY0uedCKsQn+hcJCKeCS4n09epAZyeXCZIcKf0xItH9DCsE",
|
||||||
"5sLw3z95knydFLiJhtZTQtjg0+j+PObXY7qO5Fa8XXgb4S5o24SnyLWxjt4/xxappREycTah3SSx1J+k",
|
"DwXFhEn07f1s8S3C/hN6QAZTgTPIjtGfS0pRyRSh5nNmMjQh+qOiZPJYXxAebmZhBj7udf0jv9cndcnu",
|
||||||
"K0PGFEs1EjgjVhsiCvK4ROX+gIXAi7gYcRA9uuMR7JTOkUFTBixddxCwMh9b6Fd2qhhiBaQ8z4EZPT6A",
|
"mP7pY4cr1MBLlMzRehNe/z0DZvdhl6IHIyvvhuX+FzIXlV6gXt9u0n2rLKqwmMIGobFPmFSYpTAwN1vW",
|
||||||
"dV8dVPBSwbIYPLC3ck0UnnHaolQaq11nY88toZ0HV9y6wxEaF5rtbpsWwSVS2ah9Wp/CGWcK7lWE4o0B",
|
"ScyxE7eIUW52/RD19b9bzUxy4FprjTPAmtMg6f2dsy664hp51/F2jQ3rK6p4vMPx0ibfVIfMuhMkGOfa",
|
||||||
"aEIoyJG1QkSsChdYzSTiE2RGI33pidKsGJk3kZphhfzryRaEL4mjtuYHr0gOUuG8MNqeVvIZ3CtUcEqR",
|
"tOnmSRGG//bkJHmCc6MDBW6iofWUEDb4XXR/HvPrMV1HciveLr1ldxe0bcJT5LJfR++fY4vUMiSZOEve",
|
||||||
"hh1IjfBuTCB5IS1pT9t3eCVKQGSCjvXo4wXO9XdSUmjYydrOYjY+TruAzowbfnssQZXFsZztDrPaNb6k",
|
"bvJz6u+/lSFje6LhjFgdlijI43Kw+wMWAi/iwt9BrB8dL05nKhgZNGXA0nUHASvzsYV+ZV2MIVZAyvMc",
|
||||||
"zHPGFWckRalFd3C/OKZNNsmTay9mQ0iXkAqw4vqK2CxXl/SGTYgkKaZImheRHoawsaGSMQWkOFIzIlFq",
|
"mLG+BLDuazkQvFSwrLwMrCxVU2BmnLaYAoyttbOJ7o7QzoMrbt3hCI2rOna3TTvuEqlstBlYT9A5Zwoe",
|
||||||
"Zt8CDquSsIwu+yWWszHHIrusjMJLLOBUdDnyZqvoZWOUMwJyNF6MtIJjyBZnGdFbxfSiMefq60t2Oivm",
|
"VITijdluQijIkbUdRWxBl1jNpBY+zGikLz1RmhUj8yZSWtDwrydbEL4kjtqWpCWSg1Q4L4yOrgUSBg8K",
|
||||||
"GdOw1ECBDI0XyM6btGiG7uP+0j/wt53qtPrpuT4hhNvw0lT6mZ9pXKa3oOxk3w9ywkoFyF/hCcq5VJqp",
|
"FZxSpGEHUrVd+BGFo5CWtKftO7wWJSAyQcd69PEC5/o7KSk07GRtZzHLLKddQGfGDb89lqDK4ljOdodZ",
|
||||||
"9Bv6oqzjuokQO1H3ay4Y2DdQt5vXvxCjlmVeC7baA6HdTxfHfGIAg559N4shYgaYqoh0lcFU4AxarAEZ",
|
"7RpfMsFwxhVnJEWpRXdwmjmmTTZpAWsvZkNIV5AKsErWirIjV5f0hk2IJCmmSJoXkR6GsLF8kzEFpJy0",
|
||||||
"v2uR+O18i/hDqTCNINysj4+lxqnZB6cZ6BMaW2+0EY6+kQjuC0g1LaTYChQxwbNkt6xldUuY80tNqs26",
|
"mprZt4DDqv4io8t+ieVszLHIripT/hILOMOKHHljY/SyMVIzATkaL0ZaLTVki7OM6K1ietmYc/X1Jeuq",
|
||||||
"nVWzxNBpr3uLO5iDcAasXdHHC2CozzgbCJCcziE7cn7AVXz6z62samlrK5wdO2kC8uNbSiJnWJx2G+wc",
|
"FfOMQV9qoECGxgtk501a9Hn3cX/pH/jbTuFd/fRcnxDCbXhpKv3MzzQu0ztQdrIfBjlhpQLkr/CkofDo",
|
||||||
"g9g5E5zSD+6OXaG1GZfK+6easDlNVYkp8gPMlTcDBGY+wqYox+mMsCgD5yBnzrbUnPTSRtvo5+jNhREG",
|
"i7KO6yZC7ETdr7ngFtlA3W5e/0KMWpZ5LVjYD4R2P10c84lVc559P4sholItm+AKWlh8AVodiz7xelz0",
|
||||||
"tIBqzq+5NVFYsalVqdoUTnInTxjcDSguFC+ONtqbzLTr4OZiMGJy1qgQZI4VjG5jsR+nr84Hl+dnH86v",
|
"oVSYRhBu1sfHUuPU7IPTDPQJjVmlN34jtdIJqaaFFFuBIiZ4emVxM+Y6qpwrILPXvcWdVhWd2XFX9PEC",
|
||||||
"Bn8+/9vg+PjYbJdyfXlmkIpF0bZXM3c5piSNT42n8FTPZ8doGjVTX76/uKxJOXHjjLu+R/Z+drJw2x3/",
|
"GOozzgYCJKdzyI6c93YVn/5zK6ta2toKZ8dOmoD8+JaSyBkWp90GO8cgdsEEp/SDu2NXaG3GpfJexSZs",
|
||||||
"EyNagsD0tFQzd6WjNy87zWzlg61nd6/FiMrS28gTzMh4Y9d9wL1RkZiVU5B9cRNpLGEhWUF5HJztoIiT",
|
"zlJVYor8AGeqQGDmI2yKcpzOCIsycA5y5iyCzUmvbIyUfo7eXBphQAuo5vyaW8OSFZtalapNQUD38pTB",
|
||||||
"mYoHFiglyLhUjXOsem0X5bG6blqvAMiQHZWguxkwx/AGdESinDOiuHX0BdtJh4PcXz4fD+d8Moq5uZsc",
|
"/YDiQvHiaKOV0Ey7Dm4uciYmZ40KQeZYweguFrFz9upicHVx/uHievCni78Ojo+PzXYp15dnBqlYFG17",
|
||||||
"CJpbMou3SzNBL40ttikaG9eWY32RMMxSGJVMWZv/blP5I7f1mKus1bUIvV6Lab5jdEGbbWknY/d2jjdn",
|
"NXOXY0rS+NR4Ct/p+ewYTaNm6qv3l1c1KSduUnPX98jez04Wbrvjf2JESxCYnpVq5q509OZlp5mtfLD1",
|
||||||
"bHL3odl9NUeDnBvLaeeYVtdbg2/abOB4irXOa+hbf+AbicKLIxdQFPn0Rqy1Y6e5kpfWgCettgSIkgmk",
|
"7O61GFFZeht5ghkZH/q6D7g3KhKzcgqyL24ijSUsJCsoj4OzHRRxMlPxcBClBBmXqnGOVa/tojy2WTJr",
|
||||||
"i5TqhTjj3mhJdVjF45KuWEplPL9IizPB/wuV+bnbXdhEUjsCLuLRg6cKUTDcxoLIMCFAM4lyt8JCgASm",
|
"VwBkzkqZoHtrr4OacTHnjChu3bPb2A795fPxcC5Do5ibuylqbDSLd/bWeyxRY4s7WxZzrC8ShlkKI2NZ",
|
||||||
"jnvJFrh7B8KEtM1XcNgFcV+Ec+OovjJWpArDVjFAfSUwk0ZorfYUF1daEHDlyKAFhqN6lGR9QX+6fP8X",
|
"3T0EwR+5rcdc5WOoxVX2WhwqHWNC2mxLO7kotnOXOmOTuw/N7qs5GuTcWE47x7Q6TBt80+a5wFOsdV5D",
|
||||||
"dOlBtdF+13g5su2Ma+BGHxE58nQYNwdTvABRN/7loLC9QAW2JqlSaKxM+RyEQZ9R9qaMtEbSBEB3NfO1",
|
"3/oD30gUXhy5MLDIpzdirR07zZW8tAY8abUlQJRMIF2kVC/EGfdGS6rDKh6XdMVSKuOvR1qcCV57qMzP",
|
||||||
"IrTAQl/ent02GxcNTEdrnTKrkTUmMAjM/VkISE3Q48e9Dlx3ujrEeCg30RFW8nEtfW08ZUcrwbwPQTnB",
|
"3e7CJpLaEXAZj/k8U4iC4TYWRIYJAZpJlLsVFgIkMHXcS7bA3TsQJhBxvoLDLoj7IpwbR/W1sSJVGLaK",
|
||||||
"3zHBVEYdQCuUdEgS2plk1h+3cTytR0iLP+Yg+NiVNqNH1NxF7C77jbaPrsAKP2BsRd2KUKMd3kt6d1h4",
|
"AeorgZk0Qmu1p7i40oKAa0cGLTAc1WNb6wv649X7P6MrD6qN9rvGy5FtZ1wDN/qIyJGnw7g5mOIFiLrx",
|
||||||
"E70gSsvzcQ+EUWnjBk7ZXaAKISxB8LOGgWOBiYRsi8AJd3/XjAluiVHSCrGLW/nOavHIm0N0nCmj6/i0",
|
"LweF7QUqsDVJlUJjZcrnIAz6jLI3ZaQ1/ikAuquZrxWhBRb68vbsttm4aGA6WuuUWY2HMuFcYO7PQkBq",
|
||||||
"4dPr/BbXYFN7Bpk+kN9u63jrzo4+gfOItHR5SyhF9inqr8pM9ok7LI5iEpMAaU7c/T18D+ihs4NrN+Nm",
|
"QlU/7nXgutPVIcZDuYmOsJKPa+lr4yk7WgnBfgzKCf6OCaYy6gBaoaRDktDOJLP+uI3jaT1CWvwxB8HH",
|
||||||
"wK6T1MW+1BOJm3WRu83AWRt5tDaQ1tn6zPGj+XiyqP1sB08wsdEXennZiJfGEq5vOGr/Lrj+YTTG6a37",
|
"rrQZPaLmLs562W+0fUwMVvgRI2LqVoQa7fBe0rvHwpvoBVFano97IIxKGzdwyu4CVQg8CoKfNQwcC0wk",
|
||||||
"Tf84cu993MflWVtIRLLbOho3hOFu6wwNx1ergbM6xaqTVYDB9nqOirAElm1XZ53Gl1jRmpONczDnJggc",
|
"ZFuEu7j7u2ZMcEuMklaION3Kd1aLIt8cWOVMGV3Hpw2fXue3uAab2jM0+JH8dltHyXd29AmcR6SlqztC",
|
||||||
"Musi6/PCGq2PesnW4Ur1ZL6Nl4Oba20c3SuBi9lfCdytwhCyKTQDINal2nxwCJQzUsRcMJUdqs1sv6Nx",
|
"KbJPUX9VZrJP3GFxFJOYBEhz4u7v4XtED50dXLsZNwN2naQu9qWeSLSzi7duhjvbeLG14c/O1meOH83H",
|
||||||
"KRKZGXGTkQwNrssnT55DsHU5jdTavAhLaZnB/7Y0acxHzkMN0Zg5xrMtgOPsfRGwKFGy1Oc8xZ3Z+lMo",
|
"k0XtZzt4gomNvtDLy0a8NJZwfcNR+3fB9Q+jMU7v3G/6x5F77+M+Ls/aQiKS3dYx1CF4eltnaDi+Wg2c",
|
||||||
"xYVZ1IyoiP96WcbkNpDaYjCG9tcGBq2+VO+6aGxwyYUQ8HkovK1Y93Y8jlvjsAQ0I8BIrs9in6Gll+wC",
|
"1SlWnawCDLbXc1SEJbBsuzrrNL7EitacbJyDOTeh+5BZF1mfF9ZofdRLtg5Xqqdgbrwc3Fxrox9fCVzM",
|
||||||
"vZT5QMMntaudba2xxcGwiwdTtqHqP27DFrfhMuQdACu/XBTuf2b8jmq+eU0i10pXOzVht5CNoly0MS5E",
|
"/kLgfhWGkE2hGQCxLkHqg0OgnJEi5oKp7FBtZvsdjUuReNqIm4xkaHBTnpw8h2DrchqptXkRltIyg/+y",
|
||||||
"YHbbKXCrXahhpChil8hrMp1RMp1p1Jg8Xh9h0omvbOx451hzRRSFNTuu+DDjaZnbsBFRsjHnt8YaNAep",
|
"NGnMR85DDdFIR8azLYDj7H0RsChRstRnqsWd2fpTKMWFWdSMqIj/elnG5Db83WIwhvbXBgatvlTvumhs",
|
||||||
"yLQte2qzvdkuIIbkt17Vf6mP7FWOqptio4aKrN0WuCW2FYicaClip5eDNTGiDXyaCJ6foE+Kn6BPDlTy",
|
"cMmFEPB5KLytWPcOHQkooBkBRnJ9Fvu8Or1kF+ilzAe2C4PsJv03jS0Ohl08mLINVf9xG7a4DZch7wBY",
|
||||||
"BP3McA7ZwLBqgo6Pjz9+/vx5o3ub+LQpc6+sGKtr64jB+x0oQdJLEA7Ckctm0aZ45ebdlihCSsuiTkkC",
|
"+eWicP8T4/dU881rErlWutqpCbuDbBTloo1xIQKzu06BW+1CDSNFEbtEXpPpjJLpTKPGZF/7CJNOfGUj",
|
||||||
"3/WS3tOZ/qclctBIg+suNjyfdmK/LfJBc3zfacqcsE7jtrEwhPyEpRIY+A5ZWCCfd7Dhs8vCpex0b4VL",
|
"/jtnCCiiKKzZccWHGU/L3IaNiJKNOb8z1qA5SEWmbTlvm+3NdgExJL/1qv5LfWSvclTdFBs1VGTttsAt",
|
||||||
"d514dGUGRUMqFlYhcFQQcF4hMraICxsJu51xoygoAdkei90Mq21C878IlZwhyu9AoDEvWZZoia2wQSQw",
|
"sa1A5ERLETu9HKyJEW3g00Tw/BR9UvwUfXKgkqfobwznkA0Mqybo+Pj44+fPnze6t4lPdjP3yoqxuraO",
|
||||||
"t+/ZFOLh970ISptj4reyVuFKsXbINo7UYCDYS9wqKlivPPu1xAIzRVhbZPgWuaizRcHVDCT5u8098Iv3",
|
"GLzfgRIkvQLhIBy5bBZtildu3m2JIqS0LOqUJPB9L+l9N9P/tEQOGmlw3cWG59NO7LdFFm+OHzpNmRPW",
|
||||||
"Kc9LBktzgYQxH9tTwNdBczd3Z4OSarqvh1SDlFYwX1OL14Vx1so2LN9eS3lptfxBIbiI3BQ/EqDZwGT0",
|
"adw2FoaQVbJUuATfIwsL5LNFNnx2WbiUne6tcOmuE4+uzaBoSMXCKgSOCgLOK0TGFnFpI2G3M24UBSUg",
|
||||||
"1cJxkB2O+t89e3bUHuVn3Hzx47lNc16CnZ0hjO9yqvh8nQ20E0s12CSUBK9DD495qU7GVMtjteCBUpDN",
|
"22Oxm2G1S2kOhErOEOX3INCYlyxLtMRW2CASmNv3bOL38IdeBKXNMfFbWatwpVg7ZBtHajAQ7CVuFRWs",
|
||||||
"mrf5zFp3y4VWPeRaG8Y6p/a7Ny8TlHIBMkEC56N8nKCMyNvRdJwgUiRIQV5QE4yYm5i2BGmxnaQgo0GJ",
|
"V579UmKBmSKsLTJ8iwzi2aLgagaS/N3mHvjF+0T1JYOluUDCmI/tifvroLmbu7NBSTXd10OqQUormK+p",
|
||||||
"XEbkxUtaTr0790Lw+5zfm8gwF3RVi1GI2jLiEWavyxyzgQCc6XMFOX9Ix8gv8116n578ApQuJoRt7yqn",
|
"xevCOGvFNpZvr6VswlrWpxBcRG6KHwnQbGDyMGvhOMgOR/3vnz07ao/yM26++PHcpjkvwc7OEMZ3OVV8",
|
||||||
"92mC5nmCuEAZT29BmHoomLB6aHV3Z7kD3gYctwWUVem43ewHIRowanYKhjH05qWJMhA4vUWFXwZhU/3L",
|
"vs4G2omlGmwSSoLXoYfHvFSnY6rlsVrwQCnIZs3bfGatu+VSqx5yrQ1jnVP73ZuXCUq5AJkggfNRPk5Q",
|
||||||
"VICxv224JqLXcW9pCWu3fRk4cWnT+mSJ1Myag8CU2oMHkUlz4cHwubsFoMVZf1YKASxETbSGYEgFxTrJ",
|
"RuTdaDpOECkSpCAvqAlGzE1MW4K02E5SkNGgRC4j8uIVLafenXsp+EPOH0xkmAu6qsUoRG0Z8Qiz12WO",
|
||||||
"0ax7lIOUeNrNeTwhjMjZ/vbnh7BhhwBUUTLnETOKWcCDvNVqZsvlqqDoYA3Ro9aekmuTBRwzenxZ9MRm",
|
"2UAAzvS5gpw/pGPkl/kufUhPfwZKFxPCtneV04c0QfM8QVygjKd3IEwVG0xYPbS6u7PcAW8DjtsCyqok",
|
||||||
"adgmN5yzGx0flecvfli60nKBXTqbeMNpa2NH1kzQJqKay3tkypptox+QbKT4rrSzjBMLnaQyPruzsra2",
|
"6m72gxANGDU7BcMYevPSRBkInN6hwi+DsKn+ZSrA2N82XBPR67i3tIS1274KnLi0aX2yRCqdzUFgSu3B",
|
||||||
"TSjqFuXVGTHrbebt+Nj4XjeLXxwgm2AQj/NJscgIw3TJdc0ZDBQfcBOW7X7JMdPEo/+rnvnfzMONtvOY",
|
"g8ikufBg+NzdAtDirD8vhQAWoiZaQzCkgmKd5GjWPcpBSjzt5jyeEEbkbH/782PYsEMAqiiZ84gZxSzg",
|
||||||
"5YNpoRT2C7FxlqROpUx622Zeb3w/Ho9RX1PzC0kD6lG8EXl75j2h8aSkUfXJCm3MIawqV+F/tIluIm85",
|
"Qd5pNbPlclVQdLCG6FFrT8m1yQKOGT2+LHpiszRskxvO2Y2Oj8rzFz8sXUHAwC6dTbzhtLWxI2smaBNR",
|
||||||
"XUMSK6Za5WxRrjbhsgU/cfisbiSyjBhwXAm2VVI2hv6OLl+vI3W8VIVUIwnAtvLWTygu1imDM06zUcbv",
|
"zeU9MsXottEPSDZSfFfaWcaJhU5SGZ/dWVlb2yYUdYvy6oyY9TbzdnxsfK+bxS8OkE0wiMf5pFhkhGG6",
|
||||||
"2L6xhNvWm6pCQ6wAP3Cm77hWv/W+KbkFuhiluOzI13mp9o6n5GlqhK71Bo8DhOlsFAUr06ELuMHprfcB",
|
"5LrmDAaKD7gJy3a/5Jhp4tH/Vc/8b+bhRtt5zPLBtFAK+4XYOEtSpwI0vW0zrze+H4/HqK+p+YWkAfUo",
|
||||||
"eOOC+Y7etlVTbc5QJQp93OWSr2LsbeLkIYpe+fJWtRggJxutwHuZTZaoJ8rJt4TSvWxqmwNxTB7tqLIc",
|
"3oi8O/ee0HhS0qj6ZIU25hBWFRnxP9pEN5G3nK4hiRVTrXK2KFebcNmCnzh8VjcSWUYMOK5w3iopG0N/",
|
||||||
"dHyjc324bexjpdxTqF4TMGgz9Um2pcG/EDyFrBQx8dNHPWbICMJDGz8y9AEgQx8XVFDM0Ifngx+OVkKx",
|
"R5ev15E6XqpCqpEEYFt56ycUF+uUwRmn2Sjj92zfWMJtq4RVoSFWgB8403dcq99635TcAV2MUlx25Ou8",
|
||||||
"vd9uFCKX2qq5MH0OentkvuIL37yReihSPO7Crbuzj92Q56XWKGLu0xXVbeepLFwPMVdEG5Ih8CeeCNnd",
|
"VHvHU/I0NULXeoPHAcJ0NoqClenQBdzg9M77ALxxwXxHb9uqqTZnqBKFPu5yyVcx9jZx8hClynxRsloM",
|
||||||
"XpoJPHGxCz6IIdhJBUyMTbamzG2IQ/bGUrE+7aSyZ+9mOl1JEQnG02AUrVig9Yy6dMrnchRqnmMWMZq8",
|
"kJONVuC9zCZL1BPl5DtC6V42tc2BOCaPdlRZDjq+0bmq3zb2sVLuKVSvCRi0mfok29LgXwieQlaKmPjp",
|
||||||
"4sFYZsrQuTi5Xry8pKtluMw4RIU6WTF3esZLPcCYmWRc6OqedyIgXlxN6W2o1hRYihfdS6xopb8ZWy3l",
|
"ox4zZAThoY0fGfoAkKGPCyooZujD88Hvj1ZCsb3fbhQil9pq8DB9Dnp7ZL7iC9+8kXooUjzuwq27s4/d",
|
||||||
"rJf4qjkmH5y1XLptd98WgI4Xvvn9k42lD9y6k4DuGJVceafUEgAZzzFdtEjTREAVVLZDBMmqwMk1x0Xt",
|
"kOeV1ihi7tMV1W3nqSxcDzFXRBuSIfAnngjZ3V6aCTxxsQs+iCHYSQVMjE22psxtiEP2xlKxPu2ksmfv",
|
||||||
"rsQ45ihhgAUqBP/FfjpBf8iQjfjb7Ets95tKymOa01v7uQlRqACBMrzY2ikY3HQVtKKBGRLSUksoJiHF",
|
"ZjpdSREJxtNgFK1YoPWMunLK53IUap5jFjGavOLBWGaKB7o4uV68KKirQLnMOESF6mYxd3rGSz3AmJlk",
|
||||||
"VQsALECclrFcxfdOLRrOCdyBOEF6mJaebtH7Ny/P0J/+66oe70rY4PTiDfrXP/6JznCWLa7ZhIs7LLIB",
|
"XOjqnnciIF4ST+ltqNYUWIoX3UusaKW/GVst5ayX+Ko5Jh+ctVy6bXffFoCOF7757cnG0gdu3UlAd4xK",
|
||||||
"LtUMEZNtBUzCgLBBBoWaJYhxmxfmrDdaQhOlmh0dXzNTDPrEmAVJiuw6bTKpLZheZZ72TZEvdGMCpW/0",
|
"rr1TagmAjOeYLlqkaSKgCirbIYJkVeDkmuOidldiHHOUMMACFYL/bD+doN9lyEb8bfYltvtNJeUxzemt",
|
||||||
"u76guCEm82ZF7YaVTGlqI9O6mtcu+cGVJM8UF3w1qO3MFvAe6LsckN6sL7DyntxyiWY8B4rH6P3lMbrS",
|
"/dyEKFSAQBlebO0UDG66ClrRwAwJaaklFJOQ4qoFABYgzspYruJ7pxYN5wTuQZwiPUxLT3fo/ZuX5+iP",
|
||||||
"4uWEUNAb10O+/TZs8pqZXX77LeqbmuA4VQMjFx6doFfceAxAIKnKsURYAKpK2t8RNUMcF2Sgj70psOSa",
|
"/31dj3clbHB2+Qb96x//ROc4yxY3bMLFPRbZAJe2FFsGE2ASBoQNMijULEGM27wwZ73REpoo1ezo+IaZ",
|
||||||
"2bxXifr+82dv3yRoUmqpBP30Rh5ZeBkw4xyQLCA9vmbX7IyzuUYnZzX55PnRyTUboHPrhdJf9wXD0U1b",
|
"Et6nxixIUmTXaZNJbZn7KvO0b4p8oVsTKH2r3/Vl4A0xmTcrajesZAqKG5nWVSp3yQ+ukHymuOCrQW3n",
|
||||||
"efKbY/3KWyKVRKUEdPPJ3NFJvcvC5xu7eNeaocBTwqzDq+8OGmRKzqPvnyQox/fo2ZMnR2ben5jEE0AX",
|
"tuz6QN/lgPRmfYGV9+SOSzTjOVA8Ru+vjpEpWTchFHwNum+/DZu8YWaX336L+qaSO07VwMiFR6foFTce",
|
||||||
"7y+vbPGTQqGbpXr8N6hvK/sXFC/QHWEZv7NvvyvNoYCEaz4hUYqFWKAbd9vdvECvzq9cTwCJbs6v8PQm",
|
"AxBIqnIsERaAqkYE90TNEMcFGehjbwosuWE271Wivv/8+ds3CZqUWipBP72RRxZeBsw4ByQLSI9v2A07",
|
||||||
"QRenV2evkY/fQDe+xP4N6rvi/L4ov/1MqLlTwez58+c/oJ+uzszzcxeUZJ7iLBMgpVnXuBldivrNLhEG",
|
"52yu0clZTT55fnR6wwbownqh9Nd9mXd021ZU/vZYv/KWSCVRKQHdfjJ3dFLvjfH51i7eNdQo8JQw6/Dq",
|
||||||
"UVczQO/OLmw5kAlOAfWlEoBzM8Prq6uLBPHJhKQEU01Aly//fGRziEtmItEVuhnmaXFzzTirCGFMGBYL",
|
"u4MGmUYB6IeTBOX4AT07OTky8/7EJJ4Aunx/dW2LnxQK3S51UbhFfduPoaB4ge4Jy/i9fftdaQ4FJFzL",
|
||||||
"hFmmB/NSGTuq4SVL15plXZDQC0RMBiWnEt0JXFyzip6sfoxMSg3ChtqlVrOyghOmpOVHSlJwrhjHZBc2",
|
"EIlSLMQC3brb7vYFenVx7To5SHR7cY2ntwm6PLs+f418/Aa69Y0RblHftVTwrRTsZ0LNnQpmz58//z36",
|
||||||
"u1sf14I6xpQnw6HTvI+dJ3nossBrfsSeZbfTizc1qeWk9/T4yfETo+YWwHBBeie958dPjp9bJ/DMnHdD",
|
"6frcPL9wQUnmKc4yAVKadY2b0aWo3+ztYRB1PQP07vzSlgOZ4BRQXyoBODczvL6+vkwQn0xISjDVBHT1",
|
||||||
"c0gMcK3IvLs0rRGIcPYm6530/k8JYtGsR9/sQfJzvJlBrST4mtYLLe82aojvMEE9dGPtyzHJudrcMHTw",
|
"8k9HNoe4ZCYSXaHbYZ4WtzeMs4oQxoRhsUCYZXowL02FRMtLlq41y7ogoRemWqIpw4PuBS5uWEVPVj9G",
|
||||||
"6DDW9XboMNL1bOkw0jam+PxxqcnDsydPtmpRsBRE6NWGTvpDE/URfaTeQGb7qmVmCZE7euXK8UtAwJSJ",
|
"JqUGYemKLALLCk6YkpYfKUnBuWIck13a7G59XAvqGFOeDodO8z52nuShywKv+RF7lt3OLt/UpJbT3nfH",
|
||||||
"4/qcVIJZfAsBZrVmELVIVttlAY1hhufE1MgwZnY8lcambcNMx8SbXe8HfuG2lmbvpGfFATPrMJROaeUk",
|
"J8cnRs0tgOGC9E57z49Pjp9bJ/DMnHdDc0gMcK01gLs0rRGIcPYm6532/ncJYtHsItDsHPO3eAuKWiH3",
|
||||||
"fS2chlGdmChoHRUuD1cbvY15vBVn5ZN7V5z/zfFGaIryeGwRCGp/ftAEinCNQj0vVCWAtmGE4SeSfR6G",
|
"NQ0zWt5tVH7fYYJ66Mbal2OSc7W5Yei70mGs68jRYaTrtNNhpG0n8vnjUmuOZycnWzWWWAoi9GpDJ/2h",
|
||||||
"1iMa1k2K34Dg0FJK47hwASJNlnppujMENCRbfmGpkZKlJRMN80eeLfYgo/qmQ1qrZVPLpYvAmVHVjPGW",
|
"ifqIPlJv+7N91TKzhMgdvXLl+CUgYMrEcX1OKsEsvoUAs1oLj1okq+2NgcYww3NiamQYMzueSmPTtmGm",
|
||||||
"yN+W4jOv352eVQ0MrG7Ql4RNKQxKCQnyeVNOpB5IksHmMkVhG3FKbLZY+rwnI+7afeilWyQSkHKRQXaI",
|
"Y+LNrg8Dv3BbS7N32rPigJl1GEqntHKSvhbOwqhOTBS0jgqXh6to38Y83oqz8sm9+wT86ngjtLJ5OrYI",
|
||||||
"m8HiyoToAFvUYemg2wrQriwTvG6WacrMxvuvkcHMkG6yV72I+S7Sl62E/5CCV5vYx9kub64k8P1H6Nvv",
|
"BLU/P2gCRbhGoZ4XqhJA2zDC8BPJPg9DwxgN6ybFb0BwaASmcVy4AJEmS700PTUCGpItv7DU/srSkomG",
|
||||||
"Yqt6KDzi1aYX4cU91FcSvTy/PDs6BHubmbeX95o8azzI68W9MzukE9OuiF0daT/EdezArL6S+sqrtcy+",
|
"+QPPFnuQUX3TIa3Vsqnl0kXgzKhqxnhL5G9L8ZnX787Oq7YTVjfoS8KmFAalhAT5vCknUg8kyWBzmaKw",
|
||||||
"3xZl2wYGj0fUjiIOJKwZEkQZTAhz6S8VQbsKjxsktjbJysZAWWg9oli1EZUuVquTPPL0sJ+Ootcmjh8A",
|
"jTglNhtjfd6TEXftGfXSLRIJSLnIIDvEzWBxZUJ0gC3qsHTQbQVoV5YJXjfLNGVm4/3XyGBmSDfZq156",
|
||||||
"v3YmhB2O+/a0GWA5yLDCCfJmyj8cdcZ57PgyQvq+srkvD9MkIVM1ZkcKcm1CH5R0bFWbLyzJtlKObz25",
|
"fhfpy/YveEzBq03s42yXN1cS+P4j9O13sVWdL57watOL8OIe6iuJXl5cnR8dgr3NzNvLe02etZXy14p7",
|
||||||
"P+XYmYzsSqxp1RHRroTSqIyy4cJbGtvNyhHqGHxpidNXxl+1dWzVEOA3d0k2+1s84m25RE4HOFbdjKA1",
|
"53ZIJ6ZdEbs60n6I69iBWX0l9ZVXa5l9vy7Ktm0nno6oHUUcSFizTQ0ymBDm0l8qgnYVHjdIbG2SlY2B",
|
||||||
"O6s4atlyBshFE/JSDvwTZNQypAQm9GhXe4jzSg1tCWODmmiyi68TKX2d4iS4uyTCDOEpoFtYFJiIxPXT",
|
"stB6QrFqIypdrFYneeS7w346il6bOH4A/NqZEHY47tvTZoDlIMMKJ8ibKX931BnnsePLCOn7yua+PEyT",
|
||||||
"NX9vLzybGI9GLTm2EfXFG+kNx+gMUwrC1kvEVADOFmiG56C/4YtYMHPtMMj06dLIjjCBXtbB0TwVbEXj",
|
"hEzVmB0pyDV3fVTSsVVtvrAk20o5vmHo/pRjZzKyK7GmVUdEuxJKozLKhgtvaWw3K0eoY/ClJU5fGX/V",
|
||||||
"M1+X/yFO82ax6S98oC9VbI61GzYjcmAqNNe2HkDbxIBlAWEHoG/7MYQRgztf3Phf//gnIlKW4GnI00+N",
|
"1rFVQ4Bf3SXZ7G/xhLflEjkd4Fh1M4LW7KziqGXLGSAXTchLOfBPkFHLkBKY0KNd7SHOKzW0JYwNaqLJ",
|
||||||
"dsISKip3hNtC4rbZXYPCP0laTj8P06pJSDQM44PzMN7NSDpzvUBM/4/EutUs2Zqy0rbfhm9vgUybD0PE",
|
"Lr5OpPR1ipPg7pIIM4SngO5gUWAiEtcF2fy9vfBsYjwateTYRtQXb6Q3HKNzTCkIWy8RUwE4W6AZnkO9",
|
||||||
"UzIHhpT3NRovM0PeAWwafBivHWFSAc4Qn6ApUagoKY0R6StQzQYnK/dWbAuIM7pwi5NhcURW67JNpp8/",
|
"1RMz1w6DTJ8ujewIE+hlHRzNU8FWND73dfkf4zRvFpv+wgf6UsXmWJNoMyIHpkJLdOsBtE0MWBYQdgD6",
|
||||||
"f/7DUUtzfdu5ZOu23x8fUkRpQCJ2Kru2IBlQhQ9As69AOTJI6zM7iOIKnNsSZ7KTVHtJy2nv88cIZcuq",
|
"th9DGDG498WN//WPfyIiZQmehjz91GgnLKGicke4LSRuWxQ2KPyTpOX08zCtmoREwzA+OA/j/YykM9cL",
|
||||||
"a8l0bZHxgQkwMdZB84Z1Jme+8K6d9hu5cmK3E6ZvmPLgePcfiuD9slPvlQOpth5y65q8fFliyHwLmGGt",
|
"xPT/SKxbzZKtKStt+2349hbItPkwRDwlc2BIeV+j8TIz5B3ApsGH7V7GpAKcIT5BU6JQUVIaI9JXoJoN",
|
||||||
"GE5UEn4FaqVfzAMibuVbMSu5H4P84vfH03sGA8FLlg2UIIUJqdOCT4gFcm3+keA8NyFBqMBT2MPHWi9o",
|
"TlburdgWEGd04RYnw+KIrNZlW4M/f/7cBLvF7z6bhblls/aPjymiNCARO5VdW5AMqMIHoNlXoBwZpPWZ",
|
||||||
"06qC+ACTTWf4j4QqEKY+Qr1ZoivEJZEeDSzDTMm2w3tXC3vIGNz2xVC1des3fTnetS8usXs5tg9drR7O",
|
"HURxBc5tiTPZSaq9ouW09/ljhLJl1bVkurbI+MAEmBjroHnDOpMzX3jXTvuNXDmx2wnTN0x5dLz7D0Xw",
|
||||||
"TGDO0AXJxr7y656m938rLam9qNoX0o5cTT5K5MFOXaiYJyg7tXJVOxsSz/319lVaEhvF/b+wKdFTUbst",
|
"ftWp98qBVFsPuXVNXr4sMWS+BcywVgwnKgm/ArXSL+YREbfyrZiV3I9BfvH74+k9g4HgJcsGSpDChNRp",
|
||||||
"0dRKy8BWzji/wtO2Kd2woRnjJjyIDZKhFeVgA1U0LUh+8DCojO1q8JnTbOtNQCq10/W3mgddOeVMav3c",
|
"wSfEAqU2SggJznMTEoQKPIU9fKz1gjatKogPMNl0hv9IqAJh6iPUmyW6QlwS6dHAMsyUbDu8d7Wwh4zB",
|
||||||
"xH3aqhRac05xgVOjAvuI76MEeUOWm926G6vqtCaiJaIzNxVlfQz63QWXe0ynCFUIvm7aX6kD8oXpf7VG",
|
"bV8MVVu3ftOX41374hK7l2P70NXq4cwE5gxdkGzsK7/saXr/t9KS2ouqfSHtyNXko0Qe7NSFinmCslMr",
|
||||||
"RftJ58qyJk2E/FpC+ahsEraAMNKvlSqQbv/t/z1L0F/fJSjU+DhCZqAp2rEvP3njfZsUGkjvAc0fbceX",
|
"V7WzIfHCX29fpSWxUdz/C5sSPRW12xJNrbQMbOWMi2s8bZvSDRuaMW7Cg9ggGVpRDjZQRdOC5AcPg8rY",
|
||||||
"w5mrB/R42HkVahkshxjvdMkdwE2ynPbge3TUTx0sINJ1pNY5pmoXkMHkxTUjlMIU08YkNpYbfffkBy3X",
|
"rgafO8223gSkUjtdf6t50JVTzqTWz03cp61KoTXnFBc4NSqwj/g+SpA3ZLnZrbuxqk5rIloiOnNTUdbH",
|
||||||
"mukG1fOjY3Rho/im+iPXzB6IWitdVK8+R31/ygW4HEXPO729Xc+6B/b31JvHfHH7YBuDOI9Pdbc+Foc4",
|
"oN9dcLnHdIpQheDrpv2VOiBfmP5Xa1S0n3SuLGvSRMgvJZRPyiZhCwgj/VqpAun23/6f8wT95V2CQo2P",
|
||||||
"h1FV3kIzSK1TzFIXmYOcWkPT5XpQdbluO8L+qMd9sMM6eZNMQk1DDwngeW5KIZK8zHsn30dSuR5am1gO",
|
"I2QGmqId+/KTN963SaGB9B7R/NF2fDmcuXpAT4edV6GWwXKI8U6X3AHcJMtpD75HR/3UwQIiXUdqnWOq",
|
||||||
"EixsslFLk9jOVZnaCiXZD2xd1maLCJ3JxNbZ9ai1du2pwMUMZcTVSDuEUdvnjPgPkok1BbmDfYIJlV/0",
|
"dgEZTF7cMEIpTDFtTGJjudH3J7/Xcq2ZblA9PzpGlzaKb6o/csPsgai10kX16nPU96dcgMtR9LzT29v1",
|
||||||
"OF8laB+AJjdfyKHQSjeKFkB3DuarEuGiHNEbc8MsoZKLSe4zJgbz5OPhTc/7qNzri7zvTMf1aQ/hZHxp",
|
"rHtkf0+9ecwXtw+2MYjz+FR361NxiHMYVeUtNIPUOsUsdZE5yKk1NF2uB1WX67Yj7A963Ac7rJM3ySTU",
|
||||||
"gI5EfVrT9zx40voauiggRx49IvFa23YQqYdVLncbFS8XWHvA63P5UxHsXTT9kFC4Y8jt4wDyPac0XsTO",
|
"NPSQAJ7nphQiycu8d/pDJJXrsbWJ5SDBwiYbtTSJ7VyVqa1Qkv3A1mVttojQmUxsnV2PWmvXngpczFBG",
|
||||||
"mDqXhf4vhcqaadp045Xr44fP5649a4cT58HzrqJW0VrZkP8EAD+ybXPufCaPZdqc20zeAwb9viZScWGc",
|
"XI20Qxi1fc6I/yCZWFOQO9gnmFD5RY/zVYL2AWhy84UcCq10o2gBdOdgvioRLsoRvTE3zBIquZjkPmNi",
|
||||||
"3eBZYWdHxNxCy+SetroDL21qwCUwheyGjtE5Tmf2+99IdEOyG58VbXvgC36HSIb6AmSZwzUzB9nNWy0q",
|
"ME8+Ht70vI/Kvb7I+850XJ/2EE7GlwboSNSnNX3Pgyetr6GLAnLk0RMSr7VtB5F6WOVyt1HxcoG1R7w+",
|
||||||
"mxkGb17eHCXoxoxeelcDNUE3GVY4PPnT5fu/XDPzKrLQPkavAQs1Bqz0uZUbOGvOW6Cn38tj9EeQagCT",
|
"lz8Vwd5l0w8JhTuG3D4OIN9zSuNF7Iypc1no/1KorJmmTTdeuT5++GLu2rN2OHEePe8qahWtlQ35TwDw",
|
||||||
"CRfGDUvMk3/945/XzNSVhgwVIAayHOudjkGgcTmZgEhQJngx4DQDqVwS9cXvj16YNOhX51fIweyaKY7G",
|
"E9s2585n8lSmzbnN5D1g0O9rIhUXxtkNnhV2dkTMLbRM7mmrO/DKpgZcAVPIbugYXeB0Zr//jUS3JLv1",
|
||||||
"OL2dkLgr/tLAtO2wanXhBAigQsCE3O/rsbFKVvViAwVrZ9jMtgrulQXHoKKg9glX/bCX58i9eAiz/9wT",
|
"WdG2B77g94hkqC9AljncMHOQ3b7VorKZYfDm5e1Rgm7N6KV3NVATdJthhcOTP169//MNM68iC+1j9Bqw",
|
||||||
"kJ0T9S8vz4/2YY4qOmqtn64atmsu5INHyH8lGSn/XldHyBJ9xOujoq1DOcbq1Lp1HGDS4uy4mgGaYZZR",
|
"UGPASp9buYGz5rwF+u4HeYz+AFINYDLhwrhhiXnyr3/884aZutKQoQLEQJZjvdMxCDQuJxMQCcoELwac",
|
||||||
"EMveiX6I4TM0eJTYGF7p/BRDX/wwuWaYZQiImoFAwIw13F0LoRpz34azukThI8RFLYLwmoXMQWd7Mz4Q",
|
"ZiCVS6K+/O3RC5MG/eriGjmY3TDF0RindxMSd8VfGZi2HVatLpwAAVQImJCHfT02VsmqXmygYO0Mm9lW",
|
||||||
"XwiiORNh6Ma3l7sJUX+nVHIE9+avPsjFRvQITsEEoNlwLDvd+7+8/Ru6wws7Ruotxq4C55E4r6cdf5Xu",
|
"wYOy4BhUFNQ+4aof9uoCuRcPYfafewKyc6L+1dXF0T7MUUVHrfXTVcN2zYV89Aj5ryQj5d/r6ghZok94",
|
||||||
"w+V2cF/ahVhx3BpW8N4T1M9tiVKXQR6cWIcQsj4EAqpTnyPtBfrX//v/VZqqTWzQf3JUu1WIbS3+sBq7",
|
"fVS0dSjHWJ1at44DTFqcHdczQDPMMgpi2TvRDzF8hgaPEhvDK52fYuiLHyY3DLMMAVEzEAiYsYa7ayFU",
|
||||||
"2SFSo6WHM/l2wsch7GIBxK5t3O+Q66C52xm1lz2hiYShbQn5IFnfZ2bqx0flWeh6eQBXu5kLYeQP12Ho",
|
"Y+7bcFaXKHyEuKhFEN6wkDnobG/GB+ILQTRnIgzd+vZytyHq74xKjuDB/NUHudiIHsEpmAA0G45lp3v/",
|
||||||
"xYnqdRd2zC/WZ7NoTzA+N48vAbK9jTlLooMprMRtrFwTen87ffcW1VpvrdZoZYpTPt3lVXtHbv3ikrwR",
|
"57d/Rfd4YcdIvcXYVeA8Ehf1tOOv0n243A7uS7sQK45bwwree4L6uS1R6jLIgxPrEELWh0BAdepzpL1A",
|
||||||
"FpDU9hEm7yKHaIiicakv+IMcrj4hAEk9sd6NtDWtUtdC4OUffaf/lx/QELmSQD4Sr5EqtpAK8k7EY+z5",
|
"//q//69KU7WJDfpPjmq3CrGtxR9WYzc7RGq09Hgm3074OIRdLIDYtY37DXIdNHc7o/ayJzSRMLQtIR8l",
|
||||||
"605V08Rzk7J2qbAInth+3Q979ALxnChjTLubaYHBehD6toVRW/Sd4HwnoX6Ng+jZBgdRYmqUUlNo0Uqs",
|
"6/vcTP30qDwPXS8P4Go3cyGM/OE6DL04Ub3uwo75xfpsFu0Jxhfm8RVAtrcxZ0l0MIWVuI2Va0Lvr2fv",
|
||||||
"ne313WuTSrUwtZ0mXOS91bC8pU6hv3DCvB9k5P6mxbeCFyU1El5osXrs+j0mXTbhPhPfQ6jJuNw5oeuu",
|
"3qJa663VGq1Mccqnu7xq78itX1ySN8ICkto+wuRd5BANUTQu9QV/kMPVJwQgqSfWu5G2plXqWgi8/IPv",
|
||||||
"HjJ+vWopG+FI8xDNzdO9GfKyHFtK1ZQ7J7LElPzd1XIzPVDR75DpgbqDeV8zXtXjtI3zfqQA6rVH64OB",
|
"9P/yAxoiVxLIR+I1UsUWUkHeiXiMPX/dqWqaeG5S1q4UFsET26/7YY9eIJ4TZYxp9zMtMFgPQt+2MGqL",
|
||||||
"tNmuNQJWO+CAscVmY65Vrp/WmvVNRzdEWKa3wsU+ZrxQaHsoAYu0HdKX5nHozdnNYPFrb1kJ2M8K8BXo",
|
"vhOc7yTUr3EQPdvgIEpMjVJqCi1aibWzvb57bVKpFqa204SLvLcalrfUKfRnTpj3g4zc37T4VvCipEbC",
|
||||||
"9o3upIfzv70m6hCK+o8lpQOTP2LRaYu8BiRXXuq+FwFkglzHzwaLhle2oqFPwfvRISarTku/NXS+Ne1n",
|
"Cy1Wj12/x6TLJtxn4nsINRmXOyd03dVjxq9XLWUjHGkeorl5ujdDXpVjS6macudElpiSv7tabqYHKvoN",
|
||||||
"K8AfomIHpUFuk0OPM2Qb3SLFozGqXdEY52bTaDbq2erO1EYzskVpN3jn3rlBnU6Wzm62Dvd8KJl7EFEl",
|
"Mj1QdzDva8arepy2cd6PFEC99mh9NJA227VGwGoHHDC22GzMtcr101qzvunohgjL9Fa42MeMFwptDyVg",
|
||||||
"Lqf5lpmR+ANcKl6LP2g2ULUtPnZLuN7FifeYR2ujR+3heNFOiyQcqGaiIVaTkTewc6I8UO6u13Bd0Ws7",
|
"kbZD+so8Dr05uxksfuktKwH7WQG+At2+0Z30cP6310QdQlH/saR0YPJHLDptkdeA5MpL3fcigEyQ6/jZ",
|
||||||
"Nd9XutgBUeQYSUNx27wP0yApwhIhWK77hI120LGO3LU4ki3XutLSaRNlNSAS/XRji11IMKDuMDc88uRi",
|
"YNHwylY09Cl4PzrEZNVp6deGzrem/WwF+ENU7KA0yG1y6HGGbKNbpHg0RrUrGuPcbBrNRj1b3ZnaaEa2",
|
||||||
"zMq13C19mdeW6/9WW22NRP2aulOnjZhZDAIC4ib9ekgtwtKlMI/yUjnFIBjeNefgQTCH3s2AoSrCdsUa",
|
"KO0G79w7N6jTydLZzdbhng8lcw8iqsTlNN8yMxJ/gEvFa/EHzQaqtsXHbgnXuzjxnvJobfSoPRwv2mmR",
|
||||||
"Xk+kubLa5VecTKNX+JgJNZbY1xboefbkWQc6tEbyeqXPvY22SiswagYVJRvFxubs1wi6O7027TVRih1+",
|
"hAPVTDTEajLyBnZOlAfK3fUarit6bafm+0oXOyCKHCNpKG6b92EaJEVYIgTLdZ+w0Q461pG7Fkey5VpX",
|
||||||
"0tdxrNRPRNpxKX5bCDqt4e2vschQBhSUKQDPuEKyLAouTBX3makL77rpSgT3RNp6BaEfSEjhtzEFL59H",
|
"WjptoqwGRKKfbmyxCwkG1B3mhkeeXIxZuZa7pS/z2nL932qrrZGoX1N36rQRM4tBQEDcpF8PqUVYuhTm",
|
||||||
"WKMWer4bZ3yR8HO9tEcMQW/jiFrhoUfiiFrBooD1KlRyH05wVYnXByJc+EHbyN57FNfcMaKgW/zDbymS",
|
"UV4qpxgEw7vmHDwI5tD7GTBURdiuWMPriTTXVrv8ipNp9AqfMqHGEvvaAj3PTp51oENrJK9X+tzbaKu0",
|
||||||
"wLfRf7w4gkAaB4oiKCpS8/RMwXWS2yyJ+LcPWrMt3ukESTwBtUBzTOfgjt7LV384OkanocC3Ps6LurSz",
|
"AqNmUFGyUWxszn6NoLvTa9NeE6XY4Sd9HcdK/USkHZfit4Wg0xre/hqLDGVAQZkC8IwrJMui4MJUcZ+Z",
|
||||||
"Iupcftd2WF+EXvRf/qRukmRrqeVfSywwU6ZRVbQjz2rPq6rhf63ZVa2zlfEjhTExtfZxyywHhvsab4mr",
|
"uvCum65E8ECkrVcQ+oGEFH4bU/DyeYQ1aqHnu3HGFwk/10t7whD0No6oFR56Io6oFSwKWK9CJffhBFeV",
|
||||||
"KiEJe0ZCfVdvHtAQVbBFw+omOerOa0t3h4uzCclvJYVuxf0/lBRk7yuoS68Xcsg0CbOvA9eZR6JsqmaV",
|
"eH0gwqUftI3svUdxzR0jCrrFP/yaIgl8G/2niyMIpHGgKIKiIjVPzxRcJ7nNkoh/+6A12+KdTpDEE1AL",
|
||||||
"e3WHgKqzoFw1GhSEr70wcrf+JHLNIW0RJzxRIFbzuv3J97xVH2uA+ivVyOpr3EYnexQ2v7DxD04balAJ",
|
"NMd0Du7ovXr1u6NjdBYKfOvjvKhLOyuiztX3bYf1ZehF/+VP6iZJtpZa/qXEAjNlGlVFO/Ks9ryqGv7X",
|
||||||
"6mclpoOIM3stzXRg64eugroflTywdvJvSh6GIhx7H5IwXFzlOmPkqRtzCUoRNn3cs765lgMe92F3hyi4",
|
"ml3VOlsZP1IYE1Nrn7bMcmC4r/GWuK4SkrBnJNR39eYBDVEFWzSsbpKj7ry2dHe4OJuQ/FZS6Fbc/0NJ",
|
||||||
"bheJpJsT9W8JpQN5R1Q6SxCDOYiBr7lqKtoc7XAlxGXaD5hIE+foF0EkqtMLhQz1nz15hn5XhUIeo7f8",
|
"Qfa+grr0eiGHTJMw+zpwnXkkyqZqVrlXdwioOg/KVaNBQfjaCyN3608i1xzSFnHCEwViNa/bn3zPW/Wx",
|
||||||
"DkzxI6Js+oNbOrqZUj7G9FhPN8KpOkHXPT6ZXPdutAaLMxtTabc08oPQLbgsCn/tkDyHjGAFdKG//uTo",
|
"Bqi/Uo2svsZtdLInYfNLG//gtKEGlaB+VmI6iDiz19JMB7Z+7Cqo+1HJI2sn/6bkYSjCsfchCcPFVa4z",
|
||||||
"xFxNNbDYUpxmHnSHXXwMZuuLjpjTJkaeu50bejv6EaYXDRpt8ww9nNz6dfLIqcGmTdhRgthQ7UcUksPx",
|
"Rp65MVegFGHTpz3rm2s54HEfdneIgut2kUi6OVH/jlA6kPdEpbMEMZiDGPiaq6aizdEOV0Jcpv2AiTRx",
|
||||||
"6GndV6dsnJAvGmT2/scfNUsEgtzv/BRE3g5MwO8GafkDkbdnbtxjphT7ZRxSUCbyFnkYHEheFvU5tzwa",
|
"jn4RRKI6vVDIUP/ZyTP0myoU8hi95fdgih8RZdMf3NLR7ZTyMabHeroRTtUpuunxyeSmd6s1WJzZmEq7",
|
||||||
"NXoaycj2kKRgVd/lgr1ZI826W3KOaSm5bcDL2mSdvWeqheFtZcheIsXvYhVsa24mYIfpOXTOMi3V1Kfu",
|
"pZEfhO7AZVH4a4fkOWQEK6AL/fWTo1NzNdXAYktxmnnQPXbxMZitLzpiTpsYee52bujt6EeYXjZotM0z",
|
||||||
"S1DSloEZKW51FxPJQiS6hcLeCDOT17g42qUsR5sade5aVlofWqQQzapfEPVnBAQW6WwxwHdYwNELlGKR",
|
"9Hhy69fJI2cGmzZhRwliQ7WfUEgOx6OndV+dsnFCvmiQ2fsff9QsEQhyv/NTEHk3MAG/G6TlD0Tenbtx",
|
||||||
"EYapbds34SKFrE2RWk9zX4ciVV/j4zi3mgUQvkj/iQZFuoilneq/+J4D6y6FSzemc0YgHCzTPPTUZhPe",
|
"T5lS7JdxSEGZyDvkYXAgeVnU59zyaNToaSQj20OSglV9lwv2Zo00627JOaal5LYBL2uTdfaeqRaGt5Uh",
|
||||||
"S3p3zlSU9FJBFEmjvcYfJA++SyOg35KZ3+L8Ea38nugOVbk40PB2rXhqPGJTaHB6+yD5M6fprYN5HOvr",
|
"e4kUv49VsK25mYAdpufQBcu0VFOfui9BSVsGZqS41V1MJAuR6A4KeyPMTF7j4miXshxtatSFa1lpfWiR",
|
||||||
"d25fPVy7ktO0itDEDng7dippQC8vrXRzcPC9KxXU4HcIJ4Re66hkitCuFeBbe0Qud8SvZt6jieOXpQgN",
|
"QjSrfkHUnxEQWKSzxQDfYwFHL1CKRUYYprZt34SLFLI2RWo9zX0dilR9jU/j3GoWQPgi/ScaFOkilnaq",
|
||||||
"4EAKrsTK1dXbQxCFAMnp/GHo4oOd+8Ck0Y7mFWR+FchzUKjwl2NWYkoXu6JPq6obhAY7pFtrTGt+GZmo",
|
"/+J7Dqy7FK7cmM4ZgXCwTPPQU5tNeC/p3TtTUdJLBVEkjfYaf5Q8+C6NgH5NZn6L8ye08nuiO1Tl4kDD",
|
||||||
"0/947x/yWtdYecxb3VLFoS51Mxvqm5QqFRLrChD20dFuLn077UO7HywqvmJfe0EYg2zkoBqvibjqbteY",
|
"27XiqfGITaHB6d2j5M+cpXcO5nGsr9+5ffVw7UrO0ipCEzvg7dippAG9vLTSzcHB965UUIPfIZwQeq2j",
|
||||||
"aPW1f33edccQX71v3dCkafBE9K8eKTu60WsUPnRTdTjN/+pHfn0H2M4HUtjT/vhyU3nbj6ktaPG2/TG0",
|
"kilCu1aAb+0RudwRv5p5jyaOX5YiNIADKbgSK9fXbw9BFAIkp/PHoYsPdu4Dk0Y7mleQ+VUgz0Ghwl+O",
|
||||||
"Z96/SceTXZOfrszorY+if7dkDrPNA5KOA9shGB1YhjDDdCGJK19Iqc/hMJXJI4lU2yR0PGQyld4KpKUx",
|
"WYkpXeyKPq2qbhAa7JBurTGt+WVkok7/471/zGtdY+Upb3VLFYe61M1sqG9SqlRIrCtA2EdHu7n07bSP",
|
||||||
"3eipx4AFiNNSzXonP3/UGLfd2O2HS0F7J70hLshw/tTQg9vPatsml9zv8s5DXoEpOWtK4dRt581t2LSa",
|
"7X6wqPiKfe0FYQyykYNqvCbiqrtdY6LV1/71edcdQ3z1vnVDk6bBE9G/eqTs6EavUfjQTdXhNP+LH/n1",
|
||||||
"lTgU23kNQuu3pOptRaSts0w4S3ybo1rhKNfLaHXO8+1SHdx8vMq++BS3e5gtuuJCfYtq4wNqdOeMLijU",
|
"HWA7H0hhT/vjy03lbT+mtqDF2/bH0J55/yYdT3ZNfro2o7c+iv7dkjnMNg9IOg5sh2B0YBnCDNOFJK58",
|
||||||
"oKh6K7hOjUnwUkrUzyAlGQxxqmrTQr1E06eWuEuztCB66fOsNkM431bfrztgkqVQoyQ4x6qpnBdldaKQ",
|
"IaU+h8NUJo8kUm2T0PGYyVR6K5CWxnSjpx4DFiDOSjXrnf7to8a47cZuP1wK2jvtDXFBhvPvDD24/ay2",
|
||||||
"IelIw+UJV7a6Wo7jp2jqlUxsxrL5bkZU4qoPJihk43tMNbgsBu6CC7X6nqvk8Pnj5/8OAAD//2XM3suJ",
|
"bXLJ/S7vPOQVmJKzphRO3Xbe3IZNq1mJQ7Gd1yC0fkuq3lZE2jrLhLPEtzmqFY5yvYxW57zYLtXBzcer",
|
||||||
"7gAA",
|
"7ItPcbuH2aIrLtS3qDY+oEZ3zuiCQg2KqreC69SYBC+lRP0MUpLBEKeqNi3USzR9aom7NEsLopc+z2oz",
|
||||||
|
"hPNt9f26AyZZCjVKgnOsmsp5UVYnChmSjjRcnnBlq6vlOH6Kpl7JxGYsm+9mRCWu+mCCQja+x1SDy2Lg",
|
||||||
|
"LrhQq++5Sg6fP37+/wEAAP//VvsxiT/wAAA=",
|
||||||
}
|
}
|
||||||
|
|
||||||
// GetSwagger returns the content of the embedded swagger specification file
|
// GetSwagger returns the content of the embedded swagger specification file
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"crypto/rand"
|
"crypto/rand"
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"math/big"
|
"math/big"
|
||||||
"strconv"
|
"strconv"
|
||||||
@@ -24,7 +25,12 @@ import (
|
|||||||
const (
|
const (
|
||||||
defaultLimit = 50
|
defaultLimit = 50
|
||||||
maxLimit = 200
|
maxLimit = 200
|
||||||
graphNodeCap = 500
|
// graphNodeCap bounds the whole-graph view. The cognition transactional
|
||||||
|
// types (execution, task) are audit records, not topology, and previously
|
||||||
|
// crowded out every host/lxc/service; the default whole-graph view below
|
||||||
|
// excludes them so the cap is spent on the actual fleet graph. Operators
|
||||||
|
// still reach executions/tasks via list_entities.
|
||||||
|
graphNodeCap = 2000
|
||||||
)
|
)
|
||||||
|
|
||||||
// actorInfo returns the caller's (type, label) from the request context,
|
// actorInfo returns the caller's (type, label) from the request context,
|
||||||
@@ -307,14 +313,19 @@ func (s *Server) GetGraph(ctx context.Context, req gen.GetGraphRequestObject) (g
|
|||||||
// alphabetically. Without this the cap fills with exec:* rows and
|
// alphabetically. Without this the cap fills with exec:* rows and
|
||||||
// drops every host/lxc/service/vm — and every edge those entities
|
// drops every host/lxc/service/vm — and every edge those entities
|
||||||
// connect — because edges require both endpoints in the node set.
|
// connect — because edges require both endpoints in the node set.
|
||||||
|
// Exclude the cognition transactional types (execution/task): they
|
||||||
|
// are audit records rather than topology, and at ~380 rows they
|
||||||
|
// consumed most of the old 500-node cap.
|
||||||
nodes, err = s.queryEntities(ctx, `
|
nodes, err = s.queryEntities(ctx, `
|
||||||
SELECT `+entityCols+`
|
SELECT `+entityCols+`
|
||||||
FROM entities e
|
FROM entities e
|
||||||
LEFT JOIN entity_status st ON st.entity_id = e.id
|
LEFT JOIN entity_status st ON st.entity_id = e.id
|
||||||
WHERE e.id IN (
|
WHERE e.type NOT IN ('execution','task')
|
||||||
|
AND e.id IN (
|
||||||
SELECT e2.id FROM entities e2
|
SELECT e2.id FROM entities e2
|
||||||
LEFT JOIN relationships r ON r.valid_to IS NULL
|
LEFT JOIN relationships r ON r.valid_to IS NULL
|
||||||
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
AND (r.source_id = e2.id OR r.target_id = e2.id)
|
||||||
|
WHERE e2.type NOT IN ('execution','task')
|
||||||
GROUP BY e2.id
|
GROUP BY e2.id
|
||||||
ORDER BY count(r.type) DESC, e2.slug
|
ORDER BY count(r.type) DESC, e2.slug
|
||||||
LIMIT $1
|
LIMIT $1
|
||||||
@@ -807,7 +818,7 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
|||||||
|
|
||||||
rows, err := s.pool.Query(ctx, `
|
rows, err := s.pool.Query(ctx, `
|
||||||
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
|
SELECT id, ts, actor_type, actor_id::text, action, entity_id::text,
|
||||||
method, path, status_code, detail, source_ip, correlation_id
|
method, path, status_code, detail, source_ip, correlation_id, session_id::text
|
||||||
FROM audit_log
|
FROM audit_log
|
||||||
WHERE ($1::text IS NULL OR actor_type = $1)
|
WHERE ($1::text IS NULL OR actor_type = $1)
|
||||||
AND ($2::text IS NULL OR actor_id::text = $2)
|
AND ($2::text IS NULL OR actor_id::text = $2)
|
||||||
@@ -828,10 +839,10 @@ func (s *Server) QueryAudit(ctx context.Context, req gen.QueryAuditRequestObject
|
|||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var a gen.AuditEntry
|
var a gen.AuditEntry
|
||||||
var detailBytes []byte
|
var detailBytes []byte
|
||||||
var actID, entID, method, path, sourceIP, corrID *string
|
var actID, entID, method, path, sourceIP, corrID, sessionID *string
|
||||||
var statusCode *int
|
var statusCode *int
|
||||||
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
|
if err := rows.Scan(&a.Id, &a.Ts, &a.ActorType, &actID, &a.Action, &entID,
|
||||||
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID); err != nil {
|
&method, &path, &statusCode, &detailBytes, &sourceIP, &corrID, &sessionID); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
a.ActorId = actID
|
a.ActorId = actID
|
||||||
@@ -989,6 +1000,7 @@ func (s *Server) CreateEntity(ctx context.Context, req gen.CreateEntityRequestOb
|
|||||||
entityID := inserted.ID
|
entityID := inserted.ID
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "create",
|
||||||
&entityID, "POST", "/api/v1/entities", "",
|
&entityID, "POST", "/api/v1/entities", "",
|
||||||
|
nil,
|
||||||
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
|
map[string]any{"type": req.Body.Type, "slug": slug}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -1052,49 +1064,15 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
|||||||
|
|
||||||
// Validate lifecycle transition if state is being changed.
|
// Validate lifecycle transition if state is being changed.
|
||||||
if req.Body.State != nil && *req.Body.State != "" {
|
if req.Body.State != nil && *req.Body.State != "" {
|
||||||
// Get lifecycle def for the entity's type.
|
fromState := ""
|
||||||
lc, err := sqlcgen.New(tx).GetLifecycleForType(ctx, current.Type)
|
if current.State != nil {
|
||||||
if err != nil {
|
fromState = *current.State
|
||||||
if err == pgx.ErrNoRows {
|
}
|
||||||
// No lifecycle defined — any state is allowed.
|
if err := db.ValidateTransition(ctx, tx, id, current.Type, fromState, *req.Body.State); err != nil {
|
||||||
} else {
|
if errors.Is(err, db.ErrTransitionInvalid) {
|
||||||
return nil, err
|
return nil, fmt.Errorf("%w: %v", domain.ErrInvalidTransition, err)
|
||||||
}
|
|
||||||
} else {
|
|
||||||
var transitions map[string]map[string]json.RawMessage
|
|
||||||
if err := json.Unmarshal(lc.Transitions, &transitions); err != nil {
|
|
||||||
return nil, fmt.Errorf("parse lifecycle transitions: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
fromState := ""
|
|
||||||
if current.State != nil {
|
|
||||||
fromState = *current.State
|
|
||||||
}
|
|
||||||
toState := *req.Body.State
|
|
||||||
|
|
||||||
if toState != fromState {
|
|
||||||
tos, ok := transitions[fromState]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("%w: no transitions from %q", domain.ErrInvalidTransition, fromState)
|
|
||||||
}
|
|
||||||
trans, ok := tos[toState]
|
|
||||||
if !ok {
|
|
||||||
return nil, fmt.Errorf("%w: %s → %s", domain.ErrInvalidTransition, fromState, toState)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Parse preconditions: {"requires": ["check-name", ...]}
|
|
||||||
var gate struct {
|
|
||||||
Requires []string `json:"requires"`
|
|
||||||
}
|
|
||||||
if err := json.Unmarshal(trans, &gate); err == nil && len(gate.Requires) > 0 {
|
|
||||||
for _, check := range gate.Requires {
|
|
||||||
if err := checkPrecondition(ctx, tx, id, current.Type, check); err != nil {
|
|
||||||
return nil, fmt.Errorf("%w: precondition %q not met: %v",
|
|
||||||
domain.ErrInvalidTransition, check, err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
return nil, err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1133,6 +1111,7 @@ func (s *Server) PatchEntity(ctx context.Context, req gen.PatchEntityRequestObje
|
|||||||
patchActorType, patchActor := actorInfo(ctx)
|
patchActorType, patchActor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
|
if auditErr := observability.Audit(ctx, q, patchActorType, patchActor, "patch",
|
||||||
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
|
&id, "PATCH", "/api/v1/entities/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"version": expectedVersion}); auditErr != nil {
|
map[string]any{"version": expectedVersion}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -1262,6 +1241,7 @@ func (s *Server) EnrollClient(ctx context.Context, req gen.EnrollClientRequestOb
|
|||||||
entityID := id
|
entityID := id
|
||||||
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
|
_ = observability.Audit(ctx, q, "operator", actor, "enroll",
|
||||||
&entityID, "POST", "/api/v1/clients/enroll", "",
|
&entityID, "POST", "/api/v1/clients/enroll", "",
|
||||||
|
nil,
|
||||||
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
|
map[string]any{"slug": req.Body.Slug, "mesh_ip": meshIP})
|
||||||
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
|
_ = observability.Event(ctx, q, "client.enrolled", &entityID,
|
||||||
"info", "oikos-api", "",
|
"info", "oikos-api", "",
|
||||||
@@ -1456,6 +1436,7 @@ func (s *Server) ProvisionEntity(ctx context.Context, req gen.ProvisionEntityReq
|
|||||||
_, actor := actorInfo(ctx)
|
_, actor := actorInfo(ctx)
|
||||||
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
_ = observability.Audit(ctx, q, "operator", actor, "provision",
|
||||||
&entityID, "POST", "/api/v1/entities/provision", "",
|
&entityID, "POST", "/api/v1/entities/provision", "",
|
||||||
|
nil,
|
||||||
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
|
map[string]any{"slug": req.Body.Slug, "host": hostSlug})
|
||||||
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
|
_ = observability.Event(ctx, q, "entity.provisioned", &entityID,
|
||||||
"info", "oikos-api", "",
|
"info", "oikos-api", "",
|
||||||
@@ -1546,97 +1527,5 @@ func generateAgeKeypair() (pubKey, privKey string, err error) {
|
|||||||
return pub, priv, nil
|
return pub, priv, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkPrecondition validates a named lifecycle transition precondition.
|
|
||||||
func checkPrecondition(ctx context.Context, tx pgx.Tx, entityID uuid.UUID, entityType, check string) error {
|
|
||||||
switch check {
|
|
||||||
case "no-inbound-edges":
|
|
||||||
var count int
|
|
||||||
err := tx.QueryRow(ctx,
|
|
||||||
"SELECT count(*) FROM relationships WHERE target_id = $1 AND valid_to IS NULL", entityID).Scan(&count)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if count > 0 {
|
|
||||||
return fmt.Errorf("%d inbound relationship edges remaining", count)
|
|
||||||
}
|
|
||||||
case "backups-verified":
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "backups_verified") {
|
|
||||||
return fmt.Errorf("backup verification not recorded in entity attributes")
|
|
||||||
}
|
|
||||||
case "secrets-revoked":
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "secrets_revoked") {
|
|
||||||
return fmt.Errorf("secret revocation not recorded in entity attributes")
|
|
||||||
}
|
|
||||||
case "ingress-dns-removed":
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "ingress_dns_removed") {
|
|
||||||
return fmt.Errorf("ingress/DNS removal not recorded in entity attributes")
|
|
||||||
}
|
|
||||||
case "age-key-enrolled-if-needed":
|
|
||||||
if entityType == "workstation" {
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "age_pubkey") {
|
|
||||||
return fmt.Errorf("age key not enrolled (no age_pubkey in attributes)")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
case "mesh-joined-if-needed":
|
|
||||||
if entityType == "workstation" {
|
|
||||||
var attrs string
|
|
||||||
err := tx.QueryRow(ctx, "SELECT coalesce(attributes::text,'{}') FROM entities WHERE id = $1", entityID).Scan(&attrs)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if !strings.Contains(attrs, "mesh_ip") {
|
|
||||||
return fmt.Errorf("mesh not joined (no mesh_ip in attributes)")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
case "health-check-answering":
|
|
||||||
st, err := sqlcgen.New(tx).GetEntityStatus(ctx, entityID)
|
|
||||||
if err != nil || st.Health == "unknown" || st.Health == "down" {
|
|
||||||
return fmt.Errorf("health check not answering (status: %s)", st.Health)
|
|
||||||
}
|
|
||||||
case "doc-page-complete":
|
|
||||||
var count int
|
|
||||||
err := tx.QueryRow(ctx, `
|
|
||||||
SELECT count(*) FROM relationships r
|
|
||||||
JOIN entities ke ON ke.id = r.source_id
|
|
||||||
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
|
||||||
AND r.type = 'documents' AND ke.type IN ('document','runbook','investigation')`,
|
|
||||||
entityID).Scan(&count)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if count == 0 {
|
|
||||||
return fmt.Errorf("no documentation linked to entity")
|
|
||||||
}
|
|
||||||
case "inventory-entry", "ip-reserved", "storage-pool-chosen", "cancelled-note",
|
|
||||||
"preflight-passed", "error-summary", "replacement-live-or-role-retired",
|
|
||||||
"replacement-failed", "post-verify-passed", "recovery-verified", "written-off",
|
|
||||||
"ingress-live-if-public", "doc-page-stub":
|
|
||||||
// Soft checks — always pass. These are operator-confirmed via the
|
|
||||||
// transition request itself, or are not mechanically enforceable.
|
|
||||||
default:
|
|
||||||
// Unknown preconditions are skipped (operator intent overrides).
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// ─── Helpers ───────────────────────────────────────────────────────────
|
// ─── Helpers ───────────────────────────────────────────────────────────
|
||||||
|
|||||||
@@ -155,14 +155,14 @@ func (s *Server) GetTrends(ctx context.Context, req gen.GetTrendsRequestObject)
|
|||||||
f, _ := slopeNum.Float64Value()
|
f, _ := slopeNum.Float64Value()
|
||||||
t.Slope = float32Ptr(float32(f.Float64))
|
t.Slope = float32Ptr(float32(f.Float64))
|
||||||
if f.Float64 > 0.01 {
|
if f.Float64 > 0.01 {
|
||||||
t.Direction = gen.Improving
|
t.Direction = gen.TrendDirectionImproving
|
||||||
} else if f.Float64 < -0.01 {
|
} else if f.Float64 < -0.01 {
|
||||||
t.Direction = gen.Degrading
|
t.Direction = gen.TrendDirectionDegrading
|
||||||
} else {
|
} else {
|
||||||
t.Direction = gen.Stable
|
t.Direction = gen.TrendDirectionStable
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
t.Direction = gen.Unknown
|
t.Direction = gen.TrendDirectionUnknown
|
||||||
}
|
}
|
||||||
items = append(items, t)
|
items = append(items, t)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -111,6 +111,7 @@ func (s *Server) PatchPattern(ctx context.Context, req gen.PatchPatternRequestOb
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
|
&id, "PATCH", "/api/v1/patterns/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
|
map[string]any{"status": req.Body.Status, "quarantined": req.Body.Quarantined}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -65,6 +65,7 @@ func (s *Server) CreateRelationship(ctx context.Context, req gen.CreateRelations
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "create",
|
||||||
nil, "POST", "/api/v1/relationships", "",
|
nil, "POST", "/api/v1/relationships", "",
|
||||||
|
nil,
|
||||||
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
|
map[string]any{"source": req.Body.Source, "target": req.Body.Target, "type": req.Body.Type}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
@@ -108,6 +109,7 @@ func (s *Server) EndRelationship(ctx context.Context, req gen.EndRelationshipReq
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
|
if auditErr := observability.Audit(ctx, sqlcgen.New(tx), actorType, actor, "delete",
|
||||||
nil, "DELETE", "/api/v1/relationships", "",
|
nil, "DELETE", "/api/v1/relationships", "",
|
||||||
|
nil,
|
||||||
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
|
map[string]any{"source": req.Params.Source, "target": req.Params.Target, "type": req.Params.RelType}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -237,6 +237,11 @@ func NewHandler(ctx context.Context, pool *db.Pool, cfg config.Config) http.Hand
|
|||||||
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/knowledge/orphans", s.serveKnowledgeOrphans)
|
||||||
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
|
r.With(combinedAuth(cfg, false)).Post("/api/v1/knowledge/merge", s.serveMergeKnowledge)
|
||||||
|
|
||||||
|
// Drift audit: read-only DB-side report of orphan checks, checks on
|
||||||
|
// retired targets, stuck down/unknown probes, unmonitored declared types,
|
||||||
|
// and dangling edges. Companion to the knowledge-graph-audit skill.
|
||||||
|
r.With(combinedAuth(cfg, false)).Get("/api/v1/audit/drift", s.serveAuditDrift)
|
||||||
|
|
||||||
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
// Custom (non-OpenAPI) routes: the global activity feed (recency-ordered,
|
||||||
// unlike ListExecutions which sorts by target for pagination) and the
|
// unlike ListExecutions which sorts by target for pagination) and the
|
||||||
// per-session "what did this session do" digest.
|
// per-session "what did this session do" digest.
|
||||||
|
|||||||
@@ -136,6 +136,7 @@ func (s *Server) PatchSkill(ctx context.Context, req gen.PatchSkillRequestObject
|
|||||||
actorType, actor := actorInfo(ctx)
|
actorType, actor := actorInfo(ctx)
|
||||||
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
if auditErr := observability.Audit(ctx, q, actorType, actor, "patch",
|
||||||
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
|
&id, "PATCH", "/api/v1/skills/"+req.Id, "",
|
||||||
|
nil,
|
||||||
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
|
map[string]any{"status": req.Body.Status, "pinned_version": req.Body.PinnedVersion}); auditErr != nil {
|
||||||
return nil, auditErr
|
return nil, auditErr
|
||||||
}
|
}
|
||||||
|
|||||||
251
internal/mcp/create_entity_test.go
Normal file
251
internal/mcp/create_entity_test.go
Normal file
@@ -0,0 +1,251 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
// Integration tests for the entity-mutation MCP tools (create_entity,
|
||||||
|
// update_entity_attributes), focused on the capability gap that stranded
|
||||||
|
// session 23da10db: entities mutated via MCP must derive/regenerate checks the
|
||||||
|
// same way the HTTP create/patch paths do. Guarded by OIKOS_TEST_DATABASE_URL
|
||||||
|
// (see internal/db/integration_test.go); run via `make test-db`.
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/checkdefaults"
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
|
)
|
||||||
|
|
||||||
|
// newTestPool mirrors internal/httpapi/api_test.go: a throwaway database,
|
||||||
|
// migrated and seeded with ontology/inventory/policy so create_entity's type
|
||||||
|
// validation and checkdefaults derivation have a real type tree to work
|
||||||
|
// against.
|
||||||
|
func newTestPool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
baseURL := os.Getenv("OIKOS_TEST_DATABASE_URL")
|
||||||
|
if baseURL == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_mcp_test_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
qi := strings.Index(baseURL, "?")
|
||||||
|
base, params := baseURL, ""
|
||||||
|
if qi >= 0 {
|
||||||
|
base, params = baseURL[:qi], baseURL[qi:]
|
||||||
|
}
|
||||||
|
testURL := base[:strings.LastIndex(base, "/")+1] + dbName + params
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, e := pgx.Connect(ctx, baseURL); e == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
for _, f := range []string{"ontology.yaml", "inventory.yaml", "policy.yaml"} {
|
||||||
|
content, err := os.ReadFile("../../seeds/" + f)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read seed %s: %v", f, err)
|
||||||
|
}
|
||||||
|
name := f
|
||||||
|
if err := pool.SeedIngest(ctx, name, content,
|
||||||
|
func(ctx context.Context, tx pgx.Tx, data map[string]any) error {
|
||||||
|
var err error
|
||||||
|
switch name {
|
||||||
|
case "ontology.yaml":
|
||||||
|
_, err = db.IngestOntologySeed(ctx, tx, data)
|
||||||
|
case "inventory.yaml":
|
||||||
|
_, err = db.IngestInventorySeed(ctx, tx, data)
|
||||||
|
case "policy.yaml":
|
||||||
|
_, err = db.IngestPolicySeed(ctx, tx, data)
|
||||||
|
}
|
||||||
|
return err
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("ingest %s: %v", f, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
// callTool invokes a registered tool's handler in-process and returns its
|
||||||
|
// concatenated text result.
|
||||||
|
func callTool(t *testing.T, pool *db.Pool, name string, args map[string]any) string {
|
||||||
|
t.Helper()
|
||||||
|
var handler toolHandler
|
||||||
|
for _, r := range allTools(pool, uuid.Nil) {
|
||||||
|
if r.tool.Name == name {
|
||||||
|
handler = r.handler
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if handler == nil {
|
||||||
|
t.Fatalf("tool %q not registered", name)
|
||||||
|
}
|
||||||
|
argsJSON, _ := json.Marshal(args)
|
||||||
|
res, err := handler(context.Background(), &mcp.CallToolRequest{Params: &mcp.CallToolParamsRaw{
|
||||||
|
Name: name,
|
||||||
|
Arguments: argsJSON,
|
||||||
|
}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("tool %s returned error: %v", name, err)
|
||||||
|
}
|
||||||
|
var sb strings.Builder
|
||||||
|
for _, c := range res.Content {
|
||||||
|
if tc, ok := c.(*mcp.TextContent); ok {
|
||||||
|
sb.WriteString(tc.Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return sb.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkCountFor returns the number of derived check_defs targeting slug.
|
||||||
|
func checkCountFor(t *testing.T, pool *db.Pool, slug string) int {
|
||||||
|
t.Helper()
|
||||||
|
var n int
|
||||||
|
err := pool.QueryRow(context.Background(),
|
||||||
|
`SELECT count(*) FROM check_defs cd
|
||||||
|
JOIN entities e ON e.id = cd.target_id
|
||||||
|
WHERE e.slug = $1`, slug).Scan(&n)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("count check_defs for %s: %v", slug, err)
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCreateEntity_DerivesChecks proves create_entity inserts an entity AND
|
||||||
|
// derives its default checks in one call (the HTTP create path did this; the
|
||||||
|
// MCP path previously could not create at all).
|
||||||
|
func TestCreateEntity_DerivesChecks(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
slug := "service:mcp-create-test"
|
||||||
|
|
||||||
|
out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service",
|
||||||
|
"slug": slug,
|
||||||
|
"name": "mcp-create-test",
|
||||||
|
"attributes": `{"url":"https://mcp-create-test.example"}`,
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "Created "+slug) {
|
||||||
|
t.Fatalf("create_entity result = %q, want Created %s", out, slug)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "Derived") {
|
||||||
|
t.Errorf("create_entity result = %q, want a Derived check summary", out)
|
||||||
|
}
|
||||||
|
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||||
|
t.Errorf("check_defs targeting %s = %d, want >=1 (create did not derive checks)", slug, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCreateEntity_DuplicateAndInvalid covers the guard rails: a repeat create
|
||||||
|
// is reported as "already exists" (not an error), and an unknown type is
|
||||||
|
// rejected with a clear message.
|
||||||
|
func TestCreateEntity_DuplicateAndInvalid(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||||
|
}); !strings.Contains(out, "Created service:mcp-dup") {
|
||||||
|
t.Fatalf("first create = %q", out)
|
||||||
|
}
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service", "slug": "service:mcp-dup", "name": "mcp-dup",
|
||||||
|
}); !strings.Contains(out, "already exists") {
|
||||||
|
t.Errorf("duplicate create = %q, want 'already exists'", out)
|
||||||
|
}
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "no-such-type", "slug": "no-such-type:x", "name": "x",
|
||||||
|
}); !strings.Contains(out, "not found in ontology") {
|
||||||
|
t.Errorf("unknown type = %q, want 'not found in ontology'", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdateEntityAttributes_RegeneratesChecks is the regression guard for the
|
||||||
|
// haos session: setting an entity's `monitoring` attribute via MCP must
|
||||||
|
// regenerate checks. Before this fix the MCP update path skipped
|
||||||
|
// ensureDefaultChecks, so flipping monitoring produced nothing.
|
||||||
|
func TestUpdateEntityAttributes_RegeneratesChecks(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
slug := "service:mcp-regen-test"
|
||||||
|
|
||||||
|
// Create with monitoring:none — no checks derived.
|
||||||
|
if out := callTool(t, pool, "create_entity", map[string]any{
|
||||||
|
"type": "service", "slug": slug, "name": "mcp-regen-test",
|
||||||
|
"attributes": `{"monitoring":"none","url":"https://mcp-regen.example"}`,
|
||||||
|
}); !strings.Contains(out, "Created "+slug) {
|
||||||
|
t.Fatalf("create = %q", out)
|
||||||
|
}
|
||||||
|
if got := checkCountFor(t, pool, slug); got != 0 {
|
||||||
|
t.Fatalf("check_defs with monitoring:none = %d, want 0", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Flip monitoring to [http] via update_entity_attributes — checks must
|
||||||
|
// regenerate. This is exactly what failed for service:haos.
|
||||||
|
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||||
|
"slug": slug,
|
||||||
|
"attributes": `{"monitoring":["http"]}`,
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "Updated "+slug) {
|
||||||
|
t.Fatalf("update result = %q, want Updated %s", out, slug)
|
||||||
|
}
|
||||||
|
if !strings.Contains(out, "Derived") {
|
||||||
|
t.Errorf("update result = %q, want a Derived check summary (regeneration)", out)
|
||||||
|
}
|
||||||
|
if got := checkCountFor(t, pool, slug); got < 1 {
|
||||||
|
t.Errorf("check_defs after monitoring:[http] = %d, want >=1 (MCP update did not regenerate checks)", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestUpdateEntityAttributes_NotFound keeps the existing error contract.
|
||||||
|
func TestUpdateEntityAttributes_NotFound(t *testing.T) {
|
||||||
|
pool := newTestPool(t)
|
||||||
|
out := callTool(t, pool, "update_entity_attributes", map[string]any{
|
||||||
|
"slug": "service:does-not-exist",
|
||||||
|
"attributes": `{"x":1}`,
|
||||||
|
})
|
||||||
|
if !strings.Contains(out, "not found") {
|
||||||
|
t.Errorf("update missing entity = %q, want 'not found'", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestFormatCheckResult is a pure unit test for the result-message helper, so
|
||||||
|
// the formatting contract holds even when the DB is unavailable.
|
||||||
|
func TestFormatCheckResult(t *testing.T) {
|
||||||
|
if got := formatCheckResult(checkdefaults.Result{Created: 2}); !strings.Contains(got, "Derived 2 check") {
|
||||||
|
t.Errorf("created-only = %q, want Derived 2", got)
|
||||||
|
}
|
||||||
|
got := formatCheckResult(checkdefaults.Result{Created: 1, Skipped: []checkdefaults.Skip{{Kind: "process", Reason: "no host"}}})
|
||||||
|
if !strings.Contains(got, "Derived 1 check") || !strings.Contains(got, "Skipped process") || !strings.Contains(got, "no host") {
|
||||||
|
t.Errorf("created+skipped = %q", got)
|
||||||
|
}
|
||||||
|
if got := formatCheckResult(checkdefaults.Result{Undeclared: true}); !strings.Contains(got, "no monitoring") {
|
||||||
|
t.Errorf("undeclared = %q, want no-monitoring hint", got)
|
||||||
|
}
|
||||||
|
if formatCreateResult("a", "b", checkdefaults.Result{Created: 0}) != "Created a (b)." {
|
||||||
|
t.Error("create result with no checks should have no suffix")
|
||||||
|
}
|
||||||
|
}
|
||||||
119
internal/mcp/discover.go
Normal file
119
internal/mcp/discover.go
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
package mcp
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/dtoro/oikos/internal/execlog"
|
||||||
|
"github.com/dtoro/oikos/internal/remote"
|
||||||
|
)
|
||||||
|
|
||||||
|
// discoverInfraDrift compares the live Proxmox guests (pct/qm list on every
|
||||||
|
// proxmox host) against the DB graph, surfacing drift the DB-only audit
|
||||||
|
// cannot see: guests running with no entity (missing), and entities whose
|
||||||
|
// pve_id is no longer live (ghost). This is the auto-discover/validate half
|
||||||
|
// of the knowledge-graph audit skill — read-only, reaches hosts over the same
|
||||||
|
// SSH/pct path the checks use.
|
||||||
|
func discoverInfraDrift(ctx context.Context, pool *db.Pool) any {
|
||||||
|
// 1. proxmox hosts to query.
|
||||||
|
hostRows, err := pool.Query(ctx,
|
||||||
|
`SELECT slug FROM entities WHERE type='proxmox-host' AND COALESCE(state,'active')='active'`)
|
||||||
|
if err != nil {
|
||||||
|
return map[string]any{"error": "query hosts: " + err.Error()}
|
||||||
|
}
|
||||||
|
var hosts []string
|
||||||
|
for hostRows.Next() {
|
||||||
|
var s string
|
||||||
|
if hostRows.Scan(&s) == nil {
|
||||||
|
hosts = append(hosts, s)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
hostRows.Close()
|
||||||
|
|
||||||
|
// 2. DB guests keyed by pve_id.
|
||||||
|
type dbGuest struct {
|
||||||
|
Slug string `json:"slug"`
|
||||||
|
Type string `json:"type"`
|
||||||
|
PveID string `json:"pve_id"`
|
||||||
|
Host string `json:"host"`
|
||||||
|
}
|
||||||
|
dbGuests := map[string]dbGuest{}
|
||||||
|
gr, err := pool.Query(ctx,
|
||||||
|
`SELECT slug, type, COALESCE(attributes->>'pve_id',''), COALESCE(attributes->>'host','')
|
||||||
|
FROM entities WHERE type IN ('lxc','vm')`)
|
||||||
|
if err != nil {
|
||||||
|
return map[string]any{"error": "query guests: " + err.Error()}
|
||||||
|
}
|
||||||
|
for gr.Next() {
|
||||||
|
var g dbGuest
|
||||||
|
if gr.Scan(&g.Slug, &g.Type, &g.PveID, &g.Host) == nil && g.PveID != "" {
|
||||||
|
dbGuests[g.PveID] = g
|
||||||
|
}
|
||||||
|
}
|
||||||
|
gr.Close()
|
||||||
|
|
||||||
|
// 3. enumerate live guests from every host.
|
||||||
|
live := map[string]string{} // pve_id -> "host:name"
|
||||||
|
hostErrors := map[string]string{}
|
||||||
|
for _, hs := range hosts {
|
||||||
|
et, rerr := remote.ResolveExecTarget(ctx, pool, hs, sshUser)
|
||||||
|
if rerr != nil {
|
||||||
|
hostErrors[hs] = "resolve: " + rerr.Error()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
for _, cmd := range []string{"pct list", "qm list"} {
|
||||||
|
out, eerr := sshExecStream(ctx, et.Host, et.User, et.Wrap(cmd),
|
||||||
|
execlog.Sink(func(string, []byte) {}))
|
||||||
|
if eerr != nil {
|
||||||
|
hostErrors[hs+" "+cmd] = eerr.Error()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
scanIDs(out, hs, live)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4. diff.
|
||||||
|
var missing, ghost []string
|
||||||
|
for id, hn := range live {
|
||||||
|
if _, ok := dbGuests[id]; !ok {
|
||||||
|
missing = append(missing, id+" on "+hn)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for id, g := range dbGuests {
|
||||||
|
if _, ok := live[id]; !ok {
|
||||||
|
ghost = append(ghost, g.Slug+" (pve_id="+id+")")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return map[string]any{
|
||||||
|
"hosts_queried": len(hosts),
|
||||||
|
"live_guests": len(live),
|
||||||
|
"db_guests": len(dbGuests),
|
||||||
|
"missing_entities": missing, // in Proxmox, no DB entity
|
||||||
|
"ghost_entities": ghost, // in DB, not live in Proxmox
|
||||||
|
"host_errors": hostErrors,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanIDs parses `pct list` / `qm list` output (VMID ... Name) into the live map.
|
||||||
|
func scanIDs(output, hostSlug string, live map[string]string) {
|
||||||
|
sc := bufio.NewScanner(strings.NewReader(output))
|
||||||
|
for sc.Scan() {
|
||||||
|
line := strings.TrimSpace(sc.Text())
|
||||||
|
if line == "" || strings.HasPrefix(strings.ToLower(line), "vmid") {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
fields := strings.Fields(line)
|
||||||
|
if len(fields) < 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
id := fields[0]
|
||||||
|
name := ""
|
||||||
|
if len(fields) >= 4 {
|
||||||
|
name = fields[len(fields)-1] // pct: last col is name; qm: name near end
|
||||||
|
}
|
||||||
|
live[id] = hostSlug + ":" + name
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -5,7 +5,6 @@ package mcp
|
|||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/base64"
|
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
"html"
|
"html"
|
||||||
@@ -25,6 +24,7 @@ import (
|
|||||||
"github.com/dtoro/oikos/internal/execlog"
|
"github.com/dtoro/oikos/internal/execlog"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
"github.com/dtoro/oikos/internal/policy"
|
"github.com/dtoro/oikos/internal/policy"
|
||||||
|
"github.com/dtoro/oikos/internal/remote"
|
||||||
"github.com/google/jsonschema-go/jsonschema"
|
"github.com/google/jsonschema-go/jsonschema"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"github.com/modelcontextprotocol/go-sdk/mcp"
|
"github.com/modelcontextprotocol/go-sdk/mcp"
|
||||||
@@ -72,9 +72,121 @@ func newServer(pool *db.Pool, agentID uuid.UUID) *mcp.Server {
|
|||||||
for _, t := range allTools(pool, agentID) {
|
for _, t := range allTools(pool, agentID) {
|
||||||
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
|
s.AddTool(t.tool, withActivityLogging(pool, agentID, t.tool.Name, t.handler))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Resource templates: let MCP clients browse and attach entities,
|
||||||
|
// knowledge entries, and executions as conversation resources.
|
||||||
|
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||||
|
URITemplate: "oikos://entity/{slug}",
|
||||||
|
Name: "Entity",
|
||||||
|
Description: "Oikos entity by slug (e.g. host:hubris, lxc:jellyfin)",
|
||||||
|
MIMEType: "application/json",
|
||||||
|
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||||
|
slug := matches["slug"]
|
||||||
|
var id uuid.UUID
|
||||||
|
if u, err := uuid.Parse(slug); err == nil {
|
||||||
|
id = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", slug).Scan(&id)
|
||||||
|
}
|
||||||
|
if id == uuid.Nil {
|
||||||
|
return "", fmt.Errorf("entity not found: %s", slug)
|
||||||
|
}
|
||||||
|
result := queryEntity(ctx, pool, slug)
|
||||||
|
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||||
|
}))
|
||||||
|
|
||||||
|
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||||
|
URITemplate: "oikos://knowledge/{id}",
|
||||||
|
Name: "Knowledge",
|
||||||
|
Description: "Knowledge entry by entity slug or UUID",
|
||||||
|
MIMEType: "application/json",
|
||||||
|
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||||
|
idOrSlug := matches["id"]
|
||||||
|
var entityID uuid.UUID
|
||||||
|
if u, err := uuid.Parse(idOrSlug); err == nil {
|
||||||
|
entityID = u
|
||||||
|
} else {
|
||||||
|
pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", idOrSlug).Scan(&entityID)
|
||||||
|
}
|
||||||
|
if entityID == uuid.Nil {
|
||||||
|
return "", fmt.Errorf("knowledge not found: %s", idOrSlug)
|
||||||
|
}
|
||||||
|
result := queryRows(ctx, pool, `
|
||||||
|
SELECT ke.title, ke.content, ke.tags::text, e.slug, e.type AS kind,
|
||||||
|
ke.updated_at::text
|
||||||
|
FROM knowledge_entities ke
|
||||||
|
JOIN entities e ON e.id = ke.entity_id
|
||||||
|
WHERE ke.entity_id = $1`, entityID)
|
||||||
|
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||||
|
}))
|
||||||
|
|
||||||
|
s.AddResourceTemplate(&mcp.ResourceTemplate{
|
||||||
|
URITemplate: "oikos://execution/{id}",
|
||||||
|
Name: "Execution",
|
||||||
|
Description: "Execution by UUID (returns status, result, timing)",
|
||||||
|
MIMEType: "application/json",
|
||||||
|
}, resourceHandler(pool, func(ctx context.Context, matches map[string]string) (string, error) {
|
||||||
|
result := queryRows(ctx, pool, `
|
||||||
|
SELECT e.entity_id, te.slug AS target, e.action, e.risk_class,
|
||||||
|
e.status, e.result::text, e.duration_ms,
|
||||||
|
e.started_at::text, e.completed_at::text
|
||||||
|
FROM executions e
|
||||||
|
JOIN entities te ON te.id = e.target_entity_id
|
||||||
|
WHERE e.entity_id = $1`, matches["id"])
|
||||||
|
return result.Content[0].(*mcp.TextContent).Text, nil
|
||||||
|
}))
|
||||||
|
|
||||||
return s
|
return s
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// resourceHandler adapts a simple func(ctx, params) → (string, error) into
|
||||||
|
// an MCP ResourceHandler, reading the URI matched by a ResourceTemplate.
|
||||||
|
func resourceHandler(pool *db.Pool, fn func(ctx context.Context, matches map[string]string) (string, error)) mcp.ResourceHandler {
|
||||||
|
return func(ctx context.Context, req *mcp.ReadResourceRequest) (*mcp.ReadResourceResult, error) {
|
||||||
|
uri := req.Params.URI
|
||||||
|
matches := matchURITemplate(uri)
|
||||||
|
if matches == nil {
|
||||||
|
return nil, mcp.ResourceNotFoundError(uri)
|
||||||
|
}
|
||||||
|
|
||||||
|
text, err := fn(ctx, matches)
|
||||||
|
if err != nil {
|
||||||
|
return nil, mcp.ResourceNotFoundError(uri)
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := json.MarshalIndent(json.RawMessage(text), "", " ")
|
||||||
|
if err != nil {
|
||||||
|
result = []byte(text)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &mcp.ReadResourceResult{
|
||||||
|
Contents: []*mcp.ResourceContents{{
|
||||||
|
URI: uri,
|
||||||
|
MIMEType: "application/json",
|
||||||
|
Text: string(result),
|
||||||
|
}},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// matchURITemplate extracts parameters from a URI that matches one of the
|
||||||
|
// oikos:// resource templates. Returns nil if the URI doesn't match.
|
||||||
|
func matchURITemplate(uri string) map[string]string {
|
||||||
|
// oikos://entity/{slug}
|
||||||
|
if rest, ok := strings.CutPrefix(uri, "oikos://entity/"); ok && rest != "" {
|
||||||
|
return map[string]string{"slug": rest}
|
||||||
|
}
|
||||||
|
// oikos://knowledge/{id}
|
||||||
|
if rest, ok := strings.CutPrefix(uri, "oikos://knowledge/"); ok && rest != "" {
|
||||||
|
return map[string]string{"id": rest}
|
||||||
|
}
|
||||||
|
// oikos://execution/{id}
|
||||||
|
if rest, ok := strings.CutPrefix(uri, "oikos://execution/"); ok && rest != "" {
|
||||||
|
return map[string]string{"id": rest}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
// withActivityLogging wraps a tool handler to record agent_activity rows.
|
// withActivityLogging wraps a tool handler to record agent_activity rows.
|
||||||
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
|
func withActivityLogging(pool *db.Pool, agentID uuid.UUID, toolName string, next mcp.ToolHandler) mcp.ToolHandler {
|
||||||
if agentID == uuid.Nil {
|
if agentID == uuid.Nil {
|
||||||
@@ -461,31 +573,12 @@ func sshExecStream(ctx context.Context, host, user, command string, sink execlog
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUser string, err error) {
|
// resolveHost resolves a host:<slug> to its reachable IP and SSH user. A thin
|
||||||
var attrs string
|
// wrapper over the shared resolver (internal/remote), kept so slug-based
|
||||||
err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", entitySlug).Scan(&attrs)
|
// callers keep working; the shared resolver also prefers public_ipv4 over
|
||||||
if err != nil {
|
// mesh and honors a per-entity ssh.user.
|
||||||
return "", "", fmt.Errorf("entity not found: %s", entitySlug)
|
func resolveHost(ctx context.Context, pool *db.Pool, entitySlug string) (hostIP string, sshUserOut string, err error) {
|
||||||
}
|
return remote.ResolveHost(ctx, pool, entitySlug, sshUser)
|
||||||
|
|
||||||
var m map[string]interface{}
|
|
||||||
if err := json.Unmarshal([]byte(attrs), &m); err != nil {
|
|
||||||
return "", "", fmt.Errorf("parse attributes: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if ip, ok := m["lan_ip"].(string); ok && ip != "" {
|
|
||||||
return ip, sshUser, nil
|
|
||||||
}
|
|
||||||
if mesh, ok := m["mesh"].(map[string]interface{}); ok {
|
|
||||||
for _, proto := range []string{"netbird", "tailscale"} {
|
|
||||||
if p, ok := mesh[proto].(map[string]interface{}); ok {
|
|
||||||
if ip, ok := p["ip"].(string); ok && ip != "" {
|
|
||||||
return ip, sshUser, nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return "", "", fmt.Errorf("no IP found for %s", entitySlug)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
// htmlTagRe strips HTML tags for the naive text extraction in httpGet.
|
||||||
@@ -564,109 +657,28 @@ func isPrivateHost(host string) bool {
|
|||||||
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
// connection: identity for a host, `pct exec <pve_id> -- ...` for an LXC,
|
||||||
// `qm guest exec <pve_id> -- ...` for a VM.
|
// `qm guest exec <pve_id> -- ...` for a VM.
|
||||||
//
|
//
|
||||||
// The lxc.attributes.host value is stored WITHOUT a "host:" prefix (e.g.
|
// Delegates to the shared resolver (internal/remote), the single path used by
|
||||||
// "strong", not "host:strong") — see pct_create's entity registration. The
|
// both the MCP `run` tool and the scheduler's checks. The historical notes
|
||||||
// pre-existing pct_exec handler queried resolveHost with that bare value
|
// (host attr without prefix, vm host-resolution chain, nested-quoting
|
||||||
// directly, which can never match a "host:*" slug and always fails; this
|
// handling via base64) all still hold — they now live in remote.guestWrap.
|
||||||
// prefixes it correctly.
|
|
||||||
//
|
|
||||||
// vm: support (2026-07-18): VMs in inventory.yaml carry `pve_id` and a `host`
|
|
||||||
// attribute (or a `hosts` relationship) just like LXCs, but they're reached
|
|
||||||
// via `qm guest exec` instead of `pct exec`. Previously the agent had to
|
|
||||||
// SSH-hop via `host:hubris` to reach a VM (e.g. `ssh root@<vm_ip> '...'`),
|
|
||||||
// which broke on nested shell quoting and forced manual escaping workarounds
|
|
||||||
// — see plans/2026-07-18-session-review-three-sessions.md P1.6. A VM's
|
|
||||||
// `host` attribute is optional: if absent, fall back to looking up the
|
|
||||||
// `hosts` relationship on the VM entity, then to hubris (the documented
|
|
||||||
// default Proxmox host) — same fallback chain as LXCs.
|
|
||||||
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
func resolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug string) (host, user string, wrap func(cmd string) string, err error) {
|
||||||
if strings.HasPrefix(targetSlug, "host:") {
|
et, err := remote.ResolveExecTarget(ctx, pool, targetSlug, sshUser)
|
||||||
host, user, err = resolveHost(ctx, pool, targetSlug)
|
if err != nil {
|
||||||
return host, user, func(cmd string) string { return cmd }, err
|
return "", "", nil, err
|
||||||
}
|
}
|
||||||
if strings.HasPrefix(targetSlug, "lxc:") {
|
return et.Host, et.User, et.Wrap, nil
|
||||||
var pveID, hostAttr string
|
|
||||||
// COALESCE the host column: many older LXC entities (seeded from
|
|
||||||
// inventory, not provisioned by pct_create) have pve_id but no host
|
|
||||||
// attribute at all. Scanning a SQL NULL into a plain string errors
|
|
||||||
// the whole row, wrongly reporting "missing pve_id" even when it was
|
|
||||||
// present — COALESCE avoids the NULL, "" is handled below.
|
|
||||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
|
||||||
return "", "", nil, fmt.Errorf("LXC not found or missing pve_id: %s", targetSlug)
|
|
||||||
}
|
|
||||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
|
||||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
|
||||||
id := pveID
|
|
||||||
return host, user, func(cmd string) string {
|
|
||||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
|
||||||
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", id, b64)
|
|
||||||
}, err
|
|
||||||
}
|
|
||||||
if strings.HasPrefix(targetSlug, "vm:") {
|
|
||||||
// VMs: same host-resolution chain as LXCs (attributes.host →
|
|
||||||
// `hosts` relationship → hubris default), but reached via
|
|
||||||
// `qm guest exec` instead of `pct exec`. Requires the QEMU
|
|
||||||
// guest agent running inside the VM (the standard Proxmox
|
|
||||||
// setup; ZimaOS/HAOS in this fleet already have it).
|
|
||||||
var pveID, hostAttr string
|
|
||||||
if qerr := pool.QueryRow(ctx, "SELECT attributes->>'pve_id', COALESCE(attributes->>'host', '') FROM entities WHERE slug = $1", targetSlug).Scan(&pveID, &hostAttr); qerr != nil || pveID == "" {
|
|
||||||
return "", "", nil, fmt.Errorf("VM not found or missing pve_id: %s", targetSlug)
|
|
||||||
}
|
|
||||||
hostSlug := resolveProxmoxHostSlug(ctx, pool, targetSlug, hostAttr)
|
|
||||||
host, user, err = resolveHost(ctx, pool, hostSlug)
|
|
||||||
id := pveID
|
|
||||||
return host, user, func(cmd string) string {
|
|
||||||
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
|
||||||
// `qm guest exec <id> -- /bin/bash -c '...'` returns JSON by
|
|
||||||
// default; pipe through `jq -r .out` if available, else cat.
|
|
||||||
// The base64 round-trip mirrors the LXC path so nested quoting
|
|
||||||
// (the original VM-target pain point — session 55927f0a) is
|
|
||||||
// handled identically to LXC dispatch.
|
|
||||||
return fmt.Sprintf(
|
|
||||||
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
|
||||||
id, b64, id, b64)
|
|
||||||
}, err
|
|
||||||
}
|
|
||||||
return "", "", nil, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
// resolveProxmoxHostSlug resolves the Proxmox host slug that owns a given
|
||||||
// LXC/VM target. Resolution order:
|
// LXC/VM target (see internal/remote.ResolveProxmoxHostSlug for the chain).
|
||||||
// 1. hostAttr if non-empty (the entity's attributes.host — stored without
|
// This slug-based wrapper looks up the entity id so slug callers keep working;
|
||||||
// "host:" prefix in inventory.yaml and pct_create).
|
// the shared resolver takes an id directly.
|
||||||
// 2. the `hosts` relationship on the entity (e.g. host:hubris → vm:zimaos),
|
|
||||||
// looked up in the relationships table — the canonical graph source.
|
|
||||||
// 3. "hubris" as a documented default Proxmox host fallback.
|
|
||||||
//
|
|
||||||
// Returns a slug with the "host:" prefix attached, ready for resolveHost.
|
|
||||||
// Extracted from the inline LXC path (2026-07-18) so the VM path shares the
|
|
||||||
// same chain — see plans/2026-07-18-session-review-three-sessions.md P1.6.
|
|
||||||
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
func resolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entitySlug, hostAttr string) string {
|
||||||
hostSlug := strings.TrimSpace(hostAttr)
|
var id uuid.UUID
|
||||||
if hostSlug == "" {
|
if err := pool.QueryRow(ctx, "SELECT id FROM entities WHERE slug = $1", entitySlug).Scan(&id); err != nil {
|
||||||
// Fall back to the `hosts` relationship — the graph edge from
|
id = uuid.Nil
|
||||||
// the Proxmox host to this LXC/VM. This is the canonical source
|
|
||||||
// for "who owns this VM" in inventory.yaml; the `host` attribute
|
|
||||||
// is a denormalized shortcut that not every entity has.
|
|
||||||
var relHostSlug string
|
|
||||||
// hosts relationship: source=host, target=lxc/vm. Look up the
|
|
||||||
// source slug given the target.
|
|
||||||
if err := pool.QueryRow(ctx, `
|
|
||||||
SELECT e.slug FROM relationships r
|
|
||||||
JOIN entities e ON e.id = r.source_id
|
|
||||||
WHERE r.target_id = (SELECT id FROM entities WHERE slug = $1)
|
|
||||||
AND r.type = 'hosts' AND r.valid_to IS NULL
|
|
||||||
LIMIT 1`, entitySlug).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
|
||||||
hostSlug = relHostSlug
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
if hostSlug == "" {
|
return remote.ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||||
hostSlug = "hubris" // documented default Proxmox host when unset
|
|
||||||
}
|
|
||||||
if !strings.HasPrefix(hostSlug, "host:") {
|
|
||||||
hostSlug = "host:" + hostSlug
|
|
||||||
}
|
|
||||||
return hostSlug
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// classifyAndGate is the shared classify→execute-or-queue path for every
|
// classifyAndGate is the shared classify→execute-or-queue path for every
|
||||||
@@ -728,8 +740,106 @@ func autoRun(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, comma
|
|||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// autoRunAsync starts a command in a goroutine, marking it running and returning
|
||||||
|
// immediately. The caller gets an execution_id to poll with get_execution_status.
|
||||||
|
// Used for commands containing sleep/wait/poll loops that would exceed the MCP
|
||||||
|
// client timeout (120s) — the execution continues server-side.
|
||||||
|
func autoRunAsync(ctx context.Context, pool *db.Pool, id uuid.UUID, targetSlug, command string) {
|
||||||
|
startedAt := time.Now()
|
||||||
|
if _, err := pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status='running', started_at=$2 WHERE entity_id=$1`,
|
||||||
|
id, startedAt); err != nil {
|
||||||
|
slog.Error("mcp: mark execution running (async)", "error", err, "execution_id", id)
|
||||||
|
}
|
||||||
|
|
||||||
|
host, user, wrap, err := resolveExecTarget(ctx, pool, targetSlug)
|
||||||
|
if err != nil {
|
||||||
|
pool.Exec(ctx,
|
||||||
|
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonErr("%s", err.Error()), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
slog.Error("mcp: async run resolve target", "error", err, "execution_id", id, "target", targetSlug)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
var correlationID string
|
||||||
|
if qerr := pool.QueryRow(ctx,
|
||||||
|
`SELECT correlation_id FROM executions WHERE entity_id = $1`, id).Scan(&correlationID); qerr != nil {
|
||||||
|
correlationID = ""
|
||||||
|
}
|
||||||
|
|
||||||
|
go func() {
|
||||||
|
defer func() {
|
||||||
|
if r := recover(); r != nil {
|
||||||
|
slog.Error("mcp: async run panic", "panic", r, "execution_id", id)
|
||||||
|
pool.Exec(context.Background(),
|
||||||
|
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonErr("panic: %v", r), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
|
||||||
|
sink, flush := execlog.New(context.Background(), pool, id, correlationID)
|
||||||
|
out, execErr := sshExecStream(context.Background(), host, user, wrap(command), sink)
|
||||||
|
flush()
|
||||||
|
if execErr != nil {
|
||||||
|
pool.Exec(context.Background(),
|
||||||
|
`UPDATE executions SET status='failed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonErr("%s: %s", execErr.Error(), out), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
slog.Error("mcp: async run failed", "error", execErr, "execution_id", id, "output", out)
|
||||||
|
} else {
|
||||||
|
pool.Exec(context.Background(),
|
||||||
|
`UPDATE executions SET status='completed', result=$2::jsonb, duration_ms=$3, completed_at=now() WHERE entity_id=$1`,
|
||||||
|
id, jsonOut(out), int(time.Since(startedAt).Milliseconds()))
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
|
||||||
|
// isLongRunningCommand detects shell commands containing sleep, wait, or poll
|
||||||
|
// loops that indicate the command will exceed the MCP client timeout (120s).
|
||||||
|
// These commands should use autoRunAsync to avoid the client timing out while
|
||||||
|
// the command continues server-side.
|
||||||
|
func isLongRunningCommand(cmd string) bool {
|
||||||
|
cmd = strings.TrimSpace(cmd)
|
||||||
|
|
||||||
|
// sleep with duration — `sleep 30`, `sleep 1m`, etc.
|
||||||
|
if sleepRe.MatchString(cmd) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// while/shell poll loops with sleep: `while ...; do ... sleep; done`
|
||||||
|
if pollRe.MatchString(cmd) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// standalone wait command
|
||||||
|
if waitRe.MatchString(cmd) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
sleepRe = regexp.MustCompile(`\bsleep\s+\d`)
|
||||||
|
pollRe = regexp.MustCompile(`\bwhile\b.*\bsleep\b`)
|
||||||
|
waitRe = regexp.MustCompile(`\bwait\s+\d|[&;]\s*wait\b`)
|
||||||
|
)
|
||||||
|
|
||||||
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.UUID, targetSlug, command, purpose, declaredRisk, sessionID string) *mcp.CallToolResult {
|
||||||
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
riskClass := policy.ClassifyCommand(command, declaredRisk)
|
||||||
|
|
||||||
|
// Transport-aware escalation: read-only commands on LXC targets that
|
||||||
|
// touch config paths (/opt/, /etc/) escalate to config_mutation.
|
||||||
|
// The classifier only scores the command text, not the transport layer
|
||||||
|
// — SSH-ing into a container to read /opt/ is riskier than running
|
||||||
|
// the same command locally on the Proxmox host via pct exec.
|
||||||
|
// Caught live: "cat /etc/hostname" on lxc:dns queued as config_mutation
|
||||||
|
// while "pct exec 107 -- cat /etc/hostname" on host:hubris auto-ran.
|
||||||
|
if riskClass == policy.RiskReadOnly && strings.HasPrefix(targetSlug, "lxc:") {
|
||||||
|
if strings.Contains(command, "/opt/") || strings.Contains(command, "/etc/") || strings.Contains(command, "/var/lib/") {
|
||||||
|
riskClass = policy.RiskConfigMutation
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
runParams, _ := json.Marshal(map[string]string{"command": command, "purpose": purpose})
|
||||||
actionCol := "run:" + string(runParams)
|
actionCol := "run:" + string(runParams)
|
||||||
|
|
||||||
@@ -747,6 +857,57 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
|
return textResult("No plan for this session. Call set_goal then propose_plan before run — even read-only tasks require a one-step plan. A one-step plan (\"Inspect X, report, write back\") is fine for trivial questions; the gate is about ordering, not approval. Read-only commands still auto-execute once a plan exists.")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Target validation: host-only commands (qm, pct, pvesh, iptables) must
|
||||||
|
// not be dispatched against lxc:/vm: targets — those aren't Proxmox hosts
|
||||||
|
// and don't have these tools. Caught live 2026-08-04: the agent ran
|
||||||
|
// `qm stop 100` against lxc:dns, wasting a turn.
|
||||||
|
|
||||||
|
// Command syntax validation: catch LLM-generated bash bugs before they
|
||||||
|
// hit the shell. The model sometimes inserts literal \n between commands
|
||||||
|
// or puts spaces inside flags — these always fail, so reject early.
|
||||||
|
if syntaxErr := validateCommandSyntax(command); syntaxErr != "" {
|
||||||
|
return textResult(syntaxErr)
|
||||||
|
}
|
||||||
|
|
||||||
|
if cmdPrefix, hostOnly := hostOnlyCommand(command); hostOnly && !strings.HasPrefix(targetSlug, "host:") {
|
||||||
|
hostSuggestion := resolveProxmoxHostSlug(ctx, pool, targetSlug, "")
|
||||||
|
if hostSuggestion == "" {
|
||||||
|
hostSuggestion = "host:hubris or host:strong"
|
||||||
|
}
|
||||||
|
return textResult(fmt.Sprintf("Cannot run %q on %s — %s is a Proxmox host command. Use target %s instead.",
|
||||||
|
cmdPrefix, targetSlug, cmdPrefix, hostSuggestion))
|
||||||
|
}
|
||||||
|
|
||||||
|
// systemctl and docker work on hosts and LXCs, but not VMs.
|
||||||
|
if cmdPrefix, hostLxc := hostLxcCommand(command); hostLxc {
|
||||||
|
if !strings.HasPrefix(targetSlug, "host:") && !strings.HasPrefix(targetSlug, "lxc:") {
|
||||||
|
return textResult(fmt.Sprintf("Cannot run %q on %s — %s only works on host:* or lxc:* targets.",
|
||||||
|
cmdPrefix, targetSlug, cmdPrefix))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// VM transport pre-flight: qm guest exec requires the QEMU guest agent
|
||||||
|
// to be running inside the VM. If it's not, the execution would queue
|
||||||
|
// for approval and never execute — the agent has no way to learn it's
|
||||||
|
// stuck (spotted live 2026-08-05: vm:zimaos had qemu_guest_agent=not_running,
|
||||||
|
// the run queued forever, and the agent fell back to unsafe raw SSH).
|
||||||
|
if strings.HasPrefix(targetSlug, "vm:") {
|
||||||
|
var rawAttrs []byte
|
||||||
|
if err := pool.QueryRow(ctx, `SELECT attributes FROM entities WHERE id = $1`, targetID).Scan(&rawAttrs); err == nil {
|
||||||
|
var attrs map[string]any
|
||||||
|
if json.Unmarshal(rawAttrs, &attrs) == nil {
|
||||||
|
if qga, ok := attrs["qemu_guest_agent"]; ok {
|
||||||
|
qgaStr, _ := qga.(string)
|
||||||
|
if qgaStr == "not_running" || qgaStr == "" {
|
||||||
|
return textResult(fmt.Sprintf(
|
||||||
|
"run on %s blocked: QEMU guest agent is not running (%s). qm guest exec cannot reach this VM. Start the agent inside the guest first (e.g. via SSH/systemctl start qemu-guest-agent), then re-run. If the agent is running but the entity attribute is stale, update it with update_entity_attributes(slug=%s, attributes={\"qemu_guest_agent\":\"running\"}).",
|
||||||
|
targetSlug, qgaStr, targetSlug))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Dedup: an identical pending command (same target, command, and
|
// Dedup: an identical pending command (same target, command, and
|
||||||
// purpose) blocks a re-request — stops a tool-calling loop from queuing
|
// purpose) blocks a re-request — stops a tool-calling loop from queuing
|
||||||
// the same approval repeatedly.
|
// the same approval repeatedly.
|
||||||
@@ -821,8 +982,51 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
SELECT 1 FROM relationships
|
SELECT 1 FROM relationships
|
||||||
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
|
WHERE source_id = t.id AND target_id = $1 AND type = 'involves' AND valid_to IS NULL)`,
|
||||||
id, "task:"+sessionID)
|
id, "task:"+sessionID)
|
||||||
|
// Link execution to session for auto-continuation (nomos_plan_executions
|
||||||
|
// was always empty — executions were never traceable back to sessions).
|
||||||
|
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||||
|
pool.Exec(ctx, `
|
||||||
|
INSERT INTO nomos_plan_executions (execution_id, session_id)
|
||||||
|
VALUES ($1, $2) ON CONFLICT (execution_id) DO NOTHING`, id, sid)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Auto-classify: write the classification decision to the classifications
|
||||||
|
// table (was always empty — 0 rows despite 1,884 executions). The route
|
||||||
|
// matches the auto-run vs queue-for-approval decision below.
|
||||||
|
classRoute := "escalate"
|
||||||
|
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||||
|
classRoute = "auto-act"
|
||||||
|
} else if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||||
|
classRoute = "auto-act"
|
||||||
|
} else if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||||
|
classRoute = "auto-act"
|
||||||
|
}
|
||||||
|
classReason, _ := json.Marshal(map[string]string{
|
||||||
|
"command": command, "purpose": purpose, "target": targetSlug, "declared_risk": declaredRisk,
|
||||||
|
})
|
||||||
|
classID, _ := uuid.NewV7()
|
||||||
|
pool.Exec(ctx, `INSERT INTO entities (id, slug, type, name, attributes) VALUES ($1, $2, 'classification', $3, '{}')`,
|
||||||
|
classID, "classification:"+classID.String(), "classification for "+execSlug)
|
||||||
|
pool.Exec(ctx, `INSERT INTO classifications (entity_id, action, risk_class, route, reasoning, correlation_id)
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6)`,
|
||||||
|
classID, actionCol, riskClass, classRoute, classReason, correlationID)
|
||||||
|
// Link classification to execution.
|
||||||
|
pool.Exec(ctx, `UPDATE executions SET classification_id = $2 WHERE entity_id = $1`, id, classID)
|
||||||
|
|
||||||
|
// Audit: record the execution creation with session_id for traceability.
|
||||||
|
// Every run call, whether auto-run or queued-for-approval, gets an audit
|
||||||
|
// entry so the agent's activity is traceable back to the originating session.
|
||||||
|
var auditSessionID *uuid.UUID
|
||||||
|
if sessionID != "" && sessionID != "ephemeral" {
|
||||||
|
if sid, serr := uuid.Parse(sessionID); serr == nil {
|
||||||
|
auditSessionID = &sid
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_ = observability.Audit(ctx, sqlcgen.New(pool), "agent", "nomos", "run",
|
||||||
|
&id, "POST", "/mcp", correlationID, auditSessionID,
|
||||||
|
map[string]any{"command": command, "target": targetSlug, "risk_class": riskClass, "purpose": purpose})
|
||||||
|
|
||||||
// read_only and reversible_low both run unattended, as seeds/policy.yaml
|
// read_only and reversible_low both run unattended, as seeds/policy.yaml
|
||||||
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
|
// and .agents/OIKOS.md declare ("reversible_low — restart, cache clear,
|
||||||
// sync pull. Unattended + ledger.").
|
// sync pull. Unattended + ledger.").
|
||||||
@@ -841,6 +1045,11 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
// agent still cannot talk a command DOWN: declaring reversible_low on
|
// agent still cannot talk a command DOWN: declaring reversible_low on
|
||||||
// something computed as config_mutation keeps config_mutation.
|
// something computed as config_mutation keeps config_mutation.
|
||||||
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
if riskClass == policy.RiskReadOnly || riskClass == policy.RiskReversibleLow {
|
||||||
|
if isLongRunningCommand(command) {
|
||||||
|
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||||
|
return textResult(fmt.Sprintf("run on %s (%s, async): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||||
|
targetSlug, riskClass, id, id))
|
||||||
|
}
|
||||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||||
if xerr != nil {
|
if xerr != nil {
|
||||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||||
@@ -859,6 +1068,12 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
// consent. The assent window, opened only on operator approval, is the
|
// consent. The assent window, opened only on operator approval, is the
|
||||||
// sole gate for config_mutation auto-run.)
|
// sole gate for config_mutation auto-run.)
|
||||||
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
if riskClass == policy.RiskConfigMutation && assentWindowActive(ctx, pool, agentID, sessionID) {
|
||||||
|
if isLongRunningCommand(command) {
|
||||||
|
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||||
|
slog.Info("mcp: run async via assent window", "target", targetSlug, "execution_id", id)
|
||||||
|
return textResult(fmt.Sprintf("run on %s (config_mutation, async via assent window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||||
|
targetSlug, id, id))
|
||||||
|
}
|
||||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||||
if xerr != nil {
|
if xerr != nil {
|
||||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||||
@@ -874,6 +1089,12 @@ func classifyAndGate(ctx context.Context, pool *db.Pool, agentID, targetID uuid.
|
|||||||
// operator isn't asked to re-type "I confirm" for every single command
|
// operator isn't asked to re-type "I confirm" for every single command
|
||||||
// against the thing they just confirmed.
|
// against the thing they just confirmed.
|
||||||
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
if riskClass == policy.RiskDestructive && destructiveWindowActive(ctx, pool, agentID, targetSlug, sessionID) {
|
||||||
|
if isLongRunningCommand(command) {
|
||||||
|
autoRunAsync(ctx, pool, id, targetSlug, command)
|
||||||
|
slog.Info("mcp: run async via destructive window", "target", targetSlug, "execution_id", id)
|
||||||
|
return textResult(fmt.Sprintf("run on %s (destructive, async via confirmed-target window): started — execution %s. Poll with get_execution_status(%s) for result.",
|
||||||
|
targetSlug, id, id))
|
||||||
|
}
|
||||||
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
out, xerr := autoRun(ctx, pool, id, targetSlug, command)
|
||||||
if xerr != nil {
|
if xerr != nil {
|
||||||
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
return textResult(fmt.Sprintf("run on %s: ERROR %v\n%s", targetSlug, xerr, out))
|
||||||
@@ -940,22 +1161,112 @@ func executeApprovedViaAPI(ctx context.Context, execID uuid.UUID, targetSlug, ac
|
|||||||
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
|
// for config_mutation auto-run now. See sessionHasPlan for the plan-existence
|
||||||
// check used by the P1 plan-first gate.
|
// check used by the P1 plan-first gate.
|
||||||
|
|
||||||
|
// hostOnlyCommands maps command prefixes that are only valid on Proxmox host
|
||||||
|
// targets (not LXCs or VMs). Running these against an lxc: or vm: target
|
||||||
|
// always fails with "command not found" and wastes a turn.
|
||||||
|
var hostOnlyCommands = map[string]bool{
|
||||||
|
"qm": true,
|
||||||
|
"pct": true,
|
||||||
|
"pvesh": true,
|
||||||
|
"iptables": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostLxcCommands maps command prefixes valid on host:* and lxc:* but not vm:*.
|
||||||
|
var hostLxcCommands = map[string]bool{
|
||||||
|
"systemctl": true,
|
||||||
|
"docker": true,
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostOnlyCommand checks whether the leading word of cmd is a host-only
|
||||||
|
// command. Returns the command word and true if the command can only run on
|
||||||
|
// a host: target.
|
||||||
|
func hostOnlyCommand(cmd string) (string, bool) {
|
||||||
|
trimmed := strings.TrimSpace(cmd)
|
||||||
|
parts := strings.Fields(trimmed)
|
||||||
|
if len(parts) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
first := parts[0]
|
||||||
|
// Check for shell wrappers: bash -c 'actual_cmd', sh -c 'actual_cmd'
|
||||||
|
if (first == "bash" || first == "sh") && len(parts) >= 3 && parts[1] == "-c" {
|
||||||
|
// The actual command is inside the -c argument; extract the first word.
|
||||||
|
// This handles `bash -c 'qm stop 100'` but not deeply nested wrappers.
|
||||||
|
actual := strings.Trim(strings.Join(parts[2:], " "), "'\"")
|
||||||
|
if inner := strings.Fields(actual); len(inner) > 0 {
|
||||||
|
first = inner[0]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Strip path: /usr/sbin/qm → qm
|
||||||
|
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||||
|
first = first[idx+1:]
|
||||||
|
}
|
||||||
|
return first, hostOnlyCommands[first]
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostLxcCommand checks whether the leading word of cmd is a command valid on
|
||||||
|
// host:* and lxc:* targets but not vm:*. Returns the command word and true if
|
||||||
|
// the command is restricted to host/lxc.
|
||||||
|
func hostLxcCommand(cmd string) (string, bool) {
|
||||||
|
trimmed := strings.TrimSpace(cmd)
|
||||||
|
parts := strings.Fields(trimmed)
|
||||||
|
if len(parts) == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
first := parts[0]
|
||||||
|
if idx := strings.LastIndexByte(first, '/'); idx >= 0 {
|
||||||
|
first = first[idx+1:]
|
||||||
|
}
|
||||||
|
return first, hostLxcCommands[first]
|
||||||
|
}
|
||||||
|
|
||||||
|
// validateCommandSyntax checks for common LLM-generated bash errors that always
|
||||||
|
// fail at the shell. Returns an error message or "" if the command looks valid.
|
||||||
|
func validateCommandSyntax(cmd string) string {
|
||||||
|
// Reject literal \n (the LLM sometimes writes `echo "---" && \n curl ...`
|
||||||
|
// — the \n is literal in the command string, not an actual newline).
|
||||||
|
if strings.Contains(cmd, "\\n") {
|
||||||
|
return fmt.Sprintf("Command contains literal '\\n' — use ';' or '&&' between commands, not a literal backslash-n. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reject `&& \n` patterns (the LLM writes `cmd1 && \n cmd2` — the \n is
|
||||||
|
// a literal newline that bash interprets as a command separator, but the
|
||||||
|
// leading backslash makes it a syntax error).
|
||||||
|
if andBackslashRe.MatchString(cmd) {
|
||||||
|
return fmt.Sprintf("Command contains '&&' followed by a literal backslash-newline — remove the backslash or use ';' instead. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Reject `\` at end of command with no continuation (last line ends with
|
||||||
|
// backslash but there's nothing after it).
|
||||||
|
trimmed := strings.TrimSpace(cmd)
|
||||||
|
if strings.HasSuffix(trimmed, "\\") {
|
||||||
|
return fmt.Sprintf("Command ends with a backslash but has nothing after it to continue. Remove the trailing '\\'. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Warn on common flag typos: `head - n`, `grep - i`, `tail - n`, etc.
|
||||||
|
// These are space-between-flag-and-value errors the LLM produces.
|
||||||
|
if flagSpaceRe.MatchString(cmd) {
|
||||||
|
return fmt.Sprintf("Command has a space between a flag and its value (e.g. 'head - n' instead of 'head -n'). Remove the space. Command: %q", cmd)
|
||||||
|
}
|
||||||
|
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
var andBackslashRe = regexp.MustCompile(`&&\s*\\\s*\n`)
|
||||||
|
var flagSpaceRe = regexp.MustCompile(`\b(head|tail|grep|sed|awk|sort|uniq|wc)\s+(-\w)\s+\w`)
|
||||||
|
|
||||||
// sessionHasPlan reports whether this nomos session has any plan step on
|
// sessionHasPlan reports whether this nomos session has any plan step on
|
||||||
// record (any generation, any status). Used by the P1 plan-first gate in
|
// record that isn't `replaced`. Replaced steps (from session reopen via
|
||||||
// classifyAndGate to refuse `run` before `propose_plan` has been called.
|
// store.reopenSession) don't count — the agent must propose fresh plan before
|
||||||
// A `replaced` step (from a prior plan generation that was superseded by a
|
// any `run`. Fails closed (returns true) when the query errors so a transient
|
||||||
// follow-up sub-task — see store.reopenSession) still counts: it proves the
|
// DB issue doesn't block an otherwise-valid run.
|
||||||
// agent once framed a plan for this session, and the reopen path guarantees a
|
|
||||||
// fresh `propose_plan` will run before the next `run` anyway. Fails closed
|
|
||||||
// (returns true) when the query errors so a transient DB issue doesn't block
|
|
||||||
// an otherwise-valid run.
|
|
||||||
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
|
func sessionHasPlan(ctx context.Context, pool *db.Pool, sessionID string) bool {
|
||||||
if sessionID == "" {
|
if sessionID == "" {
|
||||||
return true // no session → no gate (direct MCP call from a script)
|
return true // no session → no gate (direct MCP call from a script)
|
||||||
}
|
}
|
||||||
var count int
|
var count int
|
||||||
if err := pool.QueryRow(ctx,
|
if err := pool.QueryRow(ctx,
|
||||||
`SELECT COUNT(*) FROM session_plan_steps WHERE session_id = $1`,
|
`SELECT COUNT(*) FROM session_plan_steps
|
||||||
|
WHERE session_id = $1 AND status <> 'replaced'`,
|
||||||
sessionID).Scan(&count); err != nil {
|
sessionID).Scan(&count); err != nil {
|
||||||
return true // fail open on DB error — don't block work over a flake
|
return true // fail open on DB error — don't block work over a flake
|
||||||
}
|
}
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -16,7 +16,7 @@ import (
|
|||||||
// starts being populated when OIDC identity resolution lands.
|
// starts being populated when OIDC identity resolution lands.
|
||||||
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
||||||
action string, entityID *uuid.UUID, method, path, correlationID string,
|
action string, entityID *uuid.UUID, method, path, correlationID string,
|
||||||
detail map[string]any) error {
|
sessionID *uuid.UUID, detail map[string]any) error {
|
||||||
|
|
||||||
if detail == nil {
|
if detail == nil {
|
||||||
detail = map[string]any{}
|
detail = map[string]any{}
|
||||||
@@ -36,6 +36,7 @@ func Audit(ctx context.Context, q *sqlcgen.Queries, actorType, actorLabel,
|
|||||||
Path: &path,
|
Path: &path,
|
||||||
Detail: detailJSON,
|
Detail: detailJSON,
|
||||||
CorrelationID: corr,
|
CorrelationID: corr,
|
||||||
|
SessionID: sessionID,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -105,6 +105,15 @@ var curlLeadRe = regexp.MustCompile(`(?i)^curl\b`)
|
|||||||
// When any of these appears, the curl command is no longer read-only.
|
// When any of these appears, the curl command is no longer read-only.
|
||||||
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
|
var curlMutateRe = regexp.MustCompile(`(?i)(?:^|\s)-X\s+(?:post|put|delete|patch|connect|trace)\b|(?:^|\s)-(?:d|F|T|o)\b|(?:^|\s)--(?:data[-a-z]*|request|form|upload-file|output)\b`)
|
||||||
|
|
||||||
|
// curlDevNullOutRe matches curl output redirected to /dev/null in any of curl's
|
||||||
|
// argument forms (space, =, or attached). /dev/null is a no-op sink, so a GET
|
||||||
|
// that discards its body — the canonical reachability idiom
|
||||||
|
// `curl -o /dev/null -w '%{http_code}' URL` — is read-only. Output to any real
|
||||||
|
// path (-o /tmp/x) stays a potential mutation. Stripped before curlMutateRe so
|
||||||
|
// the remaining flags (-X, -d, ...) still classify correctly: a
|
||||||
|
// `curl -o /dev/null -X POST` stays config_mutation.
|
||||||
|
var curlDevNullOutRe = regexp.MustCompile(`(?i)(^|\s)-o\s*/dev/null(\s|$)|(^|\s)--output[=\s]\s*/dev/null(\s|$)`)
|
||||||
|
|
||||||
// redirectOutRe matches shell output redirection to a file (> or >> followed
|
// redirectOutRe matches shell output redirection to a file (> or >> followed
|
||||||
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
|
// by a path), but excludes the file-descriptor merge form `>&<digit>` (e.g.
|
||||||
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
|
// `2>&1`) which only rearranges streams and writes nothing to disk. RE2 has
|
||||||
@@ -308,6 +317,10 @@ func curlIsReadOnly(curlCmd string) bool {
|
|||||||
if !curlLeadRe.MatchString(curlCmd) {
|
if !curlLeadRe.MatchString(curlCmd) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
// -o /dev/null is a no-op sink: strip it before flag detection so the
|
||||||
|
// canonical GET-and-discard reachability probe stays read-only.
|
||||||
|
// A `curl -o /dev/null -X POST` still fails curlMutateRe after stripping.
|
||||||
|
curlCmd = curlDevNullOutRe.ReplaceAllString(curlCmd, " ")
|
||||||
if curlMutateRe.MatchString(curlCmd) {
|
if curlMutateRe.MatchString(curlCmd) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -102,6 +102,33 @@ func TestClassifyCommand_CurlPipeSh_ConfigMutation(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestClassifyCommand_CurlDevNull_ReadOnly(t *testing.T) {
|
||||||
|
// -o /dev/null is a no-op sink — the canonical GET-and-discard
|
||||||
|
// reachability idiom must stay read_only. Output to real paths stays
|
||||||
|
// config_mutation. POST/data flags after stripping still gate.
|
||||||
|
cases := []struct {
|
||||||
|
cmd string
|
||||||
|
cls string
|
||||||
|
}{
|
||||||
|
// read_only: GET with body discarded to /dev/null
|
||||||
|
{`curl -o /dev/null -w '%{http_code}' --connect-timeout 10 http://192.168.8.101:8123`, RiskReadOnly},
|
||||||
|
{`curl -sS -o /dev/null https://home.hubris.network`, RiskReadOnly},
|
||||||
|
{`curl --output /dev/null https://example.com`, RiskReadOnly},
|
||||||
|
{`curl -o /dev/null https://example.com`, RiskReadOnly},
|
||||||
|
{`curl -o/dev/null -w '%{http_code}' https://example.com`, RiskReadOnly},
|
||||||
|
// config_mutation: POST/data still caught after stripping devnull
|
||||||
|
{`curl -o /dev/null -X POST https://example.com`, RiskConfigMutation},
|
||||||
|
{`curl -o /dev/null -d '{"x":1}' https://example.com`, RiskConfigMutation},
|
||||||
|
// config_mutation: -o to real path stays config_mutation
|
||||||
|
{`curl -o /etc/caddy/Caddyfile http://example.com`, RiskConfigMutation},
|
||||||
|
}
|
||||||
|
for _, c := range cases {
|
||||||
|
if got := ClassifyCommand(c.cmd, ""); got != c.cls {
|
||||||
|
t.Errorf("ClassifyCommand(%q) = %q, want %q", c.cmd, got, c.cls)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
func TestClassifyCommand_DefaultEscalatesToConfigMutation(t *testing.T) {
|
||||||
cases := []string{
|
cases := []string{
|
||||||
"apt-get install -y nginx",
|
"apt-get install -y nginx",
|
||||||
|
|||||||
303
internal/remote/remote.go
Normal file
303
internal/remote/remote.go
Normal file
@@ -0,0 +1,303 @@
|
|||||||
|
// Package remote resolves how to execute a command on a target entity and
|
||||||
|
// turns a plain shell command into whatever must be sent over the SSH
|
||||||
|
// connection that reaches it.
|
||||||
|
//
|
||||||
|
// The canonical access model: a host or workstation is reached by direct SSH
|
||||||
|
// to its address; an LXC or VM is NEVER SSH'd into directly — it is reached
|
||||||
|
// through its owning Proxmox host via `pct exec` / `qm guest exec`. One SSH
|
||||||
|
// credential per host (the host's root key), no per-guest keys, sshd, or
|
||||||
|
// lan_ip required for execution. Network probes (http/ping) still hit a
|
||||||
|
// guest's lan_ip directly; only command execution host-hops.
|
||||||
|
//
|
||||||
|
// This is the single resolver shared by the scheduler's check execution and
|
||||||
|
// the MCP `run` tool. Previously they diverged — the scheduler SSHed guests
|
||||||
|
// directly (broken for headless/keyless/mesh-only guests), while MCP
|
||||||
|
// host-hopped (working). Keeping one path keeps them in lockstep.
|
||||||
|
package remote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// DefaultUser is the SSH user when an entity declares no ssh.user. The
|
||||||
|
// Proxmox hosts and their guests are all administered as root.
|
||||||
|
const DefaultUser = "root"
|
||||||
|
|
||||||
|
// ExecTarget is a resolved execution endpoint: the SSH address and user to
|
||||||
|
// connect to, plus Wrap, which rewrites a plain command for transport.
|
||||||
|
type ExecTarget struct {
|
||||||
|
Host string
|
||||||
|
User string
|
||||||
|
// Wrap turns a plain shell command into the form that must be sent over
|
||||||
|
// the SSH connection to this target: the identity function for a host,
|
||||||
|
// `pct exec <id> -- bash -c 'echo <b64> | base64 -d | bash'` for an LXC,
|
||||||
|
// the `qm guest exec` equivalent for a VM. The base64 round-trip keeps
|
||||||
|
// nested quoting identical across both guest kinds.
|
||||||
|
Wrap func(cmd string) string
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsGuest reports whether an entity type is reached via pct/qm exec through a
|
||||||
|
// Proxmox host rather than by direct SSH. docker-container is reached via its
|
||||||
|
// host's docker socket, not pct, so it is not a guest here.
|
||||||
|
func IsGuest(entityType string) bool {
|
||||||
|
return entityType == "lxc" || entityType == "vm"
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveHost resolves a `host:<slug>` to its reachable network address and
|
||||||
|
// SSH user. Address preference: lan_ip, then public_ipv4, then mesh IP, then
|
||||||
|
// mesh fqdn. Preferring public_ipv4 over mesh matters because the scheduler
|
||||||
|
// container has no mesh interface — a standalone-server with only a mesh IP
|
||||||
|
// (netbird-vps) was unreachable, and a public_ipv4 was sitting unused.
|
||||||
|
//
|
||||||
|
// fallbackUser is used when the entity declares no ssh.user; callers pass
|
||||||
|
// their configured default (the scheduler uses "root", the MCP run tool uses
|
||||||
|
// its configured OIKOS_SSH_USER).
|
||||||
|
func ResolveHost(ctx context.Context, pool *db.Pool, hostSlug, fallbackUser string) (addr, user string, err error) {
|
||||||
|
var raw string
|
||||||
|
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE slug = $1", hostSlug).Scan(&raw); err != nil {
|
||||||
|
return "", "", fmt.Errorf("entity not found: %s", hostSlug)
|
||||||
|
}
|
||||||
|
var m map[string]any
|
||||||
|
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||||
|
return "", "", fmt.Errorf("parse attributes for %s: %w", hostSlug, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if v, ok := m["lan_ip"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if v, ok := m["public_ipv4"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||||
|
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||||
|
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if addr == "" {
|
||||||
|
return "", "", fmt.Errorf("no IP found for %s", hostSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
user = fallbackUser
|
||||||
|
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||||
|
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||||
|
user = u
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return addr, user, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveProxmoxHostSlug resolves the Proxmox host slug that owns a guest.
|
||||||
|
// Resolution order: the hostAttr if non-empty (the entity's attributes.host,
|
||||||
|
// stored without the "host:" prefix), the `hosts` relationship on the guest
|
||||||
|
// (the canonical graph edge), then "hubris" as the documented default.
|
||||||
|
//
|
||||||
|
// entityID is the guest's entity id; the relationship lookup uses it
|
||||||
|
// directly rather than a slug subquery.
|
||||||
|
func ResolveProxmoxHostSlug(ctx context.Context, pool *db.Pool, entityID uuid.UUID, hostAttr string) string {
|
||||||
|
hostSlug := strings.TrimSpace(hostAttr)
|
||||||
|
// Only trust a clean token as a host name. The attribute is operator/
|
||||||
|
// agent-writable and has been polluted with prose before ("hubris
|
||||||
|
// (confirmed via pct config…)") — using that verbatim produces a slug that
|
||||||
|
// never resolves. Treat anything with whitespace or parens as invalid and
|
||||||
|
// fall back to the canonical `hosts` edge below.
|
||||||
|
if strings.ContainsAny(hostSlug, " \t()") {
|
||||||
|
hostSlug = ""
|
||||||
|
}
|
||||||
|
if hostSlug == "" {
|
||||||
|
var relHostSlug string
|
||||||
|
if err := pool.QueryRow(ctx, `
|
||||||
|
SELECT e.slug FROM relationships r
|
||||||
|
JOIN entities e ON e.id = r.source_id
|
||||||
|
WHERE r.target_id = $1
|
||||||
|
AND r.type = 'hosts' AND r.valid_to IS NULL
|
||||||
|
LIMIT 1`, entityID).Scan(&relHostSlug); err == nil && relHostSlug != "" {
|
||||||
|
hostSlug = relHostSlug
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if hostSlug == "" {
|
||||||
|
hostSlug = "hubris" // documented default Proxmox host when unset
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(hostSlug, "host:") {
|
||||||
|
hostSlug = "host:" + hostSlug
|
||||||
|
}
|
||||||
|
return hostSlug
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveExecTarget resolves a target slug (host:, lxc:, or vm:) to its
|
||||||
|
// execution endpoint. This is the slug-based entry used by the MCP `run` tool.
|
||||||
|
func ResolveExecTarget(ctx context.Context, pool *db.Pool, targetSlug, fallbackUser string) (ExecTarget, error) {
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(targetSlug, "host:"):
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, targetSlug, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||||
|
|
||||||
|
case strings.HasPrefix(targetSlug, "lxc:"), strings.HasPrefix(targetSlug, "vm:"):
|
||||||
|
var (
|
||||||
|
id uuid.UUID
|
||||||
|
pveID string
|
||||||
|
typ string
|
||||||
|
hostAttr string
|
||||||
|
)
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
"SELECT id, type, attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE slug = $1",
|
||||||
|
targetSlug).Scan(&id, &typ, &pveID, &hostAttr); err != nil || pveID == "" {
|
||||||
|
return ExecTarget{}, fmt.Errorf("guest not found or missing pve_id: %s", targetSlug)
|
||||||
|
}
|
||||||
|
hostSlug := ResolveProxmoxHostSlug(ctx, pool, id, hostAttr)
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(typ, pveID)}, nil
|
||||||
|
}
|
||||||
|
return ExecTarget{}, fmt.Errorf("unsupported target %q: must be host:<slug>, lxc:<slug>, or vm:<slug>", targetSlug)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ResolveExecTargetForCheck resolves an execution endpoint keyed by the
|
||||||
|
// target's id and type — the data the scheduler has at check-execution time
|
||||||
|
// (check_defs carry target_id + target_type, not a slug). Guests route via
|
||||||
|
// pct/qm exec; everything else (hosts, workstations, services resolved to
|
||||||
|
// their hosting machine) is reached by direct SSH to the entity's own address.
|
||||||
|
func ResolveExecTargetForCheck(ctx context.Context, pool *db.Pool, targetID uuid.UUID, targetType, fallbackUser string) (ExecTarget, error) {
|
||||||
|
if IsGuest(targetType) {
|
||||||
|
return resolveGuest(ctx, pool, targetID, targetType, fallbackUser)
|
||||||
|
}
|
||||||
|
|
||||||
|
// A service (or other non-compute target) has no address of its own — it
|
||||||
|
// runs on whatever compute entity provides/hosts it. Resolve that host and
|
||||||
|
// route through it: pct if the host is a guest, direct SSH (with the
|
||||||
|
// host's correct user) if it's a machine. Previously a service check baked
|
||||||
|
// its hosting LXC's lan_ip and SSHed it directly as root, which fails
|
||||||
|
// because the scheduler key isn't in each LXC — only on the Proxmox hosts.
|
||||||
|
if hostID, hostType, ok := hostingCompute(ctx, pool, targetID); ok {
|
||||||
|
if IsGuest(hostType) {
|
||||||
|
return resolveGuest(ctx, pool, hostID, hostType, fallbackUser)
|
||||||
|
}
|
||||||
|
addr, user, err := resolveHostByID(ctx, pool, hostID, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// No hosting entity found: reach the target directly at its own address
|
||||||
|
// (a host/workstation, or a service whose host wasn't resolvable).
|
||||||
|
addr, user, err := resolveHostByID(ctx, pool, targetID, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: func(cmd string) string { return cmd }}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// hostingCompute walks the provides/runs-on/hosts edges backward from a target
|
||||||
|
// to the compute entity that runs it (a service's LXC, an LXC's Proxmox host).
|
||||||
|
// Returns the host's id, type, and whether one was found. Most-specific edge
|
||||||
|
// first: provides names the runtime container directly.
|
||||||
|
func hostingCompute(ctx context.Context, pool *db.Pool, targetID uuid.UUID) (uuid.UUID, string, bool) {
|
||||||
|
var hid uuid.UUID
|
||||||
|
var htype string
|
||||||
|
err := pool.QueryRow(ctx, `
|
||||||
|
SELECT e.id, e.type FROM relationships r
|
||||||
|
JOIN entities e ON e.id = r.source_id
|
||||||
|
WHERE r.target_id = $1 AND r.valid_to IS NULL
|
||||||
|
AND r.type IN ('provides','runs-on','hosts')
|
||||||
|
ORDER BY CASE r.type WHEN 'provides' THEN 0 WHEN 'runs-on' THEN 1 ELSE 2 END
|
||||||
|
LIMIT 1`, targetID).Scan(&hid, &htype)
|
||||||
|
if err != nil {
|
||||||
|
return uuid.Nil, "", false
|
||||||
|
}
|
||||||
|
return hid, htype, true
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveGuest resolves a guest's execution endpoint: the owning Proxmox host
|
||||||
|
// (SSH'd directly) with a pct/qm exec wrapper around the command.
|
||||||
|
func resolveGuest(ctx context.Context, pool *db.Pool, guestID uuid.UUID, guestType, fallbackUser string) (ExecTarget, error) {
|
||||||
|
var pveID, hostAttr string
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||||
|
guestID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||||
|
return ExecTarget{}, fmt.Errorf("guest %s missing pve_id", guestID)
|
||||||
|
}
|
||||||
|
hostSlug := ResolveProxmoxHostSlug(ctx, pool, guestID, hostAttr)
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, hostSlug, fallbackUser)
|
||||||
|
if err != nil {
|
||||||
|
return ExecTarget{}, err
|
||||||
|
}
|
||||||
|
return ExecTarget{Host: addr, User: user, Wrap: guestWrap(guestType, pveID)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// resolveHostByID is ResolveHost keyed by entity id.
|
||||||
|
func resolveHostByID(ctx context.Context, pool *db.Pool, id uuid.UUID, fallbackUser string) (addr, user string, err error) {
|
||||||
|
var raw string
|
||||||
|
if err = pool.QueryRow(ctx, "SELECT attributes::text FROM entities WHERE id = $1", id).Scan(&raw); err != nil {
|
||||||
|
return "", "", fmt.Errorf("entity %s not found", id)
|
||||||
|
}
|
||||||
|
var m map[string]any
|
||||||
|
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||||
|
return "", "", fmt.Errorf("parse attributes: %w", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"lan_ip", "public_ipv4", "mesh_ip"} {
|
||||||
|
if v, ok := m[key].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if addr == "" {
|
||||||
|
if mesh, ok := m["mesh"].(map[string]any); ok {
|
||||||
|
if nb, ok := mesh["netbird"].(map[string]any); ok {
|
||||||
|
if v, ok := nb["ip"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
} else if v, ok := nb["fqdn"].(string); ok && v != "" {
|
||||||
|
addr = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if addr == "" {
|
||||||
|
return "", "", fmt.Errorf("no IP found for entity %s", id)
|
||||||
|
}
|
||||||
|
user = fallbackUser
|
||||||
|
if ssh, ok := m["ssh"].(map[string]any); ok {
|
||||||
|
if u, ok := ssh["user"].(string); ok && u != "" {
|
||||||
|
user = u
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if u, ok := m["user"].(string); ok && u != "" && user == fallbackUser {
|
||||||
|
// Workstations carry their login as a top-level `user` attribute
|
||||||
|
// (mac-mini: user: dtoro), not under ssh.user. Take it only when no
|
||||||
|
// explicit ssh.user was set, so a host that genuinely wants root still
|
||||||
|
// gets root.
|
||||||
|
user = u
|
||||||
|
}
|
||||||
|
return addr, user, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// guestWrap builds the pct/qm exec wrapper for a guest of the given type.
|
||||||
|
func guestWrap(entityType, pveID string) func(cmd string) string {
|
||||||
|
if entityType == "vm" {
|
||||||
|
return func(cmd string) string {
|
||||||
|
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||||
|
// `qm guest exec` returns JSON; pipe through jq for a clean stdout,
|
||||||
|
// falling back to the raw form. Mirrors the LXC base64 round-trip.
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash' | jq -r '.out // .err // empty' 2>/dev/null || qm guest exec %s -- /bin/bash -c 'echo %s | base64 -d | bash'",
|
||||||
|
pveID, b64, pveID, b64)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return func(cmd string) string {
|
||||||
|
b64 := base64.StdEncoding.EncodeToString([]byte(cmd))
|
||||||
|
return fmt.Sprintf("pct exec %s -- bash -c 'echo %s | base64 -d | bash'", pveID, b64)
|
||||||
|
}
|
||||||
|
}
|
||||||
203
internal/remote/remote_test.go
Normal file
203
internal/remote/remote_test.go
Normal file
@@ -0,0 +1,203 @@
|
|||||||
|
package remote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/base64"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/google/uuid"
|
||||||
|
)
|
||||||
|
|
||||||
|
// guestWrap and IsGuest are pure logic — always tested. The DB-backed
|
||||||
|
// resolvers are integration tests guarded by OIKOS_TEST_DATABASE_URL, the
|
||||||
|
// same convention as internal/scheduler/coverage_test.go.
|
||||||
|
|
||||||
|
func TestIsGuest(t *testing.T) {
|
||||||
|
cases := map[string]bool{
|
||||||
|
"lxc": true, "vm": true,
|
||||||
|
"proxmox-host": false, "workstation": false,
|
||||||
|
"service": false, "docker-container": false,
|
||||||
|
}
|
||||||
|
for typ, want := range cases {
|
||||||
|
if got := IsGuest(typ); got != want {
|
||||||
|
t.Errorf("IsGuest(%q) = %v, want %v", typ, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGuestWrapLXC(t *testing.T) {
|
||||||
|
w := guestWrap("lxc", "132")
|
||||||
|
out := w("/opt/oikos/checks/cpu_check.sh 'svc'")
|
||||||
|
if !strings.Contains(out, "pct exec 132 -- bash -c ") {
|
||||||
|
t.Fatalf("lxc wrap must use pct exec: %q", out)
|
||||||
|
}
|
||||||
|
if strings.Contains(out, "qm guest exec") {
|
||||||
|
t.Fatalf("lxc wrap must not use qm: %q", out)
|
||||||
|
}
|
||||||
|
// The base64 payload must round-trip to the original command.
|
||||||
|
i := strings.Index(out, "echo ")
|
||||||
|
j := strings.LastIndex(out, " | base64 -d | bash")
|
||||||
|
if i < 0 || j < 0 || j <= i {
|
||||||
|
t.Fatalf("cannot locate base64 payload in %q", out)
|
||||||
|
}
|
||||||
|
dec, err := base64.StdEncoding.DecodeString(out[i+len("echo ") : j])
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("decode payload: %v", err)
|
||||||
|
}
|
||||||
|
if string(dec) != "/opt/oikos/checks/cpu_check.sh 'svc'" {
|
||||||
|
t.Fatalf("round-trip mismatch: %q", string(dec))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGuestWrapVM(t *testing.T) {
|
||||||
|
w := guestWrap("vm", "108")
|
||||||
|
out := w("uname -a")
|
||||||
|
if !strings.Contains(out, "qm guest exec 108") {
|
||||||
|
t.Fatalf("vm wrap must use qm guest exec: %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetUnsupported(t *testing.T) {
|
||||||
|
// No DB needed: an unsupported slug prefix errors before any query.
|
||||||
|
if _, err := ResolveExecTarget(context.Background(), nil, "service:gitea", DefaultUser); err == nil {
|
||||||
|
t.Fatal("expected error for unsupported target prefix")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- integration tests (require a real Postgres) ---
|
||||||
|
|
||||||
|
func newRemotePool(t *testing.T) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
base := testDatabaseURL(t)
|
||||||
|
return createTestDB(t, base)
|
||||||
|
}
|
||||||
|
|
||||||
|
func testDatabaseURL(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
u := getenvOrDefault("OIKOS_TEST_DATABASE_URL", "")
|
||||||
|
if u == "" {
|
||||||
|
t.Skip("OIKOS_TEST_DATABASE_URL not set — skipping integration test")
|
||||||
|
}
|
||||||
|
return u
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveHostPrefersLAN(t *testing.T) {
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:x','proxmox-host','x','active','{"lan_ip":"10.0.0.1","public_ipv4":"1.2.3.4","mesh":{"netbird":{"ip":"100.64.0.1"}}}'::jsonb,1,now(),now())`,
|
||||||
|
uuid.New())
|
||||||
|
|
||||||
|
addr, user, err := ResolveHost(ctx, pool, "host:x", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveHost: %v", err)
|
||||||
|
}
|
||||||
|
if addr != "10.0.0.1" {
|
||||||
|
t.Errorf("addr = %q, want lan_ip 10.0.0.1", addr)
|
||||||
|
}
|
||||||
|
if user != "root" {
|
||||||
|
t.Errorf("user = %q, want root", user)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveHostFallsBackToPublicIPv4(t *testing.T) {
|
||||||
|
// netbird-vps: no lan_ip, has public_ipv4 + mesh ip. Must prefer
|
||||||
|
// public_ipv4 — the scheduler container has no mesh interface.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:vps','standalone-server','vps','active','{"public_ipv4":"82.165.190.79","mesh":{"netbird":{"ip":"100.122.165.149"}}}'::jsonb,1,now(),now())`,
|
||||||
|
uuid.New())
|
||||||
|
|
||||||
|
addr, _, err := ResolveHost(ctx, pool, "host:vps", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveHost: %v", err)
|
||||||
|
}
|
||||||
|
if addr != "82.165.190.79" {
|
||||||
|
t.Errorf("addr = %q, want public_ipv4 (mesh unreachable from container)", addr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetForCheckLXCRoutesViaHost(t *testing.T) {
|
||||||
|
// An LXC guest with a `hosts` edge to a proxmox host must resolve to the
|
||||||
|
// HOST's address (the host-hop target), wrapped as `pct exec`.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
hostID := uuid.New()
|
||||||
|
guestID := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'lxc:rclone','lxc','rclone','active','{"pve_id":"132"}'::jsonb,1,now(),now())`, guestID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at)
|
||||||
|
VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||||
|
|
||||||
|
et, err := ResolveExecTargetForCheck(ctx, pool, guestID, "lxc", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||||
|
}
|
||||||
|
if et.Host != "192.168.8.77" {
|
||||||
|
t.Errorf("Host = %q, want proxmox host lan_ip 192.168.8.77 (host-hop)", et.Host)
|
||||||
|
}
|
||||||
|
out := et.Wrap("/opt/oikos/checks/cpu_check.sh")
|
||||||
|
if !strings.Contains(out, "pct exec 132") {
|
||||||
|
t.Errorf("guest wrap must use pct exec 132, got %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetForCheckHostIsDirect(t *testing.T) {
|
||||||
|
// A host-like target resolves to its own address with identity wrap.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
hid := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'ws:mini','workstation','mini','active','{"lan_ip":"192.168.178.182","user":"dtoro"}'::jsonb,1,now(),now())`, hid)
|
||||||
|
|
||||||
|
et, err := ResolveExecTargetForCheck(ctx, pool, hid, "workstation", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveExecTargetForCheck: %v", err)
|
||||||
|
}
|
||||||
|
if et.Host != "192.168.178.182" {
|
||||||
|
t.Errorf("Host = %q, want 192.168.178.182", et.Host)
|
||||||
|
}
|
||||||
|
// Workstation's top-level `user` must be honored (the mac-mini fix).
|
||||||
|
if et.User != "dtoro" {
|
||||||
|
t.Errorf("User = %q, want dtoro (top-level user attr)", et.User)
|
||||||
|
}
|
||||||
|
if cmd := et.Wrap("uptime"); cmd != "uptime" {
|
||||||
|
t.Errorf("host wrap must be identity, got %q", cmd)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveExecTargetForCheckServiceRoutesViaHostingGuest(t *testing.T) {
|
||||||
|
// A service has no address of its own; it must route through its hosting
|
||||||
|
// LXC via the provides edge, host-hopping through the LXC's proxmox host.
|
||||||
|
pool := newRemotePool(t)
|
||||||
|
ctx := context.Background()
|
||||||
|
hostID := uuid.New()
|
||||||
|
guestID := uuid.New()
|
||||||
|
svcID := uuid.New()
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'host:hubris','proxmox-host','hubris','active','{"lan_ip":"192.168.8.77"}'::jsonb,1,now(),now())`, hostID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'lxc:gitea','lxc','gitea','active','{"pve_id":"104","lan_ip":"192.168.8.121"}'::jsonb,1,now(),now())`, guestID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO entities (id, slug, type, name, state, attributes, version, created_at, updated_at)
|
||||||
|
VALUES ($1,'service:gitea','service','gitea','active','{}'::jsonb,1,now(),now())`, svcID)
|
||||||
|
// provides: lxc -> service; hosts: proxmox-host -> lxc
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'provides',now(),now())`, guestID, svcID)
|
||||||
|
mustExec(t, pool, ctx, `INSERT INTO relationships (source_id, target_id, type, valid_from, created_at) VALUES ($1,$2,'hosts',now(),now())`, hostID, guestID)
|
||||||
|
|
||||||
|
et, err := ResolveExecTargetForCheck(ctx, pool, svcID, "service", DefaultUser)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ResolveExecTargetForCheck for service: %v", err)
|
||||||
|
}
|
||||||
|
// Reaches the proxmox host (host-hop), wrapped as pct exec into the guest.
|
||||||
|
if et.Host != "192.168.8.77" {
|
||||||
|
t.Errorf("Host = %q, want proxmox host 192.168.8.77 (via provides->hosts)", et.Host)
|
||||||
|
}
|
||||||
|
if out := et.Wrap("p"); !strings.Contains(out, "pct exec 104") {
|
||||||
|
t.Errorf("service check must wrap as pct exec 104, got %q", out)
|
||||||
|
}
|
||||||
|
}
|
||||||
68
internal/remote/testutil_test.go
Normal file
68
internal/remote/testutil_test.go
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
package remote
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"math/rand"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/dtoro/oikos/internal/db"
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
)
|
||||||
|
|
||||||
|
// createTestDB provisions a throwaway migrated database off baseURL, the same
|
||||||
|
// convention as internal/scheduler/coverage_test.go. The base URL must point
|
||||||
|
// at a Postgres superuser-capable connection.
|
||||||
|
func createTestDB(t *testing.T, baseURL string) *db.Pool {
|
||||||
|
t.Helper()
|
||||||
|
ctx := context.Background()
|
||||||
|
|
||||||
|
admin, err := pgx.Connect(ctx, baseURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect admin: %v", err)
|
||||||
|
}
|
||||||
|
dbName := fmt.Sprintf("oikos_rem_%08x", rand.Int63())
|
||||||
|
if _, err := admin.Exec(ctx, "CREATE DATABASE "+dbName); err != nil {
|
||||||
|
admin.Close(ctx)
|
||||||
|
t.Fatalf("create test db: %v", err)
|
||||||
|
}
|
||||||
|
admin.Close(ctx)
|
||||||
|
|
||||||
|
at := strings.LastIndex(baseURL, "/")
|
||||||
|
testURL := baseURL[:at+1] + dbName
|
||||||
|
if q := strings.Index(baseURL[at:], "?"); q >= 0 {
|
||||||
|
testURL += baseURL[at+q:]
|
||||||
|
}
|
||||||
|
|
||||||
|
pool, err := db.New(ctx, testURL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("connect test db: %v", err)
|
||||||
|
}
|
||||||
|
if err := pool.Migrate(ctx); err != nil {
|
||||||
|
t.Fatalf("migrate: %v", err)
|
||||||
|
}
|
||||||
|
t.Cleanup(func() {
|
||||||
|
pool.Close()
|
||||||
|
if admin, err := pgx.Connect(ctx, baseURL); err == nil {
|
||||||
|
admin.Exec(ctx, "DROP DATABASE IF EXISTS "+dbName+" WITH (FORCE)")
|
||||||
|
admin.Close(ctx)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return pool
|
||||||
|
}
|
||||||
|
|
||||||
|
func getenvOrDefault(key, def string) string {
|
||||||
|
if v := os.Getenv(key); v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustExec(t *testing.T, pool *db.Pool, ctx context.Context, q string, args ...any) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := pool.Exec(ctx, q, args...); err != nil {
|
||||||
|
t.Fatalf("exec %s: %v", q, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -85,7 +85,16 @@ func coverageSweep(ctx context.Context, pool *db.Pool) {
|
|||||||
case !mon.Declared:
|
case !mon.Declared:
|
||||||
undeclared++
|
undeclared++
|
||||||
case mon.None():
|
case mon.None():
|
||||||
// Explicitly unmonitorable. Nothing to say.
|
// Explicitly unmonitorable. Nothing to raise — but a type that
|
||||||
|
// USED to declare monitoring (e.g. dns-zone, [dns]→none) may have
|
||||||
|
// open `unmonitored` signals from before the change. They are no
|
||||||
|
// longer a gap, so close them; otherwise they linger forever,
|
||||||
|
// because resolveCoverageSignal only runs from the hasCheck path
|
||||||
|
// and a None() entity never gains a check.
|
||||||
|
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||||
|
resolved++
|
||||||
|
slog.Info("scheduler: type now unmonitorable, resolving stale signal", "entity", e.slug)
|
||||||
|
}
|
||||||
case e.hasCheck:
|
case e.hasCheck:
|
||||||
if resolveCoverageSignal(ctx, pool, e.id) {
|
if resolveCoverageSignal(ctx, pool, e.id) {
|
||||||
resolved++
|
resolved++
|
||||||
|
|||||||
25
internal/scheduler/reachability_test.go
Normal file
25
internal/scheduler/reachability_test.go
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
package scheduler
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// The scheduler runs in Docker on macOS, whose VM does not route ICMP to the
|
||||||
|
// LAN — every ping check reported "down" for hosts that were demonstrably up.
|
||||||
|
// tcpReachable is the fallback that keeps "is it reachable" answerable.
|
||||||
|
func TestTCPReachableAnswersWhenICMPCannot(t *testing.T) {
|
||||||
|
ctx := context.Background()
|
||||||
|
// localhost:22 is open on this machine (sshd), and port 1 is not.
|
||||||
|
if !tcpReachable(ctx, "127.0.0.1", 22, 3*time.Second) {
|
||||||
|
t.Skip("no sshd on localhost — cannot exercise the positive case")
|
||||||
|
}
|
||||||
|
if tcpReachable(ctx, "127.0.0.1", 1, 1*time.Second) {
|
||||||
|
t.Error("port 1 should not be reachable")
|
||||||
|
}
|
||||||
|
// Defaults to 22 when unset, which is what checkdefaults' ping configs use.
|
||||||
|
if !tcpReachable(ctx, "127.0.0.1", 0, 3*time.Second) {
|
||||||
|
t.Error("port 0 should default to 22")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -23,6 +23,7 @@ import (
|
|||||||
"github.com/dtoro/oikos/internal/db"
|
"github.com/dtoro/oikos/internal/db"
|
||||||
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
"github.com/dtoro/oikos/internal/db/sqlcgen"
|
||||||
"github.com/dtoro/oikos/internal/observability"
|
"github.com/dtoro/oikos/internal/observability"
|
||||||
|
"github.com/dtoro/oikos/internal/remote"
|
||||||
"github.com/google/uuid"
|
"github.com/google/uuid"
|
||||||
"golang.org/x/sync/errgroup"
|
"golang.org/x/sync/errgroup"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
@@ -111,12 +112,20 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
|||||||
q := sqlcgen.New(pool)
|
q := sqlcgen.New(pool)
|
||||||
start := time.Now()
|
start := time.Now()
|
||||||
|
|
||||||
result := executeCheck(ctx, cd)
|
result := executeCheck(ctx, pool, cd)
|
||||||
|
|
||||||
// Stamp the run before processing the result: due-ness must advance even
|
// Stamp the run before processing the result: due-ness must advance even
|
||||||
// when a check fails, or a permanently failing check would be re-run on
|
// when a check fails, or a permanently failing check would be re-run on
|
||||||
// every pass instead of at its declared interval.
|
// every pass instead of at its declared interval. last_health records THIS
|
||||||
if err := q.MarkCheckRun(ctx, cd.EntityID); err != nil {
|
// check's own verdict, which is what makes the aggregation below possible.
|
||||||
|
checkHealth := result.health
|
||||||
|
if checkHealth == "" {
|
||||||
|
checkHealth = "healthy"
|
||||||
|
}
|
||||||
|
if err := q.MarkCheckRun(ctx, sqlcgen.MarkCheckRunParams{
|
||||||
|
EntityID: cd.EntityID,
|
||||||
|
LastHealth: &checkHealth,
|
||||||
|
}); err != nil {
|
||||||
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
|
slog.Error("scheduler: mark check run", "entity", cd.EntitySlug, "error", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -150,16 +159,12 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
|||||||
|
|
||||||
if result.signalKind == "" || result.health == "healthy" {
|
if result.signalKind == "" || result.health == "healthy" {
|
||||||
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
resolveSignal(ctx, pool, cd.EntityID, targetID, cd.EntitySlug)
|
||||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
// NOT unconditionally "healthy": this check passing says nothing about
|
||||||
EntityID: targetID,
|
// the entity's other checks. Writing healthy here is what let one
|
||||||
Health: "healthy",
|
// passing probe erase a genuine failure reported by another — and,
|
||||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
// alternating with a failing probe, produced 226 health flips an hour
|
||||||
Details: []byte(`{}`),
|
// on a host that was fine throughout.
|
||||||
})
|
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||||
if prevHealth != "" && prevHealth != "healthy" {
|
|
||||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, "info",
|
|
||||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": "healthy"})
|
|
||||||
}
|
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -180,22 +185,52 @@ func runCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDef
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
|
||||||
EntityID: targetID,
|
|
||||||
Health: result.health,
|
|
||||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
|
||||||
Details: []byte(`{}`),
|
|
||||||
})
|
|
||||||
_ = sig
|
_ = sig
|
||||||
|
|
||||||
if prevHealth == "" || prevHealth == "healthy" {
|
if prevHealth == "" || prevHealth == "healthy" {
|
||||||
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
emitSchedulerEvent(ctx, pool, "signal.raised", targetID, severity,
|
||||||
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
map[string]any{"slug": cd.EntitySlug, "kind": result.signalKind, "evidence": result.evidence})
|
||||||
}
|
}
|
||||||
if prevHealth != result.health {
|
applyAggregateHealth(ctx, pool, q, targetID, cd.EntitySlug, prevHealth)
|
||||||
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
}
|
||||||
map[string]any{"slug": cd.EntitySlug, "from": prevHealth, "to": result.health})
|
|
||||||
|
// applyAggregateHealth sets the target's health to the worst verdict across
|
||||||
|
// all of its enabled checks, and emits health.changed only when that aggregate
|
||||||
|
// actually moves.
|
||||||
|
//
|
||||||
|
// Health is a property of the entity, but each check only ever observes one
|
||||||
|
// facet of it — reachability, disk, a systemd unit. Letting whichever check
|
||||||
|
// finished last overwrite the entity's health meant a host with six checks
|
||||||
|
// reported whichever facet was sampled most recently, so one failing probe and
|
||||||
|
// five passing ones oscillated forever instead of settling on "degraded".
|
||||||
|
func applyAggregateHealth(ctx context.Context, pool *db.Pool, q *sqlcgen.Queries,
|
||||||
|
targetID uuid.UUID, checkSlug, prevHealth string) {
|
||||||
|
|
||||||
|
health, err := q.WorstHealthForTarget(ctx, &targetID)
|
||||||
|
if err != nil {
|
||||||
|
slog.Error("scheduler: aggregate health", "entity", checkSlug, "error", err)
|
||||||
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
||||||
|
EntityID: targetID,
|
||||||
|
Health: health,
|
||||||
|
LastCheckAt: &[]time.Time{time.Now()}[0],
|
||||||
|
Details: []byte(`{}`),
|
||||||
|
})
|
||||||
|
|
||||||
|
if prevHealth == health {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
severity := "info"
|
||||||
|
switch health {
|
||||||
|
case "down":
|
||||||
|
severity = "critical"
|
||||||
|
case "degraded", "stale":
|
||||||
|
severity = "warning"
|
||||||
|
}
|
||||||
|
emitSchedulerEvent(ctx, pool, "health.changed", targetID, severity,
|
||||||
|
map[string]any{"slug": checkSlug, "from": prevHealth, "to": health})
|
||||||
}
|
}
|
||||||
|
|
||||||
// currentHealth reads the last recorded health for an entity, or "" if none.
|
// currentHealth reads the last recorded health for an entity, or "" if none.
|
||||||
@@ -217,7 +252,6 @@ func emitSchedulerEvent(ctx context.Context, pool *db.Pool, eventType string, en
|
|||||||
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
// checkID matches how signals are keyed (UpsertSignal uses the check's own
|
||||||
// entity id); targetID is the observed entity whose status this affects.
|
// entity id); targetID is the observed entity whose status this affects.
|
||||||
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UUID, slug string) {
|
||||||
q := sqlcgen.New(pool)
|
|
||||||
// Check if there's an open signal on this entity
|
// Check if there's an open signal on this entity
|
||||||
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
tag, err := pool.Exec(ctx, `UPDATE signals SET state = 'resolved', updated_at = now()
|
||||||
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
WHERE entity_id = $1 AND state = 'raised'`, checkID)
|
||||||
@@ -227,14 +261,12 @@ func resolveSignal(ctx context.Context, pool *db.Pool, checkID, targetID uuid.UU
|
|||||||
if tag.RowsAffected() > 0 {
|
if tag.RowsAffected() > 0 {
|
||||||
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
emitSchedulerEvent(ctx, pool, "signal.resolved", targetID, "info",
|
||||||
map[string]any{"slug": slug})
|
map[string]any{"slug": slug})
|
||||||
|
slog.Info("scheduler: signal resolved", "entity", slug)
|
||||||
}
|
}
|
||||||
_ = q.UpsertEntityStatus(ctx, sqlcgen.UpsertEntityStatusParams{
|
// Deliberately does NOT write health. Resolving THIS check's signal says
|
||||||
EntityID: targetID,
|
// nothing about the target's other checks; the caller re-derives health
|
||||||
Health: "healthy",
|
// from all of them. Forcing "healthy" here was a second path by which one
|
||||||
LastCheckAt: &[]time.Time{time.Now()}[0],
|
// passing probe erased another probe's genuine failure.
|
||||||
Details: []byte(`{}`),
|
|
||||||
})
|
|
||||||
slog.Info("scheduler: signal resolved", "entity", slug)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkResult bundles the outcome of a single check execution.
|
// checkResult bundles the outcome of a single check execution.
|
||||||
@@ -246,8 +278,10 @@ type checkResult struct {
|
|||||||
err error
|
err error
|
||||||
}
|
}
|
||||||
|
|
||||||
// executeCheck dispatches to the appropriate checker by kind.
|
// executeCheck dispatches to the appropriate checker by kind. pool is needed
|
||||||
func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
// by the ssh-script path, which resolves the target's execution endpoint
|
||||||
|
// (guests route through their Proxmox host; see internal/remote).
|
||||||
|
func executeCheck(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
switch cd.Kind {
|
switch cd.Kind {
|
||||||
case "http":
|
case "http":
|
||||||
return checkHTTP(ctx, cd)
|
return checkHTTP(ctx, cd)
|
||||||
@@ -257,12 +291,16 @@ func executeCheck(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) check
|
|||||||
return checkDisk(ctx, cd)
|
return checkDisk(ctx, cd)
|
||||||
case "cert-expiry":
|
case "cert-expiry":
|
||||||
return checkCertExpiry(ctx, cd)
|
return checkCertExpiry(ctx, cd)
|
||||||
|
case "vm-status":
|
||||||
|
return checkVMStatus(ctx, pool, cd)
|
||||||
case "ping":
|
case "ping":
|
||||||
return checkPing(ctx, cd)
|
return checkPing(ctx, cd)
|
||||||
case "ssh-script":
|
case "ssh-script":
|
||||||
return checkSSHScript(ctx, cd)
|
return checkSSHScript(ctx, pool, cd)
|
||||||
case "backup-freshness":
|
case "backup-freshness":
|
||||||
return checkBackupFreshness(ctx, cd)
|
return checkBackupFreshness(ctx, cd)
|
||||||
|
case "dns":
|
||||||
|
return checkDNS(ctx, cd)
|
||||||
default:
|
default:
|
||||||
return checkResult{health: "unknown"}
|
return checkResult{health: "unknown"}
|
||||||
}
|
}
|
||||||
@@ -448,6 +486,53 @@ func checkTCP(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResu
|
|||||||
return checkResult{health: "healthy"}
|
return checkResult{health: "healthy"}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// checkDNS verifies a DNS name resolves, catching a stale or unreachable
|
||||||
|
// zone. It looks up NS records first (a zone always has NS), falling back to
|
||||||
|
// an A/AAAA lookup for hostnames. Uses the system resolver; for split-horizon
|
||||||
|
// correctness reserve an explicit `server` in the config.
|
||||||
|
func checkDNS(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
|
cfg := struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Server string `json:"server"`
|
||||||
|
}{}
|
||||||
|
if len(cd.Config) > 0 {
|
||||||
|
_ = json.Unmarshal(cd.Config, &cfg)
|
||||||
|
}
|
||||||
|
if cfg.Name == "" {
|
||||||
|
return checkResult{health: "healthy"}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Resolve via an explicit server when supplied (split-horizon), else the
|
||||||
|
// system default resolver.
|
||||||
|
lookup := func(q string) (int, error) {
|
||||||
|
r := &net.Resolver{}
|
||||||
|
if cfg.Server != "" {
|
||||||
|
r = &net.Resolver{PreferGo: true, Dial: func(ctx context.Context, network, _ string) (net.Conn, error) {
|
||||||
|
d := net.Dialer{Timeout: 5 * time.Second}
|
||||||
|
return d.DialContext(ctx, network, net.JoinHostPort(cfg.Server, "53"))
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
ns, err := r.LookupNS(ctx, q)
|
||||||
|
if err == nil && len(ns) > 0 {
|
||||||
|
return len(ns), nil
|
||||||
|
}
|
||||||
|
addrs, err2 := r.LookupHost(ctx, q)
|
||||||
|
return len(addrs), err2
|
||||||
|
}
|
||||||
|
|
||||||
|
n, err := lookup(cfg.Name)
|
||||||
|
if err != nil || n == 0 {
|
||||||
|
return checkResult{
|
||||||
|
health: "down", signalKind: "dns",
|
||||||
|
evidence: fmt.Sprintf("DNS resolution failed for %q: %v", cfg.Name, err),
|
||||||
|
err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return checkResult{health: "healthy"}
|
||||||
|
}
|
||||||
|
|
||||||
// checkDisk performs a disk usage check.
|
// checkDisk performs a disk usage check.
|
||||||
func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
@@ -502,6 +587,12 @@ func checkDisk(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
|
// Dial is an optional explicit dial address (the TLS terminator's IP)
|
||||||
|
// for when the hostname doesn't resolve/reach from the scheduler — the
|
||||||
|
// container has no mesh interface and the host resolver doesn't know
|
||||||
|
// the split-horizon zone, so *.hubris.network dials Caddy's lab IP
|
||||||
|
// directly while SNI/cert-read still uses Host.
|
||||||
|
Dial string `json:"dial"`
|
||||||
Port int `json:"port"`
|
Port int `json:"port"`
|
||||||
WarnDays int `json:"warn_days"`
|
WarnDays int `json:"warn_days"`
|
||||||
CritDays int `json:"crit_days"`
|
CritDays int `json:"crit_days"`
|
||||||
@@ -522,9 +613,13 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
|||||||
timeout = 30 * time.Second
|
timeout = 30 * time.Second
|
||||||
}
|
}
|
||||||
|
|
||||||
addr := net.JoinHostPort(cfg.Host, fmt.Sprintf("%d", cfg.Port))
|
dialHost := cfg.Host
|
||||||
|
if cfg.Dial != "" {
|
||||||
|
dialHost = cfg.Dial
|
||||||
|
}
|
||||||
|
addr := net.JoinHostPort(dialHost, fmt.Sprintf("%d", cfg.Port))
|
||||||
|
|
||||||
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true}}
|
d := tls.Dialer{Config: &tls.Config{InsecureSkipVerify: true, ServerName: cfg.Host}}
|
||||||
conn, err := d.DialContext(ctx, "tcp", addr)
|
conn, err := d.DialContext(ctx, "tcp", addr)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
@@ -569,11 +664,61 @@ func checkCertExpiry(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) ch
|
|||||||
return checkResult{health: "healthy", metrics: metrics}
|
return checkResult{health: "healthy", metrics: metrics}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// checkVMStatus reports whether a VM is powered on, via `qm status <pve_id>`
|
||||||
|
// run on its Proxmox host. This is the right reachability probe for a VM that
|
||||||
|
// blocks ICMP (haos) and has no guest agent: it doesn't need the VM's network
|
||||||
|
// at all — "status: running" means the VM is up. The command runs on the host
|
||||||
|
// (not inside the VM), so it uses the host's address with identity wrap.
|
||||||
|
func checkVMStatus(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
|
if cd.TargetID == nil {
|
||||||
|
return checkResult{health: "unknown", evidence: "vm-status needs a target VM"}
|
||||||
|
}
|
||||||
|
var pveID, hostAttr string
|
||||||
|
if err := pool.QueryRow(ctx,
|
||||||
|
"SELECT attributes->>'pve_id', COALESCE(attributes->>'host','') FROM entities WHERE id = $1",
|
||||||
|
*cd.TargetID).Scan(&pveID, &hostAttr); err != nil || pveID == "" {
|
||||||
|
return checkResult{health: "unknown", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("vm %s has no pve_id", cd.EntitySlug)}
|
||||||
|
}
|
||||||
|
hostSlug := remote.ResolveProxmoxHostSlug(ctx, pool, *cd.TargetID, hostAttr)
|
||||||
|
addr, user, err := remote.ResolveHost(ctx, pool, hostSlug, sshUser)
|
||||||
|
if err != nil {
|
||||||
|
return checkResult{health: "down", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("resolve proxmox host for %s: %v", cd.EntitySlug, err), err: err}
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := time.Duration(cd.TimeoutS) * time.Second
|
||||||
|
if timeout <= 0 {
|
||||||
|
timeout = 15 * time.Second
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
out, err := sshExec(ctx, addr, "22", user, "qm status "+pveID, timeout)
|
||||||
|
if err != nil {
|
||||||
|
return checkResult{health: "down", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("qm status %s on %s: %v", pveID, addr, err), err: err}
|
||||||
|
}
|
||||||
|
// `qm status <id>` prints "status: running" (or stopped/paused).
|
||||||
|
if strings.Contains(string(out), "status: running") {
|
||||||
|
return checkResult{health: "healthy"}
|
||||||
|
}
|
||||||
|
trimmed := strings.TrimSpace(string(out))
|
||||||
|
if trimmed == "" {
|
||||||
|
trimmed = "(no output)"
|
||||||
|
}
|
||||||
|
return checkResult{health: "down", signalKind: "vm-status",
|
||||||
|
evidence: fmt.Sprintf("%s not running: %s", cd.EntitySlug, trimmed)}
|
||||||
|
}
|
||||||
|
|
||||||
// checkPing performs an ICMP ping check using the system ping command.
|
// checkPing performs an ICMP ping check using the system ping command.
|
||||||
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Count int `json:"count"`
|
Count int `json:"count"`
|
||||||
|
// Port for the TCP fallback below. Defaults to 22; set it for hosts
|
||||||
|
// that answer on something else (a Home Assistant VM has no sshd).
|
||||||
|
Port int `json:"port"`
|
||||||
}{}
|
}{}
|
||||||
if len(cd.Config) > 0 {
|
if len(cd.Config) > 0 {
|
||||||
_ = json.Unmarshal(cd.Config, &cfg)
|
_ = json.Unmarshal(cd.Config, &cfg)
|
||||||
@@ -607,9 +752,26 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
|
|
||||||
output, err := cmd.Output()
|
output, err := cmd.Output()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
// ICMP failing does not mean the host is down — it may mean ICMP is
|
||||||
|
// simply unavailable from here. On this deployment the scheduler runs
|
||||||
|
// in Docker on macOS, whose VM network stack does not route ICMP to
|
||||||
|
// the LAN: loopback pings succeed, every LAN ping fails, and all seven
|
||||||
|
// ping checks reported "down" for hosts that were demonstrably up
|
||||||
|
// (including the Docker host itself). Under health aggregation that one
|
||||||
|
// broken probe was enough to drag each entity to down.
|
||||||
|
//
|
||||||
|
// The question this check exists to answer is "is it reachable", and
|
||||||
|
// ICMP is only one way to ask. Fall back to a TCP connect before
|
||||||
|
// concluding anything.
|
||||||
|
if tcpReachable(ctx, cfg.Host, cfg.Port, timeout) {
|
||||||
|
return checkResult{
|
||||||
|
health: "healthy",
|
||||||
|
metrics: map[string]float64{},
|
||||||
|
}
|
||||||
|
}
|
||||||
return checkResult{
|
return checkResult{
|
||||||
health: "down", signalKind: "ping",
|
health: "down", signalKind: "ping",
|
||||||
evidence: fmt.Sprintf("ping %s: %v", cfg.Host, err),
|
evidence: fmt.Sprintf("no ICMP or TCP response from %s: %v", cfg.Host, err),
|
||||||
err: err,
|
err: err,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -623,6 +785,24 @@ func checkPing(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkRes
|
|||||||
return checkResult{health: "healthy", metrics: metrics}
|
return checkResult{health: "healthy", metrics: metrics}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// tcpReachable reports whether a TCP handshake completes, used as the
|
||||||
|
// reachability answer when ICMP is unavailable rather than unanswered.
|
||||||
|
func tcpReachable(ctx context.Context, host string, port int, timeout time.Duration) bool {
|
||||||
|
if port == 0 {
|
||||||
|
port = 22
|
||||||
|
}
|
||||||
|
if timeout <= 0 {
|
||||||
|
timeout = 5 * time.Second
|
||||||
|
}
|
||||||
|
d := net.Dialer{Timeout: timeout}
|
||||||
|
conn, err := d.DialContext(ctx, "tcp", net.JoinHostPort(host, strconv.Itoa(port)))
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
conn.Close()
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
var pingRttRe = regexp.MustCompile(`(?:rtt\s+min\/avg\/max\/mdev|round-trip\s+min\/avg\/max\/stddev)\s*=\s*[\d.]+\/([\d.]+)\/`)
|
||||||
|
|
||||||
func parsePingLatency(output []byte) float64 {
|
func parsePingLatency(output []byte) float64 {
|
||||||
@@ -637,8 +817,16 @@ func parsePingLatency(output []byte) float64 {
|
|||||||
return val
|
return val
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkSSHScript executes an allowlisted script on a remote host via SSH.
|
// checkSSHScript executes an allowlisted script on a remote target via SSH.
|
||||||
func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
//
|
||||||
|
// Routing follows the canonical access model (internal/remote): an LXC or VM
|
||||||
|
// is NEVER SSH'd into directly — it is reached through its Proxmox host via
|
||||||
|
// `pct exec`/`qm guest exec`, so a guest needs no lan_ip, sshd, or authorized
|
||||||
|
// key of its own. Hosts and workstations are reached by direct SSH, resolved
|
||||||
|
// live so a workstation's login (mac-mini: `user: dtoro`) is honored without
|
||||||
|
// a re-seed. Services and other entities fall back to the host address baked
|
||||||
|
// into check config at seed time (their hosting container's address).
|
||||||
|
func checkSSHScript(ctx context.Context, pool *db.Pool, cd sqlcgen.ListEnabledCheckDefsRow) checkResult {
|
||||||
cfg := struct {
|
cfg := struct {
|
||||||
Host string `json:"host"`
|
Host string `json:"host"`
|
||||||
Port int `json:"port"`
|
Port int `json:"port"`
|
||||||
@@ -656,12 +844,6 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
|||||||
if cfg.Host == "" || cfg.Script == "" {
|
if cfg.Host == "" || cfg.Script == "" {
|
||||||
return checkResult{health: "healthy"}
|
return checkResult{health: "healthy"}
|
||||||
}
|
}
|
||||||
if cfg.Port == 0 {
|
|
||||||
cfg.Port = 22
|
|
||||||
}
|
|
||||||
if cfg.User == "" {
|
|
||||||
cfg.User = sshUser
|
|
||||||
}
|
|
||||||
|
|
||||||
if !allowlistedScript(cfg.Script) {
|
if !allowlistedScript(cfg.Script) {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
@@ -684,13 +866,51 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
|||||||
// the remote command. The script name itself is allowlisted above.
|
// the remote command. The script name itself is allowlisted above.
|
||||||
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
|
scriptPath += " '" + strings.ReplaceAll(cfg.Args, "'", `'\''`) + "'"
|
||||||
}
|
}
|
||||||
port := strconv.Itoa(cfg.Port)
|
|
||||||
|
|
||||||
output, err := sshExec(ctx, cfg.Host, port, cfg.User, scriptPath, timeout)
|
// Resolve the execution endpoint. The resolver handles every target kind:
|
||||||
|
// LXC/VM host-hop via pct/qm exec; hosts/workstations direct at their own
|
||||||
|
// address; services route through their hosting compute entity (via the
|
||||||
|
// provides edge) so a service check reaches the right machine with the
|
||||||
|
// right user instead of baking an LXC lan_ip and SSHing it as root.
|
||||||
|
host, port, user := cfg.Host, strconv.Itoa(oru(cfg.Port, 22)), orStr(cfg.User, sshUser)
|
||||||
|
wrap := func(cmd string) string { return cmd }
|
||||||
|
targetType := ""
|
||||||
|
if cd.TargetType != nil {
|
||||||
|
targetType = *cd.TargetType
|
||||||
|
}
|
||||||
|
// target_type was omitted by older writeCheck inserts, so resolve it from
|
||||||
|
// the target entity when the column is blank — otherwise the guest routing
|
||||||
|
// below (IsGuest) never triggers and a guest check falls back to its baked
|
||||||
|
// (often mesh-only) address.
|
||||||
|
if targetType == "" && cd.TargetID != nil {
|
||||||
|
if err := pool.QueryRow(ctx, "SELECT type FROM entities WHERE id = $1", *cd.TargetID).Scan(&targetType); err != nil {
|
||||||
|
targetType = ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if cd.TargetID != nil && targetType != "" {
|
||||||
|
et, err := remote.ResolveExecTargetForCheck(ctx, pool, *cd.TargetID, targetType, sshUser)
|
||||||
|
if err != nil {
|
||||||
|
if remote.IsGuest(targetType) {
|
||||||
|
return checkResult{
|
||||||
|
health: "down", signalKind: "ssh-script",
|
||||||
|
evidence: fmt.Sprintf("route guest %s: %v", cd.EntitySlug, err), err: err,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// Non-guest: log the resolution failure so an opaque ssh "down"
|
||||||
|
// doesn't hide that the real cause was host/user resolution, then
|
||||||
|
// fall back to the baked config below.
|
||||||
|
slog.Warn("scheduler: target resolution failed, using baked config",
|
||||||
|
"entity", cd.EntitySlug, "target_type", targetType, "error", err)
|
||||||
|
} else {
|
||||||
|
host, port, user, wrap = et.Host, "22", et.User, et.Wrap
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := sshExec(ctx, host, port, user, wrap(scriptPath), timeout)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return checkResult{
|
return checkResult{
|
||||||
health: "down", signalKind: "ssh-script",
|
health: "down", signalKind: "ssh-script",
|
||||||
evidence: fmt.Sprintf("ssh %s:%s %s: %v", cfg.Host, strconv.Itoa(cfg.Port), cfg.Script, err),
|
evidence: fmt.Sprintf("ssh %s:%s %s: %v", host, port, cfg.Script, err),
|
||||||
err: err,
|
err: err,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -728,6 +948,20 @@ func checkSSHScript(ctx context.Context, cd sqlcgen.ListEnabledCheckDefsRow) che
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// oru returns v when nonzero, else def. orStr returns v when non-empty, else def.
|
||||||
|
func oru(v, def int) int {
|
||||||
|
if v != 0 {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
func orStr(v, def string) string {
|
||||||
|
if v != "" {
|
||||||
|
return v
|
||||||
|
}
|
||||||
|
return def
|
||||||
|
}
|
||||||
|
|
||||||
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
var scriptNameRe = regexp.MustCompile(`^[a-z][a-z0-9_-]+\.sh$`)
|
||||||
|
|
||||||
func allowlistedScript(name string) bool {
|
func allowlistedScript(name string) bool {
|
||||||
|
|||||||
28
migrations/027_check_last_health.up.sql
Normal file
28
migrations/027_check_last_health.up.sql
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
-- 027_check_last_health.up.sql
|
||||||
|
-- Aggregate an entity's health across its checks instead of last-writer-wins.
|
||||||
|
--
|
||||||
|
-- runCheck wrote entity_status.health on every check completion, so an
|
||||||
|
-- entity's health was simply whichever of its checks finished most recently.
|
||||||
|
-- host:hubris has 6 checks, host:strong 6 — one failing probe alternating with
|
||||||
|
-- five passing ones produced a permanent flap: 226 health.changed events for
|
||||||
|
-- host:strong in a single hour, oscillating down/healthy, while the host was
|
||||||
|
-- fine the whole time.
|
||||||
|
--
|
||||||
|
-- On this fleet the trigger is a known false positive: the scheduler's network
|
||||||
|
-- vantage point cannot ICMP host:strong, so its ping check fails while every
|
||||||
|
-- ssh-script check succeeds. Under last-writer-wins that one probe was enough
|
||||||
|
-- to declare the whole host down, twice a minute.
|
||||||
|
--
|
||||||
|
-- Storing each check's own verdict lets entity health be derived as the worst
|
||||||
|
-- current result across that entity's enabled checks — so a single failing
|
||||||
|
-- probe degrades the entity honestly without erasing what the other five say,
|
||||||
|
-- and a passing probe cannot mask a genuine failure elsewhere.
|
||||||
|
|
||||||
|
ALTER TABLE check_defs ADD COLUMN IF NOT EXISTS last_health TEXT;
|
||||||
|
|
||||||
|
COMMENT ON COLUMN check_defs.last_health IS 'This check''s own most recent verdict (healthy/degraded/down/unknown). entity_status.health is the worst of these across the target''s enabled checks.';
|
||||||
|
|
||||||
|
-- The aggregation reads every enabled check for one target on each completion.
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_check_defs_target_health
|
||||||
|
ON check_defs (target_id)
|
||||||
|
WHERE enabled AND target_id IS NOT NULL;
|
||||||
75
migrations/028_relationship_blast_direction.up.sql
Normal file
75
migrations/028_relationship_blast_direction.up.sql
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
-- 028_relationship_blast_direction.up.sql
|
||||||
|
-- Make blast_radius answer the question it is named after.
|
||||||
|
--
|
||||||
|
-- blast_radius walked source_id -> target_id for every relationship type. But
|
||||||
|
-- which end of an edge is the DEPENDENT differs per type:
|
||||||
|
--
|
||||||
|
-- machine --hosts--> container if the machine dies, the container dies
|
||||||
|
-- -> dependent is the TARGET (forward)
|
||||||
|
-- service --depends-on--> service if the target dies, the SOURCE breaks
|
||||||
|
-- -> dependent is the SOURCE (backward)
|
||||||
|
-- ingress --routes-to--> service if the service dies, the route 502s
|
||||||
|
-- -> dependent is the SOURCE (backward)
|
||||||
|
-- document --documents--> entity neither breaks the other
|
||||||
|
-- -> no runtime dependency at all
|
||||||
|
--
|
||||||
|
-- Walking everything forwards meant the answer was right for `hosts` and
|
||||||
|
-- `provides` and wrong for every backward edge, while `documents`, `involves`
|
||||||
|
-- and `targets` (2,800+ edges of pure bookkeeping) polluted the result with
|
||||||
|
-- tasks and executions that cannot "break".
|
||||||
|
--
|
||||||
|
-- Direction is therefore a property of the relationship type, declared in
|
||||||
|
-- seeds/ontology.yaml — the same shape as the `monitoring:` declaration on
|
||||||
|
-- entity types.
|
||||||
|
--
|
||||||
|
-- forward : if the SOURCE fails, the TARGET is affected
|
||||||
|
-- backward : if the TARGET fails, the SOURCE is affected
|
||||||
|
-- none : no runtime dependency (default — bookkeeping and documentation)
|
||||||
|
--
|
||||||
|
-- Defaulting to 'none' is deliberate: an undeclared edge contributes nothing
|
||||||
|
-- rather than silently producing a wrong answer, which is how the old
|
||||||
|
-- everything-is-forward behaviour went unnoticed.
|
||||||
|
|
||||||
|
ALTER TABLE relationship_types
|
||||||
|
ADD COLUMN IF NOT EXISTS blast_direction TEXT NOT NULL DEFAULT 'none'
|
||||||
|
CHECK (blast_direction IN ('forward', 'backward', 'none'));
|
||||||
|
|
||||||
|
COMMENT ON COLUMN relationship_types.blast_direction IS
|
||||||
|
'Which end of this edge depends on the other. forward = target depends on source. backward = source depends on target. none = no runtime dependency. Drives blast_radius().';
|
||||||
|
|
||||||
|
-- Walk the dependency graph in the direction each edge type declares.
|
||||||
|
--
|
||||||
|
-- Returns everything that is affected when start_id fails, with the number of
|
||||||
|
-- hops. Cycles are guarded by the path array, as before.
|
||||||
|
CREATE OR REPLACE FUNCTION blast_radius(start_id UUID, max_depth INT DEFAULT 3,
|
||||||
|
rel_types TEXT[] DEFAULT NULL)
|
||||||
|
RETURNS TABLE(entity_id UUID, depth INT) AS $$
|
||||||
|
WITH RECURSIVE walk AS (
|
||||||
|
SELECT start_id AS entity_id, 0 AS depth, ARRAY[start_id] AS path
|
||||||
|
UNION ALL
|
||||||
|
SELECT next_id, w.depth + 1, w.path || next_id
|
||||||
|
FROM walk w
|
||||||
|
JOIN LATERAL (
|
||||||
|
-- forward: this entity is the source, so the target depends on it
|
||||||
|
SELECT r.target_id AS next_id
|
||||||
|
FROM relationships r
|
||||||
|
JOIN relationship_types rt ON rt.name = r.type
|
||||||
|
WHERE r.source_id = w.entity_id
|
||||||
|
AND r.valid_to IS NULL
|
||||||
|
AND rt.blast_direction = 'forward'
|
||||||
|
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||||
|
UNION ALL
|
||||||
|
-- backward: this entity is the target, so the source depends on it
|
||||||
|
SELECT r.source_id AS next_id
|
||||||
|
FROM relationships r
|
||||||
|
JOIN relationship_types rt ON rt.name = r.type
|
||||||
|
WHERE r.target_id = w.entity_id
|
||||||
|
AND r.valid_to IS NULL
|
||||||
|
AND rt.blast_direction = 'backward'
|
||||||
|
AND (rel_types IS NULL OR r.type = ANY(rel_types))
|
||||||
|
) nxt ON TRUE
|
||||||
|
WHERE w.depth < LEAST(max_depth, 5)
|
||||||
|
AND NOT nxt.next_id = ANY(w.path)
|
||||||
|
)
|
||||||
|
SELECT entity_id, MIN(depth) FROM walk GROUP BY entity_id;
|
||||||
|
$$ LANGUAGE sql STABLE;
|
||||||
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
33
migrations/029_plan_generation_relative_seq.up.sql
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
-- 029_plan_generation_relative_seq.up.sql
|
||||||
|
-- Make plan-step seq generation-relative: 1..N within each
|
||||||
|
-- (session_id, generation). Before this, seq was globally increasing across
|
||||||
|
-- generations (gen1: 1..6, gen2: 7..12), so the model's 1-based
|
||||||
|
-- update_plan_step calls — which the prompt and schema explicitly tell it to
|
||||||
|
-- use — landed on superseded gen-1 rows after a re-plan while the live gen-2
|
||||||
|
-- work went unrecorded (or, worse, resurrected a `replaced` row as `done`).
|
||||||
|
-- The addressing key is now (session_id, generation, seq); updatePlanStep
|
||||||
|
-- resolves against MAX(generation), so a 1-based seq always maps to the
|
||||||
|
-- CURRENT plan. See plans/2026-07-30-session-review-plan-drift-and-dead-
|
||||||
|
-- activity-panel.md P0.1.
|
||||||
|
|
||||||
|
-- Renumber existing rows so seq resets to 1..N per (session, generation),
|
||||||
|
-- preserving each generation's step order.
|
||||||
|
WITH ranked AS (
|
||||||
|
SELECT id,
|
||||||
|
ROW_NUMBER() OVER (
|
||||||
|
PARTITION BY session_id, generation
|
||||||
|
ORDER BY seq, created_at
|
||||||
|
) AS new_seq
|
||||||
|
FROM session_plan_steps
|
||||||
|
)
|
||||||
|
UPDATE session_plan_steps s
|
||||||
|
SET seq = ranked.new_seq
|
||||||
|
FROM ranked
|
||||||
|
WHERE s.id = ranked.id AND s.seq <> ranked.new_seq;
|
||||||
|
|
||||||
|
-- (session_id, seq) is no longer unique once seq resets per generation; the
|
||||||
|
-- store resolves via (session_id, generation, seq). Drop the old composite
|
||||||
|
-- index (it now collides on seq) and add the generation-scoped unique index.
|
||||||
|
DROP INDEX IF EXISTS idx_plan_steps_session;
|
||||||
|
CREATE UNIQUE INDEX IF NOT EXISTS idx_plan_steps_session_gen_seq
|
||||||
|
ON session_plan_steps (session_id, generation, seq);
|
||||||
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
8
migrations/030_plan_step_replaced_reason.up.sql
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
-- 030_plan_step_replaced_reason.up.sql
|
||||||
|
-- Add replaced_reason to session_plan_steps so the agent must explain why
|
||||||
|
-- a step was replaced (wrong_diagnosis, scope_change, blocked, superseded,
|
||||||
|
-- operator_override) rather than silently replacing entire plans. The column
|
||||||
|
-- is also set by bulk-replace operations (proposePlan, setGoal, reopenSession)
|
||||||
|
-- for auditability.
|
||||||
|
|
||||||
|
ALTER TABLE session_plan_steps ADD COLUMN IF NOT EXISTS replaced_reason TEXT;
|
||||||
@@ -47,6 +47,24 @@ Read-only commands auto-run (no approval). Config_mutation commands
|
|||||||
auto-run under the assent window (after approval). Destructive commands
|
auto-run under the assent window (after approval). Destructive commands
|
||||||
always need explicit typed confirmation.
|
always need explicit typed confirmation.
|
||||||
|
|
||||||
|
**Never mark a step `done` if its tool calls errored.** If `run` timed out,
|
||||||
|
`update_entity_attributes` returned "not found", `create_relationship` returned
|
||||||
|
"source entity not found", or any tool returned an error — the step is NOT done.
|
||||||
|
Diagnose the error, try an alternative (e.g. use `create_entity` when
|
||||||
|
`update_entity_attributes` reports the entity doesn't exist), and only advance
|
||||||
|
to `done` when the step's intended work actually completed. A step whose only
|
||||||
|
tool results are errors should stay `running` — surfacing the problem to the
|
||||||
|
operator is better than silently advancing past it.
|
||||||
|
|
||||||
|
**Complete or skip steps — don't replace silently.** Use `status=replaced` only
|
||||||
|
when the entire plan generation is wrong and the step should be abandoned. When
|
||||||
|
you replace a step, provide `replaced_reason` with the cause
|
||||||
|
(`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`).
|
||||||
|
Replacing ALL steps with no reason is a session-quality violation — the plan
|
||||||
|
system's step-completion rate is a tracked metric. Advance steps you've
|
||||||
|
actually done (`status=done`) and explicitly skip ones you're abandoning
|
||||||
|
(`status=skipped`).
|
||||||
|
|
||||||
### 6. WRITE BACK + COMPLETE — `complete_task`
|
### 6. WRITE BACK + COMPLETE — `complete_task`
|
||||||
Call `update_entity_attributes` for every entity you ran `run` against
|
Call `update_entity_attributes` for every entity you ran `run` against
|
||||||
(versions, states, counts, timestamps). Call `create_relationship` for any
|
(versions, states, counts, timestamps). Call `create_relationship` for any
|
||||||
@@ -59,6 +77,15 @@ is a pure-DB Q&A that called *no* `run` at all (only get_entity/list_lxcs/
|
|||||||
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
search_knowledge): answer directly, `complete_task` with a one-line summary,
|
||||||
no writeback needed.
|
no writeback needed.
|
||||||
|
|
||||||
|
**⚠️ Before calling `complete_task(success)`, restate the user's original
|
||||||
|
goal and verify each condition yourself.** "The proxy returns 200" is NOT
|
||||||
|
the same as "the dashboard works" — Caddy can return 200 for a terminal
|
||||||
|
page (ttyd), a fallback, or a stale cached response while the actual
|
||||||
|
service is still down. If the goal was "make X reachable," verify that X
|
||||||
|
ITSELF responds — not just that the reverse proxy returned a status code.
|
||||||
|
If you can't verify the actual service (port not open, service not
|
||||||
|
responding), set `outcome=partial`, not `success`.
|
||||||
|
|
||||||
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
`complete_task` auto-closes any in-flight plan steps (pending/running → done
|
||||||
on success, → skipped on partial/failure). You do NOT need to call
|
on success, → skipped on partial/failure). You do NOT need to call
|
||||||
`update_plan_step` for every step right before completing — once your work
|
`update_plan_step` for every step right before completing — once your work
|
||||||
@@ -85,6 +112,13 @@ generation — the panel will show it as a new list), execute, write back,
|
|||||||
the panel in your reply. Never re-run `run` just to fix a display mismatch.
|
the panel in your reply. Never re-run `run` just to fix a display mismatch.
|
||||||
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
|
- Re-run a fleet-wide audit when a same-day knowledge entry already has the
|
||||||
answer → present the existing knowledge, propose a targeted refresh only.
|
answer → present the existing knowledge, propose a targeted refresh only.
|
||||||
|
- Pivot to a subsystem unrelated to the user's expressed goal without asking →
|
||||||
|
when investigation leads to a different subsystem or root cause (e.g.
|
||||||
|
debugging DHCP reservations when the goal was "make the dashboard reachable"),
|
||||||
|
call `session_questions` with the discovery and options BEFORE taking action.
|
||||||
|
Example: "The dashboard hasn't started since July 19 — this predates my work.
|
||||||
|
Do you want me to debug the dashboard service [A], skip it and stabilize the
|
||||||
|
current state [B], or stop here [C]?"
|
||||||
|
|
||||||
## Source of truth
|
## Source of truth
|
||||||
|
|
||||||
@@ -205,6 +239,21 @@ disappear.
|
|||||||
`list_lxcs` answers the same question in one call. Use it.
|
`list_lxcs` answers the same question in one call. Use it.
|
||||||
- When a bulk tool's summary isn't enough for a specific entity, call the
|
- When a bulk tool's summary isn't enough for a specific entity, call the
|
||||||
per-entity tool for that one entity — not for every entity in the fleet.
|
per-entity tool for that one entity — not for every entity in the fleet.
|
||||||
|
- **Cap pre-plan exploration:** prefer `list_entities(limit)` +
|
||||||
|
`get_entity_knowledge` (context for one entity, one call) over N+1
|
||||||
|
`get_entity`/`get_relations` chains. If you've already called
|
||||||
|
`get_entity_knowledge(slug)` and need more, call `get_entity(slug)` +
|
||||||
|
`get_relations(slug)` — not `list_entities` without a limit scanning the
|
||||||
|
whole entity table.
|
||||||
|
- **Group parallel reads:** `get_entity_knowledge`, `search_knowledge`,
|
||||||
|
`get_entity`, and `get_relations` are all read-only DB calls that can
|
||||||
|
be batched in a single tool-call block. Do not sequentialize them one
|
||||||
|
per turn when they are independent.
|
||||||
|
- **Source-reading on prod (`run cat/grep/find /opt/…`) is NOT the way to
|
||||||
|
learn how the platform works.** The MCP tools ARE the interface. If you
|
||||||
|
need to understand a check lifecycle or a scheduler behavior, search
|
||||||
|
`search_knowledge("oikos check lifecycle")` or ask the operator — do
|
||||||
|
not treat the prod host as a code repository you grep.
|
||||||
|
|
||||||
## Policy awareness
|
## Policy awareness
|
||||||
|
|
||||||
@@ -355,6 +404,26 @@ port is busy, find a free one. Only surface to the operator if you've tried
|
|||||||
reasonable alternatives and none worked. An error in one step is not a reason
|
reasonable alternatives and none worked. An error in one step is not a reason
|
||||||
to stop the entire turn — it's a reason to try a different approach.
|
to stop the entire turn — it's a reason to try a different approach.
|
||||||
|
|
||||||
|
**When you hit a genuine missing capability — STOP and ask, don't bypass:**
|
||||||
|
If a tool returns `entity … not found` when you're trying to create something
|
||||||
|
(a check, an ingress, a cert, a new service), the entity doesn't exist yet —
|
||||||
|
use `create_entity`. If you need to retire/delete an entity, use
|
||||||
|
`set_entity_state`. If you need to remove a relationship, use
|
||||||
|
`end_relationship`. If NONE of these fit and you truly lack a tool, **tell the
|
||||||
|
operator directly: "I need to X, but no MCP tool does that — can you create it
|
||||||
|
via the API?"** Do NOT pivot to `run find/grep/cat` on `/opt/homelab-context`
|
||||||
|
to reverse-engineer how the platform works — MCP tools are the interface, not
|
||||||
|
the prod source tree.
|
||||||
|
|
||||||
|
**Self-grounding — use the DB, don't invent:**
|
||||||
|
- `run` targets must be `host:<slug>`, `lxc:<slug>`, or `vm:<slug>` — never
|
||||||
|
`ws:`, raw container names, or Docker Compose service aliases.
|
||||||
|
- Never invent an IP address or subnet. Query `get_entity("service:oikos")` for
|
||||||
|
the real API address, `get_entity("host:<name>")` for a host's real LAN IP,
|
||||||
|
`list_lxcs` for container addresses. The DB is authoritative; your guess is
|
||||||
|
wrong (the homelab has multiple subnets — `192.168.8.0/24`, `192.168.178.0/24`,
|
||||||
|
etc. — and guessing the wrong one wastes turns).
|
||||||
|
|
||||||
**A hung command is not a failed command — investigate before retrying.**
|
**A hung command is not a failed command — investigate before retrying.**
|
||||||
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
|
If a `run` call times out or returns "ERROR" (e.g. SSH killed, signal,
|
||||||
gateway timeout), DO NOT immediately retry the same command with different
|
gateway timeout), DO NOT immediately retry the same command with different
|
||||||
@@ -391,6 +460,21 @@ before producing the plan. A multi-step migration proposed when the
|
|||||||
user actually wanted a one-line cleanup wastes turns and forces the
|
user actually wanted a one-line cleanup wastes turns and forces the
|
||||||
user to redirect.
|
user to redirect.
|
||||||
|
|
||||||
|
**Scope gate — ask before chasing unrelated subsystems.** When your
|
||||||
|
investigation leads to a subsystem or root cause unrelated to the
|
||||||
|
expressed goal (e.g. the user asked "why is X unreachable?" and you
|
||||||
|
find yourself debugging DHCP reservations on a DNS server, or the
|
||||||
|
dashboard logs show it hasn't started since weeks before the reported
|
||||||
|
problem), STOP and ask via `ask_operator`. Example: *"The dashboard
|
||||||
|
logs show it hasn't started since July 19 — pre-dating this incident.
|
||||||
|
Do you want me to debug the dashboard service [A], just stabilize the
|
||||||
|
IP [B], or stop here [C]?"* Chasing an unrelated subsystem without
|
||||||
|
asking is a session-quality violation — it wastes tool calls and
|
||||||
|
computes credit on a problem the operator may not want solved right
|
||||||
|
now. The `session_questions` mechanism exists for exactly this; use
|
||||||
|
it whenever the target shifts more than one degree from the stated
|
||||||
|
goal.
|
||||||
|
|
||||||
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
**Multi-goal sessions: summarize the arc, not just the last goal.**
|
||||||
When a session has more than one `set_goal` (the operator pivoted mid-
|
When a session has more than one `set_goal` (the operator pivoted mid-
|
||||||
session — e.g. "actually, just keep ludo-library"), the final
|
session — e.g. "actually, just keep ludo-library"), the final
|
||||||
|
|||||||
@@ -0,0 +1,224 @@
|
|||||||
|
# 2026-08-03 — Session review: `service:haos` monitoring + agent capability gaps
|
||||||
|
|
||||||
|
**Status:** Plan (audit complete; ready to implement).
|
||||||
|
**Reviewed session:** `23da10db-46a9-444c-bbde-ca9457bd9087` — *"Work out what
|
||||||
|
monitoring checks service:haos should have and configure them."*
|
||||||
|
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
|
||||||
|
`agent_activity`/`session_plan_steps` on the prod mac-mini (oikos prod runs here
|
||||||
|
in docker compose project `oikos`; gateway `:8092`), cross-referenced with the
|
||||||
|
code paths in `internal/mcp`, `internal/httpapi`, `internal/policy`,
|
||||||
|
`internal/checkdefaults`, `internal/db/seed.go`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Session audit (objective vs outcome)
|
||||||
|
|
||||||
|
| Dimension | Finding |
|
||||||
|
|---|---|
|
||||||
|
| Objective | Determine + configure monitoring checks for `service:haos` (HAOS VM 108, `home.hubris.network`, `192.168.8.101:8123`). |
|
||||||
|
| Outcome | ❌ **Failed/stuck.** `status=executing`, `outcome=null` ~4 min after last activity (UTC); never reached a terminal state. Only the *existing* `check:vm-status:vm:haos:0` stub got populated; the three **new** checks (`http:service`, `http:ingress`, `cert-expiry`) and their `ingress:`/`cert:` entities were never created. |
|
||||||
|
| Tool calls | **116** (vs the >30 N+1 failure signature). ~45 redundant `list_entities`/`get_entity`/`get_relations`, then a ~15-min storm of `run` doing `find`/`grep`/`cat` on prod source. |
|
||||||
|
| Plan | 8 steps proposed; steps 1–4 genuinely done; **step 5 falsely marked "done"** after both its tool calls errored `entity not found`; steps 6–8 never started. |
|
||||||
|
| Operator friction | 3 manual interventions: `status`, `proceed`, *"why dony you use the mcp?"*; plus a **44-minute approval stall** (19:52→20:36) on two trivial reachability curls. |
|
||||||
|
| Severity | **blocker** (capability gap) + **friction** (classifier, plan-state, reaping). |
|
||||||
|
|
||||||
|
### Timeline (UTC)
|
||||||
|
- **19:45–19:50** — read-only exploration; `run` correctly blocked ("No plan… call set_goal then propose_plan"). Good guard.
|
||||||
|
- **19:50** — `propose_plan` (8 steps).
|
||||||
|
- **19:52** — two `curl … -o /dev/null -w '%{http_code}'` reachability probes → both classified `config_mutation` → one queued for approval (`019fc92e…`), second blocked ("approval already pending").
|
||||||
|
- **19:52 → 20:36 (44 min)** — idle, waiting on operator approval.
|
||||||
|
- **20:36** — approval granted ("auto via assent window"); both curls → 200/200.
|
||||||
|
- **20:37** — step 4 ✅: populated `check:vm-status:vm:haos:0` + `checks` edge.
|
||||||
|
- **20:37:58** — step 5 ❌: `update_entity_attributes("check:http:service:haos:0")` → **`entity not found`**; `create_relationship` → **`source entity not found`**. *(There is no create tool.)*
|
||||||
|
- **20:38–20:47** — spiral: `search_knowledge` (empty), then `run find/grep/cat` across `/opt/homelab-context/**/*.go` to reverse-engineer check creation. Reads `checkdefaults.go`, `monitoring.go`, `default_checks.go`, `checks.go`, `coverage.go`.
|
||||||
|
- **20:42** — sets `service:haos` `monitoring: ["http"]` via `update_entity_attributes`, hoping `checkdefaults.Ensure()` auto-generates. **It does not** (see A2).
|
||||||
|
- **20:42–20:50** — tries to reach the REST API directly: `psql` on hubris (cmd 127), `docker exec` on hubris (docker absent), `curl http://192.168.178.25:8090` (wrong subnet; real net is `192.168.8.x`; exit 7), `curl http://oikos-api:8090` (MCP routes to hubris which can't resolve the mac-mini docker alias; 30s timeouts ×2), `ssh root@192.168.178.25` (no route). Final `update_entity_attributes` on `ingress:`/`cert:` → `not found`.
|
||||||
|
- **20:50:38** — last activity: a failed 30s `run`. Session goes silent, never terminates.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Root-cause findings (with code evidence)
|
||||||
|
|
||||||
|
### A1 — No entity-creation capability in the MCP toolset *(the blocker)*
|
||||||
|
`internal/mcp/tools.go` registers **37 tools**; the only entity-mutation surface is
|
||||||
|
`update_entity_attributes` (merge into an **existing** entity) and
|
||||||
|
`create_relationship` (needs **existing** source+target). Neither can create a new
|
||||||
|
entity. The capability **does** exist at the HTTP layer — `CreateEntity`
|
||||||
|
(`internal/httpapi/impl.go:865`, `POST /api/v1/entities`) — it is simply not exposed
|
||||||
|
to the agent. Every "set up / onboard / configure entity X" task that needs a new
|
||||||
|
check/ingress/cert/service hits this wall.
|
||||||
|
|
||||||
|
### A2 — MCP `update_entity_attributes` bypasses `ensureDefaultChecks`
|
||||||
|
`ensureDefaultChecks` (`internal/httpapi/default_checks.go:20`) is invoked **only**
|
||||||
|
from the HTTP handlers: `CreateEntity` (`impl.go:1012`) and `PatchEntity`
|
||||||
|
(`impl.go:1280`). `grep ensureDefaultChecks internal/mcp/` → **no matches**: the MCP
|
||||||
|
tool writes attributes straight to the store, so flipping `service:haos`
|
||||||
|
`monitoring:["http"]` never regenerated its checks. The agent's fallback strategy
|
||||||
|
was structurally doomed via MCP.
|
||||||
|
|
||||||
|
### A3 — `-o /dev/null` curl idiom misclassified as `config_mutation`
|
||||||
|
`internal/policy/command.go:106` `curlMutateRe` matches `(?:^|\s)-(?:d|F|T|o)\b` — so
|
||||||
|
`-o` (output-file) is treated as mutation. The canonical read-only reachability probe
|
||||||
|
`curl -sS -o /dev/null -w '%{http_code}' …` therefore escalates to approval. This is
|
||||||
|
the entire 44-minute stall. (`curlIsReadOnly` at `command.go:307` only passes for GET
|
||||||
|
with no `-o`/`-d`/`-X`/`>`.) A pure GET that discards the body is the single most
|
||||||
|
common health probe and shouldn't need approval.
|
||||||
|
|
||||||
|
### A4 — No platform self-knowledge doc for the check lifecycle
|
||||||
|
`search_knowledge("create check entity how to add new check monitoring")` → empty.
|
||||||
|
The agent re-derived the whole mechanism from source on prod (~15 min, dozens of
|
||||||
|
`run`). There is no agent/operator runbook explaining: check slugs are
|
||||||
|
`check:<kind>:<target>:<n>`; `check_defs` are derived from the type's `monitoring`
|
||||||
|
spec by `checkdefaults.Ensure`; Ensure runs at **seed/deploy** and on **HTTP
|
||||||
|
create/patch**, not via MCP.
|
||||||
|
|
||||||
|
### A5 — False plan progress (step marked done on failure)
|
||||||
|
At 20:37:58 both tool calls for step 5 returned `error: entity not found`, yet the
|
||||||
|
agent advanced step 5→`done`. Plan-state integrity hole: a step whose actions error
|
||||||
|
should not transition to `done`. (`session_plan_steps` confirms seq 5 = `done`.)
|
||||||
|
|
||||||
|
### A6 — No "missing-capability" escalation; self-grounding failures
|
||||||
|
On detecting the dead-end (no create tool) the agent never told the operator *"I lack
|
||||||
|
a tool to create entities — please create them"*; instead it tried to bypass its own
|
||||||
|
platform. Grounding errors: invented IP `192.168.178.25` (real LAN is `192.168.8.x`),
|
||||||
|
ran `run` against `ws:mac-mini` ("unsupported target — must be host:/lxc:/vm:"),
|
||||||
|
assumed `docker` exists on hubris, assumed the docker-alias `oikos-api` resolves from
|
||||||
|
hubris. The agent didn't query `get_entity("service:oikos")` for the real address.
|
||||||
|
|
||||||
|
### A7 — Sessions never reap from `executing`
|
||||||
|
Last activity 20:50; status still `executing` with no turn running. There is no
|
||||||
|
idle-timeout / abandoned transition when a turn ends without resolution. (Fleet-wide:
|
||||||
|
176 done / 9 failed / 1 executing; the 9 prior failures are pre-v0.15.0, mostly
|
||||||
|
approval-stalls and entity-not-found — same families.)
|
||||||
|
|
||||||
|
### A8 — N+1 tool fan-out (116 calls)
|
||||||
|
Dozens of redundant `list_entities`/`get_entity`/`get_relations` before proposing a
|
||||||
|
plan, plus the source-reading `run` storm. Above the >30-per-turn signature; indicates
|
||||||
|
weak bulk-tool use and under-constrained exploration before planning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Improvement plan (ordered)
|
||||||
|
|
||||||
|
**Scope decision (confirmed with operator):** general `create_entity` MCP tool **+
|
||||||
|
wire regen** — solves this case and the 67-entity blast radius (§4).
|
||||||
|
|
||||||
|
### Task 1 — `create_entity` MCP tool *(fixes A1; the centerpiece)*
|
||||||
|
- Register a new tool `create_entity(slug, type, name, attributes?)` in
|
||||||
|
`internal/mcp/tools.go` that **reuses** `httpapi.CreateEntity`
|
||||||
|
(`impl.go:865`) / the same store path — do not hand-roll. It must run
|
||||||
|
`ensureDefaultChecks` (free, since it goes through the create path).
|
||||||
|
- **Approval policy:** no approval required for the entity itself — it mutates the
|
||||||
|
knowledge graph, matching the existing no-approval stance of
|
||||||
|
`update_entity_attributes`/`create_relationship`/`upsert_knowledge`. (Derived checks
|
||||||
|
are safe/read-side; if a check kind is ever deemed mutating, gate *that* in the
|
||||||
|
scheduler, not here.)
|
||||||
|
- Validate `type` against `entity_types`; reject unknown slugs/types with a clear
|
||||||
|
error. Idempotent on existing slug (return the existing entity, mirroring the HTTP
|
||||||
|
`ETag`/conflict behavior).
|
||||||
|
- Expose to the agent via the tool-list build path used by `cmd/nomos/agent.go`.
|
||||||
|
|
||||||
|
### Task 2 — MCP `update_entity_attributes` triggers `ensureDefaultChecks` *(fixes A2)*
|
||||||
|
- After the attribute merge in the MCP handler, call `ensureDefaultChecks` with the
|
||||||
|
post-merge entity (same args as `impl.go:1280`). This makes "set monitoring → checks
|
||||||
|
regenerate" work via MCP, matching HTTP semantics.
|
||||||
|
- Mind the `default_checks.go:14-19` caveat: a service whose address comes from its
|
||||||
|
host edge may still produce no checks until the hosting edge exists — log/return
|
||||||
|
that as an explicit result so the agent knows to create the edge next.
|
||||||
|
|
||||||
|
### Task 3 — Classifier: read-only `curl` with `-o /dev/null` *(fixes A3)*
|
||||||
|
- In `internal/policy/command.go` `curlIsReadOnly`, treat `-o /dev/null` (and
|
||||||
|
`--output /dev/null`) as read-only — it's a no-op sink. Keep `-o <realpath>` as
|
||||||
|
mutation. Add `TestClassifyCommand_CurlDevNull_ReadOnly` next to the existing
|
||||||
|
`TestClassifyCommand_CurlPipeSh_ConfigMutation`.
|
||||||
|
- Coach complement: in `nomos/SOUL.md`, note that reachability probes should use
|
||||||
|
`curl -I` or `-o /dev/null` GETs (now read-only) rather than POSTs.
|
||||||
|
|
||||||
|
### Task 4 — Plan-state integrity: don't mark `done` on errored actions *(fixes A5)*
|
||||||
|
- In `cmd/nomos` (`agent.go`/`tasks.go` where `update_plan_step` is emitted), a step
|
||||||
|
whose turn ended with only error/`not-found` tool results must **not** auto-advance
|
||||||
|
to `done`; leave it `running`/`blocked` and surface the failure to the operator.
|
||||||
|
Minimal: if every tool call in the step returned an `error:*` result, hold the step.
|
||||||
|
|
||||||
|
### Task 5 — Stuck-session reaping *(fixes A7)*
|
||||||
|
- Add an idle sweep (extend the existing continuation/idle worker in `cmd/nomos`) that
|
||||||
|
transitions a session from `executing`→`failed` (or a new `stuck`) when no turn has
|
||||||
|
run for N minutes and no approval is pending. Emit an event so the UI (F3 terminal
|
||||||
|
handling) clears the spinner. Pick N (recommend 30 min) — confirm in review.
|
||||||
|
|
||||||
|
### Task 6 — Missing-capability escalation + grounding *(fixes A6)*
|
||||||
|
- `nomos/SOUL.md`: when a mutation tool returns `entity … not found` on a create
|
||||||
|
intent, the agent must **stop and ask the operator** (or now use `create_entity`)
|
||||||
|
rather than pivot to `run`/SSH/API-bypass. Forbidden: inventing IPs/subnets; instead
|
||||||
|
`get_entity("service:oikos")` for the real API address. `run` targets must be
|
||||||
|
`host:/lxc:/vm:` slugs (state the contract explicitly).
|
||||||
|
|
||||||
|
### Task 7 — Runbook: "how checks work / how to add monitoring" *(fixes A4)*
|
||||||
|
- Upsert a knowledge doc (via `upsert_knowledge`, linked to the `agent:nomos` and
|
||||||
|
`document:infrastructure/monitoring` entities) covering: check slug grammar,
|
||||||
|
`checkdefaults.Ensure` triggers (seed + HTTP create/patch, now also MCP), the
|
||||||
|
`monitoring` per-entity override, the host-edge caveat, and the canonical way to add
|
||||||
|
monitoring to an entity (create/patch entity → checks derive).
|
||||||
|
|
||||||
|
### Task 8 — (Lower priority) exploration budget / bulk-tool use *(A8)*
|
||||||
|
- `nomos/SOUL.md`: prefer `list_entities(limit)` + `get_entity_knowledge` bulk calls
|
||||||
|
over N+1 `get_entity`/`get_relations` fans; cap pre-plan exploration. Optional
|
||||||
|
guardrail in `agent.go` (warn at >N same-tool calls per turn).
|
||||||
|
|
||||||
|
### Recommended sequence
|
||||||
|
1 → 2 → 3 → 4 → 7 → 5 → 6 → 8. (1+2 unblock the whole task class; 3 kills the
|
||||||
|
approval stall; 4+5 fix state integrity; 7 is cheap leverage; 6+8 are persona
|
||||||
|
hardening.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Uncovered cases — the capability-gap blast radius
|
||||||
|
|
||||||
|
The existing F1–F8 plans (`2026-08-03-nomos-chat-reliability-and-ux-audit.md`,
|
||||||
|
shipped v0.15.0) and the turn-scheduler review cover **only** UI / streaming / turn
|
||||||
|
serialization / connection UX. **None** addresses agent *capability* or
|
||||||
|
MCP↔HTTP integration. This session exposes the uncovered class:
|
||||||
|
|
||||||
|
- **67 entities currently have no `check:` relationship** (DB query): 40 `lxc`, 26
|
||||||
|
`service`, 1 `vm`. Any "add monitoring to X" task fails identically until Tasks 1+2.
|
||||||
|
- **Whole task families blocked by the no-create gap:** onboarding a new host/LXC/VM,
|
||||||
|
declaring a new service/ingress/cert/dns, adding any check that doesn't already
|
||||||
|
exist, registering a relationship target that doesn't exist yet. All currently
|
||||||
|
require an operator to hand-edit `seeds/inventory.yaml` and re-seed.
|
||||||
|
- **MCP↔HTTP semantic drift (generalize A2):** audit other MCP mutation tools for
|
||||||
|
side-effects that the HTTP handlers perform but the MCP path skips (check regen,
|
||||||
|
drift-flagging, audit fields, idempotency). Each is a latent "agent did the right
|
||||||
|
thing but nothing happened" bug.
|
||||||
|
- **Classifier read-only false-positives (generalize A3):** beyond `-o /dev/null`,
|
||||||
|
review other common read-only idioms that escalate (`curl` with benign flags,
|
||||||
|
compound read-only commands) — friction compounds into approval stalls and stuck
|
||||||
|
sessions.
|
||||||
|
- **No terminal/`stuck` reaping (generalize A7):** any turn that ends unresolved
|
||||||
|
leaves the session `executing` forever; the UI never shows "done/failed".
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Validation
|
||||||
|
|
||||||
|
- **Task 1/2:** `go test ./internal/mcp/... ./internal/httpapi/...` — new test creates
|
||||||
|
`check:http:service:haos:0` via `create_entity`, asserts the entity exists **and**
|
||||||
|
that a `check_def` row was derived; then `update_entity_attributes(service:haos,
|
||||||
|
monitoring:["http"])` via MCP and assert checks regenerate (currently absent).
|
||||||
|
- **Task 3:** `go test ./internal/policy/` — `curl -sS -o /dev/null -w '%{http_code}'
|
||||||
|
URL` ⇒ `read_only`; `curl -o /tmp/x URL` ⇒ `config_mutation`.
|
||||||
|
- **Task 4:** `cmd/nomos` test — a step whose only tool result is `error:*` stays
|
||||||
|
non-`done`.
|
||||||
|
- **Task 5:** idle-sweep test — session with no turn for N min and no pending approval
|
||||||
|
⇒ `failed` (+ event emitted).
|
||||||
|
- **End-to-end re-run:** replay the haos goal against a local nomos; expect the three
|
||||||
|
checks + `ingress:`/`cert:` entities created in <15 tool calls with **zero**
|
||||||
|
approvals and a `done` outcome.
|
||||||
|
|
||||||
|
## 6. Out of scope / open questions
|
||||||
|
- Whether `create_entity` for sensitive types (e.g. `secret`, `key`) should require
|
||||||
|
approval even though it's graph-only — recommend: same no-approval stance now, add
|
||||||
|
type-specific gating later if abused.
|
||||||
|
- The exact stuck-reap window N (recommend 30 min) and whether to introduce a distinct
|
||||||
|
`stuck` status vs reuse `failed`.
|
||||||
|
- Whether to also expose a `delete_entity`/`retire_entity` MCP tool (not needed for
|
||||||
|
this case; lifecycle retirement is a separate flow).
|
||||||
@@ -1,5 +1,9 @@
|
|||||||
# 2026-07-21 Chat window full polish
|
# 2026-07-21 Chat window full polish
|
||||||
|
|
||||||
|
**Status:** Implemented. Streaming affordance, inline tool rendering, message
|
||||||
|
timestamps, code-copy buttons, and per-session store isolation all landed in
|
||||||
|
`web/src/lib/components/ChatThread.svelte` + the chat stores (v0.8.x–0.10.x).
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
After fixing the streaming reactivity bug and merging the double thinking
|
After fixing the streaming reactivity bug and merging the double thinking
|
||||||
@@ -0,0 +1,413 @@
|
|||||||
|
# Plan: Make health reflect reality + complete the knowledge graph
|
||||||
|
|
||||||
|
Status: Implemented (v0.14.x–0.16.x). Shipped across `c9a00a9` (per-entity
|
||||||
|
monitoring override), `a3914eb`/`8eb1ca2` (process check opt-in + probe_unit),
|
||||||
|
`0929c17` (discover_infra_drift), and the vm-status/layered-probe/route-via-
|
||||||
|
proxmox-host decisions now in project memory. Created 2026-07-29.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
`ws:mac-mini` reports health `down` despite being the healthy control-plane host.
|
||||||
|
Investigation showed the problem is systemic, not local: **49 enabled checks report
|
||||||
|
`down`**, almost all `ssh-script`, because the resource/updates probes assume
|
||||||
|
**scripts are deployed at `/opt/oikos/checks/` AND root SSH works on every target** —
|
||||||
|
both false for macOS, non-enrolled LXCs, and mesh-only entities. The knowledge graph
|
||||||
|
also has real gaps (unmodeled TLS certs, empty `skills` table, seed drift, a capped
|
||||||
|
topology view).
|
||||||
|
|
||||||
|
The DB is the source of truth; live state was verified via the REST API
|
||||||
|
(`Authorization: Bearer $OIKOS_MCP_BEARER_TOKEN`, token in `oikos-api-1` container env)
|
||||||
|
and `docker exec oikos-postgres-1 psql`. Direct psql access is available for cleanup.
|
||||||
|
|
||||||
|
## Decisions (confirmed with operator)
|
||||||
|
|
||||||
|
1. **Monitoring philosophy: make checks work everywhere** — via the proven `pct exec`/
|
||||||
|
`qm guest exec` host-routing the MCP `run` tool already uses (no per-guest SSH keys),
|
||||||
|
plus deploy the check scripts INTO each guest and make them macOS-aware. Hosts/workstations
|
||||||
|
use direct SSH with the correct per-target user.
|
||||||
|
2. **Canonical host-hop access** — `pct exec`/`qm guest exec` through the proxmox host is
|
||||||
|
the ONLY execution path for any LXC/VM command (scheduler + MCP `run` + agent). Direct
|
||||||
|
guest SSH is retired for execution; `lan_ip` stays for network probes only. (A1.)
|
||||||
|
3. **Auto-provision monitoring for new entities** — wire script-deploy + the
|
||||||
|
`health-check-answering` lifecycle gate into entity creation so any entity Nomos creates
|
||||||
|
becomes monitorable with zero manual steps (Track E).
|
||||||
|
4. **Lifecycle gate: skip monitoring for `deprecated`/`destroyed` targets** — no
|
||||||
|
permanent false alarms from retired things.
|
||||||
|
5. **Knowledge graph: address ALL gaps** — model TLS certificates, fix dns-zone gap,
|
||||||
|
re-export seeds, seed skills, raise graph cap.
|
||||||
|
6. **Read-only audit skill** — a `read_only` operator skill discovers live infra and diffs
|
||||||
|
it against the DB graph, producing a ranked drift report; the operator acts on findings
|
||||||
|
via existing lifecycle runbooks. No auto-fix. (Track F.)
|
||||||
|
|
||||||
|
## Findings (evidence)
|
||||||
|
|
||||||
|
### A. Health-check reality gaps (49 checks `down`)
|
||||||
|
|
||||||
|
**Root cause is a routing mismatch, verified live (tests use the scheduler's own key
|
||||||
|
`-i /etc/oikos/ssh_key`, not a default-key test):**
|
||||||
|
|
||||||
|
The MCP `run` tool already reaches every guest correctly via
|
||||||
|
`resolveExecTarget` (`internal/mcp/server.go:582`): resolve the proxmox host
|
||||||
|
(`attributes.host` → `hosts` edge → hubris default), SSH there, run
|
||||||
|
`pct exec <pve_id> -- bash -c 'echo <b64> | base64 -d | bash'` (VMs: `qm guest exec`).
|
||||||
|
That path needs **no per-guest lan_ip, no per-guest authorized_keys, no per-guest sshd**.
|
||||||
|
|
||||||
|
The **scheduler's `checkSSHScript` does not use it** — it SSHes directly to each
|
||||||
|
entity's own resolved address (`internal/scheduler/scheduler.go:758`,
|
||||||
|
`internal/checkdefaults/defaults.go:376 resolveHost`) and runs
|
||||||
|
`/opt/oikos/checks/<script>`. That is the bug. Decomposed by class:
|
||||||
|
|
||||||
|
| Class | Targets (verified) | Root cause |
|
||||||
|
|---|---|---|
|
||||||
|
| **Guests reached wrong** | `lxc:rclone` (mesh-only, no lan_ip), `lxc:nfs-export` (192.168.8.200: **ssh port 22 timeout** — no sshd), `lxc:teddycloud` (**key not authorized** — "not a homelab client"), `lxc:grimmory/romm/seanime` (strong: pct-exec reachable, **scripts not inside**) | scheduler SSHes the guest directly; should route via proxmox host `pct exec` like `resolveExecTarget`. rclone is correctly parented on hubris (`hosts` edge verified) and IS reachable via `pct exec 132` — the mesh fqdn is a red herring. |
|
||||||
|
| macOS host | `ws:mac-mini` (5 resource/updates checks `down`) | root SSH disabled (macOS); `user: dtoro` never read by resolver (`defaults.go:406` reads `attrs["ssh"]["user"]` only); scripts not deployed; scripts Linux-only |
|
||||||
|
| External / mesh-only | `host:netbird-vps` (no lan_ip; mesh unreachable from container) | `resolveHost` picks mesh IP over `public_ipv4` (`defaults.go:376`); sshd also "locked to hubris pubkey" |
|
||||||
|
| Dead route | `ingress:secrets.hubris.network` http `down` | `service:secrets-issuance` is `deprecated` but its ingress check still enabled — no lifecycle gate |
|
||||||
|
| ICMP-blocked | `vm:haos` ping `down` while up | HAOS blocks ICMP |
|
||||||
|
|
||||||
|
**Working** (prove the host-SSH model is sound): `host:hubris`, `host:strong` SSH with
|
||||||
|
the scheduler key → **SCRIPTS_PRESENT**; `lxc:gitea` direct-SSH → **SCRIPTS_PRESENT**
|
||||||
|
(it's a homelab client with root key + scripts). So the host hop is the reliable path.
|
||||||
|
|
||||||
|
**Parentage verified correct** (all `hosts` edges checked in DB): strong guests on
|
||||||
|
strong, hubris guests on hubris. No misplaced parents — the gap is routing + in-guest
|
||||||
|
script deployment, not topology.
|
||||||
|
|
||||||
|
Health aggregation itself is correct: `WorstHealthForTarget`
|
||||||
|
(`internal/db/sqlcgen/operations.sql.go:1472`) = worst enabled check. One failing
|
||||||
|
ssh-script drags an otherwise-healthy entity to `down`.
|
||||||
|
|
||||||
|
### B. Dead/stale data
|
||||||
|
|
||||||
|
- **24 orphan check_defs** + check entities, slugs `^check:(ping|ssh-script|disk):[0-9a-f]{8}$`
|
||||||
|
(e.g. `check:ssh-script:0d31fdd1`), `enabled=false`, `last_health=NULL`, `state=NULL`.
|
||||||
|
Leftover from the old `shortSlug()` collision bug (fixed in `defaults.go:263`).
|
||||||
|
- `service:secrets-issuance` = `deprecated`; `ingress:secrets.hubris.network` still
|
||||||
|
routes to it and alarms permanently.
|
||||||
|
|
||||||
|
### C. Knowledge-graph gaps
|
||||||
|
|
||||||
|
- **TLS certificates unmodeled**: `certificate` type + `uses-certificate` edge + `cert-expiry`
|
||||||
|
checker all exist, but **0** certificate entities. Cert expiry is invisible.
|
||||||
|
- **`dns-zone` declares `monitoring: [dns]`** (`seeds/ontology.yaml:382`) but no `dns`
|
||||||
|
checker exists → every zone is an `unmonitored` signal.
|
||||||
|
- **Seed drift**: 23 `dns-record` entities in DB, 0 in `seeds/inventory.yaml`.
|
||||||
|
- **`skills` table = 0** despite `.agents/skills/*/SKILL.md` on disk (runbooks = 15).
|
||||||
|
- **Graph capped at 500 nodes** (`internal/httpapi/impl.go:27 graphNodeCap = 500`);
|
||||||
|
299 `execution` + 87 `task` rows dominate, so `/graph` is not a faithful topology view.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Work breakdown
|
||||||
|
|
||||||
|
### Track A — Make ssh-script checks work everywhere (route through the proxmox host)
|
||||||
|
|
||||||
|
Core idea: stop having the scheduler SSH each guest directly. Reuse the MCP `run`
|
||||||
|
tool's proven `resolveExecTarget` pattern — reach every LXC/VM **through its proxmox
|
||||||
|
host** via `pct exec`/`qm guest exec`. This fixes rclone (no lan_ip), nfs-export
|
||||||
|
(no sshd), teddycloud (no key), and every strong guest in one stroke, because the host
|
||||||
|
hop already has working root SSH. Hosts/workstations keep direct SSH.
|
||||||
|
|
||||||
|
**A1. Canonicalize host-hop as the ONLY execution path for LXC/VM (the real fix + simplification).**
|
||||||
|
|
||||||
|
Principle: **never SSH directly into a guest to run a command.** Every LXC/VM command
|
||||||
|
execution — scheduler checks, the MCP `run` tool, and the agent — routes through the
|
||||||
|
owning proxmox host via `pct exec <pve_id> -- ...` (VMs: `qm guest exec`). One SSH
|
||||||
|
credential per host (root key, already authorized on hubris/strong), no per-guest keys,
|
||||||
|
sshd, or lan_ip needed for execution. Verified this works: `pct exec 132` reaches rclone;
|
||||||
|
the MCP `run` tool already does it for every guest (`internal/mcp/server.go:582`).
|
||||||
|
|
||||||
|
- Network probes (http/ping) keep hitting the guest's `lan_ip`/URL directly — they don't
|
||||||
|
execute inside the guest, so they're unaffected. For LXCs all checks are ssh-script, so
|
||||||
|
they all route via the host; `lan_ip` becomes optional metadata, not a monitoring prereq.
|
||||||
|
- Extract `resolveExecTarget`/`resolveProxmoxHostSlug` out of `internal/mcp` into a shared
|
||||||
|
package (e.g. `internal/remote`) so the scheduler's `checkSSHScript`
|
||||||
|
(`internal/scheduler/scheduler.go:710`) and `checkBackupFreshness` (`backup.go:79`, the
|
||||||
|
other direct-SSH path) and the MCP `run` tool share ONE resolver. Today they diverge —
|
||||||
|
the scheduler SSHes guests directly (broken), MCP host-hops (works).
|
||||||
|
- `checkSSHScript`/`checkBackupFreshness`: when the target is `lxc:`/`vm:`, resolve the
|
||||||
|
proxmox host and wrap the invocation as `pct exec <pve_id> -- bash -c 'echo <b64> |
|
||||||
|
base64 -d | bash'` (VMs: the `qm guest exec` form at `server.go:625`). For `host:`/`ws:`
|
||||||
|
keep direct SSH (they ARE the host).
|
||||||
|
- **Risk class:** `config_mutation` (changes how probes reach every guest) → operator
|
||||||
|
approval. Verify one LXC end-to-end (rclone) before fanning out.
|
||||||
|
|
||||||
|
**A2. Deploy check scripts INTO guests (via `pct push`), not just to the host.**
|
||||||
|
- Verified: scripts exist on hubris/strong (the hosts) but `NO_SCRIPTS` inside grimmory,
|
||||||
|
romm, seanime, rclone. A `pct exec`-routed check still runs inside the guest, so the
|
||||||
|
scripts must live in the guest.
|
||||||
|
- Add a fleet-deploy tool (`tools/deploy-checks.sh`): for each LXC, from its proxmox
|
||||||
|
host, `pct push <id> checks/<script> /opt/oikos/checks/<script>` + chmod 755 (loop the
|
||||||
|
`checks/*.sh` set). For VMs, scp/agent; for hosts/workstations, run `checks/install.sh`.
|
||||||
|
- Backfill once now (all guests + mac-mini). See Track E for the automated version.
|
||||||
|
|
||||||
|
**A3. Fix per-target SSH user + resolver (hosts/workstations only).**
|
||||||
|
- `internal/checkdefaults/defaults.go:406 resolveSSHUser`: also read top-level
|
||||||
|
`attrs["user"]` (workstations carry `user: dtoro`, not `ssh.user`). Returns `dtoro`
|
||||||
|
for mac-mini. Re-derive mac-mini's check_defs so config carries the user.
|
||||||
|
- **Do NOT enable root SSH on mac-mini** — use `dtoro` (keeps macOS hardening).
|
||||||
|
|
||||||
|
**A4. macOS-aware check scripts.**
|
||||||
|
- `checks/cpu_check.sh:5` `top -bn1` (Linux) → branch on `uname -s == Darwin`
|
||||||
|
(`top -l 1`/`sysctl`). Same for `memory_check.sh`, `load_check.sh`, `disk_usage_check.sh`
|
||||||
|
(`df` differs), `updates_check.sh` (already apt-guarded; on Darwin report `healthy`
|
||||||
|
with `security_updates=0` or read `softwareupdate --list`).
|
||||||
|
- Each must still emit `{"health":..,"metrics":{..}}` JSON
|
||||||
|
(`internal/scheduler/scheduler.go:767`).
|
||||||
|
|
||||||
|
**A5. Reachability for external/mesh-only hosts.**
|
||||||
|
- `internal/checkdefaults/defaults.go:376 resolveHost`: prefer `public_ipv4` over mesh IP
|
||||||
|
for `standalone-server`/external so `host:netbird-vps` (82.165.190.79) is probeable.
|
||||||
|
Note sshd is "locked to hubris pubkey" (`inventory.yaml:89`) — either add the scheduler
|
||||||
|
key or proxy via hubris. Confirm before assuming direct SSH works.
|
||||||
|
- `ws:republic-laptop`: roving laptop on mesh only. ping-`down` when asleep is real;
|
||||||
|
keep ping-only and accept transient `down`, or set `monitoring: none`. (Decision in
|
||||||
|
Open Questions.)
|
||||||
|
- `lxc:rclone` no longer a special case — handled by A1's pct routing.
|
||||||
|
|
||||||
|
**A6. ICMP-blocked VMs.**
|
||||||
|
- `vm:haos` ping `down` while up: optional `tcp`-ping fallback in `checkPing`
|
||||||
|
(`internal/scheduler/scheduler.go:604`) for VMs that block ICMP, gated by an attribute.
|
||||||
|
Lower priority — confirm haos blocks ICMP before building.
|
||||||
|
|
||||||
|
### Track B — Lifecycle monitoring gate
|
||||||
|
|
||||||
|
**B1. Skip monitoring for deprecated/destroyed targets.**
|
||||||
|
- Disable (set `enabled=false`) and skip-scheduling `check_defs` whose `target` entity
|
||||||
|
`state` ∈ {`deprecated`,`destroyed`}.
|
||||||
|
- Implement by joining target state in `ListEnabledCheckDefs`
|
||||||
|
(`internal/db/sqlcgen/operations.sql.go`, the `ListEnabledCheckDefs` query) — exclude rows
|
||||||
|
whose target is retired — **or** in a `housekeeping` sweep
|
||||||
|
(`internal/scheduler/scheduler.go:302`) that disables them. Prefer the query filter
|
||||||
|
(no write needed at runtime).
|
||||||
|
- Matches `policy.yaml` lifecycle philosophy (`destroyed.refuse: all`); extend the comment.
|
||||||
|
- Effect: dead `ingress:secrets.hubris.network` alarm goes silent automatically.
|
||||||
|
|
||||||
|
### Track C — Dead-data cleanup
|
||||||
|
|
||||||
|
**C1. Delete 24 orphan check_defs + check entities.**
|
||||||
|
- Direct SQL (have psql access): delete `check_defs` then `entities` matching
|
||||||
|
`slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$'`. Confirm `state IS NULL` /
|
||||||
|
`enabled=false` first (already verified).
|
||||||
|
- Wrap as a one-shot migration or `scripts/cleanup-orphan-checks.sh`. **Risk class:** read
|
||||||
|
the rows first; this is `config_mutation` → operator approval.
|
||||||
|
|
||||||
|
**C2. Retire the secrets route.**
|
||||||
|
- With B1 in place the alarm silences. Optionally set `ingress:secrets.hubris.network`
|
||||||
|
→ `deprecated`/`destroyed` and remove its `routes-to` edge to service:secrets-issuance
|
||||||
|
(or keep as archaeology). Decide with operator.
|
||||||
|
|
||||||
|
**C3. Destroy 7 stray test LXCs (active cruft in the graph).**
|
||||||
|
- DB shows these with live `hosts` edges on strong, never cleaned up:
|
||||||
|
`lxc:preflight-test`, `lxc:preflight-test2`, `lxc:test-autocontinue`,
|
||||||
|
`lxc:test-decompose3`, `lxc:test-livewatch`, `lxc:test-livewatch2`, `lxc:typetype`.
|
||||||
|
- First confirm they're really gone in Proxmox (`pct list` on strong); if so, set their
|
||||||
|
entity state → `destroyed` (move to archaeology) and drop the `hosts` edges. If any
|
||||||
|
container still exists, destroy via `pct destroy` first (destructive → approval).
|
||||||
|
- They currently generate checks and pollute the graph/health view.
|
||||||
|
|
||||||
|
### Track D — Knowledge graph
|
||||||
|
|
||||||
|
**D1. Model TLS certificates.**
|
||||||
|
- Seed `certificate` entities (one per `*.hubris.network` route, or per Caddy-managed
|
||||||
|
cert) + `uses-certificate` edges from each `ingress-route`.
|
||||||
|
- Source real data: read Caddy's cert store (LXC 121) expiry via the existing `cert-expiry`
|
||||||
|
checker's discovery, or seed from Caddyfile and backfill `expires` live.
|
||||||
|
- Wires the `cert-expiry` checker (`internal/scheduler/scheduler.go`, `cert-expiry` kind)
|
||||||
|
against real entities instead of nothing.
|
||||||
|
|
||||||
|
**D2. dns-zone monitoring gap.**
|
||||||
|
- `seeds/ontology.yaml:382`: change `dns-zone` `monitoring: [dns]` → `monitoring: none`
|
||||||
|
with a comment "no dns checker yet; revisit when implemented". Stops the per-zone
|
||||||
|
`unmonitored` noise. Re-seed.
|
||||||
|
|
||||||
|
**D3. Re-export seeds to fix drift.**
|
||||||
|
- Run `oikos export` (or the export endpoint) so the 23 runtime `dns-record` entities +
|
||||||
|
other runtime-created topology land in `seeds/inventory.yaml`. Diff, review, commit.
|
||||||
|
|
||||||
|
**D4. Seed skills from disk.**
|
||||||
|
- Ingest `.agents/skills/*/SKILL.md` as `skill` entities (mirror how runbooks seed → 15
|
||||||
|
exist). Add to the knowledge seed ingest path (`internal/db/seed.go`) or a one-shot
|
||||||
|
ingest. `get_skills()` then returns data.
|
||||||
|
|
||||||
|
**D5. Raise graph node cap.**
|
||||||
|
- `internal/httpapi/impl.go:27 graphNodeCap = 500` → raise (e.g. 5000) **and/or**
|
||||||
|
paginate `/api/v1/graph`. Ensure the query stays performant (it already limits by default;
|
||||||
|
confirm no full-table risk). Optionally exclude cognition rows (`execution`/`task`) from
|
||||||
|
the default topology view via a `?layer=infrastructure` filter so infra isn't crowded out.
|
||||||
|
|
||||||
|
### Track E — Auto-provision monitoring when a new entity is created
|
||||||
|
|
||||||
|
Goal: the operator's request — "make sure this is handled automatically in the future
|
||||||
|
when the agent creates new entities." Today `ensureDefaultChecks`
|
||||||
|
(`internal/httpapi/default_checks.go:9`) writes check_defs on entity creation but does
|
||||||
|
NOT make the target probe-ready (no script deploy, no host-routing). Its own comment
|
||||||
|
admits the gap. A new entity should become monitorable with zero manual steps.
|
||||||
|
|
||||||
|
**E1. Hook script-deploy into entity creation / provisioning.**
|
||||||
|
- Extend `ensureDefaultChecks` (called on entity create, `default_checks.go`) so that,
|
||||||
|
after writing check_defs, it also ensures the target can answer:
|
||||||
|
- **LXC/VM**: `pct push` the `checks/*.sh` set into the guest from its proxmox host
|
||||||
|
(reuse the host resolution from A1). Idempotent (skip if present + unchanged).
|
||||||
|
- **host/workstation**: ensure scripts at `/opt/oikos/checks/` (run `checks/install.sh`
|
||||||
|
over SSH; locally on mac-mini).
|
||||||
|
- Because the check itself is routed via `pct exec` (Track A), no per-guest SSH key or
|
||||||
|
sshd is needed — host hop + in-guest scripts are the only prerequisites, both now
|
||||||
|
automated. mac-mini still needs its `dtoro` key (A3) once.
|
||||||
|
|
||||||
|
**E2. Tie into the lifecycle `provisioning → active` gate.**
|
||||||
|
- The ontology already requires `health-check-answering` for `provisioning → active`
|
||||||
|
(`seeds/ontology.yaml:39`, checked by `internal/ontology/validate.go:167`).
|
||||||
|
- Make that gate actually run one check against the new entity and require a non-`down`
|
||||||
|
verdict before the transition is allowed. This closes the loop: an entity isn't "active"
|
||||||
|
(and isn't trusted for blast-radius/auto decisions) until monitoring proves it answers.
|
||||||
|
|
||||||
|
**E3. Re-run on re-seed / attribute change.**
|
||||||
|
- `checkdefaults.Ensure` already re-derives check config from the seed on re-ingest
|
||||||
|
(`internal/checkdefaults/defaults.go:301`, seed wins, `enabled` preserved). Mirror that
|
||||||
|
for script deploy: when `pve_id`/`host`/address attributes change, re-target the check
|
||||||
|
and re-deploy scripts to the new guest.
|
||||||
|
|
||||||
|
**Net effect:** a new LXC provisioned by Nomos (via `pct_create`, which registers the
|
||||||
|
entity + `hosts` edge, `internal/httpapi/actuator.go:615`) automatically gets
|
||||||
|
script-pushed + check_defs + a passing `health-check-answering` gate before going active.
|
||||||
|
|
||||||
|
### Track F — Read-only knowledge-graph audit skill
|
||||||
|
|
||||||
|
Goal: the operator's request — a skill that auto-discovers live infra and validates the
|
||||||
|
knowledge graph (entities, parentage, checks, scripts, seeds, certs) against reality,
|
||||||
|
producing a ranked drift report. **Read-only; no auto-fix** — the operator routes each
|
||||||
|
finding to the relevant lifecycle runbook.
|
||||||
|
|
||||||
|
**Precedent (reuse, don't duplicate):** existing drift/quality machinery is fragmented and
|
||||||
|
knowledge-content focused. The audit orchestrates these + fills the topology/script gaps:
|
||||||
|
- `internal/httpapi/knowledge_drift.go` — duplicate notes, orphan notes, tag splits (already endpoints).
|
||||||
|
- `internal/scheduler/coverage.go coverageSweep` — unmonitored declared types (re-use its logic/SQL).
|
||||||
|
- MCP discovery: `list_lxcs` (`internal/mcp/tools.go:478`), `get_lxc_state`, `list_entities`,
|
||||||
|
`get_relations`, `http_get`. These already enumerate live LXC/VM state from the proxmox host.
|
||||||
|
|
||||||
|
**F1. Add an on-demand audit primitive (MCP tool + endpoint).**
|
||||||
|
- New MCP tool `audit_knowledge_graph` (+ `GET /api/v1/audit/drift`) — read-only, runs the
|
||||||
|
discovery+diff in one pass and returns a ranked report. Each finding = `{category, severity,
|
||||||
|
entities, evidence, suggested_runbook}`.
|
||||||
|
- Discovery sources (all via the canonical host-hop / existing tools): `pct list` + `pct
|
||||||
|
config` on hubris & strong (guests, `net0` IP, onboot state); `qm list` (VMs); Caddy admin
|
||||||
|
API / Caddyfile (routes → certs); docker `ps` on compose hosts; the `checks/*.sh` set vs
|
||||||
|
what's deployed at `/opt/oikos/checks/` per target.
|
||||||
|
- Report categories (the gaps this investigation found):
|
||||||
|
1. **Ghost entities** — in DB but not in Proxmox (e.g. stray `lxc:test-*`).
|
||||||
|
2. **Missing entities** — in Proxmox/Caddy/docker but no DB entity.
|
||||||
|
3. **Misplaced parent** — `hosts` edge disagrees with where the guest actually runs (the
|
||||||
|
rclone class — though rclone's parent is correct; this catches real migrations).
|
||||||
|
4. **Orphan/dead checks** — `check_defs` whose target is deprecated/destroyed, or random-slug
|
||||||
|
orphans (`^check:(ping|ssh-script|disk):[0-9a-f]{8}$`).
|
||||||
|
5. **Undeployed scripts** — checks expect `/opt/oikos/checks/<script>` but it's absent in
|
||||||
|
the guest (the strong-guest/rclone class).
|
||||||
|
6. **Unmonitored declared types** — reuse `coverageSweep` SQL (dns-zone today, agents).
|
||||||
|
7. **Seed drift** — entities/edges in DB but not in `seeds/inventory.yaml` (23 dns-records),
|
||||||
|
via `oikos export` diff.
|
||||||
|
8. **Unmodeled certs** — Caddy serves a cert with no `certificate` entity + `uses-certificate` edge.
|
||||||
|
9. **Knowledge rot** — delegate to the existing `knowledge_drift` endpoints (duplicates/orphans/tags).
|
||||||
|
|
||||||
|
**F2. Author the skill.**
|
||||||
|
- `.agents/skills/knowledge-graph-audit/SKILL.md` — front-matter
|
||||||
|
`risk_class: read_only`, `inputs: [scope?]`, `verification: "drift report returns ok"`.
|
||||||
|
Body: run `audit_knowledge_graph`, read the ranked report, and for each category point at
|
||||||
|
the remediation runbook (`lifecycle-deprecate-node`, `lifecycle-destroy-node`,
|
||||||
|
`config-change-deploy` for scripts, `lifecycle-migrate-node` for parents, this plan's
|
||||||
|
tracks for cert/seed/graph-cap work). No mutating steps.
|
||||||
|
- Seed a matching `runbook:knowledge-graph-audit` entity in `seeds/knowledge.yaml`
|
||||||
|
(bound by `applies_to_type`) so `search_knowledge`/`get_skills` surface it (also fixes the
|
||||||
|
empty-skills-table gap, Track D4).
|
||||||
|
|
||||||
|
**F3. Optional: periodic sweep (later).** Wrap categories 4/6 as a scheduler housekeeping
|
||||||
|
sweep that raises `drift` signals, mirroring `coverageSweep`. Out of scope for this plan
|
||||||
|
unless the operator wants continuous drift signals; the on-demand skill is the deliverable.
|
||||||
|
|
||||||
|
**Risk class:** `read_only`. The audit only reads (pct list/config, docker ps, Caddy API,
|
||||||
|
DB selects, an `oikos export` to a temp file). No writes. Safe to run unattended.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
After each track, verify via API (read-only, no approval):
|
||||||
|
|
||||||
|
- `GET /api/v1/entities/ws:mac-mini` → `health` ∈ {healthy,degraded} (not `down`).
|
||||||
|
- `GET /api/v1/entities/lxc:rclone` → `health` healthy (proves pct-routing through hubris;
|
||||||
|
rclone currently unreachable because it resolves to a mesh fqdn). Verify its checks now
|
||||||
|
route via `pct exec 132` on hubris.
|
||||||
|
- Strong guests (`lxc:grimmory`, `lxc:romm`, `lxc:seanime`) → ssh-script checks healthy
|
||||||
|
after scripts pushed inside + routed via strong's `pct exec`.
|
||||||
|
- `GET /api/v1/checks?include_disabled=false` → `down` count drops from 49 to the
|
||||||
|
genuinely-down set (republic-laptop asleep, real outages only). Re-run the per-class table.
|
||||||
|
- `GET /api/v1/entities/service:secrets-issuance` + its ingress → no enabled check.
|
||||||
|
- Orphan cleanup: `SELECT count(*) FROM check_defs cd JOIN entities e ON e.id=cd.entity_id
|
||||||
|
WHERE e.slug ~ '^check:(ping|ssh-script|disk):[0-9a-f]{8}$';` → 0.
|
||||||
|
- Test LXCs (C3): `SELECT count(*) FROM entities WHERE slug IN
|
||||||
|
('lxc:preflight-test','lxc:test-livewatch',...) AND state<>'destroyed';` → 0.
|
||||||
|
- Provision a throwaway LXC via Nomos → it auto-gets scripts + check_defs + passes
|
||||||
|
`health-check-answering` before reaching `active` (E1/E2).
|
||||||
|
- `GET /api/v1/entities?type=certificate&limit=1` → >0; cert-expiry checks created.
|
||||||
|
- `GET /api/v1/entities?type=skill&limit=50` → >0.
|
||||||
|
- `GET /api/v1/graph` node count > 500 (or infra fully represented with a layer filter).
|
||||||
|
- `oikos export` diff shows dns-record entities present; `git diff seeds/inventory.yaml`.
|
||||||
|
- Scheduler logs: `checkdefaults: declared check not created` warnings gone for dns-zone.
|
||||||
|
- **Canonical access (A1):** no scheduler code path SSHes a guest directly —
|
||||||
|
`grep -rn "sshExec" internal/scheduler` shows it only for `host:`/`ws:` targets; LXC/VM
|
||||||
|
go through the shared `pct exec`/`qm guest exec` resolver.
|
||||||
|
- **Audit skill (F1/F2):** `audit_knowledge_graph` MCP tool returns a ranked report with
|
||||||
|
the 9 categories; running it against current state reproduces this plan's findings
|
||||||
|
(orphan checks, stray test LXCs, undeployed scripts, seed drift, 0 certs). The skill
|
||||||
|
is read-only — confirm it performs no DB writes (audit-log shows only reads).
|
||||||
|
|
||||||
|
Unit/integration tests to add/update:
|
||||||
|
- `internal/checkdefaults` / shared `internal/remote` resolver: LXC/VM check routes via
|
||||||
|
`pct exec`/`qm guest exec` to the resolved proxmox host; resolver reads top-level `user`;
|
||||||
|
`public_ipv4` preferred for standalone-server (`defaults_test.go`).
|
||||||
|
- `internal/scheduler`: `ListEnabledCheckDefs` excludes deprecated/destroyed targets
|
||||||
|
(new test); `coverage_test.go` still green; `sshExec` no longer called for guest slugs.
|
||||||
|
- macOS script branches: assert JSON shape unchanged on `Darwin` (shunit2 or a smoke run).
|
||||||
|
- E1: new-entity creation triggers script push (mock pct/SSH in test).
|
||||||
|
- F1: `audit_knowledge_graph` against a fixture DB+mock discovery returns the expected
|
||||||
|
category counts (ghost, missing, orphan, undeployed, drift).
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **Canonical host-hop (A1)** makes each proxmox host the single SSH dependency for all its
|
||||||
|
guests. This is already true (pct exec requires the host up) and is a net improvement
|
||||||
|
(one credential vs many), but a host outage now fails all its guest checks together —
|
||||||
|
which is the *correct* blast radius (guests are unreachable when their host is down).
|
||||||
|
- **Routing change (A1)** alters how probes reach every guest — `config_mutation`. Verify
|
||||||
|
one LXC end-to-end (rclone via `pct exec 132`) before fanning out. Extracting
|
||||||
|
`resolveExecTarget` into a shared package keeps scheduler + MCP in lockstep.
|
||||||
|
- **Script push into guests (A2/E1)** writes to guest filesystems — `config_mutation`.
|
||||||
|
Idempotent + content-checked; never clobber a same-named operator script without diffing.
|
||||||
|
- **mac-mini root SSH**: do NOT enable root login; use `dtoro` (A3) — keeps macOS hardening.
|
||||||
|
- **`netbird-vps` sshd locked to hubris pubkey**: may need the scheduler key added or
|
||||||
|
proxying via hubris; confirm before assuming direct SSH works (A5).
|
||||||
|
- **`health-check-answering` gate (E2)** could block a legitimately-active entity whose
|
||||||
|
only working check is ICMP-blocked (haos). Allow the gate to pass on any non-`down`
|
||||||
|
reachable probe, or grant an operator override.
|
||||||
|
- **Audit skill (F1)** discovers infra via `pct`/Caddy/docker reads — keep it strictly
|
||||||
|
read_only; ensure discovery commands are in the read-only allowlist (no state change).
|
||||||
|
- **Seed re-export** can surface large diffs (cognition entities) — scope export to
|
||||||
|
topology entities, or review carefully before commit. Bump `VERSION` per repo rule.
|
||||||
|
- **Graph cap raise**: large node sets may slow the graph render; pair with a layer filter.
|
||||||
|
|
||||||
|
## Open questions (none blocking; confirm during implementation)
|
||||||
|
|
||||||
|
- republic-laptop: mesh-only roving laptop — keep ping-only (accept transient `down`) or
|
||||||
|
`monitoring: none`? (A5)
|
||||||
|
- secrets ingress: keep as archaeology or destroy the route? (C2)
|
||||||
|
- certificates: seed statically from Caddyfile, or auto-discover live from Caddy store? (D1)
|
||||||
|
- netbird-vps: add scheduler key to its sshd, or always proxy through hubris? (A5)
|
||||||
|
- Audit discovery for docker hosts/stacks: enumerate via `docker ps`, or model compose
|
||||||
|
stacks only? (F1)
|
||||||
|
|
||||||
|
## Suggested order
|
||||||
|
|
||||||
|
A1 (canonical host-hop routing — unblocks rclone + all guests) → A2 (push scripts into
|
||||||
|
guests) → A3 → A4 (mac-mini) → A5 → E1/E2 (automate for new entities) → B1 → C1 → C3 → C2
|
||||||
|
→ D2 (quick, silences dns noise) → F1/F2 (audit skill — also validates the above worked)
|
||||||
|
→ D1 → D4 → D3 → D5. Validate after each track.
|
||||||
@@ -0,0 +1,293 @@
|
|||||||
|
# 2026-07-30 — Session review: plan drift & a dead activity panel
|
||||||
|
|
||||||
|
**Status:** Done — 2026-08-03. Shipped in `467589d` (v0.14.1), deployed to
|
||||||
|
production. The two operator-reported complaints are resolved and verified on
|
||||||
|
the bug-report session itself (`398f5eda`); see [Resolution](#resolution-2026-08-03)
|
||||||
|
at the end. Items P0.1, P0.2 (fix 1+2), P1.1, P1.2 are complete; P0.2 fix 3,
|
||||||
|
P2.1, P2.2 are deferred (the reported symptoms no longer reproduce).
|
||||||
|
|
||||||
|
**Scope:** The five most-recently-active `agent:nomos` sessions by
|
||||||
|
`last_active_at`, pulled from the live Postgres on 2026-07-30, plus the
|
||||||
|
code paths they exercise (`cmd/nomos/store.go`, `cmd/nomos/tasks.go`,
|
||||||
|
`web/src/lib/stores/{activity,workspace,chat}.ts`,
|
||||||
|
`web/src/lib/components/UnifiedTimeline.svelte`).
|
||||||
|
**Trigger:** Operator report — "the plan was off, the activity sidepanel
|
||||||
|
was not kept up to date and feels off, not live."
|
||||||
|
|
||||||
|
Both complaints are real, both reproduce deterministically, and both have
|
||||||
|
a single-line root cause. They are *not* the same bug, but they compound:
|
||||||
|
the plan bug produces the exact event stream that the activity panel
|
||||||
|
silently discards.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Sessions reviewed
|
||||||
|
|
||||||
|
| # | sid | goal (short) | outcome | activity rows | plan gens | re-planned? |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| 1 | `398f5eda` | hubris recurring network outage → EEE mitigation | success | 48 | 2 | **yes** |
|
||||||
|
| 2 | `0a49ba3d` | triage active signals on host:strong | success | 30 | 1 | no |
|
||||||
|
| 3 | `9368633d` | sensor temperatures on host:strong | success | 12 | 1 | no |
|
||||||
|
| 4 | `2065a29a` | temps → pivot to "fun fact about chickens" | success | 18 | 2 | **yes** |
|
||||||
|
| 5 | `bad26076` | greeting / responsiveness test | success | 6 | 1 | no |
|
||||||
|
|
||||||
|
**Score: 5 success / 0 partial / 0 failed.** The agent's *reasoning* was
|
||||||
|
fine in all five. Every defect below is in the bookkeeping and the
|
||||||
|
rendering — the parts the operator actually looks at.
|
||||||
|
|
||||||
|
**The correlation that matters: both sessions that re-planned (`398f5eda`,
|
||||||
|
`2065a29a`) recorded a corrupt plan. Neither of the three that didn't
|
||||||
|
re-plan did.** Re-planning was a 100% failure path (pre-fix).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P0.1 — `update_plan_step` addresses the wrong plan generation
|
||||||
|
|
||||||
|
This is "the plan was off," and it was fully deterministic.
|
||||||
|
|
||||||
|
`proposePlan` numbered a new generation's steps *continuing* from the old
|
||||||
|
one (`store.go:937`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
seq := startSeq + i + 1 // startSeq = MAX(seq) of all prior steps
|
||||||
|
```
|
||||||
|
|
||||||
|
So on generation 2 of `398f5eda`, the six new steps landed at **seq 7–12**.
|
||||||
|
|
||||||
|
But the tool result the model got back never mentioned those numbers
|
||||||
|
(`tasks.go:282`):
|
||||||
|
|
||||||
|
```
|
||||||
|
"Plan set (6 steps). If all steps are read-only, execute now — …"
|
||||||
|
```
|
||||||
|
|
||||||
|
…while `update_plan_step`'s schema told it (`tasks.go:81`):
|
||||||
|
|
||||||
|
```go
|
||||||
|
"seq": "1-based step number from propose_plan."
|
||||||
|
```
|
||||||
|
|
||||||
|
The model had no way to learn the real seq numbers and was explicitly told
|
||||||
|
to use 1-based ones. It did exactly that.
|
||||||
|
|
||||||
|
**What the DB recorded for `398f5eda`:**
|
||||||
|
|
||||||
|
```
|
||||||
|
20:07:40 propose_plan → gen 2 created at seq 7..12
|
||||||
|
gen 1 (seq 1..6) marked `replaced`
|
||||||
|
20:08:10 update_plan_step seq=1 running ← hits gen-1 step 1
|
||||||
|
20:08:18 update_plan_step seq=1 done ← resurrects a `replaced` row
|
||||||
|
20:08:18 update_plan_step seq=2 running
|
||||||
|
…
|
||||||
|
20:09:57 complete_task
|
||||||
|
```
|
||||||
|
|
||||||
|
Result — the persisted plan was a lie in three separate ways:
|
||||||
|
|
||||||
|
- **Steps 1–5 (the abandoned "force 1Gbps" plan) show `done`** with real
|
||||||
|
start/finish timestamps. Work that was never performed was recorded as
|
||||||
|
performed. `updatePlanStep` wrote status by seq with no guard, so it
|
||||||
|
happily flipped `replaced` → `running` → `done`.
|
||||||
|
- **Steps 7–12 (the actual EEE work that ran) had `started_at = NULL`**
|
||||||
|
and were bulk-closed to `done` by `completeTask`'s auto-close sweep
|
||||||
|
(`store.go:1096`) at 20:09:57 — all six sharing one timestamp.
|
||||||
|
- **The panel shows 12 steps**, because `getPlanSteps` returned every
|
||||||
|
generation unfiltered (`store.go:1356`) and the frontend never reads the
|
||||||
|
`generation` field at all (`grep generation web/src` → zero hits outside
|
||||||
|
the API type).
|
||||||
|
|
||||||
|
`2065a29a` had the identical signature: gen 2 at seq 4–5, gen-1 steps 1
|
||||||
|
and 2 flipped to `done`/`skipped` four seconds later.
|
||||||
|
|
||||||
|
### Fix (implemented)
|
||||||
|
|
||||||
|
1. **Make seq generation-relative.** `proposePlan` resets seq to `1..N` per
|
||||||
|
generation; `(session_id, generation, seq)` is the addressing key.
|
||||||
|
`updatePlanStep` resolves against `MAX(generation)`. This matches what
|
||||||
|
the model naturally does and what every prompt already says.
|
||||||
|
2. **Return the seq numbers to the model.** The `propose_plan` result now
|
||||||
|
enumerates them (`1=…; 2=…`).
|
||||||
|
3. **Refuse writes to superseded rows.** `updatePlanStep` addresses only
|
||||||
|
the current generation; a stale/out-of-range seq returns
|
||||||
|
`errPlanStepNotFound` (never resurrects a `replaced` row).
|
||||||
|
4. **Filter by generation on read.** `getPlanSteps` returns only
|
||||||
|
`MAX(generation)` by default; `?all=true` for the audit/eval view.
|
||||||
|
5. **Stamp `started_at` in the auto-close sweep.** `completeTask` closing
|
||||||
|
a step sets `started_at = COALESCE(started_at, now())`.
|
||||||
|
|
||||||
|
A migration (`029`) renumbers existing rows to per-generation `1..N` and
|
||||||
|
replaces the `(session_id, seq)` index with a unique
|
||||||
|
`(session_id, generation, seq)`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P0.2 — The activity panel invents its own timestamps
|
||||||
|
|
||||||
|
This is "not live / feels off," and it was worse than a staleness bug: the
|
||||||
|
times on screen were **fabricated at render time**.
|
||||||
|
|
||||||
|
`activity.ts:118` — every tool entry:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
timestamp: now - ($msgs.length - mi) * 1000
|
||||||
|
```
|
||||||
|
|
||||||
|
`now` was `Date.now()` captured at the top of `computeActivityLog`. So a
|
||||||
|
tool call's displayed time was *"the moment this function last ran, minus
|
||||||
|
one second per message from the end."* Not when the call happened.
|
||||||
|
|
||||||
|
Three consequences, all of which read as "not live":
|
||||||
|
|
||||||
|
- **The clock was wrong.** `UnifiedTimeline` rendered these through
|
||||||
|
`hhmm()` / `hhmmss()`, so opening yesterday's session showed every step
|
||||||
|
timestamped *right now*, one second apart.
|
||||||
|
- **It churned every 3 seconds.** The message poller re-set `messages`
|
||||||
|
unconditionally on every tick, which re-derived `activityLog`, which
|
||||||
|
re-captured `now`. Every entry's timestamp marched forward 3s at a time,
|
||||||
|
forever. Motion with no information.
|
||||||
|
- **Real and fake timestamps sorted together.** Plan steps used the
|
||||||
|
genuine `started_at`; tool calls used the synthetic value; the final
|
||||||
|
sort mixed them. Steps with no `started_at` fell back to `now` — **97 of
|
||||||
|
339 non-pending steps in the DB (29%) had `started_at = NULL`** — so they
|
||||||
|
landed at the bottom of the timeline regardless of when they ran.
|
||||||
|
|
||||||
|
The real data already existed and was already served: `agent_activity`
|
||||||
|
holds true `ts`, `duration_ms`, `success`, and
|
||||||
|
`correlation_id = session_id`, exposed at `GET /agent-activity`. The panel
|
||||||
|
ignored it and reconstructed a worse version from the message blob.
|
||||||
|
|
||||||
|
### Fix (implemented — fix 1 + 2)
|
||||||
|
|
||||||
|
1. **Carry real timestamps on tool calls.** `computeActivityLog` uses each
|
||||||
|
tool call's message `created_at` (a true persisted time). The
|
||||||
|
`now - (len - mi) * 1000` expression is gone entirely.
|
||||||
|
2. **Only fall back to wall-clock for genuinely-live entries, and freeze
|
||||||
|
it once assigned** — a `Map<id, timestamp>` outside the derivation, so
|
||||||
|
re-deriving never moves an existing entry. This is what kills the churn.
|
||||||
|
|
||||||
|
Deferred to a later pass: backing the panel with `agent_activity` for
|
||||||
|
historical sessions (fix 3, unlocks `duration_ms`) — the two reported
|
||||||
|
symptoms (wrong clock, churn) no longer reproduce without it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P1.1 — Plan-step events for a superseded generation were silently dropped
|
||||||
|
|
||||||
|
The frontend half of P0.1, and the reason the panel *froze* rather than
|
||||||
|
merely showing wrong steps.
|
||||||
|
|
||||||
|
On `plan.proposed` with `appended: false`, the store replaced its step
|
||||||
|
list wholesale — so after the re-plan it held seq 7–12. Every subsequent
|
||||||
|
`plan.step.started` / `plan.step.finished` carried seq 1–5 and a gen-1
|
||||||
|
`step_id`, and `applyPlanStepEventTo` bailed on no match:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
if (i === -1) return steps
|
||||||
|
```
|
||||||
|
|
||||||
|
So for the entire second half of `398f5eda` — the half where all the real
|
||||||
|
work happened — **the panel showed six pending steps and nothing ever
|
||||||
|
moved.** Then `completeTask` closed them in the DB while emitting only
|
||||||
|
`task.status`, no per-step events, so they stayed pending on screen even
|
||||||
|
after the session finished.
|
||||||
|
|
||||||
|
### Fix (implemented)
|
||||||
|
|
||||||
|
- Fixing P0.1 removed the cause (the events now carry the correct
|
||||||
|
generation-relative seq + the panel's current steps match). The `i === -1`
|
||||||
|
branch now `console.warn`s and increments an exported
|
||||||
|
`droppedPlanStepEvents` counter instead of returning silently, so the
|
||||||
|
next divergence is visible instead of looking like a dead UI.
|
||||||
|
- **`completeTask`'s auto-close sweep now emits `plan.step.finished` per
|
||||||
|
closed step** (scoped to the current generation). General rule enforced:
|
||||||
|
no plan-step status change without a corresponding event.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P1.2 — Every plan carried a duplicate writeback step
|
||||||
|
|
||||||
|
In `398f5eda` gen 2, step 11 was the model's own writeback step and step 12
|
||||||
|
was the auto-appended one. The detector substring-matched the literal tool
|
||||||
|
names `update_entity_attributes` / `create_relationship` in the title or
|
||||||
|
detail; the model wrote a natural-language equivalent, so the match failed
|
||||||
|
and a redundant step was appended. Same pattern in `0a49ba3d` and
|
||||||
|
`9368633d`.
|
||||||
|
|
||||||
|
### Fix (implemented)
|
||||||
|
|
||||||
|
Broadened the detector to a case-insensitive check for `write back` /
|
||||||
|
`writeback` / `upsert_knowledge` in the title or detail, on top of the
|
||||||
|
existing tool-name match.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P2.1 — Long unexplained stalls, invisible in the UI *(deferred)*
|
||||||
|
|
||||||
|
- `bad26076`: a greeting took **16 minutes** wall-clock with 6 activity rows.
|
||||||
|
- `2065a29a`: step 1 showed `started_at` → `finished_at` spanning **16 minutes**
|
||||||
|
for a `sensors` call that returned in milliseconds.
|
||||||
|
|
||||||
|
The work took under a second; the step was *open* for 16 minutes. The panel
|
||||||
|
has no way to distinguish "working" from "waiting for a nudge." Surfaces a
|
||||||
|
step's idle time: mark a `running` step *stalled* when it has had no
|
||||||
|
`agent_activity` row for >60s. Deferred — needs the `agent_activity`-backed
|
||||||
|
panel (P0.2 fix 3).
|
||||||
|
|
||||||
|
## P2.2 — `agent_activity` is a single-type table *(deferred — decision)*
|
||||||
|
|
||||||
|
All rows are `activity_type = 'tool_call'`. Either start emitting the other
|
||||||
|
types the schema anticipates (`reasoning`, `plan`, `error`) or drop the
|
||||||
|
dimension. Worth a decision, not urgent.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended sequence (executed)
|
||||||
|
|
||||||
|
| Order | Item | Status |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | P0.1 fix 3 + 4 (refuse superseded writes, filter on read) | done |
|
||||||
|
| 2 | P0.2 fix 1 + 2 (real timestamps, frozen fallback) | done |
|
||||||
|
| 3 | P1.1 (emit events from the auto-close sweep) | done |
|
||||||
|
| 4 | P0.1 fix 2 (generation-relative seq) + migration | done |
|
||||||
|
| 5 | P1.2, P2.1 | P1.2 done; P2.1 deferred |
|
||||||
|
| 6 | P0.2 fix 3 (back the panel with `agent_activity`) | deferred |
|
||||||
|
| 7 | P2.2 | deferred |
|
||||||
|
|
||||||
|
## Regression coverage (added)
|
||||||
|
|
||||||
|
- `store_test.go`: `TestUpdatePlanStep_GenerationRelative` — re-plan →
|
||||||
|
`update_plan_step(seq=1)` must address gen-2 and never resurrect a
|
||||||
|
superseded gen-1 `replaced` row; out-of-range seq → `errPlanStepNotFound`.
|
||||||
|
- `store_test.go`: `TestCompleteTask_AutoCloseEmitsEvents` — auto-close
|
||||||
|
emits one `plan.step.finished` per closed step and stamps `started_at`.
|
||||||
|
- `store_test.go`: `TestProposePlan_RefuseInFlight` — updated for
|
||||||
|
generation-relative seq + `?all=true`.
|
||||||
|
- `web/src/lib/stores/activity.test.ts`: `computeActivityLog` is pure w.r.t.
|
||||||
|
wall-clock (two calls 50ms apart → identical output), persisted tool calls
|
||||||
|
use real `created_at`, live entries freeze instead of churning.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolution (2026-08-03)
|
||||||
|
|
||||||
|
Shipped in commit `467589d` (VERSION `0.14.0` → `0.14.1`), pushed to
|
||||||
|
`origin/main`, deployed via the Gitea webhook (`scripts/deploy.sh`):
|
||||||
|
pg_dump → pull → `docker compose build` → `up -d` → health check (healthy).
|
||||||
|
|
||||||
|
Verification on the bug-report session `398f5eda` post-migration:
|
||||||
|
|
||||||
|
```
|
||||||
|
gen 1: seq 1..6 (the abandoned "force 1Gbps" plan — superseded)
|
||||||
|
gen 2: seq 1..6 (the real EEE work — was seq 7..12, now normalized to 1..6)
|
||||||
|
```
|
||||||
|
|
||||||
|
- `schema_migrations` v29 applied; old `idx_plan_steps_session` dropped,
|
||||||
|
unique `idx_plan_steps_session_gen_seq` in place.
|
||||||
|
- Containers recreated; `healthz` and `/agent/sessions/:id/plan` HTTP 200.
|
||||||
|
- Full `cmd/nomos` suite (23 tests) + web suite (70 tests) green; `go vet`
|
||||||
|
clean; ESLint/Prettier clean.
|
||||||
|
|
||||||
|
Note: historical `started_at = NULL` on already-completed steps (visible on
|
||||||
|
`398f5eda` gen 2) is left as-is — backfilling would fabricate times. Going
|
||||||
|
forward `completeTask` stamps `started_at`, and the frontend freezes
|
||||||
|
NULL-started steps stably so they no longer churn.
|
||||||
503
plans/done/2026-08-03-cyberspace-style-adoption.md
Normal file
503
plans/done/2026-08-03-cyberspace-style-adoption.md
Normal file
@@ -0,0 +1,503 @@
|
|||||||
|
# 2026-08-03 — Adopt cyberspace.online terminal aesthetic + dithered images
|
||||||
|
|
||||||
|
**Status:** Implemented in v0.16.0 (`757ef2f`). Shipped as a **full theme
|
||||||
|
replacement** (Terracotta/Carbon → cyberspace BBS/terminal style), not the
|
||||||
|
opt-in addition originally drafted below — the operator chose full replacement
|
||||||
|
during execution (see decision `theme.replace_with_cyberspace`). The `<RasterImage>`
|
||||||
|
Atkinson-dithering component and the warm-cream/JetBrains-Mono look landed as
|
||||||
|
drafted; only the "opt-in vs replace" scope changed.
|
||||||
|
|
||||||
|
Adopt the look of https://cyberspace.online/ (a BBS / "social media
|
||||||
|
de-imagined" terminal aesthetic) as a **new, opt-in theme family** in oikos,
|
||||||
|
with **both light and dark variants**, plus a reusable **`<RasterImage>`**
|
||||||
|
component that renders images to a `<canvas>` with Atkinson dithering (the
|
||||||
|
"kinda dithered" image style). The existing Terracotta/Carbon themes stay the
|
||||||
|
default; this adds, it does not replace.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## TL;DR
|
||||||
|
|
||||||
|
1. Add a third theme family — **"Cyberspace Dark"** and **"Cyberspace Light"** —
|
||||||
|
wired through the same `--background` / `--foreground` / … token layer every
|
||||||
|
component already uses, so nothing in the UI tree changes; only the tokens
|
||||||
|
get new values. Square corners (`--radius: 0`), warm cream-on-black, mono
|
||||||
|
everything.
|
||||||
|
2. Extend `web/src/lib/stores/theme.svelte.ts` from a 2-state `'light'|'dark'`
|
||||||
|
toggle to a named-theme model, keeping `.dark` class behavior for
|
||||||
|
compatibility.
|
||||||
|
3. Self-host JetBrains Mono (body) + a pixel/terminal face (VT323 or Departure
|
||||||
|
Mono) for the logo/headings accents, replacing the Google Fonts `<link>`.
|
||||||
|
4. Build `web/src/lib/components/RasterImage.svelte`: draws any image to a
|
||||||
|
`<canvas>` reduced to a 2-color (theme `fg`/`bg`) palette via **Atkinson
|
||||||
|
dithering**, with an `<img>` fallback and a skeleton placeholder — exactly
|
||||||
|
the cyberspace pattern. Re-renders when the theme changes (palette flips).
|
||||||
|
5. Optional cosmetic idioms (terminal-box focus ring, braille spinner, `<s>`
|
||||||
|
strike lists) as small additive utilities, not a redesign.
|
||||||
|
|
||||||
|
The whole thing is **non-breaking and incremental**: each step ships behind the
|
||||||
|
existing theme picker, so Terracotta/Carbon users see nothing until they opt in.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Extracted style spec (source of truth from cyberspace.online)
|
||||||
|
|
||||||
|
Captured from the live site's SSR HTML + inline boot script. This is the
|
||||||
|
reference the tokens below are derived from.
|
||||||
|
|
||||||
|
### 1.1 Color model
|
||||||
|
|
||||||
|
Cyberspace defines **exactly three colors per theme** — `fg`, `bg`, `fgDim` —
|
||||||
|
applied to CSS custom properties. Everything else (borders, primary, cards) is
|
||||||
|
*derived* from those three. There are 11 named themes total; the two we care
|
||||||
|
about:
|
||||||
|
|
||||||
|
| Theme | `fg` (text) | `bg` (canvas) | `fgDim` (muted) |
|
||||||
|
|---------|--------------|---------------|-----------------|
|
||||||
|
| Dark | `#efe5c0` | `#000000` | `#a89984` |
|
||||||
|
| Light | `#000000` | `#efe5c0` | `#3a3a3a` |
|
||||||
|
|
||||||
|
Note the elegance: **light and dark are exact inverses** — they share the same
|
||||||
|
warm cream (`#efe5c0`, a Gruvbox-ish paper tone) and just swap which side of it
|
||||||
|
is ink vs. paper. The muted tone `#a89984` is straight out of the Gruvbox
|
||||||
|
palette. This is why both themes read as "the same site" despite opposite
|
||||||
|
polarity.
|
||||||
|
|
||||||
|
Boot-time fallback (the site's original/GRiD theme) is amber `#FF9810` on
|
||||||
|
`#120900` — useful as a *third* optional accent if we ever want a true-phosphor
|
||||||
|
variant.
|
||||||
|
|
||||||
|
### 1.2 Type
|
||||||
|
|
||||||
|
- **Body / mono:** JetBrains Mono (self-hosted `.woff2`, Regular).
|
||||||
|
- **Boot + logo accents:** Departure Mono (self-hosted `.woff2`). A quirky
|
||||||
|
monospace; VT323 (Google, free) is a close, easy substitute.
|
||||||
|
- **Stylized wordmark** (`ᑕ¥βєяรקค¢є`, class `.font-vt`): a terminal/pixel face.
|
||||||
|
Rule lives in their external `entry.*.css` (not in the SSR dump); VT323 is the
|
||||||
|
safe assumption.
|
||||||
|
|
||||||
|
cyberspace sets `font-mono` on the root wrapper — the **entire UI is
|
||||||
|
monospace**. There is no proportional body face. Headings use the same mono
|
||||||
|
family at larger size / normal weight.
|
||||||
|
|
||||||
|
### 1.3 Layout & component idioms
|
||||||
|
|
||||||
|
- **Left rail nav:** fixed, icon-only when minimized (~80px), expands on click.
|
||||||
|
Square buttons, Phosphor icons, uppercase `text-xs` labels.
|
||||||
|
- **`.terminal-box`:** the universal card. Bordered (`border border-border`),
|
||||||
|
**square corners** (`rounded-none` everywhere — `--radius` is effectively 0),
|
||||||
|
and on focus/emphasis gets `ring-2 ring-fg` (a 2px ring in the foreground
|
||||||
|
color).
|
||||||
|
- **Emphasis by inversion:** active/primary state is `bg-fg text-bg` — fill with
|
||||||
|
foreground ink, text becomes the canvas color. No separate "accent" hue; the
|
||||||
|
accent *is* fg.
|
||||||
|
- **Strikethrough as a feature list:** `<s>Ads</s> <s>Videos</s> …` — crossed-out
|
||||||
|
`<s>` elements spell out what the product removes. Cheap, on-brand.
|
||||||
|
- **Braille spinner:** `BrailleSpinner` component animates braille block chars
|
||||||
|
(`⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏`) for loading states instead of a circle.
|
||||||
|
- **Max content width** `max-w-4xl`, centered; generous vertical rhythm; thin
|
||||||
|
2px scrollbars colored `--color-border`.
|
||||||
|
- Borders are `1px solid` in a border color derived from `fg`/`fgDim` at low
|
||||||
|
alpha (their `--color-border` is not literally in the dump, but every
|
||||||
|
bordered surface uses it, and it tracks `fg`).
|
||||||
|
|
||||||
|
### 1.4 The dithered image (`RasterImage`) — what we actually know
|
||||||
|
|
||||||
|
From the SSR HTML the component is unambiguous about its *shape*, silent on its
|
||||||
|
*algorithm* (the dither JS is in an external `/_nuxt/*.js` bundle not present in
|
||||||
|
the page dump):
|
||||||
|
|
||||||
|
- Renders a **`<canvas>`** as primary output, with an **`<img>` fallback** as a
|
||||||
|
sibling. Parent selectors `[&>canvas]:max-w-full [&>canvas]:h-auto` and
|
||||||
|
`[&>img]:…` size both responsively.
|
||||||
|
- Emits a **`.raster-image-skeleton`** placeholder (empty div, `background:
|
||||||
|
var(--color-bg)`) during SSR/before hydration — no flash of the raw photo.
|
||||||
|
- Scoped styles (`data-v-4d61df89`): `.raster-image { display:block }`,
|
||||||
|
`.raster-image-skeleton { display:block; background:var(--color-bg) }`.
|
||||||
|
|
||||||
|
**Inferred technique** (standard for this look): Canvas 2D → `drawImage` →
|
||||||
|
`getImageData` → per-pixel luminance reduction to a 2-color palette (`fg`/`bg`)
|
||||||
|
with an **error-diffusion** pass (Atkinson or Floyd–Steinberg) → `putImageData`.
|
||||||
|
This produces the characteristic speckled 1-bit halftone. Target is almost
|
||||||
|
certainly the theme's own fg/bg, which is *why* the dithered art recolors
|
||||||
|
correctly when you flip themes.
|
||||||
|
|
||||||
|
We will implement Atkinson (see §4) — it's the classic Mac/BBS dither, slightly
|
||||||
|
softer than Floyd–Steinberg, and matches "kinda dithered" precisely.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Recommended approach: opt-in theme family (not a rebrand)
|
||||||
|
|
||||||
|
oikos today = Art-Nouveau / terracotta / rounded / serif-heading (Inknut
|
||||||
|
Antiqua), floating-window desktop shell. cyberspace = BBS / mono / square /
|
||||||
|
cream-on-black. These are **opposite poles**; a flat rebrand would discard the
|
||||||
|
existing art direction and rework every component's rounding/spacing.
|
||||||
|
|
||||||
|
**Decision: add cyberspace as a new theme family, selectable in the existing
|
||||||
|
theme picker.** This is low-risk, reversible, and lets the dithered images +
|
||||||
|
terminal idioms land incrementally. The full-rebrand alternative is documented
|
||||||
|
in §7 for if you later decide to make it the default.
|
||||||
|
|
||||||
|
Because every oikos component consumes colors through the Tailwind v4 token
|
||||||
|
layer (`--background`, `--foreground`, `--card`, `--border`, `--primary`, …)
|
||||||
|
defined in `web/src/app.css` `@theme inline`, a new theme is **just a new set
|
||||||
|
of values for those same custom properties** — zero component edits required
|
||||||
|
for the recolor. That indirection is the whole reason this is cheap.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Theme token additions (`web/src/app.css`)
|
||||||
|
|
||||||
|
Add two new blocks alongside the existing `:root` (Terracotta) and `.dark`
|
||||||
|
(Carbon). They set the *same* token names to cyberspace's values, plus pin
|
||||||
|
`--radius: 0` for square corners and remap fonts (see §5).
|
||||||
|
|
||||||
|
Driven by a `data-theme` attribute on `<html>` (set by the store, §6), so all
|
||||||
|
four states — Terracotta, Carbon, Cyberspace Dark, Cyberspace Light — coexist:
|
||||||
|
|
||||||
|
```css
|
||||||
|
/* ── Cyberspace Dark (cream on black) ── */
|
||||||
|
:root[data-theme='cyber-dark'] {
|
||||||
|
--radius: 0px;
|
||||||
|
--background: #000000;
|
||||||
|
--foreground: #efe5c0;
|
||||||
|
--card: #000000; /* cyberspace has no card tint; cards are just bordered bg */
|
||||||
|
--card-foreground: #efe5c0;
|
||||||
|
--popover: #000000;
|
||||||
|
--popover-foreground: #efe5c0;
|
||||||
|
--primary: #efe5c0; /* emphasis = fg ink */
|
||||||
|
--primary-foreground: #000000; /* inverted */
|
||||||
|
--secondary: #1a1a1a;
|
||||||
|
--secondary-foreground: #efe5c0;
|
||||||
|
--muted: #141414;
|
||||||
|
--muted-foreground: #a89984; /* fgDim */
|
||||||
|
--accent: #efe5c0;
|
||||||
|
--accent-foreground: #000000;
|
||||||
|
--destructive: #cc241d; /* Gruvbox red, sits in the same palette */
|
||||||
|
--destructive-foreground: #efe5c0;
|
||||||
|
--border: color-mix(in oklab, #efe5c0 22%, transparent); /* fg-derived hairline */
|
||||||
|
--input: color-mix(in oklab, #efe5c0 28%, transparent);
|
||||||
|
--ring: #efe5c0; /* the ring-2 ring-fg look */
|
||||||
|
--sidebar: #000000;
|
||||||
|
--sidebar-foreground: #efe5c0;
|
||||||
|
--sidebar-primary: #efe5c0;
|
||||||
|
--sidebar-primary-foreground: #000000;
|
||||||
|
--sidebar-accent: #1a1a1a;
|
||||||
|
--sidebar-accent-foreground: #efe5c0;
|
||||||
|
--sidebar-border: color-mix(in oklab, #efe5c0 22%, transparent);
|
||||||
|
--sidebar-ring: #efe5c0;
|
||||||
|
--chart-1: #efe5c0; --chart-2: #a89984; --chart-3: #fabd2f;
|
||||||
|
--chart-4: #b8bb26; --chart-5: #83a598; /* Gruvbox for charts */
|
||||||
|
--success: #b8bb26; --warning: #fabd2f;
|
||||||
|
|
||||||
|
/* oikos semantic aliases (app.css :root block) */
|
||||||
|
--bg: var(--background); --bg-surface: var(--card); --bg-deeper: #050505;
|
||||||
|
--bg-hover: var(--secondary); --bg-active: var(--accent);
|
||||||
|
--text: var(--foreground); --text-muted: var(--muted-foreground);
|
||||||
|
--accent-blue: #83a598; --accent-green: var(--success);
|
||||||
|
--accent-red: var(--destructive); --accent-orange: var(--warning);
|
||||||
|
|
||||||
|
/* terminal face for this theme only (see §5) */
|
||||||
|
--font-sans: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||||
|
--font-mono: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||||
|
--font-heading: 'VT323', 'JetBrains Mono', monospace; /* pixel wordmark feel */
|
||||||
|
}
|
||||||
|
|
||||||
|
/* ── Cyberspace Light (black on cream paper) — exact inverse ── */
|
||||||
|
:root[data-theme='cyber-light'] {
|
||||||
|
--radius: 0px;
|
||||||
|
--background: #efe5c0;
|
||||||
|
--foreground: #000000;
|
||||||
|
--card: #efe5c0;
|
||||||
|
--card-foreground: #000000;
|
||||||
|
--popover: #efe5c0;
|
||||||
|
--popover-foreground: #000000;
|
||||||
|
--primary: #000000;
|
||||||
|
--primary-foreground: #efe5c0;
|
||||||
|
--secondary: #e0d6b0;
|
||||||
|
--secondary-foreground: #000000;
|
||||||
|
--muted: #e6dcc0;
|
||||||
|
--muted-foreground: #3a3a3a; /* fgDim */
|
||||||
|
--accent: #000000;
|
||||||
|
--accent-foreground: #efe5c0;
|
||||||
|
--destructive: #9d0006;
|
||||||
|
--destructive-foreground: #efe5c0;
|
||||||
|
--border: color-mix(in oklab, #000000 22%, transparent);
|
||||||
|
--input: color-mix(in oklab, #000000 28%, transparent);
|
||||||
|
--ring: #000000;
|
||||||
|
--sidebar: #efe5c0;
|
||||||
|
--sidebar-foreground: #000000;
|
||||||
|
--sidebar-primary: #000000;
|
||||||
|
--sidebar-primary-foreground: #efe5c0;
|
||||||
|
--sidebar-accent: #e0d6b0;
|
||||||
|
--sidebar-accent-foreground: #000000;
|
||||||
|
--sidebar-border: color-mix(in oklab, #000000 22%, transparent);
|
||||||
|
--sidebar-ring: #000000;
|
||||||
|
--chart-1: #000000; --chart-2: #3a3a3a; --chart-3: #b57614;
|
||||||
|
--chart-4: #79740e; --chart-5: #076678;
|
||||||
|
--success: #79740e; --warning: #b57614;
|
||||||
|
|
||||||
|
--bg: var(--background); --bg-surface: var(--card); --bg-deeper: #e6dcc0;
|
||||||
|
--bg-hover: var(--secondary); --bg-active: var(--accent);
|
||||||
|
--text: var(--foreground); --text-muted: var(--muted-foreground);
|
||||||
|
--accent-blue: #076678; --accent-green: var(--success);
|
||||||
|
--accent-red: var(--destructive); --accent-orange: var(--warning);
|
||||||
|
|
||||||
|
--font-sans: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||||
|
--font-mono: 'JetBrains Mono', ui-monospace, Menlo, monospace;
|
||||||
|
--font-heading: 'VT323', 'JetBrains Mono', monospace;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Two notes:
|
||||||
|
|
||||||
|
- **`.dark` vs `data-theme`.** The current store flips `.dark` on `<html>`. To
|
||||||
|
keep Carbon working unchanged, leave `.dark` logic alone and layer
|
||||||
|
`data-theme` on top: when a cyberspace theme is active the store sets
|
||||||
|
`data-theme` and **removes** `.dark` (cyberspace themes are self-contained —
|
||||||
|
they set both polarities explicitly). See §6.
|
||||||
|
- **Borders from `fg`.** cyberspace's hairline tracks the foreground, not a
|
||||||
|
fixed gray. `color-mix(in oklab, <fg> 22%, transparent)` reproduces that and
|
||||||
|
auto-flips between the two themes. Tune the % after visual review.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. The dithered image component (`RasterImage.svelte`)
|
||||||
|
|
||||||
|
**File:** `web/src/lib/components/RasterImage.svelte` (sibling of the existing
|
||||||
|
`Spinner.svelte`).
|
||||||
|
|
||||||
|
### 4.1 API
|
||||||
|
|
||||||
|
```svelte
|
||||||
|
<RasterImage src={entity.iconUrl} alt="host icon" width={320} />
|
||||||
|
<!-- optional: scale (downsample factor), threshold bias, mono palette override -->
|
||||||
|
```
|
||||||
|
|
||||||
|
- `src`, `alt` — as `<img>`.
|
||||||
|
- `width` — render width in CSS px; canvas is sized to this × natural aspect.
|
||||||
|
Downscaling before dithering is what sells the "lo-fi" look (defaults ~256–
|
||||||
|
320). Expose `scale` (0–1) to control.
|
||||||
|
- Reads the active theme's `--foreground` / `--background` via
|
||||||
|
`getComputedStyle(document.documentElement)` so the dither palette **follows
|
||||||
|
the theme** (cream/black in cyber-dark, black/cream in cyber-light, and
|
||||||
|
perfectly sensible in Terracotta/Carbon too).
|
||||||
|
|
||||||
|
### 4.2 Behavior
|
||||||
|
|
||||||
|
1. Show `.raster-image-skeleton` (empty, `background: var(--background)`) until
|
||||||
|
the source image loads — matches cyberspace's no-flash placeholder.
|
||||||
|
2. On load: create an offscreen canvas at `width × (h/w*width)`, `drawImage`
|
||||||
|
(with `imageSmoothingEnabled = true` for the downscale), pull
|
||||||
|
`getImageData`.
|
||||||
|
3. Run **Atkinson dithering** to 2 colors:
|
||||||
|
- For each pixel: luminance `Y = 0.299R + 0.587G + 0.114B`.
|
||||||
|
- Threshold at 128 (+ optional `bias`), snap to either `fg` or `bg`.
|
||||||
|
- Push **1/8 of the quantization error** to each of 6 neighbors (Atkinson's
|
||||||
|
kernel): right, below-left, below, below-right, and two pixels down on the
|
||||||
|
next-next row. (Atkinson diffuses less than Floyd–Steinberg → softer, more
|
||||||
|
"screen-printed" — exactly the cyberspace feel.)
|
||||||
|
- Write `fg`/`bg` (read from CSS vars at render time) into the buffer.
|
||||||
|
4. `putImageData`. Canvas is the visible output; the loaded `<img>` is kept as
|
||||||
|
`aria-hidden` fallback for no-JS / copy-image / accessibility.
|
||||||
|
5. **Re-dither on theme change**: subscribe to the theme store; when it flips,
|
||||||
|
re-read `--foreground`/`--background` and re-run steps 3–4 (cheap — the
|
||||||
|
decoded `ImageBitmap` is cached, only the palette pass reruns). This is the
|
||||||
|
detail that makes the art flip polarity with the theme toggle.
|
||||||
|
6. **Respect `prefers-reduced-data` / reduced motion?** Dithering is not motion,
|
||||||
|
but offer a `plain` prop to skip the canvas and render the raw `<img>` for
|
||||||
|
users who want crisp photos (e.g. entity detail screens where legibility
|
||||||
|
beats aesthetic).
|
||||||
|
|
||||||
|
### 4.3 Reference dither kernel (Atkinson)
|
||||||
|
|
||||||
|
```
|
||||||
|
* → 1/8 1/8
|
||||||
|
1/8 1/8 1/8 (current pixel = *)
|
||||||
|
1/8 1/8 (* is at top-left of this 4×? — see standard Atkinson spread)
|
||||||
|
```
|
||||||
|
|
||||||
|
Spread pattern (error e from pixel at (x,y) distributed):
|
||||||
|
|
||||||
|
```
|
||||||
|
px x+1 (1/8) x+2 (1/8)
|
||||||
|
x-1 (1/8) x (1/8) x+1 (1/8)
|
||||||
|
x+1 (1/8) x+2 (1/8) [next row offsets]
|
||||||
|
```
|
||||||
|
|
||||||
|
Concretely, 6 neighbors each get `e/8`: `(x+1,y)`, `(x+2,y)`, `(x-1,y+1)`,
|
||||||
|
`(x,y+1)`, `(x+1,y+1)`, `(x,y+2)`. (Clamp at edges — drop, don't wrap.)
|
||||||
|
|
||||||
|
### 4.4 Where to use it
|
||||||
|
|
||||||
|
- Entity icons / host thumbnails in the KB and entity desktop (the obvious win).
|
||||||
|
- Mascot or login/Config background art (`ConfigBackground.svelte` already
|
||||||
|
exists — a dithered backdrop there would be striking).
|
||||||
|
- Any user-uploaded image in chat/knowledge where we want the "de-imagined"
|
||||||
|
tone. Keep it **opt-in per call site** via the `plain` prop — don't dither
|
||||||
|
diagrams/screenshots that need to stay readable.
|
||||||
|
|
||||||
|
### 4.5 Cross-origin caveat
|
||||||
|
|
||||||
|
`getImageData` throws on tainted canvases. If `src` is cross-origin and the
|
||||||
|
server doesn't send CORS headers, fall back to the plain `<img>` (log once).
|
||||||
|
For self-hosted assets (the common case here) it's a non-issue.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Fonts: self-host JetBrains Mono + VT323
|
||||||
|
|
||||||
|
cyberspace self-hosts both faces as `.woff2`. oikos currently pulls DM Sans /
|
||||||
|
DM Mono / Inknut Antiqua from Google Fonts via a `<link>` in
|
||||||
|
`web/index.html:10`.
|
||||||
|
|
||||||
|
- Drop `JetBrainsMono-Regular.woff2` and `VT323-Regular.woff2` under
|
||||||
|
`web/static/fonts/` (or `web/public/fonts/` — match where static assets are
|
||||||
|
served from; check `vite.config`).
|
||||||
|
- Add `@font-face` blocks at the top of `app.css` with `font-display: swap`.
|
||||||
|
- For the cyberspace themes only, the `--font-sans`/`--font-mono`/`--font-heading`
|
||||||
|
overrides in §3 remap the families — Terracotta/Carbon keep DM Sans/Inknut
|
||||||
|
untouched. This is the key trick: **font choice is part of the theme**, not a
|
||||||
|
global swap, so the two art directions don't fight.
|
||||||
|
- Leave the Google Fonts `<link>` in place for now (Terracotta/Carbon still need
|
||||||
|
it); add a follow-up to self-host those too if we want to kill the external
|
||||||
|
request entirely. Out of scope for this plan.
|
||||||
|
|
||||||
|
VT323 vs Departure Mono: VT323 is free on Google Fonts and trivial to self-host;
|
||||||
|
Departure Mono is the authentic cyberspace face but needs a license check.
|
||||||
|
**Recommend VT323** to start; swap to Departure Mono later if you want exact
|
||||||
|
fidelity.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Theme store changes (`web/src/lib/stores/theme.svelte.ts`)
|
||||||
|
|
||||||
|
Current: `Theme = 'light' | 'dark'`, flips `.dark` class. Extend to a named set
|
||||||
|
while preserving the existing API (callers of `toggleTheme`/`getTheme` keep
|
||||||
|
working):
|
||||||
|
|
||||||
|
```ts
|
||||||
|
export type ThemeName = 'terracotta' | 'carbon' | 'cyber-dark' | 'cyber-light'
|
||||||
|
// Back-compat aliases used by existing callers:
|
||||||
|
// 'light' -> 'terracotta', 'dark' -> 'carbon'
|
||||||
|
```
|
||||||
|
|
||||||
|
- Store key stays `oikos-theme`; migrate old `'light'`/`'dark'` values on read.
|
||||||
|
- `applyClass` becomes `applyTheme`: sets `data-theme` on `<html>` and toggles
|
||||||
|
`.dark` **only** for `carbon` (so Terracotta and both cyberspace themes run
|
||||||
|
with no `.dark`). This is important: the `.dark` block in `app.css` must not
|
||||||
|
layer on top of the cyberspace token blocks — cyberspace sets its own
|
||||||
|
polarities.
|
||||||
|
- Update `THEME_LABELS` to the four names; update whatever UI surfaces the
|
||||||
|
picker (search for `THEME_LABELS` / `toggleTheme` usages — likely
|
||||||
|
`Settings.svelte` or the desktop shell's chrome) to a 4-option control instead
|
||||||
|
of a binary toggle.
|
||||||
|
|
||||||
|
**Watch out:** any code that assumes `document.documentElement.classList.contains('dark')`
|
||||||
|
≡ "dark colors" will be wrong for `cyber-dark`. Audit `grep -rn "classList.*dark\|\.dark" web/src` and prefer reading `getTheme()`/`data-theme` instead.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Optional cosmetic idioms (additive utilities)
|
||||||
|
|
||||||
|
Small, theme-aware utilities in `app.css` — usable in any theme but idiomatic
|
||||||
|
for cyberspace:
|
||||||
|
|
||||||
|
- `.terminal-box` — `{ border:1px solid var(--border); border-radius:0 }` plus a
|
||||||
|
`.terminal-box:focus-within { box-shadow: 0 0 0 2px var(--ring) }` to mirror
|
||||||
|
the `ring-2 ring-fg` focus. Lets cards opt into the terminal look without a
|
||||||
|
component rewrite.
|
||||||
|
- `.braille-spinner` — keyframe cycling `⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏` as `::after` content, colored
|
||||||
|
`var(--muted-foreground)`. Alternative to `Spinner.svelte` for loading states
|
||||||
|
under cyberspace themes.
|
||||||
|
- `.font-vt` — `{ font-family: var(--font-heading) }` so the wordmark class
|
||||||
|
cyberspace uses maps to our heading var (VT323 under cyber themes, Inknut
|
||||||
|
under Terracotta). Drop-in for any stylized title.
|
||||||
|
- `.strike-list` — `li > s { color: var(--muted-foreground) }` convenience for
|
||||||
|
the crossed-out feature-list pattern in marketing/empty states.
|
||||||
|
|
||||||
|
None of these are required for the theme to work; they're palette for the
|
||||||
|
"de-imagined" voice where we want it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Implementation order (incremental, each step shippable)
|
||||||
|
|
||||||
|
1. **Fonts** (§5) — self-host JetBrains Mono + VT323, `@font-face` in app.css.
|
||||||
|
No visual change yet (only cyberspace themes reference them).
|
||||||
|
2. **Tokens** (§3) — add the two `:root[data-theme='cyber-*']` blocks.
|
||||||
|
3. **Store** (§6) — extend `theme.svelte.ts` to named themes + `data-theme`;
|
||||||
|
update the picker UI. **At this point both cyberspace themes are live and
|
||||||
|
fully recolor the whole app** — the cheapest milestone, biggest visible win.
|
||||||
|
4. **`RasterImage.svelte`** (§4) — build + wire into entity icons and
|
||||||
|
`ConfigBackground`. This is the "dithered image" deliverable.
|
||||||
|
5. **Idioms** (§7) — terminal-box, braille spinner, etc., applied opportunistically.
|
||||||
|
|
||||||
|
Each step is independently mergeable. Step 3 alone satisfies "light + dark
|
||||||
|
cyberspace themes"; step 4 satisfies "dithered images."
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. Verification
|
||||||
|
|
||||||
|
- `cd web && npm run build` (or the repo's build command — confirm in
|
||||||
|
`web/package.json`) — Tailwind v4 must accept the new `data-theme` selectors
|
||||||
|
and `color-mix()` (both standard; no config change expected).
|
||||||
|
- `npm run check` / `svelte-check` for the store + component TS.
|
||||||
|
- Manual: cycle all four themes in the picker; confirm no `.dark` bleed on
|
||||||
|
`cyber-light`; confirm `RasterImage` re-dithers on theme flip; confirm
|
||||||
|
`prefers-reduced-data`/`plain` prop shows crisp image; confirm cross-origin
|
||||||
|
`src` degrades to `<img>` without console errors.
|
||||||
|
- Lighthouse / a11y: 1-bit dithered images still need a real `alt` (kept on the
|
||||||
|
fallback `<img>`); contrast on `#a89984`-on-black passes WCAG AA for body text
|
||||||
|
(ratio ≈ 7.4:1) — fine.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 10. Alternatives considered
|
||||||
|
|
||||||
|
- **Full rebrand (replace Terracotta/Carbon).** Highest visual payoff, highest
|
||||||
|
cost: every component's rounding/serif/spacing was authored for the Art
|
||||||
|
ouveau
|
||||||
|
direction; square + mono would need a component-level sweep, not just tokens.
|
||||||
|
Defer unless you want cyberspace as *the* oikos look — then do it as a
|
||||||
|
follow-up that deletes Terracotta/Carbon and makes `cyber-dark` the sole
|
||||||
|
default.
|
||||||
|
- **CSS-only image dither (filters / SVG turbulence).** Cheaper, but can't do
|
||||||
|
true 1-bit error diffusion or recolor to theme fg/bg. Rejected — the canvas
|
||||||
|
pass is the whole point and is ~60 lines.
|
||||||
|
- **Ordered (Bayer) dither instead of Atkinson.** More regular/grid-like
|
||||||
|
("newspaper halftone"). Atkinson is softer and more terminal-like; keep
|
||||||
|
Bayer as a `algorithm='bayer'` prop option later if wanted.
|
||||||
|
- **Server-side dithering.** Could pre-dither icons at ingest. Rejected for
|
||||||
|
now — client canvas keeps one source of truth (the original image) and lets
|
||||||
|
the palette follow the live theme, which a baked asset can't.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 11. Non-goals / out of scope
|
||||||
|
|
||||||
|
- Replicating cyberspace's sidebar-rail *layout* (oikos uses a floating-window
|
||||||
|
desktop shell; the rail is a different app model). We take the *visual*
|
||||||
|
language, not the IA.
|
||||||
|
- Porting the 9 other novelty themes (C64, Matrix, VT320, …). Two (light/dark)
|
||||||
|
satisfy the request; the token model makes adding more trivial later.
|
||||||
|
- Removing the Google Fonts dependency for Terracotta/Carbon (follow-up).
|
||||||
|
- Licensing/redistributing Departure Mono (use VT323 unless cleared).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 12. Risks
|
||||||
|
|
||||||
|
- **`.dark` coupling.** Existing code may equate `.dark` with "dark UI".
|
||||||
|
Mitigation: audit in step 3; the grep is small.
|
||||||
|
- **Dither perf on large images.** Atkinson is O(n) and runs on a downscaled
|
||||||
|
canvas (≤~320px wide), so per-image cost is negligible; but batch-rendering
|
||||||
|
many entity icons on first paint could jank. Mitigation: dither lazily (on
|
||||||
|
intersection) and cache the result on the element.
|
||||||
|
- **Tainted canvas** on cross-origin images → silent fallback to `<img>`
|
||||||
|
(already handled in the design).
|
||||||
|
- **Token drift.** If a component hardcodes a color instead of using a token,
|
||||||
|
it won't recolor under cyberspace. This is the same risk Carbon already has;
|
||||||
|
no new exposure, just more visible under a stronger theme.
|
||||||
184
plans/done/2026-08-03-nomos-chat-changes-review.md
Normal file
184
plans/done/2026-08-03-nomos-chat-changes-review.md
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
# 2026-08-03 — Review: nomos chat reliability/UX changes (F1–F7)
|
||||||
|
|
||||||
|
**Status:** Implemented (P0, P1, P2 all done). See
|
||||||
|
[Resolution](#resolution) at the end.
|
||||||
|
|
||||||
|
A critical self-review of the uncommitted F1–F7 changeset
|
||||||
|
(`plans/2026-08-03-nomos-chat-reliability-and-ux-audit.md` Resolution). The
|
||||||
|
change set is mostly sound and builds/tests green, but **F1 introduced one
|
||||||
|
real lost-work regression** by changing the contract of `resumeSession` (it can
|
||||||
|
now skip) without updating two callers that mutate state *before* calling it.
|
||||||
|
That must be fixed before this ships.
|
||||||
|
|
||||||
|
## What was changed (for orientation)
|
||||||
|
- F1 `cmd/nomos/turngate.go` (+test): per-session single-flight; `resumeSession`
|
||||||
|
acquires non-blocking and **skips** if a turn is active; `handleChat` live path
|
||||||
|
acquires with a 5s wait.
|
||||||
|
- F2/F3 `web/src/lib/stores/chat.ts`: humanized errors, `clearTurnState` on
|
||||||
|
terminal `task.status`, turn-free reconnect.
|
||||||
|
- F4 streaming in global `activityLog` + inline `ToolCallCard`.
|
||||||
|
- F5 artifact/knowledge deep links; F6 step-first headline; F7 stable layout.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P0 — F1 loses finished-execution continuations (must fix before shipping)
|
||||||
|
|
||||||
|
**Bug.** `processContinuations` (`cmd/nomos/continue.go:166-167`) calls
|
||||||
|
`a.store.markContinued(ctx, p.ExecID)` **before** dispatching
|
||||||
|
`continueSession → resumeSession`. `markContinued` sets `continued_at`, and
|
||||||
|
`pendingContinuations` (`store.go:1763`) filters `WHERE continued_at IS NULL` —
|
||||||
|
so a marked execution is **never re-queued**.
|
||||||
|
|
||||||
|
Before F1, `resumeSession` always ran, so marking-first was safe. F1 made
|
||||||
|
`resumeSession` skip when a turn is already active for the session. Now:
|
||||||
|
|
||||||
|
- **Two executions for one session finish near-simultaneously** (the common
|
||||||
|
multi-step case): the loop marks BOTH, spawns two goroutines; goroutine 1
|
||||||
|
acquires and runs, goroutine 2's `resumeSession` **skips** → execution 2 is
|
||||||
|
marked continued but its result is **never fed back to the agent. Lost.**
|
||||||
|
- **A live turn is streaming when an async execution finishes**: continuation
|
||||||
|
marks + dispatches; `resumeSession` skips (live turn holds the permit) →
|
||||||
|
result lost.
|
||||||
|
|
||||||
|
This silently drops auto-continuation — worse than the interleaving F1 set out
|
||||||
|
to fix.
|
||||||
|
|
||||||
|
**Fix.** Make `resumeSession` report whether it actually ran, and mark-continued
|
||||||
|
only after a successful run; on a busy-skip, leave the execution pending for the
|
||||||
|
next worker tick.
|
||||||
|
|
||||||
|
1. `cmd/nomos/continue.go` — change `resumeSession` to return `bool`:
|
||||||
|
```go
|
||||||
|
func (a *agent) resumeSession(ctx context.Context, sessionID, note string) bool {
|
||||||
|
if !a.gate.acquire(sessionID, 0) {
|
||||||
|
slog.Info("nomos: turn already active, skipping background resume", "session", sessionID)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
defer a.gate.release(sessionID)
|
||||||
|
…existing body…
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
2. `continueSession` — mark only after a real run; on skip, leave pending:
|
||||||
|
```go
|
||||||
|
func (a *agent) continueSession(ctx context.Context, p pendingContinuation) {
|
||||||
|
slog.Info("nomos: auto-continuing session", "session", p.SessionID, "execution", p.ExecID, "status", p.Status)
|
||||||
|
if !a.resumeSession(ctx, p.SessionID, buildContinuationNote(p)) {
|
||||||
|
slog.Info("nomos: continuation deferred — a turn is active; will retry next tick", "session", p.SessionID, "execution", p.ExecID)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
a.store.markContinued(ctx, p.ExecID)
|
||||||
|
}
|
||||||
|
```
|
||||||
|
3. `processContinuations` — **delete** the `a.store.markContinued(ctx, p.ExecID)`
|
||||||
|
line at `continue.go:166` (the dispatch `safego.Go(... continueSession ...)`
|
||||||
|
stays). The `markContinued` at `:162` (the no-assent-window branch, which
|
||||||
|
saves a note and does **not** call resumeSession) stays as-is — that path
|
||||||
|
intentionally consumes the item.
|
||||||
|
4. Update every other `resumeSession` caller to ignore the new return value
|
||||||
|
(`/resume`, `handleAnswerQuestion`, the empty-message reconnect in
|
||||||
|
`handleChat`) — they don't need the bool; a bare call discards it. No behavior
|
||||||
|
change for them (their skip semantics are already correct/desired).
|
||||||
|
|
||||||
|
**Why this preserves the original "no re-continue loop" guarantee:** a
|
||||||
|
`resumeSession` that *runs* always returns `true` (even on its internal LLM
|
||||||
|
failure path — it has already persisted a failure note), so it gets marked and
|
||||||
|
won't loop. Only a *busy-skip* returns `false` and stays pending, which is
|
||||||
|
correct (retry once the turn frees). Crash-safety also improves: a crash between
|
||||||
|
acquire and mark leaves the item un-marked → re-queued on restart.
|
||||||
|
|
||||||
|
**Validation:**
|
||||||
|
- New test: two `pendingContinuation`s for one session dispatched concurrently;
|
||||||
|
assert both are eventually processed (both `continued_at` set) and at no point
|
||||||
|
do two `resumeSession` bodies overlap (reuse the `turnGate` single-flight
|
||||||
|
pattern, or assert via a shared counter in a stubbed `chatWith`).
|
||||||
|
- Existing `cmd/nomos` suite stays green; `go vet` clean.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P1 — F1 can false-auto-close a merely-busy session (low risk, fix for robustness)
|
||||||
|
|
||||||
|
**Bug.** `processIdleSweep` (`continue.go:78-89`) bumps `completion_nudges`
|
||||||
|
**before** calling `resumeSession`. If `resumeSession` skips (busy), the nudge is
|
||||||
|
counted as unanswered; the next sweep sees `CompletionNudges >= 1` and
|
||||||
|
**auto-closes** a session that was just busy.
|
||||||
|
|
||||||
|
**Likelihood is low** because `staleGoalSessions` (`store.go:1336`) filters
|
||||||
|
`last_active_at < now() - threshold` and an active turn keeps updating
|
||||||
|
`last_active_at` — so a busy session shouldn't appear stale. But the coupling is
|
||||||
|
the same shape as P0 and worth closing.
|
||||||
|
|
||||||
|
**Fix.** Gate the bump on the run, mirroring P0:
|
||||||
|
```go
|
||||||
|
safego.Go("nomos:idle-nudge:"+s.ID, func() {
|
||||||
|
note := …
|
||||||
|
if a.resumeSession(ctx, s.ID, note) {
|
||||||
|
if err := a.store.bumpCompletionNudge(ctx, s.ID); err != nil { … }
|
||||||
|
}
|
||||||
|
})
|
||||||
|
```
|
||||||
|
(If skipped, leave `completion_nudges` at 0 so a genuinely-stale sweep nudges
|
||||||
|
again later.)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## P2 — Minor / hygiene (optional, can ship without)
|
||||||
|
|
||||||
|
- **Redundant catch-up turn on reconnect.** When the live turn *already ended*
|
||||||
|
before a dropped-SSE reconnect fires, the empty-message path still runs a
|
||||||
|
"report your state" `resumeSession` turn the operator didn't ask for. F1 makes
|
||||||
|
it non-concurrent (good) but it's still a spare turn. Consider: in
|
||||||
|
`handleChat`'s empty-message branch, skip the `resumeSession` if the session
|
||||||
|
is already terminal (`done`/`failed`/`abandoned`) or had activity within the
|
||||||
|
last few seconds — just return 202 and let the poller catch up.
|
||||||
|
- **Top-level side-effect on import.** `chat.ts` now calls `subscribeEvents()` +
|
||||||
|
`liveEvents.subscribe(...)` at module top level. It works (and `vitest` stays
|
||||||
|
green because tests mock `./chat`), but a hidden SSE-connect-on-import is
|
||||||
|
fragile for future tests. Prefer a lazy `ensureChatEventSync()` called from
|
||||||
|
the window mount path, matching how `workspace.ts` subscribes inside
|
||||||
|
`startWorkspace` rather than at import.
|
||||||
|
- **F7 follow-up (already documented):** the `NewTaskChat → SessionChatWindow`
|
||||||
|
window-swap on first send still flashes; an in-place handoff would remove it.
|
||||||
|
- **Pre-existing, not introduced:** `a.chat` retries the LLM stream on
|
||||||
|
`ctx`-cancellation (client disconnect) up to 3×, holding the turn permit a few
|
||||||
|
extra seconds. Out of scope here.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
- F8 (ordering toggle + live background tool-delta streaming) — deferred in the
|
||||||
|
original plan; its main symptom is removed by F1.
|
||||||
|
- `run` execution deep-links (need an execution-view opener).
|
||||||
|
|
||||||
|
## Recommended order
|
||||||
|
1. **P0** (lost continuations) — blocks shipping F1.
|
||||||
|
2. **P1** (idle-sweep nudge gate) — small, same pattern.
|
||||||
|
3. P2 items as time allows.
|
||||||
|
4. Re-run `go test ./cmd/nomos/`, `go vet`, web `vitest`, `vite build`; keep
|
||||||
|
`VERSION` at `0.15.0` (these are correctness fixes to the same changeset, not
|
||||||
|
a new bump) — or bump patch to `0.15.1` if shipped as a follow-up commit.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolution
|
||||||
|
|
||||||
|
All review items implemented. The whole batch (F1–F7 + these review fixes)
|
||||||
|
remains one uncommitted changeset at `VERSION 0.15.0`.
|
||||||
|
|
||||||
|
| Item | Fix | Where |
|
||||||
|
|---|---|---|
|
||||||
|
| **P0** | `resumeSession` returns `bool` (false on busy-skip). `continueSession` marks an execution `continued` **only after** the turn ran; on a skip it defers and the next worker tick retries (item stays pending). Removed the pre-dispatch `markContinued` in `processContinuations`. Other callers (`/resume`, answer-question, reconnect) ignore the return. | `cmd/nomos/continue.go` |
|
||||||
|
| **P0 test** | `TestResumeSession_SkipsWhenBusy`, `TestContinueSession_DefersWhenBusy` — DB-free contract tests proving the skip path returns false without running the body (nil provider would panic otherwise). | `cmd/nomos/continue_test.go` |
|
||||||
|
| **P1** | Idle sweep bumps `completion_nudges` only after `resumeSession` actually runs, so a busy-skip can't be counted as an unanswered nudge → no false auto-close. | `cmd/nomos/continue.go` (`processIdleSweep`) |
|
||||||
|
| **P2.1** | Empty-message reconnect (now defensive — the frontend no longer POSTs empty messages post-F2) skips a terminal session instead of spawning a spare "report state" turn. | `cmd/nomos/main.go` (`handleChat`) |
|
||||||
|
| **P2.2** | Event subscription armed lazily from `chatFor()` (`ensureChatEventSync`) instead of at module import — no SSE-connect-on-import side-effect. | `web/src/lib/stores/chat.ts` |
|
||||||
|
|
||||||
|
**Verification:** `go test -count=1 ./cmd/nomos/` green (incl. the two new
|
||||||
|
contract tests); `go vet` clean. Web `vitest` 70/70; `vite build` succeeds; no
|
||||||
|
new `tsc`/eslint errors in any touched file.
|
||||||
|
|
||||||
|
**Note on the P0 end-to-end test:** the full "two continuations both processed,
|
||||||
|
no overlap" scenario needs a live LLM provider (chatWith isn't stubbable without
|
||||||
|
a refactor) and was therefore covered at the contract level (the skip returns
|
||||||
|
false without running the body) plus the existing `turnGate` single-flight test
|
||||||
|
for serialization, rather than as a DB integration test.
|
||||||
356
plans/done/2026-08-03-nomos-chat-reliability-and-ux-audit.md
Normal file
356
plans/done/2026-08-03-nomos-chat-reliability-and-ux-audit.md
Normal file
@@ -0,0 +1,356 @@
|
|||||||
|
# 2026-08-03 — Nomos chat: reliability & predictability audit
|
||||||
|
|
||||||
|
**Status:** Implemented (F1–F7) in v0.15.0; F8 deferred. See
|
||||||
|
[Resolution](#resolution-2026-08-03) at the end.
|
||||||
|
|
||||||
|
**Scope:** The live chat/task UX across one production session, audited through
|
||||||
|
the code paths behind each operator-reported symptom —
|
||||||
|
`cmd/nomos/{main.go,agent.go,continue.go,store.go}`,
|
||||||
|
`web/src/lib/stores/{chat,activity,execstream,events,workspace}.ts`,
|
||||||
|
`web/src/lib/components/{ChatThread,AgentTrace,ToolCallCard,UnifiedTimeline,TaskContextPanel,SessionChatWindow}.svelte`.
|
||||||
|
**Trigger:** Operator report — streaming invisible in the tool card; the
|
||||||
|
activity/plan panel wrong about parallel/nested runs and timestamps with no clear
|
||||||
|
sequence; no links to artifacts/knowledge referenced in chat; agent "thinking"
|
||||||
|
flickers/overwrites itself; layout jumps when a chat goes from empty to content;
|
||||||
|
"Agent connection lost / Error in input stream" messages that aren't actionable
|
||||||
|
and don't self-resolve; overall flaky/disconnected feel where the task never
|
||||||
|
cleanly ended.
|
||||||
|
|
||||||
|
The prior round (`2026-07-30-session-review-plan-drift-and-dead-activity-panel.md`,
|
||||||
|
shipped in `467589d`) fixed the plan-seq and fabricated-timestamp rendering bugs.
|
||||||
|
This round's symptoms are a different layer: **turn orchestration, streaming
|
||||||
|
wiring, and connection-state UX**. One architectural gap (F1) is the common
|
||||||
|
cause behind several of them.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The one root cause that compounds everything: F1
|
||||||
|
|
||||||
|
### F1 — No per-session turn serialization (concurrent turns corrupt the view)
|
||||||
|
|
||||||
|
`handleChat` runs `a.chat(ctx, ...)` directly in the HTTP request goroutine, and
|
||||||
|
every "resume" path (`resumeSession`, the reconnect empty-message path, the
|
||||||
|
auto-continuation worker, the idle sweep, answer-question) launches **another
|
||||||
|
goroutine** (`safego.Go`) running a full turn. There is **no mutex keyed on
|
||||||
|
`sessionID`** anywhere. The codebase already knows this is a hazard —
|
||||||
|
`agent.go:316-323` marks approved executions `continued` specifically because
|
||||||
|
"two concurrent LLM calls for the same session cause empty responses and race
|
||||||
|
conditions" — but the fix is per-path patching, not a general lock.
|
||||||
|
|
||||||
|
What this produces, deterministically:
|
||||||
|
|
||||||
|
- A network blip on the browser↔nomos stream fires `handleDisconnect`
|
||||||
|
(`chat.ts:383`), which POSTs an **empty-message reconnect** →
|
||||||
|
`main.go:194-206` spawns `resumeSession` as a **new goroutine**. If the
|
||||||
|
original turn is still alive (or finishes its current tool call), **two turns
|
||||||
|
now run for one session**: interleaved `tool_use`/`text_delta` events, a
|
||||||
|
re-proposed plan, and "the agent is repeating itself."
|
||||||
|
- The activity timeline (`activity.ts:119-185`) groups tools under a plan step
|
||||||
|
by *inferring* `currentStepSeq` from `update_plan_step` calls in the message
|
||||||
|
stream. Two interleaved turns make that inference wrong → tools land under the
|
||||||
|
wrong step, steps appear to nest/parallelize that never did, the sequence
|
||||||
|
reads as garbage. This is the "parallel runs / nesting / no clear sequence"
|
||||||
|
report.
|
||||||
|
- Two turns appending to the same session's messages is also the source of the
|
||||||
|
duplicate-tool-call/empty-response class of bugs the prior plan docs keep
|
||||||
|
patching individually.
|
||||||
|
|
||||||
|
**This is why the experience "felt flaky and disconnected" and "the task didn't
|
||||||
|
end":** the panel is faithfully rendering a corrupted, interleaved event stream.
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **One in-flight turn per session, server-side.** Add a per-`sessionID`
|
||||||
|
turn mutex (a `sync.Map[string]*singleflight` or a keyed `sync.Mutex`) in
|
||||||
|
`handleChat`/`resumeSession`/`continue.go`. A second attempt to start a turn
|
||||||
|
for a session that already has one running must **queue** (preferred — the
|
||||||
|
operator's message waits its turn) or **return 409 "turn in progress"** (the
|
||||||
|
frontend then just re-polls; no new goroutine). This single change removes
|
||||||
|
the interleaving that drives F2/F3/F8.
|
||||||
|
2. **Make the empty-message reconnect a no-op when a turn is already running.**
|
||||||
|
Today it *always* spawns `resumeSession`. Gate it on "is any turn active for
|
||||||
|
this session?" — if yes, return 202 and let the existing turn + the poller do
|
||||||
|
the work. A blip should never *create* work.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F2 — Reconnect spawns a new turn and surfaces raw, non-actionable errors
|
||||||
|
|
||||||
|
`chat.ts:383-426` `handleDisconnect`: on a dropped SSE it sets
|
||||||
|
`connectionState='disconnected'`, starts the 3s poller, shows
|
||||||
|
`"Agent connection lost. The task is still running — retrying…"`, then calls
|
||||||
|
`streamChat('', sid, …)` up to 3× — each of which is the empty-message POST that
|
||||||
|
triggers F1's new `resumeSession` goroutine. Separately, the LLM stream errors
|
||||||
|
surface verbatim: `agent.go:388` does `emitError("llm: %v", err)`, so an
|
||||||
|
OpenRouter transport break reaches the operator as `llm: error in input stream:
|
||||||
|
…` (the openai-go SDK's SSE-reader text), shown raw in `ChatThread`'s error bar.
|
||||||
|
|
||||||
|
Combined with F1, this is the exact "messages not actionable and not
|
||||||
|
self-resolving" + "task didn't end" experience: a blip both invents a duplicate
|
||||||
|
turn and paints a scary, unfixable error that lingers.
|
||||||
|
|
||||||
|
Secondary defects in the same path:
|
||||||
|
|
||||||
|
- `streaming` stays `true` for the entire reconnect window, so the composer is
|
||||||
|
disabled and the poller's `if (streaming && connected) return` guard
|
||||||
|
(`chat.ts:177`) suppresses updates except while disconnected — fragile.
|
||||||
|
- The **per-window** error path (`sendSessionMessage`, `startTask`) does **not**
|
||||||
|
auto-reconnect at all — it only polls. Its `onReconnect` in
|
||||||
|
`SessionChatWindow.svelte:110` is `() => loadSessionChat(sessionId)`, which
|
||||||
|
just *re-fetches the transcript* and never re-attaches to a live stream. And
|
||||||
|
the global `reconnect()` (`chat.ts:428`) keys off the **global**
|
||||||
|
`currentSession`, so a floating window's Reconnect button can target the wrong
|
||||||
|
session. Two different, both-broken reconnect behaviors.
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **Stop the empty-message-reconnect from creating turns** (depends on F1.2).
|
||||||
|
Reconnect should mean "catch up," not "run more."
|
||||||
|
2. **Humanize + bucket error strings.** Map known transport errors to
|
||||||
|
operator-readable, actionable copy with a single primary action:
|
||||||
|
- `llm: …input stream…` / 502/503/timeout → "The model connection dropped.
|
||||||
|
The task is still running in the background — it'll catch up
|
||||||
|
automatically." (auto-dismiss when the next event/poll lands)
|
||||||
|
- `HTTP 401/403` → "Session expired — reconnect." (action: re-auth)
|
||||||
|
- unknown → show the raw text but behind a "Details" toggle, not as the
|
||||||
|
headline.
|
||||||
|
3. **Make errors self-resolving.** Clear the error + connection-lost banner the
|
||||||
|
moment the poller sees a newer message or any live event for the session
|
||||||
|
arrives (wire `eventsConnected` / a session-scoped event into the banner's
|
||||||
|
visibility). Today the banner stays until manual dismiss even after recovery.
|
||||||
|
4. **Unify reconnect.** One `reconnect(sessionId)` that (a) re-fetches the
|
||||||
|
transcript, (b) if no turn is active, is a pure no-op refresh; used by both
|
||||||
|
the main view and windows. Drop the global-`currentSession` coupling.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F3 — The UI can't tell when a turn truly ended (so it never looks "done")
|
||||||
|
|
||||||
|
When the SSE stream ends without a `done` event, `streamChat`'s `onDone`
|
||||||
|
(`chat.ts:355-368`) calls `handleDisconnect`. Even if the backend turn then
|
||||||
|
finishes and persists its final message, the frontend only learns via the 3s
|
||||||
|
poller re-setting `messages` — but nothing transitions `streaming`→`false` or
|
||||||
|
`connectionState`→`connected` from that path, so the spinner/indicator and the
|
||||||
|
"connection lost" banner can persist indefinitely. That is "the task didn't
|
||||||
|
end / backend connection was lost."
|
||||||
|
|
||||||
|
The backend does emit a terminal signal — `task.status` events on
|
||||||
|
`complete_task`/auto-complete (`workspace.ts:82-88` `STATUS_AFFECTING`) — but
|
||||||
|
nothing in the chat store reacts to a terminal `task.status` to force
|
||||||
|
`streaming=false` + clear the banner. The signal exists; the chat ignores it.
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **Treat a terminal `task.status` (done/failed) for the viewed session as
|
||||||
|
authoritative end-of-turn** in `chat.ts`: set `streaming=false`,
|
||||||
|
`connectionState='connected'`, dismiss any connection-lost error. The poller
|
||||||
|
already refreshes messages; this just closes the loop on the *state* flags.
|
||||||
|
2. **Add a `task.completed` / `turn.ended` SSE event** from the backend on every
|
||||||
|
terminal path (today `done` is a chat-stream-only event; background turns
|
||||||
|
have no equivalent). The always-on events stream already reaches the panel —
|
||||||
|
route the same signal to the chat store so background-completed turns clear
|
||||||
|
the UI without waiting on a poll.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F4 — Command streaming isn't shown where the operator looks
|
||||||
|
|
||||||
|
Streaming **exists** (`execstream.ts` `liveExecutionOutputFor`, fed by
|
||||||
|
`fetchExecutionLogs` via the always-on events stream) and the
|
||||||
|
`UnifiedTimeline` **does** render `tool.liveOutput` with tail-pinned scroll
|
||||||
|
(`UnifiedTimeline.svelte:451-457`). But:
|
||||||
|
|
||||||
|
- The **global** `activityLog` (`activity.ts:236`) — used by the main Chat page's
|
||||||
|
panel — never calls `withLiveOutput`. Only the **per-window**
|
||||||
|
`activityLogFor(sessionId)` (`activity.ts:271`) attaches live output. So the
|
||||||
|
main chat view's timeline shows no streaming at all.
|
||||||
|
- The **inline chat tool cards** — `ToolCallCard.svelte` (rendered inside
|
||||||
|
`AgentTrace.svelte`) — show only args/result/error. They never read
|
||||||
|
`liveOutput`. Expanding a running `run` call in the transcript (the natural
|
||||||
|
place to "check the tool") shows nothing live; output appears all at once when
|
||||||
|
the `tool_result` lands.
|
||||||
|
|
||||||
|
This is the report: "I expected checking on the tool to let me see the
|
||||||
|
streaming."
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **Wire live output into the global `activityLog`** so the main chat panel
|
||||||
|
streams too (call `withLiveOutput` in the `activityLog` derivation, same as
|
||||||
|
`activityLogFor`).
|
||||||
|
2. **Show streaming in the inline tool card.** Pass the session's live-output
|
||||||
|
store into `AgentTrace`/`ToolCallCard` (or attach `liveOutput` to the running
|
||||||
|
`run` tool entry the way the timeline does) and render a tail-pinned `<pre>`
|
||||||
|
while the call is `tool_use`/running. Reuse the UnifiedTimeline's scroll-pin
|
||||||
|
pattern. Gated runs (queued-for-approval) should instead show a "queued —
|
||||||
|
watch in entity detail" affordance (per `execstream.ts` header comment).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F5 — Artifacts and knowledge referenced in chat aren't navigable
|
||||||
|
|
||||||
|
When the agent records knowledge, the activity panel shows `Recorded: <title>`
|
||||||
|
(`activity.ts:188-203`) but it's plain text — no link. The backend already
|
||||||
|
emits `knowledge.recorded` and links the note to the task
|
||||||
|
(`store.go:1572 linkKnowledgeToTask`, `agent.go:594`), and the Wiki reader
|
||||||
|
exists (`web/src/lib/components/knowledge/WikiReader.svelte`). Nothing connects
|
||||||
|
them. Same for `get_entity`/`run` results: slugs and execution ids appear in
|
||||||
|
tool output but aren't clickable to open the entity window or execution view.
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **Make activity/tool entries link-bearing.** Add an optional
|
||||||
|
`link?: { kind: 'knowledge'|'entity'|'execution', id: string }` to
|
||||||
|
`ActivityEntry`. Populate it from `upsert_knowledge` (title→knowledge id from
|
||||||
|
the result), `get_entity` (slug), and `run` (execution id). Render a
|
||||||
|
clickable chip that opens the right surface: knowledge → Wiki reader (new tab
|
||||||
|
/ window), entity → entity detail window, execution → execution log pane
|
||||||
|
(already fetched by `EntityDetailContent.svelte`).
|
||||||
|
2. **Render entity/knowledge mentions in assistant markdown as links** when they
|
||||||
|
resolve to known slugs (lightweight: a post-process pass on rendered text, or
|
||||||
|
let the model emit explicit `[slug](entity:…)` markers it already has tools to
|
||||||
|
discover).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F6 — "Thinking" is an unstable single-line headline, not a predictable trace
|
||||||
|
|
||||||
|
`ChatThread`'s `indicatorLabel` (`ChatThread.svelte:83-89`) returns the **first**
|
||||||
|
running activity entry's description; `AgentTrace`'s `headline` mirrors it. As
|
||||||
|
tools fire sequentially the running entry changes, so the one line rewrites
|
||||||
|
itself every call — "the thinking overwrites itself." There is no persistent,
|
||||||
|
additive reasoning surface, and no predictable turn structure (plan → steps →
|
||||||
|
answer) the operator can learn to read. Claude-Code-style predictability is
|
||||||
|
absent.
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **A stable, additive per-turn reasoning block.** Keep the collapsed trace as
|
||||||
|
a *summary* ("Step 2 of 4 · running `run`"), but when expanded show an
|
||||||
|
**append-only** log of (a) the model's intermediate `text` (reasoning before
|
||||||
|
each tool call — already emitted at `agent.go:458-460` and persisted) and
|
||||||
|
(b) each tool call as a fixed row, instead of a single mutating headline.
|
||||||
|
2. **Predictable turn shape.** Enforce/cue a consistent sequence in the UI —
|
||||||
|
Goal → Plan → Steps (each with its tools nested) → Final answer — and render
|
||||||
|
each phase as a stable section that fills in rather than a line that
|
||||||
|
overwrites. The UnifiedTimeline already models most of this; surface the same
|
||||||
|
model in the inline trace so chat and panel tell one story.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F7 — Layout jumps when a chat goes from empty to content
|
||||||
|
|
||||||
|
`SessionChatWindow.svelte:58-63` gates the right rail on `hasContext`: empty
|
||||||
|
task → `ChatThread` full-width; first activity/touched entity → switches to
|
||||||
|
`Splitpanes` with the `TaskContextPanel` rail. The swap is instant and
|
||||||
|
**reflows the chat column width** the moment the first event lands — "switching
|
||||||
|
from empty to chat with something, the layout was off." Compounded by the
|
||||||
|
`NewTaskChat` → real `SessionChatWindow` window-swap on first send
|
||||||
|
(`NewTaskChat.svelte:17-22`).
|
||||||
|
|
||||||
|
### Fix (proposed)
|
||||||
|
|
||||||
|
1. **Reserve the rail's space from the start** (collapse to a thin sliver / icon
|
||||||
|
rail when empty) instead of mounting it on demand, so adding content doesn't
|
||||||
|
change the chat column width. Or animate the rail in.
|
||||||
|
2. **Avoid the window swap on first send** — let the new-task window *become* the
|
||||||
|
session window in place once the id is assigned (same component, swap the
|
||||||
|
store source) rather than close+open.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## F8 — Activity/plan ordering & parallelism *(largely a symptom of F1)*
|
||||||
|
|
||||||
|
With F1 fixed (no interleaved turns) the heuristic step-grouping in
|
||||||
|
`activity.ts` becomes reliable again. Remaining standalone items:
|
||||||
|
|
||||||
|
- The timeline is **newest-first** with ts-0 goal/pending parked at the bottom
|
||||||
|
(`UnifiedTimeline.svelte:119-127`); for a long task this can read as
|
||||||
|
"sequence is off." Consider an explicit **oldest-first / seq-ordered** mode
|
||||||
|
toggle, and always show the step number prominently so order is unambiguous
|
||||||
|
regardless of sort.
|
||||||
|
- Background/auto-continued turns still rely on the 3s poller for their result
|
||||||
|
to appear; until F3's terminal event lands, the panel can lag. The
|
||||||
|
always-on events stream already carries `plan.*` and `entity.touched` live —
|
||||||
|
extend it to carry per-tool `tool.*` deltas for background turns so the panel
|
||||||
|
is live, not polled, during autonomous work.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Recommended sequence
|
||||||
|
|
||||||
|
| Order | Item | Why first |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | **F1** per-session turn mutex + no-op reconnect-when-busy | Removes the interleaving that is the root cause of F2/F3/F8 symptoms; everything else is cosmetics on top of a corrupted stream. |
|
||||||
|
| 2 | **F3** terminal-event → clear chat state | Once turns can't double, make "the task ended" unambiguous so the UI stops lingering. |
|
||||||
|
| 3 | **F2** humanized/self-resolving errors + unified reconnect | Turns the scary, sticky "connection lost / input stream" into recoverable, auto-clearing UX. |
|
||||||
|
| 4 | **F4** streaming in the global log + inline tool card | Highest-visibility "I can't see what it's doing" fix; small, isolated change. |
|
||||||
|
| 5 | **F6** stable additive reasoning trace | Predictability of the interaction model (the Claude-Code feel). |
|
||||||
|
| 6 | **F5** artifact/knowledge deep links | Navigation completeness. |
|
||||||
|
| 7 | **F7** layout stability | Polish. |
|
||||||
|
| 8 | **F8** ordering mode + live background deltas | Polish, partly free after F1. |
|
||||||
|
|
||||||
|
## Verification hooks (when implementing)
|
||||||
|
|
||||||
|
- `cmd/nomos`: a test that starts two turns for the same session and asserts the
|
||||||
|
second queues/is-rejected (no interleaved `tool_use` order in persisted
|
||||||
|
messages).
|
||||||
|
- `web/src/lib/stores`: extend `activity.test.ts`/`execstream.test.ts` — global
|
||||||
|
`activityLog` now carries `liveOutput`; tool-card live output renders while
|
||||||
|
`tool_use` and clears on `tool_result`.
|
||||||
|
- A reconnect/integration test: drop the SSE mid-turn, assert (a) no duplicate
|
||||||
|
`resumeSession` goroutine, (b) banner auto-clears on next event, (c)
|
||||||
|
`streaming` returns to false on terminal `task.status`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Note on method
|
||||||
|
|
||||||
|
This audit was done against the **code paths** behind the reported symptoms, not
|
||||||
|
a single session transcript (no MCP/DB access from this session). To tie a
|
||||||
|
specific finding to a specific past session, pull the session via
|
||||||
|
`docker exec oikos-postgres-1 psql -U oikos oikos -c "select id,goal,outcome
|
||||||
|
from agent_sessions order by last_active_at desc limit 5"` and cross-reference
|
||||||
|
its `agent_activity` rows / persisted messages against the F1 interleaving
|
||||||
|
signature (two assistant turns' tool ids interleaved in one message shell).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolution (2026-08-03)
|
||||||
|
|
||||||
|
Implemented F1–F7 in v0.15.0 (`VERSION 0.14.2 → 0.15.0`). F8 deferred (its
|
||||||
|
primary symptom — interleaved/out-of-order entries — is removed by F1; the
|
||||||
|
ordering toggle and live background tool-delta streaming remain as nice-to-
|
||||||
|
haves).
|
||||||
|
|
||||||
|
| Item | What shipped | Where |
|
||||||
|
|---|---|---|
|
||||||
|
| **F1** | Per-session single-flight turn gate (`turnGate`): at most one in-flight turn per session. Background resume paths (`resumeSession` — covers the continuation worker, idle sweep, answer-question, /resume, and the empty-message reconnect) skip non-blocking when busy; the live chat path waits briefly then bails with an actionable error instead of stacking a second turn. | `cmd/nomos/turngate.go` (+`turngate_test.go`), wired in `agent.go` (struct/init), `continue.go` (`resumeSession`), `main.go` (`handleChat`). |
|
||||||
|
| **F3** | Terminal `task.status` events (done/failed/abandoned/awaiting_input) now clear a stuck chat view's `streaming`/`connectionState` and dismiss the connection-lost toasts — the authoritative "turn ended" signal the UI was ignoring. Poller safety net catches the edge where the event fired during the disconnect window. | `web/src/lib/stores/chat.ts` (`clearTurnState`, liveEvents subscription, `startSessionPolling`). |
|
||||||
|
| **F2** | Raw errors humanized ("The model connection dropped. The task keeps running…") and bucketed; one connection surface per drop (not banner+toast+raw error); errors self-clear via F3. The turn-spawning reconnect attempt loop is gone (dead global path simplified to a turn-free refresh); window "Reconnect" re-fetches + resets state. | `web/src/lib/stores/chat.ts` (`humanizeChatError`, error handlers, `loadSessionChat`, `handleDisconnect`/`reconnect`), `web/src/lib/components/ChatThread.svelte` (banner copy). |
|
||||||
|
| **F4** | Command streaming now shows (a) in the **global** activity timeline (live output wired into `activityLog`, was only per-window) and (b) in the **inline chat tool card** — expanding a running `run` shows live output auto-opened and tail-pinned. | `web/src/lib/types.ts` (`liveOutput`), `web/src/lib/stores/activity.ts` (`currentLiveOutput`), `web/src/lib/components/ChatThread.svelte` (`toolsWithLive`), `web/src/lib/components/ToolCallCard.svelte`. |
|
||||||
|
| **F6** | The "thinking" headline is now step-first (stable across a step's many tool calls) instead of rewriting per command; falls back to the current tool / "thinking…" only when no step is active. | `web/src/lib/components/ChatThread.svelte` (`indicatorLabel`). |
|
||||||
|
| **F5** | Activity entries now carry a deep link: recorded knowledge docs and `get_entity` lookups get an "open artifact" chip that opens the entity/knowledge window directly. | `web/src/lib/stores/activity.ts` (`link`, `knowledgeLinkFromResult`, `entityLinkFromArgs`), `web/src/lib/components/UnifiedTimeline.svelte`. |
|
||||||
|
| **F7** | The empty→content layout reflow is gone: `SessionChatWindow` now has one stable `Splitpanes`+`ChatThread` from open (no more destroy/remount of the thread or column reflow when the rail appears). | `web/src/lib/components/SessionChatWindow.svelte`. |
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
- `go test ./cmd/nomos/` green (incl. new `turngate_test.go`: non-blocking skip,
|
||||||
|
blocking-waits-for-release, timeout, and a 50-goroutine single-flight
|
||||||
|
concurrency test asserting max in-flight = 1). `go vet` clean.
|
||||||
|
- Web `vitest` 70/70 green (incl. `activity.test.ts`/`execstream.test.ts`); the
|
||||||
|
`activity.test.ts` chat mock gained `currentSession` for the new
|
||||||
|
`currentLiveOutput` derivation.
|
||||||
|
- `vite build` succeeds (all Svelte components compile). Pre-existing `tsc`
|
||||||
|
strictness errors in unrelated files (`ui/*`, `oidc.ts`, `windows.ts`,
|
||||||
|
`workspace.ts`) are unchanged; no new errors in any touched file.
|
||||||
|
|
||||||
|
**Follow-ups (not in this pass):**
|
||||||
|
- F8: oldest-first ordering toggle; emit per-tool `tool.*` events on the
|
||||||
|
always-on stream during background `resumeSession` turns so the panel is live
|
||||||
|
(not 3s-polled) during autonomous work.
|
||||||
|
- F5: `run` execution deep-links (open the entity detail's execution pane) —
|
||||||
|
needs an execution-view opener; knowledge/entity links shipped first as the
|
||||||
|
explicit complaint.
|
||||||
|
- F7: the `NewTaskChat → SessionChatWindow` window-swap on first send (a
|
||||||
|
windows.ts open/close) still causes a brief flash; an in-place handoff
|
||||||
|
(same window, swap store source) would remove it.
|
||||||
184
plans/done/2026-08-03-nomos-chat-working-visibility.md
Normal file
184
plans/done/2026-08-03-nomos-chat-working-visibility.md
Normal file
@@ -0,0 +1,184 @@
|
|||||||
|
# 2026-08-03 — Nomos chat: working-visibility, message queue, generation-aware timeline
|
||||||
|
|
||||||
|
**Status:** Implemented (F1–F4) in v0.17.0. See
|
||||||
|
[Resolution](#resolution-2026-08-03) at the end.
|
||||||
|
|
||||||
|
## Context (grounded in last-session logs + DB, not just code)
|
||||||
|
|
||||||
|
Operator report: *"On the chat window I can't tell the agent is working; it's
|
||||||
|
making tool calls but no feedback. Typing returns 'Nomos is still finishing a
|
||||||
|
previous step…'. Activity not up to date. Several plans at once, some don't
|
||||||
|
execute."*
|
||||||
|
|
||||||
|
Verified against runtime state:
|
||||||
|
|
||||||
|
- **Last session `23da10db`** ran ONE live turn for **6m33s** (21 iterations,
|
||||||
|
19:45:36→19:52:03, correlation `679566cb`). At 19:48:00 the operator typed
|
||||||
|
`status`; at **19:48:05 the turn gate deferred it** (`turn already active,
|
||||||
|
deferring operator message`). The operator could type at all only because the
|
||||||
|
client had already lost the stream (`streaming=false`) while the server kept
|
||||||
|
running — i.e. the client showed an *idle* window over a *working* turn. It
|
||||||
|
ended `awaiting_input`.
|
||||||
|
- **Turn runtimes are long**: sessions in the DB run 15-27 min
|
||||||
|
(e.g. `44df8802` 24:24, `4319b9f8` 27:05). `handleChat` (main.go:170) has **no
|
||||||
|
SSE keepalive**; inter-iteration gaps reach 20-40s, so a proxy/browser idle
|
||||||
|
close mid-turn resets `streaming` while the turn continues on
|
||||||
|
`context.Background()` (pctx).
|
||||||
|
- **Re-proposing is real**: `44df8802` has **generation 1 (5 steps, all
|
||||||
|
`replaced`) → generation 2 (25 steps, done)**, with **2 `propose_plan` + 52
|
||||||
|
`update_plan_step`** calls persisted. The activity timeline renders every one
|
||||||
|
of those across both generations.
|
||||||
|
|
||||||
|
## Root causes
|
||||||
|
|
||||||
|
- **G1 — "working" == `streaming`.** Every working-indication in the chat window
|
||||||
|
(AgentTrace running status, indicator headline, stream cursor, panel spinner,
|
||||||
|
`disabled={streaming}` input) is gated on the live SSE flag. A background turn
|
||||||
|
(`resumeSession`/continuation worker) has no stream; a desynced long live turn
|
||||||
|
has a dead stream. In both cases `streaming=false` while the server is actively
|
||||||
|
working. The **session `status`** (`planning`/`executing`/`awaiting_input`) is
|
||||||
|
the reliable "server is running a turn" signal and is already live-refreshed
|
||||||
|
(`workspace.ts` `taskFor`, `STATUS_AFFECTING`), but the chat UI never uses it.
|
||||||
|
- **G2 — busy-turn message is rejected, not queued.** main.go:292-302: the turn
|
||||||
|
gate waits 5s then emits the "still finishing a previous step" error and
|
||||||
|
returns. The user message *is* persisted (main.go:270) but is **inert** — the
|
||||||
|
user must manually re-send.
|
||||||
|
- **G3 — activity is poll/event laggy.** Tool-level activity derives from
|
||||||
|
`messages`, refreshed only by the 3s poller; plan steps are **events-only**
|
||||||
|
(`workspace.ts` `hydrateSession`) with no poll, so a missed `plan.proposed`
|
||||||
|
event leaves the panel stuck on a stale generation.
|
||||||
|
- **G4 — timeline is generation-unaware.** `activity.ts` `computeActivityLog`
|
||||||
|
walks **all** messages' tool calls, so a re-proposed task renders N
|
||||||
|
"Proposed plan" entries and attributes tools to steps via `currentStepSeq`
|
||||||
|
inferred from `update_plan_step` calls across **every** generation — tools land
|
||||||
|
under the wrong (current-gen) step or under steps that were `replaced`. This is
|
||||||
|
the "several plans / some steps never run" view.
|
||||||
|
|
||||||
|
## Fixes (ordered)
|
||||||
|
|
||||||
|
### F1 — Status-driven `working` signal (fixes G1)
|
||||||
|
Add a derived store `taskWorking(sessionId)` = `$streaming OR status ∈
|
||||||
|
{planning, executing}` (explicitly **not** `awaiting_input` — that is paused for
|
||||||
|
input), plus a global `currentWorking` for the main view backed by `currentTask`.
|
||||||
|
Use it wherever `streaming` currently drives "is it working":
|
||||||
|
- `ChatThread.svelte`: `traceStatus` last-message = `working ? 'running' : …`;
|
||||||
|
`indicatorLabel` and the AgentTrace `status`/`label` props.
|
||||||
|
- `TaskContextPanel.svelte:137` spinner and `UnifiedTimeline` `streaming` prop →
|
||||||
|
`working`.
|
||||||
|
- Keep a separate `streaming` for the literal "live text deltas are arriving"
|
||||||
|
cursor; `working` is the superset for indicators/input.
|
||||||
|
- Input stays **enabled** while `working` (the user must be able to interject);
|
||||||
|
the send path queues when busy (F2). Show a muted "Nomos is working…" hint in
|
||||||
|
the composer when `working && !streaming`.
|
||||||
|
|
||||||
|
### F2 — Queue operator messages; auto-run when free (fixes G2)
|
||||||
|
- Server: in-memory per-session FIFO on the `agent` struct (mirrors `turnGate`),
|
||||||
|
`{message, reply}` entries. `handleChat`: when the gate is busy, **enqueue**
|
||||||
|
instead of rejecting, and emit a `queued` SSE event (replaces today's error at
|
||||||
|
main.go:294-302). Persist the user message as today (already done pre-acquire).
|
||||||
|
- Drain: arm a per-session drainer that, on gate release, acquires again and runs
|
||||||
|
the next queued message as a normal turn (same persist/emit path as
|
||||||
|
`handleChat`). Strictly one-at-a-time under the gate — this cannot stack turns
|
||||||
|
(the hazard v0.15.0 F1 removed); background `resumeSession` keeps its
|
||||||
|
non-blocking skip and never touches the queue.
|
||||||
|
- If the session is terminal (`done`/`failed`) or `awaiting_input` when a queued
|
||||||
|
message runs, `reopenSession`/answer handling applies as for any follow-up.
|
||||||
|
- Frontend: on the `queued` event show an inline "Queued — will run when the
|
||||||
|
current step finishes" chip on that user bubble; clear it when the turn's real
|
||||||
|
events begin. Drop the humanized "still finishing" error for the busy case.
|
||||||
|
|
||||||
|
### F3 — SSE keepalive on `handleChat` (prevents the G1 desync at the source)
|
||||||
|
Wrap `a.chat(...)` in a goroutine + `select` with a **10-15s ticker** that writes
|
||||||
|
an SSE comment (`:keepalive\n\n`) and flushes, so 20-40s inter-iteration gaps no
|
||||||
|
longer trip proxy/browser idle timeouts. Stop the ticker when `a.chat` returns.
|
||||||
|
(EventSource ignores comment lines by spec — safe.)
|
||||||
|
|
||||||
|
### F4 — Generation-aware timeline + self-healing plan panel (fixes G3/G4)
|
||||||
|
- `activity.ts` `computeActivityLog`: find the **last** `propose_plan` in the
|
||||||
|
message stream; ignore `propose_plan`/`update_plan_step` calls **before** it
|
||||||
|
for both rendering and `currentStepSeq` inference. Render at most one
|
||||||
|
"Proposed plan" entry (the current generation). Steps continue to come from
|
||||||
|
`$steps` (already current-gen via `fetchPlan` MAX(generation)). Optionally emit
|
||||||
|
a single "Plan revised" entry when >1 generation exists.
|
||||||
|
- Plan-panel resilience: on any `STATUS_AFFECTING` event (and on reconnect),
|
||||||
|
re-fetch the plan (`fetchPlan`) in addition to the live `plan.proposed` handler,
|
||||||
|
so a missed event self-heals instead of leaving a stale generation.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
- `go test ./cmd/nomos/`: extend `turngate_test.go`/new `messagequeue_test.go` —
|
||||||
|
queued message runs strictly after release; FIFO order preserved across 3
|
||||||
|
queued sends; a background `resumeSession` busy-skip does **not** consume or
|
||||||
|
starve the queue; queued message runs even if session went `awaiting_input`.
|
||||||
|
- Web `vitest`: `activity.test.ts` — add a 2-generation fixture (2× propose_plan,
|
||||||
|
interleaved update_plan_step) asserting exactly one "Proposed plan" and correct
|
||||||
|
step attribution to gen-2 steps; `chat`/store test — `working` is true from
|
||||||
|
`status==='executing'` even with `streaming=false`; `queued` event renders the
|
||||||
|
queued chip and clears on first tool_use.
|
||||||
|
- Manual: (a) start a long task, **reload the window mid-turn** → the working
|
||||||
|
indicator stays on (status-driven); (b) send a message mid-turn → "Queued" →
|
||||||
|
runs after the turn; (c) open `44df8802`-style 2-gen session → timeline shows
|
||||||
|
one plan, no ghost proposals.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **F2 must not reintroduce concurrent turns.** The queue drains one-at-a-time
|
||||||
|
under the gate; background resume remains non-blocking and queue-agnostic.
|
||||||
|
Existing `turngate_test.go` concurrency assertion (max in-flight = 1) must stay
|
||||||
|
green.
|
||||||
|
- **Status-driven `working` could stick on** if a terminal event is missed.
|
||||||
|
Mitigated by the existing terminal `task.status` → `clearTurnState` recovery
|
||||||
|
plus a `loadSessions` refresh on reconnect (F4).
|
||||||
|
- **Keepalive comments** must stay SSE comments (`:` prefix) so they aren't
|
||||||
|
parsed as events.
|
||||||
|
|
||||||
|
## Out of scope / follow-ups
|
||||||
|
|
||||||
|
- Model efficiency: the 8+ pure-exploration iterations (repeated
|
||||||
|
`list_entities`/`get_relations`) that inflate turn length to 15-27 min —
|
||||||
|
prompt/iteration-budget tuning, separate effort.
|
||||||
|
- F8 from the prior plan (oldest-first timeline toggle; per-tool `tool.*` events
|
||||||
|
for background turns). F1's status-driven `working` makes background work
|
||||||
|
visible without live per-tool deltas, so this remains lower priority.
|
||||||
|
|
||||||
|
## Open implementation note
|
||||||
|
|
||||||
|
Host the per-session message queue on the `agent` struct (in-memory `map[string]
|
||||||
|
[]queuedMsg` + per-session drainer goroutine), mirroring `turnGate`. No DB table
|
||||||
|
needed — messages are already persisted by `handleChat` before enqueue; the queue
|
||||||
|
only schedules *when* a turn runs, not *whether* the message is stored.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Resolution (2026-08-03)
|
||||||
|
|
||||||
|
Implemented F1–F4 in v0.15.1 → v0.17.0 (the intermediate 0.16.0 was the
|
||||||
|
cyberspace-aesthetic commit, landed via auto-pull during this work).
|
||||||
|
|
||||||
|
| Item | What shipped | Where |
|
||||||
|
|---|---|---|
|
||||||
|
| **F1** | Status-driven `working` signal (`taskWorking(sessionId)` / `currentWorking`) = live stream OR session status ∈ {planning, executing}. Drives the chat trace running state, the "thinking" headline, the activity spinner, and the timeline `streaming` prop — so a background/long/desynced turn still looks alive (the "can't tell it's working" symptom). The composer stays enabled during background work so the operator can interject. | `web/src/lib/stores/workspace.ts` (`isWorking`, `taskWorking`, `currentWorking`), `ChatThread.svelte` (`working` prop, `traceStatus`, indicator), `TaskContextPanel.svelte`, `SessionChatWindow.svelte`, `NewTaskChat.svelte`. |
|
||||||
|
| **F2** | Operator messages sent during an in-flight turn are now QUEUED and auto-run when the gate frees, replacing the "still finishing a previous step… send it again" rejection. Per-session in-memory FIFO drained strictly one-at-a-time under the turn gate (no concurrent-turn reintroduction). A `queued` SSE event tells the client, which drops the optimistic bubble and shows a "Queued — will run when it finishes the current step" hint (derived from `working` + last-message shape, so it survives the poller). | `cmd/nomos/messagequeue.go` (+`messagequeue_test.go`), `agent.go` (queue field), `main.go` (`runChatTurn`, `drainQueued`, handleChat queue path), `continue.go` (resumeSession drains on release), `web/src/lib/types.ts` (`ChatQueuedEvent`), `chat.ts` (`queued` handling in sendSessionMessage/startTask). |
|
||||||
|
| **F3** | SSE keepalive: a 12s `:keepalive` comment ticker during `handleChat` so 20-40s inter-iteration gaps no longer trip a proxy/browser idle timeout (the desync root cause). All SSE writes (events + keepalive) serialized through one mutex — `http.ResponseWriter` is not concurrency-safe. | `cmd/nomos/main.go` (`writeMu`/`writeEvent`, keepalive goroutine). |
|
||||||
|
| **F4** | Generation-aware activity timeline: only the LAST `propose_plan` renders as "Proposed plan"; superseded ones collapse to a single "Earlier plan revised" marker, and step-attribution only follows the current generation's `update_plan_step` calls. Plus plan-panel self-heal: the plan is refetched (debounced) on any task-lifecycle event so a missed `plan.proposed` no longer freezes the panel on a stale generation. | `web/src/lib/stores/activity.ts` (`computeActivityLog`), `workspace.ts` (`schedulePlanRefetch`). |
|
||||||
|
|
||||||
|
**Verification:**
|
||||||
|
- `go vet ./cmd/nomos/` clean; `go test ./cmd/nomos/` green, incl. new
|
||||||
|
`messagequeue_test.go` (FIFO, requeueFront, per-session isolation, concurrency,
|
||||||
|
drainQueued no-op-on-empty, drainQueued requeues-when-busy). Existing
|
||||||
|
`turngate_test.go`/`continue_test.go` still green (single-flight guarantee
|
||||||
|
intact).
|
||||||
|
- Web `vitest` 72/72 green (added 2 F4 generation-awareness tests to
|
||||||
|
`activity.test.ts`: one "Proposed plan" + revised marker + current-gen-only
|
||||||
|
step attribution; plan-less Q&A attributes nothing).
|
||||||
|
- `vite build` succeeds. `tsc --noEmit` shows only the pre-existing baseline
|
||||||
|
errors (`ui/*`, `oidc.ts`, `windows.ts`, `workspace.ts:123/201/221`) noted in
|
||||||
|
v0.15.0 — no new errors from this change. ESLint: no new errors (the one new
|
||||||
|
`svelte/valid-compile` on `chatWorking` got the same disable its siblings have).
|
||||||
|
|
||||||
|
**Follow-ups (not in this pass):**
|
||||||
|
- Model efficiency: the long (15-27 min) exploration-heavy turns that made the
|
||||||
|
desync so painful — prompt / iteration-budget tuning, separate effort.
|
||||||
|
- F8 from the prior plan (oldest-first timeline toggle; per-tool `tool.*` events
|
||||||
|
for background turns). F1's status-driven `working` makes background work
|
||||||
|
visible without live per-tool deltas, so this stays lower priority.
|
||||||
191
plans/done/2026-08-04-chat-window-overhaul.md
Normal file
191
plans/done/2026-08-04-chat-window-overhaul.md
Normal file
@@ -0,0 +1,191 @@
|
|||||||
|
# 2026-08-04 — Chat interaction overhaul: inline progressive stream (Claude Code style)
|
||||||
|
|
||||||
|
**Status:** Planned — not started. (Refocused from the earlier feature-heavy
|
||||||
|
draft; backend features deferred — see "Deferred".)
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Streamline agent interactions — thinking, plan, tool usage, responses — into
|
||||||
|
**one linear progressive inline stream per turn** (the Claude Code / Cline /
|
||||||
|
Roo pattern), instead of the current split where the transcript shows a
|
||||||
|
collapsed trace and the real live activity lives in a separate rail timeline.
|
||||||
|
The right rail becomes **graph-only** (and auto-zooms to fit all entities).
|
||||||
|
|
||||||
|
## Locked decisions (operator interview)
|
||||||
|
|
||||||
|
| Decision | Choice |
|
||||||
|
|---|---|
|
||||||
|
| Live activity layout | **Inline stream (Claude Code)** — one progressive column per turn; rail keeps ONLY the Scope graph; Activity timeline tab removed |
|
||||||
|
| Tool-call detail | **Per-tool progressive lines** — each tool its own compact live line (spinner → one-line result summary), expandable to raw |
|
||||||
|
| Feature phases | **Defer** — edit/resubmit, @mentions, attachments are later phases; this plan is interaction-focused + graph auto-zoom |
|
||||||
|
|
||||||
|
## Diagnosis (grounded in current code)
|
||||||
|
|
||||||
|
- The transcript (`ChatThread` → `AgentTrace`) collapses a whole turn's tool
|
||||||
|
calls into one line ("Proposed plan" / "N tool calls"), raw-JSON detail on
|
||||||
|
expand. Not progressive; you can't see what's happening without expanding.
|
||||||
|
- The actual live plan + tool timeline lives in the **right rail**
|
||||||
|
(`TaskContextPanel` → `UnifiedTimeline`): newest-first backbone + tool stubs.
|
||||||
|
So "what is the agent doing" is in a **second place** — a cognitive split.
|
||||||
|
- `UnifiedTimeline` is imported **only** by `TaskContextPanel` (grep confirms),
|
||||||
|
so removing the Activity pane is self-contained.
|
||||||
|
- The `activityLog` **store** stays required: it feeds inline labels
|
||||||
|
(`toolActivityLabel`), live `run` output (`toolsWithLive`), and the mascot
|
||||||
|
(`mascot/stimuli.ts`). Only the timeline *view* is removed.
|
||||||
|
- Tool events already arrive separately (`tool_use` then `tool_result` in
|
||||||
|
`chat.ts`), and the activity log already carries humanized labels + per-tool
|
||||||
|
`stepSeq` attribution. So progressive per-tool lines + step grouping are a
|
||||||
|
**presentation** change, not a data/model change.
|
||||||
|
- `run` results are free-form text (e.g. `"run on lxc:caddy: ERROR exit status
|
||||||
|
1"`) → one-line result summaries are best-effort text parsing, no backend.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
### D1 — One progressive inline stream per turn
|
||||||
|
Replace `AgentTrace` (one collapsed blob per turn) with a new
|
||||||
|
**`TurnTrace.svelte`** rendered inline for each assistant turn, top-to-bottom:
|
||||||
|
1. **Live plan checklist** (only on the most-recent/running turn — see D3).
|
||||||
|
2. **Tool lines grouped by plan step** (D2), then orphan tools (no step).
|
||||||
|
3. **Streamed text answer** (existing `markdown-body prose-chat`), with the
|
||||||
|
blinking cursor while streaming (existing).
|
||||||
|
4. A compact **"Thinking" line** while `working` and before any output: reuses
|
||||||
|
the existing `indicatorLabel` (running step → tool → "Agent is thinking…").
|
||||||
|
Fades once text/tools arrive; reappears between steps.
|
||||||
|
|
||||||
|
### D2 — Per-tool progressive lines (the Claude-Code signature)
|
||||||
|
One `ToolLine.svelte` per tool call (replaces `ToolCallCard`'s row style):
|
||||||
|
- Left: state icon — spinner while `tool_use`-only, ✓ on result, ✗ on error.
|
||||||
|
- Label: existing `toolActivityLabel(tool)` (humanized action).
|
||||||
|
- **One-line result summary** on completion — new `toolResultSummary(tool)`
|
||||||
|
in `activity.ts` (see plumbing). E.g.:
|
||||||
|
- `run` → `exit 0 · <first line>` (parse "exit status N" / "ERROR")
|
||||||
|
- `get_entity` → `host:hubris (healthy)`; `get_health_summary` → `healthy X · degraded Y · down Z`
|
||||||
|
- `list_entities`/`list_lxcs` → `N entities`; `get_relations` → `N relations`
|
||||||
|
- `search_knowledge` → `N results`; `upsert_knowledge` → `recorded document:…`
|
||||||
|
- `update_plan_step` → `step <seq> → <status>`; `propose_plan` → `N steps`
|
||||||
|
- default → first non-empty line of stringified result (≤80ch); `done` if empty
|
||||||
|
- Live `run` output: while streaming, the line auto-expands a pinned-tail mini
|
||||||
|
pane (reuse the `liveOutput` path from `toolsWithLive`).
|
||||||
|
- Click → expand raw args/result (border-driven `<pre>`, cyberspace-square).
|
||||||
|
- Border-driven, no rounded/shadow (per `border_driven_language`).
|
||||||
|
|
||||||
|
### D3 — Live plan checklist (TodoWrite-style)
|
||||||
|
On the **running/last** turn, render the current-generation `planSteps`
|
||||||
|
(already generation-aware via `workspace.ts`) as a checklist: pending = hollow,
|
||||||
|
running = spinner + highlight, done = ✓, failed = ✗, blocked = pause. Steps
|
||||||
|
check off live as `plan.step.*` events land. This is the unified timeline's
|
||||||
|
plan view, moved inline and scoped to the active turn. Past turns render only
|
||||||
|
their tool lines + text (the plan is session-level; the running turn carries
|
||||||
|
its current state, mirroring how TodoWrite re-displays state each turn). On a
|
||||||
|
terminal task state (`done`/`failed`), the checklist collapses to one line:
|
||||||
|
`Plan complete — N steps` / `Plan failed — step K`.
|
||||||
|
|
||||||
|
### D4 — Rail → graph only
|
||||||
|
`TaskContextPanel`: remove the Activity pane and the `UnifiedTimeline` import;
|
||||||
|
the panel becomes the Scope graph full-height (keep the collapsible "Scope"
|
||||||
|
header + the `nowTouching` strip). The graph is now the rail's entire job, so
|
||||||
|
auto-fit (D6) matters more. `activityLog*` stores remain imported only where
|
||||||
|
the inline stream/mascot need them.
|
||||||
|
|
||||||
|
### D5 — Cyberspace cohesion of the stream
|
||||||
|
Apply alongside the rewrite so the new inline view is on-system from day one:
|
||||||
|
- Transcript → **terminal log rows** (square, full-width, `YOU`/`NOMOS`
|
||||||
|
role-tags, hairline `divide-y` separators; no bubbles, no soft shadow).
|
||||||
|
Delete `.user-msg { box-shadow }`.
|
||||||
|
- Tool lines + expanded `<pre>`: border-driven, square, opaque.
|
||||||
|
- Composer: opaque `bg-background`, square (remove `rounded-2xl`/`bg-card/50`).
|
||||||
|
- Rewrite the stale "Art Nouveau" `<style>` comments → "cyberspace/terminal".
|
||||||
|
- Per `central_css_override`: drive surface styling centrally in `app.css`
|
||||||
|
where it's a primitive concern; no ad-hoc `rounded-*`/`shadow-*`/`backdrop-blur`.
|
||||||
|
|
||||||
|
### D6 — Graph auto-fit + drag-pan (`SessionGraph.svelte`) (carried over)
|
||||||
|
- Wrap nodes+links in `<g transform="translate(tx,ty) scale(s)">`; fit the bbox
|
||||||
|
of all nodes (radius + label + padding) into `cw`/`ch`; cap `s ∈ [0.2, 2.5]`.
|
||||||
|
- Re-fit on: mount, node-set change, container resize, sim-settle
|
||||||
|
(`alpha > 0.05`), background double-click. **Not** every tick (fights pan).
|
||||||
|
A `userPanned` flag pauses auto-follow after a manual pan until next
|
||||||
|
membership/resize/double-click.
|
||||||
|
- Background drag = pan (`tx`/`ty`); node drag converts screen→graph via the
|
||||||
|
inverse transform before setting `fx`/`fy`. Dot-grid stays in screen space.
|
||||||
|
- Keep: open-on-click, `touched` pulse, health-diff label, selection ring.
|
||||||
|
Respect `scrollIntoView` pitfall (transform, not scroll).
|
||||||
|
|
||||||
|
## Phased task list (each independently shippable; all frontend)
|
||||||
|
|
||||||
|
- **P1 — Inline progressive stream.** `TurnTrace.svelte` + `ToolLine.svelte`;
|
||||||
|
wire into `ChatThread` per turn; "Thinking" line; tool→step grouping via
|
||||||
|
activity-log `stepSeq` matched by tool id; keep `toolsWithLive` for `run`.
|
||||||
|
- **P2 — Live plan checklist.** Inline current-gen `planSteps` on the running
|
||||||
|
turn; collapse-to-summary at terminal state.
|
||||||
|
- **P3 — Rail → graph only.** Strip Activity pane + `UnifiedTimeline` from
|
||||||
|
`TaskContextPanel`; verify no other importers (grep: only TaskContextPanel).
|
||||||
|
- **P4 — Cyberspace cohesion.** Terminal log rows; remove rounded/shadow/
|
||||||
|
translucency; square composer; centralize in `app.css`; fix stale comments.
|
||||||
|
- **P5 — Graph auto-fit + drag-pan.** D6.
|
||||||
|
- **Polish (small, frontend-only):** per-message/tool **copy**; **scroll-to-
|
||||||
|
bottom** button (uses `container.scrollTo`, never `scrollIntoView`).
|
||||||
|
|
||||||
|
## Plumbing specifics (grounded, no backend)
|
||||||
|
|
||||||
|
- New `toolResultSummary(t: ToolCallResult): string` in `activity.ts`, beside
|
||||||
|
`toolActivityLabel`. Per-name switch (D2 list), graceful fallback.
|
||||||
|
- Tool→step grouping: build `id → stepSeq` from the activity log once per turn;
|
||||||
|
tools with no step render as orphans.
|
||||||
|
- Reuse: `planSteps` (generation-aware), `indicatorLabel`, `toolsWithLive`,
|
||||||
|
`toolActivityLabel`, `liveOutput` streaming path.
|
||||||
|
|
||||||
|
## Constraints honored (saved decisions)
|
||||||
|
|
||||||
|
- `design_system.central_css_override`, `border_driven_language`: square,
|
||||||
|
hairline, opaque, focus-by-color, no soft shadows/glows.
|
||||||
|
- `chat_thread.pane_layout`: dynamic status (Thinking line, live checklist)
|
||||||
|
lives in the **message Pane**, never the input Pane.
|
||||||
|
- `wmkit.scrollintoview_reflow_pitfall`: `container.scrollTo` for scroll-to-
|
||||||
|
bottom; transform (not scroll) for graph pan.
|
||||||
|
|
||||||
|
## Risks
|
||||||
|
|
||||||
|
- **Removing the rail timeline loses the "overview" view.** Mitigation: the
|
||||||
|
inline checklist + per-turn tool lines carry the same info progressively; the
|
||||||
|
graph still shows fleet scope. If operators miss the overview, a collapsed
|
||||||
|
"full timeline" can return as a toggle (follow-up).
|
||||||
|
- **Auto-fit vs manual pan** — handled by `userPanned` + settle-alpha gate.
|
||||||
|
- **Inline stream length on long turns** (15–27 min, many tools) — progressive
|
||||||
|
lines can get long; mitigate by auto-collapsing finished steps (keep the
|
||||||
|
running step + its tools expanded, prior steps as one-line summaries).
|
||||||
|
- **Best-effort result summaries** may misformat unusual payloads — fallback is
|
||||||
|
always a truncated raw line + expandable raw detail, never a blank.
|
||||||
|
|
||||||
|
## Validation
|
||||||
|
|
||||||
|
- `npm run lint`, `tsc --noEmit` (no NEW errors beyond the known baseline in
|
||||||
|
`ui/*`, `oidc.ts`, `windows.ts`, `workspace.ts`), `vite build`, `vitest`
|
||||||
|
(add a `toolResultSummary` unit test per tool name + fallback).
|
||||||
|
- Manual matrix: (a) start a long task → Thinking line → plan checklist
|
||||||
|
appears and checks off live → each tool streams as its own line with a
|
||||||
|
one-line summary → text streams; (b) reload mid-turn → working still shows;
|
||||||
|
(c) `run` tool → live output pins to tail then collapses to summary; (d)
|
||||||
|
graph auto-fits at settle + on new entity + drag-pan + double-click reset;
|
||||||
|
(e) no rounded/soft-shadow remains on chat surfaces; (f) rail shows graph only.
|
||||||
|
|
||||||
|
## Deferred (later phases, after this lands + validates)
|
||||||
|
|
||||||
|
- **Edit-and-resubmit** — `truncateFrom` store method + `POST /sessions/{id}/edit`
|
||||||
|
(extract `streamTurn` from `handleChat`); reuse `reopenSession` (already
|
||||||
|
exists, store.go:838 — marks prior `session_plan_steps` `replaced`, clears
|
||||||
|
outcome) for the reset. Reject edit while the gate is busy (HTTP 409); edit
|
||||||
|
cannot queue (truncation must be atomic). Regenerate = no-op-edit case.
|
||||||
|
- **@entity mentions** — small `GET /api/v1/entities/search?q=` + composer
|
||||||
|
autocomplete inserting `type:name` slugs the agent/graph already parse.
|
||||||
|
- **Attachments** — multipart upload + `agent_attachments` table + configured
|
||||||
|
`OIKOS_ATTACHMENTS_DIR` (explicit volume, not relative) + capped text inlining.
|
||||||
|
- **Continue button** — needs `/resume` to `reopenSession` first for terminal
|
||||||
|
sessions (today `/resume` does not reopen `done`/`failed`; `handleChat`'s
|
||||||
|
follow-up path does). Small backend tweak.
|
||||||
|
- **Image vision** pending provider confirmation.
|
||||||
|
|
||||||
|
## Out of scope / follow-ups
|
||||||
|
|
||||||
|
- A collapsible "full timeline" overview toggle if the rail removal is missed.
|
||||||
|
- `read_attachment` MCP tool (lazy full-content fetch, lower context than inlining).
|
||||||
|
- Oldest-first timeline toggle / per-tool `tool.*` events for background turns.
|
||||||
145
plans/done/2026-08-04-hermes-mcp-client-integration.md
Normal file
145
plans/done/2026-08-04-hermes-mcp-client-integration.md
Normal file
@@ -0,0 +1,145 @@
|
|||||||
|
# 2026-08-04 — Hermes MCP client integration: native tool surface for oikos
|
||||||
|
|
||||||
|
**Status:** Plan.
|
||||||
|
**Context:** Hermes Agent (mac-mini workstation) now connects to oikos's MCP server
|
||||||
|
as a native MCP client (`mcp_servers.oikos` in `~/.hermes/config.yaml`). All 37+ MCP
|
||||||
|
tools are available as `mcp__oikos__*` first-class Hermes tool calls — no more raw
|
||||||
|
curl with batch-initialize SSE parsing. The integration works; this plan tightens the
|
||||||
|
remaining seams.
|
||||||
|
|
||||||
|
**Trigger:** First-use retrospective identified three areas that make the integration
|
||||||
|
harder to use than it should be.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Motivation
|
||||||
|
|
||||||
|
The oikos MCP server (`internal/mcp/`) speaks Streamable HTTP at
|
||||||
|
`https://mcp.hubris.network/mcp`. Hermes Agent's native MCP client connects to it on
|
||||||
|
startup, discovers tools, and registers them as callable functions. This replaces the
|
||||||
|
previous pattern where agents fired raw curl requests with batch `initialize` +
|
||||||
|
`tools/call` envelopes.
|
||||||
|
|
||||||
|
Three friction points observed:
|
||||||
|
|
||||||
|
- **No lightweight connectivity check.** The `/healthz` HTTP endpoint exists but isn't
|
||||||
|
exposed at the MCP protocol layer. An agent that wants to verify the MCP server is
|
||||||
|
reachable must call a real tool (e.g. `list_entities` with a limit of 1) — every call
|
||||||
|
carries the Streamable HTTP session-initialization overhead.
|
||||||
|
- **Bearer token in plaintext.** `~/.hermes/config.yaml` stores the token directly in the
|
||||||
|
`mcp_servers.oikos.headers.Authorization` value. Hermes does not support env-var
|
||||||
|
interpolation in MCP server configs, so the token can't live only in `.env`.
|
||||||
|
- **Zero-visibility streaming overhead.** Streamable HTTP batches `initialize` +
|
||||||
|
`tools/call` per request. This adds ~2KB of transport per tool call that the agent
|
||||||
|
never sees. For a single `get_health_summary` call this is negligible; for a 10-tool
|
||||||
|
exploration pass it's 20KB of invisible overhead.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Changes
|
||||||
|
|
||||||
|
### I — MCP health/ping tool (`mcp__oikos__ping`)
|
||||||
|
|
||||||
|
**Why:** Agents need a zero-cost connectivity check before calling production tools.
|
||||||
|
Currently every check incurs the full Streamable HTTP initialize + tools/call round-trip.
|
||||||
|
|
||||||
|
**What:**
|
||||||
|
|
||||||
|
Add a `ping` tool that returns `{"ok": true, "server": "oikos", "version": "dev"}`.
|
||||||
|
No arguments. No DB hit. No auth check (already protected by the MCP transport's auth
|
||||||
|
layer — the request won't arrive if the bearer token is missing).
|
||||||
|
|
||||||
|
```go
|
||||||
|
// internal/mcp/tools.go
|
||||||
|
{
|
||||||
|
Name: "ping",
|
||||||
|
Description: "Lightweight connectivity check. Returns immediately with server identity, no DB hit.",
|
||||||
|
InputSchema: jsonschema.Must(nil), // no params
|
||||||
|
Handler: func(ctx context.Context, args json.RawMessage, caller CallerInfo) (json.RawMessage, error) {
|
||||||
|
return json.RawMessage(`{"ok":true,"server":"oikos","version":"` + version.Version + `"}`), nil
|
||||||
|
},
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Risk class:** read-only. No auth, no DB, no state. Auto-approves.
|
||||||
|
|
||||||
|
**Test:** `hermes mcp test oikos` (from the Hermes CLI) verifies MCP server reachability
|
||||||
|
independently; the `ping` tool gives agent code the same signal programmatically.
|
||||||
|
|
||||||
|
### II — Tool name documentation in server metadata
|
||||||
|
|
||||||
|
**Why:** Hermes prefixes MCP tools as `mcp_{server}_{tool}`, so `get_health_summary`
|
||||||
|
becomes `mcp__oikos__get_health_summary`. Agents discover tool names at runtime via
|
||||||
|
`tools/list`, but there's no short summary of what each tool group does that survives
|
||||||
|
into the MCP tool description.
|
||||||
|
|
||||||
|
**What:**
|
||||||
|
|
||||||
|
Audit and tighten every tool's `Description` field in `internal/mcp/tools.go` so the
|
||||||
|
first 8–12 words are a searchable one-liner an agent can pattern-match against.
|
||||||
|
Current descriptions that are vague or redundant get a prefix rewrite:
|
||||||
|
|
||||||
|
| Tool | Current description | Revised |
|
||||||
|
|------|-------------------|---------|
|
||||||
|
| `get_entity` | "Get entity metadata" | "Look up one entity by slug or UUID — type, state, attributes, health" |
|
||||||
|
| `list_entities` | "List entities" | "Browse entities by type, state, or name substring — paginated" |
|
||||||
|
| `upsert_knowledge` | "Record what you learned" | "Write a document/investigation/runbook to the knowledge graph — idempotent" |
|
||||||
|
| `run` | "Run ANY shell command" | "Execute a shell command on any host/LXC/VM — auto-classified by risk" |
|
||||||
|
|
||||||
|
Existing tools pass through unchanged if their description is already crisp. ~15 tools
|
||||||
|
get description rewrites.
|
||||||
|
|
||||||
|
**Risk class:** read-only (config change). No runtime effect.
|
||||||
|
|
||||||
|
### III — Env-var interpolation docs for Hermes config (oikos-side documentation)
|
||||||
|
|
||||||
|
**Why:** The bearer token lives in `~/.hermes/config.yaml` in plaintext because Hermes
|
||||||
|
does not support `${VAR}` interpolation in MCP server configs. This is a Hermes
|
||||||
|
upstream feature request, not an oikos change — but oikos should document the
|
||||||
|
workaround and track the upstream ask.
|
||||||
|
|
||||||
|
**What:**
|
||||||
|
|
||||||
|
Add a `### Hermes MCP client` subsection to `docs/infrastructure/mcp-server.md` (or
|
||||||
|
create it if it doesn't exist) that covers:
|
||||||
|
|
||||||
|
1. The config block to add to `~/.hermes/config.yaml` (already done — record it
|
||||||
|
for the next person).
|
||||||
|
2. The token exposure caveat: Hermes doesn't support env-var interpolation in
|
||||||
|
`mcp_servers` `headers` yet (upstream issue nousresearch/hermes-agent#TODO — file
|
||||||
|
once).
|
||||||
|
3. Workaround: `hermes config set security.redact_secrets true` (already default) so
|
||||||
|
the token value is stripped from tool output and logs even if it appears in
|
||||||
|
diagnostic text.
|
||||||
|
4. How to verify the connection: `hermes mcp list` → `hermes mcp test oikos`.
|
||||||
|
|
||||||
|
**Risk class:** docs-only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Open questions
|
||||||
|
|
||||||
|
| Question | Decision |
|
||||||
|
|----------|----------|
|
||||||
|
| Should `ping` bypass auth entirely or still require a valid bearer token? | **Still requires auth.** The MCP transport layer validates the token before routing to `ping` — no special treatment needed. If the token is missing, the request never reaches the handler. |
|
||||||
|
| Who files the Hermes upstream feature request for `${VAR}` interpolation? | **Oikos operator** (dtoro). The need is specific to this deployment. File at https://github.com/NousResearch/hermes-agent/issues. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Not doing (yet)
|
||||||
|
|
||||||
|
- **Persistent MCP sessions** — Streamable HTTP stateless mode is fine for the
|
||||||
|
current tool-call volume (~1–5 calls per agent turn). Persistent sessions would
|
||||||
|
save ~2KB per call but add connection lifecycle complexity. Revisit if per-turn
|
||||||
|
tool calls exceed 20.
|
||||||
|
- **`tools/list` caching** — Hermes already caches tool discovery at session start.
|
||||||
|
The 37-tool list is ~4KB; caching adds complexity for negligible savings.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Verification
|
||||||
|
|
||||||
|
1. `curl -X POST https://mcp.hubris.network/mcp ... -d '...ping...'` returns `{"ok":true,"server":"oikos","version":"dev"}`
|
||||||
|
2. `hermes mcp list` shows `ping` among oikos tools
|
||||||
|
3. `hermes doctor` passes
|
||||||
|
4. Tool descriptions are crisp: `hermes mcp list` output for oikos shows prefixed summaries
|
||||||
369
plans/done/2026-08-04-session-audit-agent-reliability.md
Normal file
369
plans/done/2026-08-04-session-audit-agent-reliability.md
Normal file
@@ -0,0 +1,369 @@
|
|||||||
|
# 2026-08-04 — Session audit: agent reliability, plan system, and learning loop gaps
|
||||||
|
|
||||||
|
**Status:** Done — all 12 tasks implemented, tested, deployed in v0.21.0. Verified in production
|
||||||
|
with live session tests (classifications, feedback, token tracking, execution linkage all confirmed).
|
||||||
|
**Reviewed sessions:** Past 5 completed plus ZimaOS continuation (268895a5)
|
||||||
|
**Method:** Direct Postgres read of `agent_sessions`/`agent_messages`/
|
||||||
|
`agent_activity`/`session_plan_steps`/`executions`/`classifications`/`feedback`/
|
||||||
|
`patterns`/`skills`/`approvals`/`nomos_plan_executions`/`audit_log` on the prod
|
||||||
|
mac-mini. Cross-referenced with `internal/audit/`, `internal/mcp/`,
|
||||||
|
`internal/httpapi/`, `internal/policy/`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Sessions audited
|
||||||
|
|
||||||
|
| # | Session | Turns | Run calls | Failures | Plan steps (done/total) | Duration | Outcome |
|
||||||
|
|---|---------|-------|-----------|----------|-------------------------|----------|---------|
|
||||||
|
| S1 | Pocket-pascal deploy (a433b386) | 8 | 111 | 1 | 10/11 | 1.5h | Success (truncated by turn limit) |
|
||||||
|
| S2 | SSH re-investigation (6f0ade08) | 4 | 42 | 0 | 0/4 | 11m | Success (all steps replaced) |
|
||||||
|
| S3 | Webhook HMAC (0f509508) | 2 | 40 | 0 | 5/5 | 2m | Success (clean; best session) |
|
||||||
|
| S4 | Check scripts (d458a5f8) | 23 | 96 | 4 | 0/14 | 2h | Success (3 plan gens; 0 steps done) |
|
||||||
|
| S5 | ZimaOS outage (268895a5) | 18+10 | 209 | 8 | 0/5 | 2h30m | **Marked success; dashboard still broken** |
|
||||||
|
|
||||||
|
**Headline:** 85% session success rate, but plan adherence is ~38% (15/39 steps
|
||||||
|
ever reached `done`). The ZimaOS session was the worst: marked `success` while
|
||||||
|
the dashboard was still down, then burned 81 more calls and 30 minutes chasing
|
||||||
|
irrelevant DHCP reservations.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Cross-cutting findings
|
||||||
|
|
||||||
|
### F1 — MCP 30s client timeout kills every long-running command (P0)
|
||||||
|
|
||||||
|
All **9** `run` failures across the audit are `Post "http://api:8090/mcp":
|
||||||
|
context deadline exceeded` at exactly 30s. Commands with `sleep 30`, `qm
|
||||||
|
shutdown`+wait, or async poll loops always hit this. The agent retries with
|
||||||
|
longer sleeps and hits the same wall.
|
||||||
|
|
||||||
|
**Root cause:** `cmd/nomos` MCP client uses a 30s timeout; the `run` tool
|
||||||
|
blocks synchronously waiting for command completion. No async path exists for
|
||||||
|
long-running commands.
|
||||||
|
|
||||||
|
### F2 — Premature success: `complete_task` fires before the goal is actually met (P0)
|
||||||
|
|
||||||
|
Session 268895a5 called `complete_task(success)` at 18:29 with summary "zimaos
|
||||||
|
returns 200." The endpoint was serving ttyd (terminal), not the ZimaOS
|
||||||
|
dashboard. The agent conflated "HTTPS 200" with "dashboard works." The user had
|
||||||
|
to resume the session.
|
||||||
|
|
||||||
|
**Root cause:** No pre-completion validation. The agent can mark success with a
|
||||||
|
summary that doesn't match reality. `complete_task` is a write-and-forget
|
||||||
|
operation with no state check.
|
||||||
|
|
||||||
|
### F3 — Wrong target: `run` doesn't validate the target can execute the command (P0)
|
||||||
|
|
||||||
|
In the ZimaOS continuation, `qm stop 100 --skiplock` was executed on
|
||||||
|
`lxc:dns`. The command failed (`qm: command not found`) because the agent
|
||||||
|
copied the command from a previous run but forgot to change the target.
|
||||||
|
Similarly, `cmd/nomos` tried `docker exec` on `host:hubris` (no docker).
|
||||||
|
|
||||||
|
**Root cause:** `run(target, command)` in `internal/mcp/server.go` doesn't
|
||||||
|
validate that the target type can execute the given command. A simple
|
||||||
|
allowlist would catch `qm`/`pct`/`pvesh` on non-host targets.
|
||||||
|
|
||||||
|
### F4 — Plan system is decorative: 62% of steps never reach `done` (P1)
|
||||||
|
|
||||||
|
Across 5 sessions: 39 plan steps. 24 (62%) were `replaced`, 15 (38%) reached
|
||||||
|
`done`. Session d458a5f8 had 3 complete plan regenerations with **zero**
|
||||||
|
completed steps. The agent replaces plans instead of completing or explicitly
|
||||||
|
skipping steps.
|
||||||
|
|
||||||
|
**Root cause:** Plan steps carry status (`pending`/`running`/`done`/`failed`/
|
||||||
|
`skipped`/`blocked`/`replaced`) but `replaced` has no `replaced_reason`
|
||||||
|
field. The model can silently replace every step and the system doesn't flag
|
||||||
|
it. No constraint ties `propose_plan` to existing plan state.
|
||||||
|
|
||||||
|
### F5 — No plan on session resume: continuation sessions run ad-hoc (P1)
|
||||||
|
|
||||||
|
The ZimaOS continuation (18:29→19:01) had 81 calls with **zero**
|
||||||
|
`propose_plan` calls. The agent ran ad-hoc tool calls with no structure.
|
||||||
|
|
||||||
|
**Root cause:** When a session resumes, the `must_have_plan` guard is already
|
||||||
|
satisfied by the old (completed) plan. The agent doesn't re-plan on resume.
|
||||||
|
|
||||||
|
### F6 — Scope expansion / rabbit holes: agent chases irrelevant sub-goals (P2)
|
||||||
|
|
||||||
|
In the ZimaOS continuation, the agent spent ~40 calls trying to fix Technitium
|
||||||
|
DHCP reservations — a completely different subsystem from the goal ("make the
|
||||||
|
dashboard reachable"). The ZimaOS dashboard hadn't started since July 19
|
||||||
|
(3-week-old issue), making the DHCP reservation effort moot. The agent never
|
||||||
|
surfaced a question like "This is pre-existing — should I still fix DHCP?"
|
||||||
|
|
||||||
|
**Root cause:** No scope gate. When the agent pivots to a subsystem unrelated
|
||||||
|
to the stated goal, nothing stops it. The `session_questions` mechanism exists
|
||||||
|
(2 calls in 193 sessions) but the model never uses it.
|
||||||
|
|
||||||
|
### F7 — Command generation errors: malformed bash from the LLM (P2)
|
||||||
|
|
||||||
|
In the ZimaOS continuation, the agent generated:
|
||||||
|
- `head - n` instead of `head -n` → bash syntax error
|
||||||
|
- `echo "---" && \n curl ...` → literal `\n` in command → ambiguous redirect
|
||||||
|
|
||||||
|
**Root cause:** The LLM generates bash commands inline in content blocks. No
|
||||||
|
syntax validation, no escape-character handling. The `run` tool should reject
|
||||||
|
malformed commands before execution.
|
||||||
|
|
||||||
|
### F8 — Learning pipeline completely empty (P4)
|
||||||
|
|
||||||
|
| Table | Rows |
|
||||||
|
|-------|------|
|
||||||
|
| `classifications` | **0** |
|
||||||
|
| `feedback` | **0** |
|
||||||
|
| `patterns` | **0** |
|
||||||
|
| `skills` | **0** |
|
||||||
|
|
||||||
|
Despite 1,884 executions and 263 approvals, the system learns nothing from
|
||||||
|
outcomes. The ZimaOS session discovered that the dashboard hadn't started since
|
||||||
|
July 19 — this was never persisted. The HMAC trailing-newline discovery was
|
||||||
|
persisted manually; if the agent forgot `upsert_knowledge`, it would be lost.
|
||||||
|
|
||||||
|
### F9 — Observability gaps (P3)
|
||||||
|
|
||||||
|
| Gap | Detail |
|
||||||
|
|-----|--------|
|
||||||
|
| Token tracking | `agent_activity.token_count` is NULL for every row |
|
||||||
|
| Execution linkage | `nomos_plan_executions` is empty; `audit_log.session_id` is null |
|
||||||
|
| Plan quality | No metric for step completion rate (currently 38%) |
|
||||||
|
| MCP timeout rate | No counter for `run` calls that hit the client timeout |
|
||||||
|
|
||||||
|
### F10 — Execution success rate is 74% (P2)
|
||||||
|
|
||||||
|
1,884 executions: 1,400 completed (74%), 295 failed (15.6%), 152 cancelled
|
||||||
|
(8%), 37 denied (2%). One in four execution attempts doesn't complete.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Improvement plan (ordered by impact/effort)
|
||||||
|
|
||||||
|
### Task 1 — Prevent premature `complete_task(success)` *(fixes F2)*
|
||||||
|
|
||||||
|
**`internal/mcp/tools.go`** — In the `complete_task` handler, when
|
||||||
|
`outcome=success`: require the summary field to contain a verifiable state
|
||||||
|
assertion. Minimum: if the goal mentions a URL, check that the summary doesn't
|
||||||
|
contradict known state. Lightweight: log a warning if the summary says "returns
|
||||||
|
200" but the last `ping_service` or `run` result says otherwise.
|
||||||
|
|
||||||
|
**Coach:** `nomos/SOUL.md` — explicit rule: *"Before calling
|
||||||
|
complete_task(success), restate the user's original goal in your own words and
|
||||||
|
verify each condition. If any condition is 'probably works' rather than
|
||||||
|
'verified,' ask the operator or set outcome=partial."*
|
||||||
|
|
||||||
|
### Task 2 — Target validation in `run` *(fixes F3)*
|
||||||
|
|
||||||
|
**`internal/mcp/server.go`** — In the `run` handler, before dispatching:
|
||||||
|
validate that the command prefix matches the target type.
|
||||||
|
|
||||||
|
```
|
||||||
|
pct/qm/pvesh/iptables → only host:* targets
|
||||||
|
systemctl/docker → host:* or lxc:* targets
|
||||||
|
curl/nmap/ss → any target
|
||||||
|
```
|
||||||
|
|
||||||
|
If mismatched, return a clear error: *"Cannot run `qm` on lxc:dns — `qm` is a
|
||||||
|
Proxmox host command. Use target host:hubris or host:strong."* Do not classify
|
||||||
|
or execute.
|
||||||
|
|
||||||
|
**Test:** `TestClassifyCommand_WrongTarget` → commands with host-only prefixes
|
||||||
|
on LXC targets return error without execution.
|
||||||
|
|
||||||
|
### Task 3 — Raise MCP client timeout; add async path for long-running commands *(fixes F1)*
|
||||||
|
|
||||||
|
**`cmd/nomos`** — Raise the MCP client timeout from 30s to 120s.
|
||||||
|
|
||||||
|
**`internal/mcp/server.go`** — For `run` commands that the classifier
|
||||||
|
determines will exceed the client timeout (presence of `sleep`, `wait`,
|
||||||
|
`timeout` in the command), return immediately with an `execution_id` and status
|
||||||
|
`running`. The agent already has `get_execution_status` — use it:
|
||||||
|
|
||||||
|
1. Classify the command; if it contains `sleep`, `wait`, or shell constructs
|
||||||
|
that imply polling, flag it as `async_potential`.
|
||||||
|
2. Start the command, return the `execution_id` immediately.
|
||||||
|
3. Agent polls with `get_execution_status(execution_id)`.
|
||||||
|
4. If the client timeout is hit mid-poll, the execution continues on the server
|
||||||
|
— it's not lost.
|
||||||
|
|
||||||
|
**Test:** `run` with `sleep 60; echo done` on host:hubris → returns
|
||||||
|
immediately (not 30s timeout), `get_execution_status` eventually returns
|
||||||
|
`completed`.
|
||||||
|
|
||||||
|
### Task 4 — Plan step integrity: require `replaced_reason` on replacement *(fixes F4)*
|
||||||
|
|
||||||
|
**`session_plan_steps` migration** — Add `replaced_reason TEXT` column.
|
||||||
|
|
||||||
|
**`cmd/nomos`** — When the agent emits `update_plan_step` with
|
||||||
|
`status=replaced`, require a non-empty `replaced_reason`. Valid reasons:
|
||||||
|
`wrong_diagnosis`, `scope_change`, `blocked`, `superseded`, `operator_override`.
|
||||||
|
|
||||||
|
**Coach:** `nomos/SOUL.md` — explicit rule: *"Complete (status=done) or
|
||||||
|
explicitly skip (status=skipped) steps. Use status=replaced only when the
|
||||||
|
entire plan generation is wrong; include the reason. Replacing all steps with
|
||||||
|
no reason is a session-quality violation."*
|
||||||
|
|
||||||
|
### Task 5 — Force `propose_plan` on session resume *(fixes F5)*
|
||||||
|
|
||||||
|
**`cmd/nomos`** — When a session with status `done` or `failed` receives a new
|
||||||
|
user message, reset the plan state: clear step status, require a new
|
||||||
|
`propose_plan` call before any `run` calls. The "must have plan" guard should
|
||||||
|
consider the resumed session as plan-less until a fresh `propose_plan` is
|
||||||
|
called.
|
||||||
|
|
||||||
|
**Guard:** `set_goal` + `propose_plan` must be called before any `run` in a
|
||||||
|
resumed session. Reuse the existing "No plan — call set_goal then propose_plan"
|
||||||
|
error from `internal/mcp/server.go`.
|
||||||
|
|
||||||
|
### Task 6 — Scope gate: surface `session_questions` on context switch *(fixes F6)*
|
||||||
|
|
||||||
|
**Coach:** `nomos/SOUL.md` — explicit rule: *"Before pivoting to a subsystem
|
||||||
|
not mentioned in the user's goal, ask via session_questions. Example: 'The
|
||||||
|
dashboard logs show it hasn't started since July 19. Do you want me to debug
|
||||||
|
the dashboard service itself [A], skip it and just stabilize the IP [B], or
|
||||||
|
stop here [C]?'"*
|
||||||
|
|
||||||
|
**`cmd/nomos` prompt** — Add to the system prompt: *"When the investigation
|
||||||
|
leads to a subsystem or root cause unrelated to the expressed goal, surface a
|
||||||
|
session_question before taking action."*
|
||||||
|
|
||||||
|
### Task 7 — Auto-upsert knowledge on session close *(fixes F8)*
|
||||||
|
|
||||||
|
**`cmd/nomos`** — On `complete_task` (any outcome: success, partial, failure),
|
||||||
|
auto-generate a knowledge entry:
|
||||||
|
|
||||||
|
```
|
||||||
|
title: "<date>: <goal summary>"
|
||||||
|
content: "## Outcome\n<outcome>\n## Root cause\n<extracted>\n## What was done\n<summary>\n## What was left\n<unresolved>"
|
||||||
|
tags: [session:<id>]
|
||||||
|
about: [entities involved]
|
||||||
|
```
|
||||||
|
|
||||||
|
This ensures every session leaves a trace regardless of whether the agent
|
||||||
|
remembered to call `upsert_knowledge`.
|
||||||
|
|
||||||
|
### Task 8 — Token tracking *(fixes F9)*
|
||||||
|
|
||||||
|
**`cmd/nomos`** — After each LLM call, extract `usage.prompt_tokens`,
|
||||||
|
`usage.completion_tokens`, `usage.total_tokens` from the response and write to
|
||||||
|
`agent_activity.token_count`. Currently the field exists but is never populated
|
||||||
|
(NULL for all rows).
|
||||||
|
|
||||||
|
### Task 9 — Execution linkage *(fixes F9)*
|
||||||
|
|
||||||
|
**`internal/mcp/server.go`** — When `run` creates an execution, write a row
|
||||||
|
into `nomos_plan_executions` linking `session_id`, `plan_step_seq`, and
|
||||||
|
`execution_id`.
|
||||||
|
|
||||||
|
**`internal/mcp/server.go`** — Pass `session_id` (from MCP request headers)
|
||||||
|
into `audit_log` writes. Currently `audit_log.session_id` is NULL — the
|
||||||
|
`createAuditLog` function in `internal/httpapi/impl.go` receives the
|
||||||
|
correlation_id but not the session_id from the MCP path.
|
||||||
|
|
||||||
|
### Task 10 — Plan quality metric *(fixes F9)*
|
||||||
|
|
||||||
|
**`cmd/nomos`** — At session close, compute: `completed_steps /
|
||||||
|
total_steps_per_plan` (currently ~38%). Log as a metric or write as a session
|
||||||
|
attribute. Track over time to measure plan-adherence improvements from Tasks
|
||||||
|
4+5.
|
||||||
|
|
||||||
|
### Task 11 — Auto-classify every `run` call *(fixes F8)*
|
||||||
|
|
||||||
|
**`internal/mcp/server.go`** — The `run` handler already calls the classifier
|
||||||
|
(`classifyCommand` in `internal/policy/command.go`) to determine risk_class and
|
||||||
|
approval route. Write the result to the `classifications` table. Currently the
|
||||||
|
table is empty (0 rows) despite 1,884 executions being classified.
|
||||||
|
|
||||||
|
### Task 12 — Auto-feedback on session close *(fixes F8)*
|
||||||
|
|
||||||
|
**`cmd/nomos`** — On `complete_task`, generate a `feedback` entry:
|
||||||
|
|
||||||
|
```
|
||||||
|
session_id: <id>
|
||||||
|
outcome: <outcome>
|
||||||
|
observation: <summary>
|
||||||
|
lesson: <extracted from complete_task.summary>
|
||||||
|
side_effects: <entities created/modified during session>
|
||||||
|
```
|
||||||
|
|
||||||
|
**`cmd/oikos`** — Add a daily cron or scheduler job that reads recent
|
||||||
|
`feedback` entries and extracts `patterns` (recurring root causes, same-fix
|
||||||
|
applied multiple times, known-broken services). Seed the pattern table.
|
||||||
|
|
||||||
|
### Task 13 — Command syntax validation in `run` *(fixes F7)*
|
||||||
|
|
||||||
|
**`internal/mcp/server.go`** — Before executing a `run` command, do
|
||||||
|
lightweight bash syntax validation:
|
||||||
|
|
||||||
|
```
|
||||||
|
- Reject literal \n in commands (should be ; or &&)
|
||||||
|
- Reject commands where the last line ends with \ (backslash-continuation)
|
||||||
|
but no next line
|
||||||
|
- Warn on common typos: "head - n", "grep - i", spaces before flags
|
||||||
|
- Reject `&& \n` patterns (the LLM sometimes inserts literal \n between && chains)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Task 14 — Stuck-session reaping (from prior plan; re-confirmed)
|
||||||
|
|
||||||
|
This session exhibited the same idle zombie pattern (23da10db — no closed_at,
|
||||||
|
status `failed` but outcome `failure`). Task 5 from the 2026-08-03 plan is
|
||||||
|
still open. Copying here for completeness.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Recommended sequence
|
||||||
|
|
||||||
|
```
|
||||||
|
P0 (blocks operational waste):
|
||||||
|
1 → 2 → 3
|
||||||
|
|
||||||
|
P1 (fixes plan architecture):
|
||||||
|
4 → 5
|
||||||
|
|
||||||
|
P2 (cognitive guardrails):
|
||||||
|
6 → 7 → 13
|
||||||
|
|
||||||
|
P3 (observability):
|
||||||
|
8 → 9 → 10
|
||||||
|
|
||||||
|
P4 (learning loop):
|
||||||
|
11 → 12
|
||||||
|
```
|
||||||
|
|
||||||
|
Sequence rationale: Tasks 1-3 stop the worst outcomes (premature success,
|
||||||
|
wrong-target execution, MCP timeouts). Tasks 4-5 make the plan system actually
|
||||||
|
useful instead of decorative. Tasks 6-7 add guardrails that prevent the ZimaOS
|
||||||
|
rabbit-hole class of failure. Tasks 8-10 give us visibility into whether any of
|
||||||
|
the previous tasks are working. Tasks 11-12 close the learning loop.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Validation
|
||||||
|
|
||||||
|
| Task | Test |
|
||||||
|
|------|------|
|
||||||
|
| 1 | Session with goal "make X reachable" where last ping shows 502 → `complete_task(success)` is rejected or warns |
|
||||||
|
| 2 | `run("lxc:dns", "qm stop 100")` → error: "qm is a Proxmox host command" |
|
||||||
|
| 3 | `run` with `sleep 45; echo done` → returns execution_id immediately, `get_execution_status` shows final result |
|
||||||
|
| 4 | `update_plan_step(status=replaced)` with no reason → rejected; with reason → accepted |
|
||||||
|
| 5 | Resumed session calls `run` before `propose_plan` → blocked: "No plan — call propose_plan" |
|
||||||
|
| 6 | Agent pivots to unrelated subsystem → `session_questions` is called before action |
|
||||||
|
| 7 | `complete_task` → knowledge entry created automatically with session link |
|
||||||
|
| 8 | `agent_activity.token_count` is non-NULL after any LLM call |
|
||||||
|
| 9 | `nomos_plan_executions` has rows linking session + step + execution |
|
||||||
|
| 10 | Session close writes `plan_adherence` attribute (step-completion %) |
|
||||||
|
| 11 | `classifications` table has 1 row per `run` call with risk_class + route |
|
||||||
|
| 12 | `complete_task` → auto `feedback` entry; daily pattern job finds recurring issues |
|
||||||
|
| 13 | `run` with `head - n /etc/hosts` → rejected with clear error about malformed command |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Out of scope / open questions
|
||||||
|
|
||||||
|
- Whether to raise `auto_act` from `off` for `reversible_low` actions (separate
|
||||||
|
policy decision; would reduce approval pileup without code changes).
|
||||||
|
- Whether to add a `delete_entity` MCP tool for lifecycle management (separate
|
||||||
|
from this reliability plan).
|
||||||
|
- The exact TTL for stuck-session reaping (30 min recommended, confirmed in
|
||||||
|
2026-08-03 plan).
|
||||||
|
- Whether `run` async mode should be opt-in (command contains sleep/wait) or
|
||||||
|
universal (every run returns immediately, agent always polls). Recommend
|
||||||
|
opt-in for now — most commands complete in <5s.
|
||||||
304
plans/done/2026-08-04-unified-mcp-agents.md
Normal file
304
plans/done/2026-08-04-unified-mcp-agents.md
Normal file
@@ -0,0 +1,304 @@
|
|||||||
|
# 2026-08-04 — Unified plan: external MCP agents + Nomos reliability
|
||||||
|
|
||||||
|
**Status:** Complete. All 5 stages implemented (2026-08-04).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Context
|
||||||
|
|
||||||
|
Two audiences, two gaps:
|
||||||
|
|
||||||
|
| Audience | Current state | Goal |
|
||||||
|
|---|---|---|
|
||||||
|
| **Nomos** (internal agent) | 85% session success, but 38% plan adherence, 9 timeout failures, premature success, empty learning table | Reliable, self-correcting, leaves a trace |
|
||||||
|
| **External agents** (Claude, Goose, etc.) | 40 MCP tools — heavy on observe, light on act. Can't manage signals, checks, executions, or knowledge beyond `upsert`. | Full Oikos surface: observe + act + curate |
|
||||||
|
|
||||||
|
The two plans share infrastructure (`internal/mcp/server.go`) and have cross-task
|
||||||
|
dependencies. This document merges them into one sequenced plan.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Cross-plan dependencies
|
||||||
|
|
||||||
|
Three audit tasks are **prerequisites** for external-agent mutation tools:
|
||||||
|
|
||||||
|
| Audit task | Enables | Why |
|
||||||
|
|---|---|---|
|
||||||
|
| **Task 3** — async `run` + timeout | `cancel_execution`, `list_executions` with live status | Without async `run`, every command >30s hits a client timeout. The agent loses track of the execution and can't cancel it. |
|
||||||
|
| **Task 9** — execution linkage | `list_executions` filtered by session/entity | `nomos_plan_executions` table is empty; `audit_log.session_id` is NULL. Without linkage, execution queries are blind. |
|
||||||
|
| **Task 2** — target validation in `run` | All Phase 2 mutation tools | Can't let external agents mutate state on targets that can't execute the command (audit found `qm` run on `lxc:dns`). |
|
||||||
|
|
||||||
|
Two items were **dropped** from the original MCP expansion:
|
||||||
|
|
||||||
|
- **`approve_execution` / `deny_execution`** — separation-of-duties violation. An
|
||||||
|
MCP agent approving its own queued commands breaks the approval model. The
|
||||||
|
correct fix is raising `auto_act` for `reversible_low` (policy change, zero
|
||||||
|
code — listed in the audit plan's out-of-scope). The existing assent-window
|
||||||
|
path in `run()` already auto-approves reversible-low commands.
|
||||||
|
- **Execution log streaming** — MCP has no push model. `get_execution_status`
|
||||||
|
returns the latest output; full streaming stays WebUI-only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Unified sequence (5 stages, 31 tasks)
|
||||||
|
|
||||||
|
### Stage 1: Foundation — shared infrastructure (3 tasks)
|
||||||
|
|
||||||
|
These unlock everything downstream. Do first.
|
||||||
|
|
||||||
|
**T1 — Target validation in `run`** *(audit Task 2)*
|
||||||
|
`internal/mcp/server.go`: before dispatching a `run` command, validate prefix
|
||||||
|
against target type. `qm`/`pct`/`pvesh` → `host:*` only. `systemctl`/`docker` →
|
||||||
|
`host:*` or `lxc:*`. Mismatch returns error without execution.
|
||||||
|
|
||||||
|
**T2 — Async `run` + 120s timeout** *(audit Task 3)*
|
||||||
|
`cmd/nomos`: raise MCP client timeout 30s → 120s.
|
||||||
|
`internal/mcp/server.go`: when classifier detects `sleep`/`wait`/poll loops in
|
||||||
|
the command, start execution and return `execution_id` immediately. Agent polls
|
||||||
|
with `get_execution_status`. Execution continues server-side even if client
|
||||||
|
timeout.
|
||||||
|
|
||||||
|
**T3 — Execution linkage** *(audit Task 9)*
|
||||||
|
`internal/mcp/server.go`: when `run` creates an execution, write into
|
||||||
|
`nomos_plan_executions` (session_id + plan_step_seq + execution_id). Pass
|
||||||
|
`session_id` from MCP request headers into `audit_log` writes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Stage 2: External agent observe (8 tasks)
|
||||||
|
|
||||||
|
All read-only MCP tools. Zero risk, ship fast. Unblocks external agents from
|
||||||
|
understanding system state.
|
||||||
|
|
||||||
|
**T4 — `get_dashboard_summary`**
|
||||||
|
Fleet health counts, signals by severity, pending approvals, event rate. One
|
||||||
|
call instead of 4. Wraps existing `GetDashboardSummary` DB query.
|
||||||
|
|
||||||
|
**T5 — `get_ontology`**
|
||||||
|
Entity types, relationship types, lifecycle states, monitoring specs. Agents
|
||||||
|
need this to reason about the schema.
|
||||||
|
|
||||||
|
**T6 — `list_checks`**
|
||||||
|
Per-entity health checks with verdict, last run, probe output. Filter by entity
|
||||||
|
slug or check state.
|
||||||
|
|
||||||
|
**T7 — `list_executions`**
|
||||||
|
Cursor-paginated execution history. Filter by entity slug, status, risk class.
|
||||||
|
Depends on T3 (execution linkage) for session/entity filtering.
|
||||||
|
|
||||||
|
**T8 — `get_knowledge_revisions`**
|
||||||
|
Version history for a knowledge entity. Agent can see what changed and when.
|
||||||
|
|
||||||
|
**T9 — `get_knowledge_duplicates`**
|
||||||
|
Near-duplicate knowledge entries via trigram clustering. Wraps existing
|
||||||
|
`knowledgeDuplicates` query.
|
||||||
|
|
||||||
|
**T10 — `get_knowledge_orphans`**
|
||||||
|
Knowledge entries not linked to any entity. Agent can suggest cleanup.
|
||||||
|
|
||||||
|
**T11 — `list_knowledge_tags`**
|
||||||
|
All tags with counts. Agent can see the taxonomy.
|
||||||
|
|
||||||
|
**T12 — `list_entity_sessions`** *(Phase 3 polish)*
|
||||||
|
Active Nomos sessions linked to an entity. Depends on T3 (execution linkage).
|
||||||
|
|
||||||
|
**T13 — `find_entities_by`** *(Phase 3 polish)*
|
||||||
|
Search entities by attribute (IP, port, version, tag). More flexible than
|
||||||
|
`list_entities` (type/state only).
|
||||||
|
|
||||||
|
**T14 — MCP resources**
|
||||||
|
Expose entities, knowledge entries, and executions as MCP resource templates:
|
||||||
|
`oikos://entity/{slug}`, `oikos://knowledge/{id}`, `oikos://execution/{id}`.
|
||||||
|
MCP clients that support resources (Claude Desktop, Goose) can browse and attach
|
||||||
|
them to conversations.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Stage 3: Nomos reliability (6 tasks)
|
||||||
|
|
||||||
|
Fixes the worst failure modes found in the 5-session audit.
|
||||||
|
|
||||||
|
**T15 — Prevent premature `complete_task(success)`** *(audit Task 1)*
|
||||||
|
`internal/mcp/tools.go`: when `complete_task` with `outcome=success`, verify the
|
||||||
|
summary doesn't contradict known state. If goal mentions a URL but the last
|
||||||
|
probe shows non-200, log a warning.
|
||||||
|
`nomos/SOUL.md`: explicit rule — restate goal, verify every condition before
|
||||||
|
calling success. If any condition is "probably works," use `outcome=partial`.
|
||||||
|
|
||||||
|
**T16 — Plan step integrity: require `replaced_reason`** *(audit Task 4)*
|
||||||
|
Schema: add `replaced_reason TEXT` to `session_plan_steps`.
|
||||||
|
`cmd/nomos`: when agent emits `update_plan_step(status=replaced)`, require
|
||||||
|
non-empty reason (enum: `wrong_diagnosis`, `scope_change`, `blocked`,
|
||||||
|
`superseded`, `operator_override`).
|
||||||
|
`nomos/SOUL.md`: explicit rule — complete or skip steps. Replacing all steps
|
||||||
|
with no reason is a session-quality violation.
|
||||||
|
|
||||||
|
**T17 — Force `propose_plan` on session resume** *(audit Task 5)*
|
||||||
|
`cmd/nomos`: when a session with status `done`/`failed` receives a new user
|
||||||
|
message, reset plan state. Require fresh `propose_plan` before any `run`.
|
||||||
|
The "must have plan" guard treats resumed sessions as plan-less.
|
||||||
|
|
||||||
|
**T18 — Scope gate: surface `session_questions` on context switch** *(audit Task 6)*
|
||||||
|
`cmd/nomos` system prompt: "When investigation leads to a subsystem unrelated to
|
||||||
|
the expressed goal, call `session_questions` before taking action."
|
||||||
|
`nomos/SOUL.md`: explicit rule — ask before pivoting.
|
||||||
|
|
||||||
|
**T19 — Command syntax validation in `run`** *(audit Task 13)*
|
||||||
|
`internal/mcp/server.go`: before executing, reject literal `\n` in commands,
|
||||||
|
backslash-continuation on last line, `head - n`/`grep - i` space-before-flag
|
||||||
|
typos, and `&& \n` patterns from LLM formatting errors.
|
||||||
|
|
||||||
|
**T20 — Stuck-session reaping** *(audit Task 14, from 2026-08-03 plan)*
|
||||||
|
Reap sessions with `closed_at IS NULL` and no message in 30 minutes. Set
|
||||||
|
status=failed, outcome=failure.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Stage 4: External agent act (9 tasks)
|
||||||
|
|
||||||
|
Mutation MCP tools. Each writes audit log + emits event. Requires Stage 1
|
||||||
|
infrastructure (T1 target validation, T2 async run, T3 execution linkage).
|
||||||
|
Follows existing direct-DB patterns — no HTTP API calls.
|
||||||
|
|
||||||
|
**T21 — `ack_signal(signal_id)`**
|
||||||
|
Acknowledge an open signal. Agent investigating an alert marks it acknowledged.
|
||||||
|
|
||||||
|
**T22 — `resolve_signal(signal_id, resolution?)`**
|
||||||
|
Resolve a signal with optional resolution note.
|
||||||
|
|
||||||
|
**T23 — `mute_signal(signal_id, duration?)`**
|
||||||
|
Temporarily mute a signal. Optional duration (default 1h).
|
||||||
|
|
||||||
|
**T24 — `cancel_execution(execution_id, reason)`**
|
||||||
|
Cancel a queued/running execution. Depends on T2 (async `run` returns
|
||||||
|
`execution_id`) and T3 (execution linkage for audit context).
|
||||||
|
|
||||||
|
**T25 — `update_check(check_id, enabled)`**
|
||||||
|
Enable/disable a health check. Agent suppresses a noisy probe.
|
||||||
|
|
||||||
|
**T26 — `delete_knowledge(knowledge_id)`**
|
||||||
|
Soft-delete a knowledge entry (move to trash, restorable).
|
||||||
|
|
||||||
|
**T27 — `restore_knowledge(knowledge_id)`**
|
||||||
|
Restore a trashed knowledge entry.
|
||||||
|
|
||||||
|
**T28 — `merge_knowledge(source_id, target_id)`**
|
||||||
|
Fold one knowledge entry into another. Source gets soft-deleted, content
|
||||||
|
appended to target.
|
||||||
|
|
||||||
|
**T29 — `rename_knowledge_tag(old_name, new_name)`**
|
||||||
|
Bulk-rename a tag across all knowledge entries.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Stage 5: Close the learning loop (5 tasks)
|
||||||
|
|
||||||
|
Turn execution data into persistent knowledge. Currently all learning tables are
|
||||||
|
empty (0 classifications, 0 feedback, 0 patterns, 0 skills).
|
||||||
|
|
||||||
|
**T30 — Auto-classify every `run` → `classifications` table** *(audit Task 11)*
|
||||||
|
`internal/mcp/server.go`: `run` already calls `classifyCommand`. Write the
|
||||||
|
result to the `classifications` table (risk_class + route + patterns matched).
|
||||||
|
Currently 0 rows despite 1,884 executions.
|
||||||
|
|
||||||
|
**T31 — Auto-upsert knowledge on session close** *(audit Task 7)*
|
||||||
|
`cmd/nomos`: on `complete_task` (any outcome), auto-generate a knowledge entry:
|
||||||
|
title=`<date>: <goal>`, content with Outcome/Root cause/What was done/Unresolved
|
||||||
|
sections, tags=`[session:<id>]`, linked to involved entities.
|
||||||
|
|
||||||
|
**T32 — Auto-feedback on session close** *(audit Task 12)*
|
||||||
|
`cmd/nomos`: on `complete_task`, generate a `feedback` entry: session_id,
|
||||||
|
outcome, observation, lesson, side_effects. Daily cron job reads recent feedback
|
||||||
|
and extracts patterns (recurring root causes, same-fix-applied-multiple-times).
|
||||||
|
|
||||||
|
**T33 — Token tracking** *(audit Task 8)*
|
||||||
|
`cmd/nomos`: after each LLM call, extract `usage.total_tokens` from the response
|
||||||
|
and write to `agent_activity.token_count`. Currently NULL for all rows.
|
||||||
|
|
||||||
|
**T34 — Plan quality metric** *(audit Task 10)*
|
||||||
|
`cmd/nomos`: at session close, compute `completed_steps / total_steps` (currently
|
||||||
|
~38%). Write as session attribute. Track over time to measure impact of T16+T17.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Validation
|
||||||
|
|
||||||
|
| Task | Test |
|
||||||
|
|---|---|
|
||||||
|
| T1 | `run("lxc:dns", "qm stop 100")` → error: "qm is a Proxmox host command" |
|
||||||
|
| T2 | `run` with `sleep 45; echo done` → returns `execution_id` immediately; `get_execution_status` eventually shows completed |
|
||||||
|
| T3 | After `run`, `nomos_plan_executions` has row linking session + step + execution |
|
||||||
|
| T4 | `get_dashboard_summary()` returns health counts, signal counts, approval count in one call |
|
||||||
|
| T5 | `get_ontology()` returns entity_types, relationship_types, lifecycle_states |
|
||||||
|
| T6 | `list_checks(entity_slug="lxc:jellyfin")` returns all checks with verdict + last run |
|
||||||
|
| T7 | `list_executions(entity_slug="host:hubris", limit=10)` returns cursor-paginated list |
|
||||||
|
| T8 | `get_knowledge_revisions(id)` returns ordered revision list with timestamps |
|
||||||
|
| T9 | `get_knowledge_duplicates()` returns clusters with similarity scores |
|
||||||
|
| T10 | `get_knowledge_orphans()` returns knowledge entries with zero entity links |
|
||||||
|
| T11 | `list_knowledge_tags()` returns {name, count} for all tags |
|
||||||
|
| T12 | `list_entity_sessions("lxc:jellyfin")` returns active sessions with goal + status |
|
||||||
|
| T13 | `find_entities_by(ip="10.0.0.5")` returns matching entities |
|
||||||
|
| T14 | MCP client can browse `oikos://entity/*` resources |
|
||||||
|
| T15 | Session with goal "make X reachable" where last ping shows 502 → `complete_task(success)` warns or rejects |
|
||||||
|
| T16 | `update_plan_step(status=replaced)` with no reason → rejected |
|
||||||
|
| T17 | Resumed session calls `run` before `propose_plan` → blocked |
|
||||||
|
| T18 | Agent pivots to unrelated subsystem → `session_questions` is called |
|
||||||
|
| T19 | `run` with `head - n /etc/hosts` → rejected with syntax error |
|
||||||
|
| T20 | Session idle for 30+ min with no `closed_at` → reaped (status=failed) |
|
||||||
|
| T21 | `ack_signal(id)` → signal status transitions to acknowledged, audit logged |
|
||||||
|
| T22 | `resolve_signal(id, "fixed DNS")` → resolved with note |
|
||||||
|
| T23 | `mute_signal(id, 3600)` → muted for 1 hour, auto-unmutes |
|
||||||
|
| T24 | `cancel_execution(id, "wrong target")` → execution cancelled, audit logged |
|
||||||
|
| T25 | `update_check(id, false)` → check disabled, scheduler stops probing |
|
||||||
|
| T26 | `delete_knowledge(id)` → soft-deleted (trashed), restorable |
|
||||||
|
| T27 | `restore_knowledge(id)` → restored from trash, reappears in list |
|
||||||
|
| T28 | `merge_knowledge(src, dst)` → src deleted, content appended to dst |
|
||||||
|
| T29 | `rename_knowledge_tag("old", "new")` → all entries updated |
|
||||||
|
| T30 | After any `run`, `classifications` has row with risk_class + route |
|
||||||
|
| T31 | `complete_task` → knowledge entry created automatically with session link |
|
||||||
|
| T32 | `complete_task` → feedback entry created; daily job extracts pattern if same root cause appears ≥3 times |
|
||||||
|
| T33 | `agent_activity.token_count` is non-NULL after LLM call |
|
||||||
|
| T34 | Session close writes `plan_adherence` attribute (% steps completed) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Files touched
|
||||||
|
|
||||||
|
| File | Tasks |
|
||||||
|
|---|---|
|
||||||
|
| `internal/mcp/server.go` | T1, T2, T3, T19, T24, T30 |
|
||||||
|
| `internal/mcp/tools.go` | T4–T14, T21–T29 |
|
||||||
|
| `internal/mcp/discover.go` | (no changes — references for T10/T11 patterns) |
|
||||||
|
| `cmd/nomos/main.go` (or config) | T2 (timeout), T16, T17, T18, T31, T32, T33, T34 |
|
||||||
|
| `nomos/SOUL.md` | T15, T16, T18 |
|
||||||
|
| `internal/httpapi/` | T3 (audit_log.session_id plumbing) |
|
||||||
|
| DB migrations | T16 (replaced_reason column) |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. What stays WebUI-only
|
||||||
|
|
||||||
|
| Feature | Reason |
|
||||||
|
|---|---|
|
||||||
|
| FleetMap visual graph | Canvas rendering — not an MCP concern |
|
||||||
|
| uPlot metric charts | Raw data available via `query_metrics`/`get_trend` |
|
||||||
|
| Desktop shell, Cluck, App Store | Pure UI layer |
|
||||||
|
| SSE event streaming | MCP has no push model; polling covers it |
|
||||||
|
| Execution log streaming | `get_execution_status` returns latest output |
|
||||||
|
| Nomos session chat | MCP is a tool interface, not a chat agent |
|
||||||
|
| Knowledge wiki editor (revision browse, cleanup UI) | MCP tools expose the data + mutations; UI provides the editing experience |
|
||||||
|
| Approval queue with Approve/Deny buttons | Approvals stay operator-gated via WebUI/Matrix |
|
||||||
|
| Client enrollment flow | Enrollment is IP-gated, not an MCP tool |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Out of scope
|
||||||
|
|
||||||
|
- **`approve_execution` / `deny_execution` MCP tools** — dropped. The fix is raising
|
||||||
|
`auto_act` for `reversible_low` (policy change, no code).
|
||||||
|
- **`delete_entity` MCP tool** — separate lifecycle management concern.
|
||||||
|
- **Per-client bearer tokens** — open item tracked in CLIENTS.md. Until they
|
||||||
|
exist, external agents share the same `OIKOS_MCP_BEARER_TOKEN`.
|
||||||
|
- **Exactly-once pattern extraction from feedback** — T32 seeds the pipeline;
|
||||||
|
the full pattern-mining algorithm (TF-IDF clustering, causal inference from
|
||||||
|
event timelines) is future work.
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
# 2026-08-05 — Agent execution safety: QEMU guest agent guardrails + host-mutation gate
|
||||||
|
|
||||||
|
**Status:** Plan.
|
||||||
|
**Context:** ZimaOS NFS recovery session (2026-08-04/05) surfaced three systemic
|
||||||
|
failures in how agents drive oikos mutations. The `run` tool queued an execution
|
||||||
|
against a VM whose QEMU guest agent was down — it sat `pending_approval` forever,
|
||||||
|
never executed, and the agent silently fell back to raw SSH. That same raw-SSH
|
||||||
|
fallback was then used to `apt-get install nfs-kernel-server` directly on the
|
||||||
|
hubris **PVE host**, crashing it and taking the whole homelab subnet down for
|
||||||
|
~15 minutes.
|
||||||
|
|
||||||
|
**Trigger:** Incident `investigation:nomos/incident-hubris-crash-from-nfs-kernel-server-on-pve-host-2026-08-05`
|
||||||
|
(2026-08-05) + session audit. The crash was caused by an agent bypassing the
|
||||||
|
`run` approval gate, which exists precisely to catch that kind of mistake.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Motivation
|
||||||
|
|
||||||
|
The OODA loop's Act phase is the security boundary (ADR 0012: "Hermes has no
|
||||||
|
direct SSH access... all mutations go through the execution queue"). This session
|
||||||
|
proved the boundary has three leaks:
|
||||||
|
|
||||||
|
1. **`run` on a VM with a dead QEMU guest agent queues silently.** The execution
|
||||||
|
is classified `config_mutation`, queued for approval, and *never fails* — it
|
||||||
|
just sits in `pending_approval` while the agent assumes progress. There is no
|
||||||
|
feedback that the underlying transport (`qm guest exec`) cannot work.
|
||||||
|
|
||||||
|
2. **No guardrail against host-level package/kernel mutations.** `apt-get install`
|
||||||
|
targeting a `proxmox-host` entity is classified `config_mutation` and gated —
|
||||||
|
*if* the agent routes it through `run`. When the first `run` call stalls
|
||||||
|
(leak #1), the agent falls back to raw SSH, which has no classification at all.
|
||||||
|
The crash was the direct result of that fallback.
|
||||||
|
|
||||||
|
3. **Agents are trusted to self-report the `health` attribute.** `update_entity_attributes`
|
||||||
|
let the agent set `health:"healthy"` on `lxc:nfs-export`, which derived 4
|
||||||
|
spurious health checks. Health is scheduler-owned; agents shouldn't write it.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Changes
|
||||||
|
|
||||||
|
### I — `run` pre-flights the execution transport before queueing
|
||||||
|
|
||||||
|
**Why:** A queued execution that can never run is worse than a failed one — it
|
||||||
|
looks like progress, stalls the agent, and (this session) pushed the agent into
|
||||||
|
the unsafe raw-SSH fallback.
|
||||||
|
|
||||||
|
**What:**
|
||||||
|
|
||||||
|
In the `run` tool handler (`internal/mcp/`), before inserting the execution row:
|
||||||
|
|
||||||
|
- If target type is `vm`, read the target entity's attributes. If
|
||||||
|
`qemu_guest_agent` is missing or `not_running`, return an immediate error:
|
||||||
|
`"run on vm:zimaos blocked: QEMU guest agent is not running (qm guest exec
|
||||||
|
unavailable). Start the agent first or use a different target."`
|
||||||
|
- Same check for `lxc` targets whose `pct exec` path is known-broken (optional —
|
||||||
|
start with VM only).
|
||||||
|
|
||||||
|
This converts "queued forever" into a fast, actionable failure the agent can
|
||||||
|
recover from immediately.
|
||||||
|
|
||||||
|
**Risk class:** read-only (validation only, no execution row created).
|
||||||
|
|
||||||
|
**Test:** Unit test with a fake VM entity that has `qemu_guest_agent: not_running`
|
||||||
|
→ assert the tool returns the blocking error and inserts no execution row.
|
||||||
|
|
||||||
|
### II — Host-mutation command guardrail in `run` classification
|
||||||
|
|
||||||
|
**Why:** `apt-get install` on a Proxmox host is the exact class of mutation that
|
||||||
|
must always hit the approval gate. The classifier already escalates `apt`/kernel
|
||||||
|
touches; this makes the escalation explicit and documented so agents stop
|
||||||
|
second-guessing it.
|
||||||
|
|
||||||
|
**What:**
|
||||||
|
|
||||||
|
- In `seeds/policy.yaml`, add an explicit rule: `proxmox-host` targets + commands
|
||||||
|
matching `(apt-get install|apt install|dpkg|modprobe|kernel)` ⇒ `config_mutation`
|
||||||
|
(operator approval required), never `reversible_low`.
|
||||||
|
- Extend the classifier to also flag `update-rc.d`, `systemctl enable` on host
|
||||||
|
targets if not already covered.
|
||||||
|
- Add a note in the `run` tool description: "Host-level package/kernel mutations
|
||||||
|
always require operator approval."
|
||||||
|
|
||||||
|
**Risk class:** policy change — knowledge/DB, deploy via seed ingest.
|
||||||
|
|
||||||
|
**Test:** `classify_command("apt-get install -y nfs-kernel-server", declared_risk=read_only)`
|
||||||
|
on `host:hubris` → must return `config_mutation`, not read_only. Add a fixture test
|
||||||
|
in the classifier suite.
|
||||||
|
|
||||||
|
### III — `health` attribute is read-only for agents
|
||||||
|
|
||||||
|
**Why:** This session's `update_entity_attributes({"health":"healthy"})` on
|
||||||
|
`lxc:nfs-export` derived 4 checks. Health is computed by the scheduler from probe
|
||||||
|
results; an agent asserting it creates false monitoring.
|
||||||
|
|
||||||
|
**What:**
|
||||||
|
|
||||||
|
- In `update_entity_attributes` handler: reject (or strip with a warning) the
|
||||||
|
`health` key. Return a message: `"health is scheduler-owned; attribute ignored.
|
||||||
|
Use get_health_summary/list_checks to observe it."`
|
||||||
|
- Document in the tool description: `"Do not write health — it is derived from
|
||||||
|
probes."`
|
||||||
|
|
||||||
|
**Risk class:** knowledge-graph mutation (existing), no infra impact.
|
||||||
|
|
||||||
|
**Test:** `update_entity_attributes(slug=lxc:nfs-export, attributes={"health":"healthy"})`
|
||||||
|
→ response shows health ignored, other keys merged.
|
||||||
|
|
||||||
|
### IV — Agent-side: record discovered dependency edges
|
||||||
|
|
||||||
|
**Why:** The session discovered `vm:zimaos` depends on `lxc:nfs-export` for
|
||||||
|
`/media/library`, but no `depends-on` edge was recorded. A future agent
|
||||||
|
investigating a ZimaOS mount failure would have no graph signal pointing at the
|
||||||
|
NFS server.
|
||||||
|
|
||||||
|
**What (agent behaviour, not code):** After confirming a runtime dependency, call
|
||||||
|
`create_relationship(source, target, type)` immediately. Concretely this session:
|
||||||
|
`create_relationship("vm:zimaos", "lxc:nfs-export", "depends-on")`.
|
||||||
|
|
||||||
|
**Where to enforce:** Update the homelab-context `HERMES.md` / SOUL.md agent
|
||||||
|
instructions with a one-line rule: "When you discover a dependency between two
|
||||||
|
entities (a service consumes another's export/mount/API), record it with
|
||||||
|
`create_relationship` in the same session." Plus a runbook in
|
||||||
|
`.agents/skills/` if one doesn't exist.
|
||||||
|
|
||||||
|
**Risk class:** knowledge-graph mutation, auto-approves.
|
||||||
|
|
||||||
|
**Test:** Manual — after recording the edge, `get_relations("vm:zimaos")` shows
|
||||||
|
`depends-on → lxc:nfs-export`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Rollout
|
||||||
|
|
||||||
|
| Step | Item | When |
|
||||||
|
| ---- | ---- | ---- |
|
||||||
|
| 1 | II — policy.yaml classifier rule + tests | next seed ingest |
|
||||||
|
| 2 | I — `run` VM transport pre-flight + test | next mcp server deploy |
|
||||||
|
| 3 | III — health read-only guard + test | same deploy as I |
|
||||||
|
| 4 | IV — agent instruction update in homelab-context | commit + sync |
|
||||||
|
| 5 | Verify: re-run `classify_command` + manual `run` on vm:zimaos (agent now up) | after deploy |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Out of scope
|
||||||
|
|
||||||
|
- Per-client MCP bearer tokens (separate track, ADR 0012 note).
|
||||||
|
- `request_execution` re-introduction — the unified `run` primitive stays.
|
||||||
|
- Automating the Technitium DHCP reservation UI (this session's leftover — the
|
||||||
|
reservation for `BC:24:11:22:C2:F2 → 192.168.8.102` was added manually in the
|
||||||
|
web UI; consider a `runbook:technitium-dhcp-reservation` doc in a follow-up).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Changelog
|
||||||
|
|
||||||
|
- 2026-08-05 — plan created from ZimaOS/NFS session audit + hubris crash incident.
|
||||||
@@ -17,11 +17,10 @@ went sideways, open an investigation.
|
|||||||
| 2026-07-14 | [Activity gaps](2026-07-14-activity-gaps.md) | In Progress |
|
| 2026-07-14 | [Activity gaps](2026-07-14-activity-gaps.md) | In Progress |
|
||||||
| 2026-07-14 | [Activity timeline](2026-07-14-activity-timeline.md) | In Progress |
|
| 2026-07-14 | [Activity timeline](2026-07-14-activity-timeline.md) | In Progress |
|
||||||
| 2026-07-17 | [Codebase review, lint audit, and documentation maintenance](2026-07-17-codebase-review-and-cleanup.md) | Report delivered — doc/tooling fixes applied; code refactors pending |
|
| 2026-07-17 | [Codebase review, lint audit, and documentation maintenance](2026-07-17-codebase-review-and-cleanup.md) | Report delivered — doc/tooling fixes applied; code refactors pending |
|
||||||
| 2026-07-18 | [Session review: three recent sessions](2026-07-18-session-review-three-sessions.md) | Implemented in v0.7.12 — P0.1/P0.2/P1.3/P1.4/P1.5/P1.6/P1.8/P2.10; P1.7 and P2.9 deferred (retry cap covers) |
|
|
||||||
| 2026-07-20 | [Desktop mascot ("Cluck")](2026-07-20-desktop-mascot.md) | Implemented in v0.8.0 — see deviation note; physics/window-interaction follow-ups tracked separately |
|
|
||||||
| 2026-07-20 | [Session review: past 10 sessions](2026-07-20-session-review-ten-sessions.md) | Implemented in v0.7.13 — all P0/P1/P2 items landed |
|
|
||||||
| 2026-07-20 | [Mascot physics/window-interaction audit](2026-07-20-mascot-physics-audit.md) | P0–P2 implemented; P3 ("cool stuff") ideas open |
|
| 2026-07-20 | [Mascot physics/window-interaction audit](2026-07-20-mascot-physics-audit.md) | P0–P2 implemented; P3 ("cool stuff") ideas open |
|
||||||
| 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready |
|
| 2026-07-21 | [Frontend as OS + Apps — architecture audit & refactor](2026-07-21-frontend-os-apps-architecture.md) | Planned — Phase 1 ready |
|
||||||
|
| 2026-08-04 | [Hermes MCP client integration](done/2026-08-04-hermes-mcp-client-integration.md) | Done — deployed |
|
||||||
|
| 2026-08-05 | [Agent execution safety: QEMU guest agent gate + host-mutation guard](done/2026-08-05-agent-execution-safety-qemu-guest-agent-gate.md) | Done — implemented (1b9c761) |
|
||||||
|
|
||||||
## Done
|
## Done
|
||||||
|
|
||||||
@@ -59,6 +58,17 @@ See [`done/`](done/) for executed plans:
|
|||||||
| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](done/2026-07-14-post-fix-session-remainders.md) |
|
| 2026-07-14 | [Post-fix session remainders: empty responses & plan drift](done/2026-07-14-post-fix-session-remainders.md) |
|
||||||
| 2026-07-15 | [Plan-first and iteration](done/2026-07-15-plan-first-and-iteration.md) |
|
| 2026-07-15 | [Plan-first and iteration](done/2026-07-15-plan-first-and-iteration.md) |
|
||||||
| 2026-07-15 | [WhatsApp session audit](done/2026-07-15-whatsapp-session-audit.md) |
|
| 2026-07-15 | [WhatsApp session audit](done/2026-07-15-whatsapp-session-audit.md) |
|
||||||
|
| 2026-07-18 | [Session review: three recent sessions](done/2026-07-18-session-review-three-sessions.md) |
|
||||||
|
| 2026-07-20 | [Desktop mascot ("Cluck")](done/2026-07-20-desktop-mascot.md) |
|
||||||
|
| 2026-07-20 | [Session review: past 10 sessions](done/2026-07-20-session-review-ten-sessions.md) |
|
||||||
|
| 2026-07-21 | [Chat window full polish](done/2026-07-21-chat-full-polish.md) |
|
||||||
|
| 2026-07-29 | [Make health reflect reality + complete the knowledge graph](done/2026-07-29-health-check-reality-and-knowledge-graph.md) |
|
||||||
|
| 2026-07-30 | [Session review: plan drift & dead activity panel](done/2026-07-30-session-review-plan-drift-and-dead-activity-panel.md) |
|
||||||
|
| 2026-08-03 | [Nomos chat changes review (P0/P1/P2)](done/2026-08-03-nomos-chat-changes-review.md) |
|
||||||
|
| 2026-08-03 | [Nomos chat: reliability & predictability audit](done/2026-08-03-nomos-chat-reliability-and-ux-audit.md) |
|
||||||
|
| 2026-08-03 | [Adopt cyberspace.online terminal aesthetic + dithered images](done/2026-08-03-cyberspace-style-adoption.md) |
|
||||||
|
| 2026-08-03 | [Nomos chat: working-visibility, message queue, generation-aware timeline](done/2026-08-03-nomos-chat-working-visibility.md) |
|
||||||
|
| 2026-08-04 | [Chat interaction overhaul: inline progressive stream + thinking blocks](done/2026-08-04-chat-window-overhaul.md) |
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
|
|||||||
70
scripts/cleanup-orphan-checks.sh
Executable file
70
scripts/cleanup-orphan-checks.sh
Executable file
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# cleanup-orphan-checks.sh — remove orphan check entities + check_defs.
|
||||||
|
#
|
||||||
|
# These are leftovers from the old shortSlug() collision bug: check entities
|
||||||
|
# with truncated 8-hex slugs (e.g. check:ssh-script:0d31fdd1) that have no
|
||||||
|
# live target and are disabled. They pollute the entity table and the checks
|
||||||
|
# view. The audit_knowledge_graph tool reports them as `orphan_checks`.
|
||||||
|
#
|
||||||
|
# Risk class: config_mutation (deletes rows). DRY-RUN by default; pass --apply
|
||||||
|
# to actually delete. Review the listed slugs first — they must all match the
|
||||||
|
# legacy random-slug pattern and be disabled.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# cleanup-orphan-checks.sh # dry-run: list what would be deleted
|
||||||
|
# cleanup-orphan-checks.sh --apply # delete check_defs rows, then entities
|
||||||
|
#
|
||||||
|
# Connects via the OIKOS_TEST... no — via the running postgres container by
|
||||||
|
# default, or OIKOS_PSQL if set.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
|
||||||
|
PATTERN='^check:(ping|ssh-script|disk):[0-9a-f]{8}$'
|
||||||
|
|
||||||
|
# Orphan = matches the legacy random-slug pattern AND has no enabled check_def
|
||||||
|
# pointing at a real target. A random-slug check that IS enabled and has a live
|
||||||
|
# target is a working check with a bad slug — keep it (deleting would drop
|
||||||
|
# monitoring), and flag it for a slug fix instead.
|
||||||
|
ORPHAN_PRED="e.type='check' AND e.slug ~ '$PATTERN'
|
||||||
|
AND NOT EXISTS (SELECT 1 FROM check_defs cd
|
||||||
|
WHERE cd.entity_id = e.id AND cd.enabled AND cd.target_id IS NOT NULL)"
|
||||||
|
|
||||||
|
echo "== orphan checks matching /$PATTERN/ (no enabled check_def w/ target) =="
|
||||||
|
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
|
||||||
|
|
||||||
|
echo "== details (slug, state, enabled) =="
|
||||||
|
$PSQL_CMD -F ' | ' -Ac "
|
||||||
|
SELECT e.slug, COALESCE(e.state,'(null)'),
|
||||||
|
COALESCE((SELECT cd.enabled::text FROM check_defs cd WHERE cd.entity_id=e.id LIMIT 1),'no-check_def')
|
||||||
|
FROM entities e
|
||||||
|
WHERE $ORPHAN_PRED
|
||||||
|
ORDER BY e.slug;" | head -60
|
||||||
|
|
||||||
|
if [ "${1:-}" != "--apply" ]; then
|
||||||
|
echo
|
||||||
|
echo "DRY RUN — no rows deleted. Re-run with --apply to delete:"
|
||||||
|
echo " check_defs whose check entity is an orphan, then those entities."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "== applying (config_mutation) =="
|
||||||
|
# check_defs first, then the dependent rows an entity owns (status, signals,
|
||||||
|
# metrics), then the orphan check entities. FKs prevent a plain entity delete.
|
||||||
|
$PSQL_CMD -v ON_ERROR_STOP=1 <<SQL
|
||||||
|
BEGIN;
|
||||||
|
DELETE FROM check_defs WHERE entity_id IN (SELECT id FROM entities e WHERE $ORPHAN_PRED);
|
||||||
|
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||||
|
DELETE FROM metric_samples WHERE entity_id IN (SELECT id FROM ids);
|
||||||
|
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||||
|
DELETE FROM signals WHERE entity_id IN (SELECT id FROM ids) OR target_entity_id IN (SELECT id FROM ids);
|
||||||
|
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||||
|
DELETE FROM entity_status WHERE entity_id IN (SELECT id FROM ids);
|
||||||
|
WITH ids AS (SELECT id FROM entities e WHERE $ORPHAN_PRED)
|
||||||
|
DELETE FROM relationships WHERE source_id IN (SELECT id FROM ids) OR target_id IN (SELECT id FROM ids);
|
||||||
|
DELETE FROM entities e WHERE $ORPHAN_PRED;
|
||||||
|
COMMIT;
|
||||||
|
SQL
|
||||||
|
|
||||||
|
echo "== remaining orphans (should be 0) =="
|
||||||
|
$PSQL_CMD -tAc "SELECT count(*) FROM entities e WHERE $ORPHAN_PRED;"
|
||||||
79
scripts/fix-zimaos-nfs.sh
Normal file
79
scripts/fix-zimaos-nfs.sh
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Fix NFS mounts on ZimaOS — both library shares
|
||||||
|
# Run: bash /tmp/fix-zimaos-nfs.sh
|
||||||
|
# You'll be prompted for sudo password once
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "=== Step 1: Grant passwordless sudo for mount/umount ==="
|
||||||
|
echo "dtoro ALL=(ALL) NOPASSWD: /usr/sbin/mount.nfs, /usr/sbin/umount.nfs, /bin/mount, /bin/umount, /usr/bin/mount" | sudo tee /etc/sudoers.d/zimaos-nfs > /dev/null
|
||||||
|
sudo chmod 440 /etc/sudoers.d/zimaos-nfs
|
||||||
|
echo " ✓ sudoers drop-in created"
|
||||||
|
|
||||||
|
echo "=== Step 2: Mount library from hubris (nfs-export LXC) ==="
|
||||||
|
sudo mkdir -p /media/library
|
||||||
|
sudo mount -t nfs -o nfsvers=4,rw,hard,intr 192.168.8.200:/mnt/library /media/library
|
||||||
|
echo " ✓ /media/library ← 192.168.8.200:/mnt/library"
|
||||||
|
|
||||||
|
echo "=== Step 3: Mount ludo-library from strong ==="
|
||||||
|
sudo mkdir -p /media/ludo-library
|
||||||
|
sudo mount -t nfs -o nfsvers=4,rw,hard,intr 192.168.8.241:/mnt/media_local /media/ludo-library
|
||||||
|
echo " ✓ /media/ludo-library ← 192.168.8.241:/mnt/media_local"
|
||||||
|
|
||||||
|
echo "=== Step 4: Verify ==="
|
||||||
|
echo ""
|
||||||
|
df -h | grep -E 'nfs|192.168'
|
||||||
|
echo ""
|
||||||
|
echo "--- /media/library contents ---"
|
||||||
|
ls /media/library/ | head -10
|
||||||
|
echo ""
|
||||||
|
echo "--- /media/ludo-library contents ---"
|
||||||
|
ls /media/ludo-library/ | head -10
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== Step 5: Persistent systemd mount units ==="
|
||||||
|
|
||||||
|
# Library mount unit
|
||||||
|
sudo tee /etc/systemd/system/media-library.mount > /dev/null << 'MOUNTUNIT'
|
||||||
|
[Unit]
|
||||||
|
Description=Mount nfs-export:/mnt/library as library
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Mount]
|
||||||
|
What=192.168.8.200:/mnt/library
|
||||||
|
Where=/media/library
|
||||||
|
Type=nfs
|
||||||
|
Options=nfsvers=4,rw,hard,intr
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
MOUNTUNIT
|
||||||
|
|
||||||
|
# Ludo-library mount unit
|
||||||
|
sudo tee /etc/systemd/system/media-ludo\x2dlibrary.mount > /dev/null << 'MOUNTUNIT2'
|
||||||
|
[Unit]
|
||||||
|
Description=Mount strong:/mnt/media_local as ludo-library
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Mount]
|
||||||
|
What=192.168.8.241:/mnt/media_local
|
||||||
|
Where=/media/ludo-library
|
||||||
|
Type=nfs
|
||||||
|
Options=nfsvers=4,rw,hard,intr
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
MOUNTUNIT2
|
||||||
|
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable media-library.mount
|
||||||
|
sudo systemctl enable media-ludo\x2dlibrary.mount
|
||||||
|
echo " ✓ Both systemd mount units created and enabled"
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "=== DONE ==="
|
||||||
|
echo "Both NFS mounts active and persistent across reboots:"
|
||||||
|
echo " /media/library ← 192.168.8.200:/mnt/library (hubris nfs-export LXC)"
|
||||||
|
echo " /media/ludo-library ← 192.168.8.241:/mnt/media_local (strong)"
|
||||||
31
scripts/report-stray-test-lxcs.sh
Executable file
31
scripts/report-stray-test-lxcs.sh
Executable file
@@ -0,0 +1,31 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# report-stray-test-lxcs.sh — list leftover test/scratch LXC entities.
|
||||||
|
#
|
||||||
|
# Provisioning experiments leave active `lxc:test-*` / `lxc:preflight-*`
|
||||||
|
# entities in the graph long after the containers are gone or repurposed.
|
||||||
|
# They generate checks and pollute health/graph views. This reports them and
|
||||||
|
# their DB state + which Proxmox host each is parented on, so the operator can
|
||||||
|
# confirm the container is really gone and retire the entity via the
|
||||||
|
# lifecycle-destroy-node runbook (a destructive, approval-gated action).
|
||||||
|
#
|
||||||
|
# Read-only. Pair with: lifecycle-destroy-node (mark destroyed) or
|
||||||
|
# lifecycle-deprecate-node.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
PSQL_CMD="${OIKOS_PSQL:-docker exec -i oikos-postgres-1 psql -U oikos -d oikos}"
|
||||||
|
|
||||||
|
echo "== stray test/scratch LXC entities =="
|
||||||
|
$PSQL_CMD -F ' | ' -Ac "
|
||||||
|
SELECT e.slug, COALESCE(e.state,'active') AS state,
|
||||||
|
e.attributes->>'pve_id' AS pve_id,
|
||||||
|
COALESCE(h.slug,'(no host)') AS host
|
||||||
|
FROM entities e
|
||||||
|
LEFT JOIN relationships r ON r.target_id = e.id AND r.type='hosts' AND r.valid_to IS NULL
|
||||||
|
LEFT JOIN entities h ON h.id = r.source_id
|
||||||
|
WHERE e.type='lxc' AND e.slug ~ '^lxc:(test|preflight)'
|
||||||
|
ORDER BY e.slug;"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "Next: for each, confirm the container is gone in Proxmox (pct list on its"
|
||||||
|
echo "host), then retire via lifecycle-destroy-node (destructive) or mark"
|
||||||
|
echo "deprecated. If a container still exists, pct destroy it first."
|
||||||
@@ -87,6 +87,12 @@ entities:
|
|||||||
mesh: {netbird: {ip: 100.122.165.149, fqdn: netbird-ionos.netbird.selfhosted}}
|
mesh: {netbird: {ip: 100.122.165.149, fqdn: netbird-ionos.netbird.selfhosted}}
|
||||||
ssh: {user: root}
|
ssh: {user: root}
|
||||||
note: netbird mgmt+signal+relay+dashboard + coturn; sshd locked to hubris pubkey
|
note: netbird mgmt+signal+relay+dashboard + coturn; sshd locked to hubris pubkey
|
||||||
|
monitoring: [http] # public HTTPS probe via https://mcp.hubris.network
|
||||||
|
# (2026-08-05: was `none` — the VPS went silent for
|
||||||
|
# 7 days because nothing probed it. The standalone-server
|
||||||
|
# type inherits [ping,resource,updates] from machine, but
|
||||||
|
# SSH/ICMP don't reach it from the lab; an HTTP probe on
|
||||||
|
# the public endpoint is the reachable liveness signal).
|
||||||
- slug: "ws:mac-mini"
|
- slug: "ws:mac-mini"
|
||||||
type: workstation
|
type: workstation
|
||||||
name: mac-mini
|
name: mac-mini
|
||||||
@@ -238,8 +244,8 @@ entities:
|
|||||||
doc_page: knowledge/wiki/containers/101-jellyfin.md,
|
doc_page: knowledge/wiki/containers/101-jellyfin.md,
|
||||||
risk_notes: "native Authentik OIDC (no forward-auth gate); VAAPI depends on GPU passthrough on strong"}}
|
risk_notes: "native Authentik OIDC (no forward-auth gate); VAAPI depends on GPU passthrough on strong"}}
|
||||||
- {slug: "service:nextcloud", type: service, name: nextcloud,
|
- {slug: "service:nextcloud", type: service, name: nextcloud,
|
||||||
attributes: {url: "https://cloud.hubris.network",
|
attributes: {url: "https://cloud.hubris.network",
|
||||||
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
|
doc_page: knowledge/wiki/containers/114-nextcloud.md}}
|
||||||
- {slug: "service:paperless", type: service, name: paperless,
|
- {slug: "service:paperless", type: service, name: paperless,
|
||||||
attributes: {url: "https://paperless.hubris.network",
|
attributes: {url: "https://paperless.hubris.network",
|
||||||
doc_page: knowledge/wiki/containers/103-paperless.md,
|
doc_page: knowledge/wiki/containers/103-paperless.md,
|
||||||
@@ -249,8 +255,8 @@ entities:
|
|||||||
doc_page: knowledge/wiki/containers/118-elementsynapse.md,
|
doc_page: knowledge/wiki/containers/118-elementsynapse.md,
|
||||||
risk_notes: "alert/approval channel for Oikos — outage silences agent escalation"}}
|
risk_notes: "alert/approval channel for Oikos — outage silences agent escalation"}}
|
||||||
- {slug: "service:photos", type: service, name: photos,
|
- {slug: "service:photos", type: service, name: photos,
|
||||||
attributes: {url: "https://photos.hubris.network",
|
attributes: {url: "https://photos.hubris.network",
|
||||||
doc_page: knowledge/wiki/containers/120-mule-images.md}}
|
doc_page: knowledge/wiki/containers/120-mule-images.md}}
|
||||||
- {slug: "service:arr-stack", type: service, name: arr_stack,
|
- {slug: "service:arr-stack", type: service, name: arr_stack,
|
||||||
attributes: {doc_page: knowledge/wiki/containers/122-arriman.md,
|
attributes: {doc_page: knowledge/wiki/containers/122-arriman.md,
|
||||||
note: "jellyseerr / qbit / sab on docker compose"}}
|
note: "jellyseerr / qbit / sab on docker compose"}}
|
||||||
@@ -265,11 +271,21 @@ entities:
|
|||||||
attributes: {url: "https://zimaos.hubris.network",
|
attributes: {url: "https://zimaos.hubris.network",
|
||||||
doc_page: knowledge/wiki/vms/100-zimaos.md}}
|
doc_page: knowledge/wiki/vms/100-zimaos.md}}
|
||||||
- {slug: "service:haos", type: service, name: haos,
|
- {slug: "service:haos", type: service, name: haos,
|
||||||
attributes: {doc_page: knowledge/wiki/vms/108-haos.md}}
|
attributes: {doc_page: knowledge/wiki/vms/108-haos.md,
|
||||||
|
monitoring: none}} # redundant: vm:haos covers liveness via vm-status; haos blocks SSH so a process check can't reach it
|
||||||
- {slug: "service:teddycloud", type: service, name: teddycloud,
|
- {slug: "service:teddycloud", type: service, name: teddycloud,
|
||||||
attributes: {url: "https://teddy.hubris.network",
|
attributes: {url: "https://teddy.hubris.network",
|
||||||
doc_page: knowledge/wiki/containers/131-teddycloud.md,
|
doc_page: knowledge/wiki/containers/131-teddycloud.md,
|
||||||
risk_notes: "no forward-auth gate — reachable by anyone on LAN/mesh"}}
|
risk_notes: "no forward-auth gate — reachable by anyone on LAN/mesh"}}
|
||||||
|
# The Go control plane itself: api/scheduler/notifier/web on the mac-mini,
|
||||||
|
# and what mcp.hubris.network fronts since the cutover. It existed in the
|
||||||
|
# database (created outside the seed) but was never declared here, so a
|
||||||
|
# fresh seed could not resolve the routes-to edge below.
|
||||||
|
- {slug: "service:oikos", type: service, name: oikos,
|
||||||
|
attributes: {url: "https://oikos.hubris.network",
|
||||||
|
host: "ws:mac-mini",
|
||||||
|
ports: {api: 8090, web: 8091, nomos_gateway: 8092},
|
||||||
|
note: "homelab automation platform — api/scheduler/notifier/web on mac-mini docker compose (project name oikos)"}}
|
||||||
- {slug: "service:homelab-mcp", type: service, name: homelab_mcp,
|
- {slug: "service:homelab-mcp", type: service, name: homelab_mcp,
|
||||||
attributes: {port: 9810, systemd_unit: homelab-mcp,
|
attributes: {port: 9810, systemd_unit: homelab-mcp,
|
||||||
endpoint: "https://mcp.hubris.network/mcp",
|
endpoint: "https://mcp.hubris.network/mcp",
|
||||||
@@ -330,6 +346,32 @@ entities:
|
|||||||
- {slug: "ingress:sab.hubris.network", type: ingress-route, name: sab.hubris.network,
|
- {slug: "ingress:sab.hubris.network", type: ingress-route, name: sab.hubris.network,
|
||||||
attributes: {forward_auth: true}}
|
attributes: {forward_auth: true}}
|
||||||
|
|
||||||
|
# ─── TLS certificates (Caddy-managed, *.hubris.network) ───────────
|
||||||
|
# Each cert's expiry is probed by dialing Caddy's lab IP (`dial`) with SNI
|
||||||
|
# set to the hostname — the scheduler container has no mesh/split-horizon
|
||||||
|
# DNS, so it can't resolve *.hubris.network, but it CAN reach Caddy on the
|
||||||
|
# lab LAN.
|
||||||
|
- {slug: "cert:proxmox.hubris.network", type: certificate, name: proxmox.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:git.hubris.network", type: certificate, name: git.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:auth.hubris.network", type: certificate, name: auth.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:media.hubris.network", type: certificate, name: media.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:cloud.hubris.network", type: certificate, name: cloud.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:paperless.hubris.network", type: certificate, name: paperless.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:matrix.hubris.network", type: certificate, name: matrix.hubris.network}
|
||||||
|
- {slug: "cert:photos.hubris.network", type: certificate, name: photos.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:artifacto.hubris.network", type: certificate, name: artifacto.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:trmnl.hubris.network", type: certificate, name: trmnl.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:zimaos.hubris.network", type: certificate, name: zimaos.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:teddy.hubris.network", type: certificate, name: teddy.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:mcp.hubris.network", type: certificate, name: mcp.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:house.hubris.network", type: certificate, name: house.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:books.hubris.network", type: certificate, name: books.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:seanime.hubris.network", type: certificate, name: seanime.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:roms.hubris.network", type: certificate, name: roms.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:jellyseerr.hubris.network", type: certificate, name: jellyseerr.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:qbit.hubris.network", type: certificate, name: qbit.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
- {slug: "cert:sab.hubris.network", type: certificate, name: sab.hubris.network, attributes: {dial: "192.168.8.175"}}
|
||||||
|
|
||||||
# ─── Governance ────────────────────────────────────────────────────
|
# ─── Governance ────────────────────────────────────────────────────
|
||||||
- {slug: "person:dtoro", type: person, name: dtoro,
|
- {slug: "person:dtoro", type: person, name: dtoro,
|
||||||
attributes: {matrix_id: "@dtoro:avispero"}}
|
attributes: {matrix_id: "@dtoro:avispero"}}
|
||||||
@@ -451,7 +493,54 @@ relationships:
|
|||||||
# fronts the Go api. Nomos recorded this correctly on 2026-07-12; the seed
|
# fronts the Go api. Nomos recorded this correctly on 2026-07-12; the seed
|
||||||
# was the stale one, and re-asserting the old edge alongside it is what made
|
# was the stale one, and re-asserting the old edge alongside it is what made
|
||||||
# ingress:mcp a cardinality violation.
|
# ingress:mcp a cardinality violation.
|
||||||
|
- {source: "ws:mac-mini", target: "service:oikos", type: provides}
|
||||||
- {source: "ingress:mcp.hubris.network", target: "service:oikos", type: routes-to}
|
- {source: "ingress:mcp.hubris.network", target: "service:oikos", type: routes-to}
|
||||||
|
# Every public hostname is terminated by caddy. Without these the
|
||||||
|
# reverse proxy — the single widest point of failure in the lab —
|
||||||
|
# had a blast radius of one.
|
||||||
|
- {source: "ingress:proxmox.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:git.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:auth.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:media.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:cloud.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:paperless.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:matrix.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:photos.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:artifacto.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:trmnl.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:zimaos.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:teddy.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:mcp.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:secrets.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:house.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:books.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:seanime.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:roms.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:jellyseerr.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:qbit.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
- {source: "ingress:sab.hubris.network", target: "service:caddy", type: served-by}
|
||||||
|
# Each public route is served with its TLS certificate.
|
||||||
|
- {source: "ingress:proxmox.hubris.network", target: "cert:proxmox.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:git.hubris.network", target: "cert:git.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:auth.hubris.network", target: "cert:auth.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:media.hubris.network", target: "cert:media.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:cloud.hubris.network", target: "cert:cloud.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:paperless.hubris.network", target: "cert:paperless.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:matrix.hubris.network", target: "cert:matrix.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:photos.hubris.network", target: "cert:photos.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:artifacto.hubris.network", target: "cert:artifacto.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:trmnl.hubris.network", target: "cert:trmnl.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:zimaos.hubris.network", target: "cert:zimaos.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:teddy.hubris.network", target: "cert:teddy.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:mcp.hubris.network", target: "cert:mcp.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:house.hubris.network", target: "cert:house.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:books.hubris.network", target: "cert:books.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:seanime.hubris.network", target: "cert:seanime.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:roms.hubris.network", target: "cert:roms.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:jellyseerr.hubris.network", target: "cert:jellyseerr.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:qbit.hubris.network", target: "cert:qbit.hubris.network", type: uses-certificate}
|
||||||
|
- {source: "ingress:sab.hubris.network", target: "cert:sab.hubris.network", type: uses-certificate}
|
||||||
|
|
||||||
- {source: "ingress:secrets.hubris.network", target: "service:secrets-issuance", type: routes-to}
|
- {source: "ingress:secrets.hubris.network", target: "service:secrets-issuance", type: routes-to}
|
||||||
- {source: "ingress:house.hubris.network", target: "service:house", type: routes-to}
|
- {source: "ingress:house.hubris.network", target: "service:house", type: routes-to}
|
||||||
- {source: "ingress:books.hubris.network", target: "service:grimmory", type: routes-to}
|
- {source: "ingress:books.hubris.network", target: "service:grimmory", type: routes-to}
|
||||||
|
|||||||
@@ -6372,6 +6372,23 @@ investigations:
|
|||||||
tags:
|
tags:
|
||||||
- investigation
|
- investigation
|
||||||
runbooks:
|
runbooks:
|
||||||
|
- slug: knowledge-graph-audit
|
||||||
|
name: Knowledge-graph audit
|
||||||
|
risk_class: read_only
|
||||||
|
entity_type: entity
|
||||||
|
procedure: {}
|
||||||
|
content: "---\nname: knowledge-graph-audit\nrisk_class: read_only\ninputs: []\nverification:\
|
||||||
|
\ \"audit_knowledge_graph returns a report with summary.total_findings\"\ndocs_update_checklist:\
|
||||||
|
\ []\n---\n\n# Knowledge-graph audit\n\nRead-only validation that the knowledge graph\
|
||||||
|
\ and its monitoring reflect reality.\nCall MCP `audit_knowledge_graph` (or `GET /api/v1/audit/drift`)\
|
||||||
|
\ for a ranked report:\norphan check entities, checks on deprecated/destroyed targets,\
|
||||||
|
\ probes stuck down/unknown, unmonitored declared types, and dangling edges. Each\
|
||||||
|
\ finding carries a `suggested_runbook`. Triage critical (down_checks) first; confirm\
|
||||||
|
\ each with `get_entity`/`get_relations` before acting. This skill makes no changes\
|
||||||
|
\ \u2014 route confirmed findings to their remediation runbook and re-run the audit\
|
||||||
|
\ to verify. Live-infra discovery (pct/docker/certs vs DB, misplaced parents, undeployed\
|
||||||
|
\ scripts, unmodeled certs, seed drift via `oikos export`) is a documented manual\
|
||||||
|
\ follow-up until that machinery lands.\n"
|
||||||
- slug: client-enrollment
|
- slug: client-enrollment
|
||||||
name: Client enrollment
|
name: Client enrollment
|
||||||
risk_class: read_only
|
risk_class: read_only
|
||||||
|
|||||||
@@ -241,6 +241,11 @@ entity_types:
|
|||||||
layer: infrastructure
|
layer: infrastructure
|
||||||
lifecycle: infrastructure
|
lifecycle: infrastructure
|
||||||
description: Machine outside PVE management (e.g. external VPS).
|
description: Machine outside PVE management (e.g. external VPS).
|
||||||
|
# Override inherited [ping, resource, updates] — standalone servers may
|
||||||
|
# not be SSH/ICMP-reachable from the scheduler. HTTP is the least-
|
||||||
|
# common-denominator liveness signal. Entities with full SSH access
|
||||||
|
# can override to [ping, resource, http].
|
||||||
|
monitoring: [http]
|
||||||
attributes:
|
attributes:
|
||||||
type: object
|
type: object
|
||||||
properties:
|
properties:
|
||||||
@@ -272,7 +277,9 @@ entity_types:
|
|||||||
layer: infrastructure
|
layer: infrastructure
|
||||||
lifecycle: infrastructure
|
lifecycle: infrastructure
|
||||||
description: Virtual machine.
|
description: Virtual machine.
|
||||||
monitoring: [ping] # no guest agent assumed; reachability only
|
monitoring: [vm-status] # `qm status` from the host: powered-on liveness
|
||||||
|
# that works even when the VM blocks ICMP and has
|
||||||
|
# no guest agent (haos). ping is unreliable for VMs.
|
||||||
attributes:
|
attributes:
|
||||||
type: object
|
type: object
|
||||||
properties:
|
properties:
|
||||||
@@ -379,8 +386,13 @@ entity_types:
|
|||||||
layer: infrastructure
|
layer: infrastructure
|
||||||
lifecycle: infrastructure
|
lifecycle: infrastructure
|
||||||
description: DNS zone (e.g. split-horizon hubris.network).
|
description: DNS zone (e.g. split-horizon hubris.network).
|
||||||
monitoring: [dns] # NOTE: no `dns` checker exists yet — this is a
|
monitoring: [dns] # resolves the zone's apex via the configured
|
||||||
# real gap and coverageSweep will report it
|
# resolver, verifying the zone is authoritatively
|
||||||
|
# reachable. (2026-08-05: was `none` — the DNS layer
|
||||||
|
# had zero checks, so a stale record like
|
||||||
|
# matrix→82.165.190.79 went unnoticed. Requires
|
||||||
|
# ontology re-ingest; coverageSweep clears stale
|
||||||
|
# signals after.)
|
||||||
attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
|
attributes: {type: object, properties: {zone: {type: string}, authority: {type: string}}}
|
||||||
dns-record:
|
dns-record:
|
||||||
parent: entity
|
parent: entity
|
||||||
@@ -699,36 +711,42 @@ relationship_types:
|
|||||||
target: compute-entity
|
target: compute-entity
|
||||||
cardinality: one-to-many
|
cardinality: one-to-many
|
||||||
description: Machine hosts a VM/container (hubris hosts lxc:apps).
|
description: Machine hosts a VM/container (hubris hosts lxc:apps).
|
||||||
|
blast_direction: forward
|
||||||
runs-hypervisor:
|
runs-hypervisor:
|
||||||
inverse: hypervisor-on
|
inverse: hypervisor-on
|
||||||
source: machine
|
source: machine
|
||||||
target: hypervisor
|
target: hypervisor
|
||||||
cardinality: one-to-one
|
cardinality: one-to-one
|
||||||
description: Machine runs hypervisor software.
|
description: Machine runs hypervisor software.
|
||||||
|
blast_direction: forward
|
||||||
member-of:
|
member-of:
|
||||||
inverse: has-member
|
inverse: has-member
|
||||||
source: proxmox-host
|
source: proxmox-host
|
||||||
target: cluster
|
target: cluster
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: PVE host belongs to a cluster.
|
description: PVE host belongs to a cluster.
|
||||||
|
blast_direction: backward
|
||||||
part-of:
|
part-of:
|
||||||
inverse: comprises
|
inverse: comprises
|
||||||
source: docker-container
|
source: docker-container
|
||||||
target: compose-stack
|
target: compose-stack
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Docker container belongs to a compose stack.
|
description: Docker container belongs to a compose stack.
|
||||||
|
blast_direction: backward
|
||||||
provides:
|
provides:
|
||||||
inverse: provided-by
|
inverse: provided-by
|
||||||
source: compute-entity
|
source: compute-entity
|
||||||
target: service
|
target: service
|
||||||
cardinality: one-to-many
|
cardinality: one-to-many
|
||||||
description: Compute entity provides a service (lxc:gitea provides service:gitea).
|
description: Compute entity provides a service (lxc:gitea provides service:gitea).
|
||||||
|
blast_direction: forward
|
||||||
runs:
|
runs:
|
||||||
inverse: run-by
|
inverse: run-by
|
||||||
source: service
|
source: service
|
||||||
target: application
|
target: application
|
||||||
cardinality: one-to-many
|
cardinality: one-to-many
|
||||||
description: Service runs an application.
|
description: Service runs an application.
|
||||||
|
blast_direction: backward
|
||||||
configured-by:
|
configured-by:
|
||||||
inverse: configures
|
inverse: configures
|
||||||
source: entity
|
source: entity
|
||||||
@@ -741,36 +759,52 @@ relationship_types:
|
|||||||
target: entity
|
target: entity
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Pipeline deploys to a service/host.
|
description: Pipeline deploys to a service/host.
|
||||||
|
blast_direction: forward
|
||||||
routes-to:
|
routes-to:
|
||||||
inverse: routed-via
|
inverse: routed-via
|
||||||
source: ingress-route
|
source: ingress-route
|
||||||
target: service
|
target: service
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Public hostname routes to a service.
|
description: Public hostname routes to a service.
|
||||||
|
blast_direction: backward
|
||||||
|
served-by:
|
||||||
|
inverse: serves
|
||||||
|
source: ingress-route
|
||||||
|
target: service
|
||||||
|
cardinality: many-to-one
|
||||||
|
description: Ingress route is terminated by this reverse proxy. Distinct
|
||||||
|
from routes-to, which names the BACKEND the route forwards to — without
|
||||||
|
this edge the proxy's blast radius is invisible, and lxc:caddy reported
|
||||||
|
one affected entity despite terminating every *.hubris.network route.
|
||||||
|
blast_direction: backward
|
||||||
secured-by:
|
secured-by:
|
||||||
inverse: secures
|
inverse: secures
|
||||||
source: ingress-route
|
source: ingress-route
|
||||||
target: identity-provider
|
target: identity-provider
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Route gated by forward-auth.
|
description: Route gated by forward-auth.
|
||||||
|
blast_direction: backward
|
||||||
uses-certificate:
|
uses-certificate:
|
||||||
inverse: certifies
|
inverse: certifies
|
||||||
source: ingress-route
|
source: ingress-route
|
||||||
target: certificate
|
target: certificate
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Route served with this certificate.
|
description: Route served with this certificate.
|
||||||
|
blast_direction: backward
|
||||||
authenticates-via:
|
authenticates-via:
|
||||||
inverse: authenticates-service
|
inverse: authenticates-service
|
||||||
source: service
|
source: service
|
||||||
target: identity-provider
|
target: identity-provider
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Service uses native OIDC (jellyfin authenticates-via authentik).
|
description: Service uses native OIDC (jellyfin authenticates-via authentik).
|
||||||
|
blast_direction: backward
|
||||||
in-zone:
|
in-zone:
|
||||||
inverse: contains-record
|
inverse: contains-record
|
||||||
source: dns-record
|
source: dns-record
|
||||||
target: dns-zone
|
target: dns-zone
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Record belongs to a zone.
|
description: Record belongs to a zone.
|
||||||
|
blast_direction: backward
|
||||||
resolves-to:
|
resolves-to:
|
||||||
inverse: resolved-from
|
inverse: resolved-from
|
||||||
source: dns-record
|
source: dns-record
|
||||||
@@ -783,24 +817,28 @@ relationship_types:
|
|||||||
target: service
|
target: service
|
||||||
cardinality: many-to-many
|
cardinality: many-to-many
|
||||||
description: Runtime dependency (blast-radius edge).
|
description: Runtime dependency (blast-radius edge).
|
||||||
|
blast_direction: backward
|
||||||
connects-via:
|
connects-via:
|
||||||
inverse: connects
|
inverse: connects
|
||||||
source: compute-entity
|
source: compute-entity
|
||||||
target: network
|
target: network
|
||||||
cardinality: many-to-many
|
cardinality: many-to-many
|
||||||
description: Coarse network membership (host on LAN / mesh).
|
description: Coarse network membership (host on LAN / mesh).
|
||||||
|
blast_direction: backward
|
||||||
has-interface:
|
has-interface:
|
||||||
inverse: interface-of
|
inverse: interface-of
|
||||||
source: compute-entity
|
source: compute-entity
|
||||||
target: network-interface
|
target: network-interface
|
||||||
cardinality: one-to-many
|
cardinality: one-to-many
|
||||||
description: Optional per-interface refinement.
|
description: Optional per-interface refinement.
|
||||||
|
blast_direction: backward
|
||||||
interface-on:
|
interface-on:
|
||||||
inverse: has-endpoint
|
inverse: has-endpoint
|
||||||
source: network-interface
|
source: network-interface
|
||||||
target: network
|
target: network
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Interface attaches to a network.
|
description: Interface attaches to a network.
|
||||||
|
blast_direction: backward
|
||||||
|
|
||||||
# Storage
|
# Storage
|
||||||
mounts:
|
mounts:
|
||||||
@@ -810,24 +848,28 @@ relationship_types:
|
|||||||
cardinality: many-to-many
|
cardinality: many-to-many
|
||||||
description: Compute entity mounts a volume. Edge attributes carry
|
description: Compute entity mounts a volume. Edge attributes carry
|
||||||
mount_point and options.
|
mount_point and options.
|
||||||
|
blast_direction: backward
|
||||||
stores-on:
|
stores-on:
|
||||||
inverse: stores-for
|
inverse: stores-for
|
||||||
source: compute-entity
|
source: compute-entity
|
||||||
target: storage-pool
|
target: storage-pool
|
||||||
cardinality: many-to-many
|
cardinality: many-to-many
|
||||||
description: Rootfs/data lives on a pool.
|
description: Rootfs/data lives on a pool.
|
||||||
|
blast_direction: backward
|
||||||
contains:
|
contains:
|
||||||
inverse: contained-in
|
inverse: contained-in
|
||||||
source: storage-pool
|
source: storage-pool
|
||||||
target: volume
|
target: volume
|
||||||
cardinality: one-to-many
|
cardinality: one-to-many
|
||||||
description: Pool contains a volume.
|
description: Pool contains a volume.
|
||||||
|
blast_direction: forward
|
||||||
holds-dataset:
|
holds-dataset:
|
||||||
inverse: dataset-on
|
inverse: dataset-on
|
||||||
source: volume
|
source: volume
|
||||||
target: dataset
|
target: dataset
|
||||||
cardinality: one-to-many
|
cardinality: one-to-many
|
||||||
description: Volume holds a tracked dataset.
|
description: Volume holds a tracked dataset.
|
||||||
|
blast_direction: backward
|
||||||
backs-up-to:
|
backs-up-to:
|
||||||
inverse: backup-of
|
inverse: backup-of
|
||||||
source: entity
|
source: entity
|
||||||
@@ -842,12 +884,14 @@ relationship_types:
|
|||||||
target: ups
|
target: ups
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Machine on UPS power.
|
description: Machine on UPS power.
|
||||||
|
blast_direction: backward
|
||||||
located-at:
|
located-at:
|
||||||
inverse: location-of
|
inverse: location-of
|
||||||
source: machine
|
source: machine
|
||||||
target: site
|
target: site
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Machine's physical site.
|
description: Machine's physical site.
|
||||||
|
blast_direction: backward
|
||||||
registered-with:
|
registered-with:
|
||||||
inverse: registrar-of
|
inverse: registrar-of
|
||||||
source: domain-registration
|
source: domain-registration
|
||||||
@@ -880,12 +924,14 @@ relationship_types:
|
|||||||
target: secret
|
target: secret
|
||||||
cardinality: many-to-one
|
cardinality: many-to-one
|
||||||
description: Grant covers a secret.
|
description: Grant covers a secret.
|
||||||
|
blast_direction: forward
|
||||||
can-decrypt:
|
can-decrypt:
|
||||||
inverse: readable-by
|
inverse: readable-by
|
||||||
source: compute-entity
|
source: compute-entity
|
||||||
target: secret
|
target: secret
|
||||||
cardinality: many-to-many
|
cardinality: many-to-many
|
||||||
description: Host can decrypt a secret (legacy SOPS; Infisical grants later).
|
description: Host can decrypt a secret (legacy SOPS; Infisical grants later).
|
||||||
|
blast_direction: backward
|
||||||
|
|
||||||
# Cognition
|
# Cognition
|
||||||
checks:
|
checks:
|
||||||
|
|||||||
@@ -70,6 +70,11 @@ approval_rules:
|
|||||||
- {entity_type: docker-container, action: restart, risk_class: reversible_low, autonomy_level: auto}
|
- {entity_type: docker-container, action: restart, risk_class: reversible_low, autonomy_level: auto}
|
||||||
- {entity_type: machine, action: apt-upgrade, risk_class: config_mutation, autonomy_level: escalate}
|
- {entity_type: machine, action: apt-upgrade, risk_class: config_mutation, autonomy_level: escalate}
|
||||||
- {entity_type: machine, action: reboot, risk_class: config_mutation, autonomy_level: escalate}
|
- {entity_type: machine, action: reboot, risk_class: config_mutation, autonomy_level: escalate}
|
||||||
|
# host-level package/kernel install (apt-get install, dpkg, modprobe, systemctl enable)
|
||||||
|
# always classifies as config_mutation — the classifier defaults to config_mutation
|
||||||
|
# for any command not in the read-only allowlist, so apt-get install reaches this
|
||||||
|
# tier naturally. Documented explicitly here so agents stop second-guessing:
|
||||||
|
# host mutations always need operator approval.
|
||||||
- {entity_type: machine, action: format-disk, risk_class: destructive, autonomy_level: never}
|
- {entity_type: machine, action: format-disk, risk_class: destructive, autonomy_level: never}
|
||||||
- {entity_type: config-repo, action: edit, risk_class: config_mutation, autonomy_level: escalate}
|
- {entity_type: config-repo, action: edit, risk_class: config_mutation, autonomy_level: escalate}
|
||||||
- {entity_type: deploy-pipeline, action: trigger, risk_class: config_mutation, autonomy_level: escalate}
|
- {entity_type: deploy-pipeline, action: trigger, risk_class: config_mutation, autonomy_level: escalate}
|
||||||
|
|||||||
94
tools/deploy-checks.sh
Executable file
94
tools/deploy-checks.sh
Executable file
@@ -0,0 +1,94 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# deploy-checks.sh — push the check scripts into every monitored target.
|
||||||
|
#
|
||||||
|
# Background: an ssh-script check runs the script INSIDE the target, so the
|
||||||
|
# script must exist at /opt/oikos/checks/ on the target itself — not just on
|
||||||
|
# the proxmox host. The scheduler routes LXC/VM checks through the host via
|
||||||
|
# `pct exec`/`qm guest exec`, so it never SSHes a guest directly, but the
|
||||||
|
# script still has to be present inside the guest. This script deploys them.
|
||||||
|
#
|
||||||
|
# Run from a Proxmox host (it uses `pct`/`qm`) to populate every local guest,
|
||||||
|
# and/or pass --host to install on a host/workstation over SSH.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# deploy-checks.sh # on a proxmox host: push to every LXC/VM here
|
||||||
|
# deploy-checks.sh --host ws:mac-mini # ssh-install scripts on a host/workstation
|
||||||
|
# deploy-checks.sh --checks /path # override the source checks dir
|
||||||
|
#
|
||||||
|
# Idempotent: skips a script whose deployed copy is byte-identical.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
CHECKS_DIR="${OIKOS_CHECK_DIR:-${HOMELAB_CONTEXT_DIR:-/opt/homelab}/checks}"
|
||||||
|
DEST=/opt/oikos/checks
|
||||||
|
|
||||||
|
die() { echo "deploy-checks: $*" >&2; exit 1; }
|
||||||
|
|
||||||
|
deploy_to_guest() {
|
||||||
|
local id="$1" vm="$2" # vm=0 for LXC, 1 for VM
|
||||||
|
local kind=pct; [ "$vm" = "1" ] && kind=qm
|
||||||
|
echo "[deploy-checks] $kind $id"
|
||||||
|
# Ensure the destination dir exists inside the guest.
|
||||||
|
if [ "$kind" = "pct" ]; then
|
||||||
|
pct exec "$id" -- mkdir -p "$DEST" 2>/dev/null || { echo " skip (pct exec failed)"; return; }
|
||||||
|
else
|
||||||
|
# qm guest exec returns JSON; best-effort for VMs (guest agent required).
|
||||||
|
qm guest exec "$id" -- mkdir -p "$DEST" >/dev/null 2>&1 || { echo " skip (qm guest exec failed)"; return; }
|
||||||
|
fi
|
||||||
|
local pushed=0 skipped=0
|
||||||
|
for script in "$CHECKS_DIR"/*.sh; do
|
||||||
|
local name; name=$(basename "$script")
|
||||||
|
[ "$name" = "deploy-checks.sh" ] && continue
|
||||||
|
[ "$name" = "install.sh" ] && continue
|
||||||
|
if [ "$kind" = "pct" ]; then
|
||||||
|
pct push "$id" "$script" "$DEST/$name" --perms 755 2>/dev/null && pushed=$((pushed+1)) || skipped=$((skipped+1))
|
||||||
|
else
|
||||||
|
# qm has no push; copy via the guest agent file write if available.
|
||||||
|
qm guest exec "$id" -- /bin/sh -c "cat > $DEST/$name" < "$script" >/dev/null 2>&1 && pushed=$((pushed+1)) || skipped=$((skipped+1))
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
echo " pushed=$pushed skipped=$skipped"
|
||||||
|
}
|
||||||
|
|
||||||
|
deploy_to_host() {
|
||||||
|
local target="$1" # user@ip or slug resolved by caller
|
||||||
|
echo "[deploy-checks] host $target"
|
||||||
|
ssh -o BatchMode=yes -o StrictHostKeyChecking=no "$target" "bash -s" < "$CHECKS_DIR/install.sh" \
|
||||||
|
|| echo " WARNING: install on $target failed"
|
||||||
|
}
|
||||||
|
|
||||||
|
if [ ! -d "$CHECKS_DIR" ]; then die "checks dir not found: $CHECKS_DIR"; fi
|
||||||
|
|
||||||
|
# Host/workstation install mode.
|
||||||
|
if [ "${1:-}" = "--host" ]; then
|
||||||
|
[ $# -ge 2 ] || die "--host needs a target (user@ip)"
|
||||||
|
deploy_to_host "$2"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Proxmox-host mode: push to every local LXC and VM.
|
||||||
|
if command -v pct >/dev/null 2>&1; then
|
||||||
|
# LXC containers: ID and status. Skip stopped ones.
|
||||||
|
while IFS= read -r line; do
|
||||||
|
[ -z "$line" ] && continue
|
||||||
|
id=$(awk '{print $1}' <<<"$line")
|
||||||
|
status=$(awk '{print $2}' <<<"$line")
|
||||||
|
[ "$status" = "running" ] || { echo "[deploy-checks] skip LXC $id ($status)"; continue; }
|
||||||
|
deploy_to_guest "$id" 0
|
||||||
|
done < <(pct list 2>/dev/null | tail -n +2)
|
||||||
|
else
|
||||||
|
echo "deploy-checks: 'pct' not found — not a Proxmox host."
|
||||||
|
echo " On a host/workstation, use: deploy-checks.sh --host user@ip"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if command -v qm >/dev/null 2>&1; then
|
||||||
|
while IFS= read -r line; do
|
||||||
|
[ -z "$line" ] && continue
|
||||||
|
id=$(awk '{print $1}' <<<"$line")
|
||||||
|
status=$(awk '{print $2}' <<<"$line")
|
||||||
|
[ "$status" = "running" ] || continue
|
||||||
|
deploy_to_guest "$id" 1
|
||||||
|
done < <(qm list 2>/dev/null | tail -n +2)
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "[deploy-checks] done"
|
||||||
@@ -1,13 +1,35 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# setup-checks.sh — deploy check scripts to /opt/oikos/checks on each host.
|
# setup-checks.sh — deploy check scripts to /opt/oikos/checks.
|
||||||
# Auto-setup hook: tools/setup-*.sh runs after every git pull.
|
# Auto-setup hook: tools/setup-*.sh runs after every git pull.
|
||||||
|
#
|
||||||
|
# On a plain host/workstation this installs the scripts locally (the pulling
|
||||||
|
# host). On a Proxmox host it ALSO pushes the scripts into every running LXC/VM
|
||||||
|
# guest, because an ssh-script check runs the script INSIDE the target — a
|
||||||
|
# script present only on the host does nothing for a guest reached via
|
||||||
|
# pct/qm exec. Guest deployment is delegated to deploy-checks.sh.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
|
CLONE_DIR="${HOMELAB_CONTEXT_DIR:-/opt/homelab}"
|
||||||
CHECK_SETUP="$CLONE_DIR/checks/install.sh"
|
CHECK_SETUP="$CLONE_DIR/checks/install.sh"
|
||||||
|
DEPLOY="$CLONE_DIR/tools/deploy-checks.sh"
|
||||||
|
|
||||||
if [ -f "$CHECK_SETUP" ]; then
|
if [ -f "$CHECK_SETUP" ]; then
|
||||||
bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
|
bash "$CHECK_SETUP" || echo "[setup-checks] WARNING: install.sh exited with code $?"
|
||||||
else
|
else
|
||||||
echo "[setup-checks] no checks/install.sh found, skipping"
|
echo "[setup-checks] no checks/install.sh found, skipping"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# On a Proxmox host, keep every guest's scripts in sync too. Best-effort: a
|
||||||
|
# failing push to one guest must not abort the whole hook. Warn (not skip
|
||||||
|
# silently) if deploy-checks.sh itself is absent — without it guests never get
|
||||||
|
# scripts and the pct-exec routing reports every guest check down.
|
||||||
|
if command -v pct >/dev/null 2>&1; then
|
||||||
|
if [ ! -f "$DEPLOY" ]; then
|
||||||
|
echo "[setup-checks] WARNING: $DEPLOY missing — guest scripts will go stale. Commit tools/deploy-checks.sh alongside this hook."
|
||||||
|
elif [ ! -x "$DEPLOY" ]; then
|
||||||
|
echo "[setup-checks] WARNING: $DEPLOY not executable — running via bash"
|
||||||
|
bash "$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
|
||||||
|
else
|
||||||
|
"$DEPLOY" || echo "[setup-checks] WARNING: guest deploy exited non-zero (scripts may be stale on some guests)"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|||||||
@@ -4,12 +4,6 @@
|
|||||||
<meta charset="UTF-8" />
|
<meta charset="UTF-8" />
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
<title>Oikos</title>
|
<title>Oikos</title>
|
||||||
<link rel="preconnect" href="https://fonts.googleapis.com" />
|
|
||||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
|
|
||||||
<link
|
|
||||||
href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,100..1000;1,9..40,100..1000&family=DM+Mono:ital,wght@0,300;0,400;0,500;1,300;1,400;1,500&family=Inknut+Antiqua:wght@300;400;500;600;700&display=swap"
|
|
||||||
rel="stylesheet"
|
|
||||||
/>
|
|
||||||
<link rel="icon" type="image/svg+xml" href="favicon.svg" />
|
<link rel="icon" type="image/svg+xml" href="favicon.svg" />
|
||||||
<link rel="apple-touch-icon" href="apple-touch-icon.png" />
|
<link rel="apple-touch-icon" href="apple-touch-icon.png" />
|
||||||
<link rel="icon" type="image/png" sizes="192x192" href="android-chrome-192.png" />
|
<link rel="icon" type="image/png" sizes="192x192" href="android-chrome-192.png" />
|
||||||
|
|||||||
653
web/package-lock.json
generated
653
web/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -42,6 +42,7 @@
|
|||||||
"vitest": "^2.0.0"
|
"vitest": "^2.0.0"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
|
"@joan/procedural-glyph-engine": "file:../vendor",
|
||||||
"@surdeddd/wmkit": "^0.3.0",
|
"@surdeddd/wmkit": "^0.3.0",
|
||||||
"clsx": "^2.1.1",
|
"clsx": "^2.1.1",
|
||||||
"d3-force": "^3.0.0",
|
"d3-force": "^3.0.0",
|
||||||
|
|||||||
BIN
web/public/fonts/JetBrainsMono-Bold.woff2
Normal file
BIN
web/public/fonts/JetBrainsMono-Bold.woff2
Normal file
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user